AI can tell where your photos were taken

"Hi! Tomorrow is Mother's Day and my mom passed away about eight years ago. I don't know if you can tell where this photo was taken because I was very young and don't remember." Image credit: georainbolt

Trevor Rainbolt, 25, is a well-known expert in determining the geographical location of photos. He often receives requests from netizens to help him find the locations where old photos were taken. With his excellent judgment, he has helped many people.

Ryan Bolt is a professional player of GeoGuessr , a game that allows people to guess the location of a place by looking at the photo. His popularity has also made many people know about this game. His outstanding performance is inseparable from long-term training. He has played this game for more than 10,000 hours, and he can play it for four to eight hours a day during the epidemic.

However, in May this year, he lost to AI.

This powerful AI player was developed by three graduate students at Stanford University in about two months. During training, it guessed the country where the photo was taken with an accuracy rate of 92%. The average score of playing GeoGuessr was as high as 4525 (out of 5000 points), ranking in the top 0.01% of players worldwide.

AI sometimes gets close to full marks | Image source: georainbolt

Facing this AI opponent, Ryanbolt was no match for it. However, the privacy risks that this AI may bring also make many people feel uneasy.

How to determine the image position

Like Ryanbolt, the three developers of AI are also veteran players of GeoGuessr.

GeoGuessr is an online geographic guessing game with about 50 million players. It was developed by a Swedish software engineer in 2013. When entering the game, players will be randomly placed somewhere in Google Street View, and then guess their geographical location based on the information on the screen. Finally, players need to place a pin somewhere on the world map to mark their guessed location. Most of the players of the game are very young, and some of the best players are only 14 years old.

The image that appeared in the first round when Ryanbolt played against AI, you can also guess it | Image source: georainbolt

The accuracy and speed of the player's guess determines the score. The closer the guess is to the actual location, the more points the player wins. In addition, the faster the guess is, the higher the score will be.

First round scores | Image source: georainbolt

Any details in the Google Street View images provided by the game are the basis for players to judge. Clues such as the soil on the ground, telephone poles, street signs, road markings, people's clothes, plants, visible landscapes, etc. in the photos can be used to judge the location. Of course, the player's personal intuition is also important.

As for judging skills, Reinbolt would recommend looking for bollards and utility poles in the picture first, which have very unique designs in each country or region. For example, Danish bollards have yellow tops, while German ones are black.

For example, the materials used for telephone poles are also particular. Some are made of wood, some are made of concrete. The shapes of telephone poles are also different, and the stickers on the poles are also a clue. "95% of Australia's concrete poles are in Victoria," said Rainbolt.

Professional players will learn multiple languages ​​and remember different things in different countries, such as which countries use triple white road lines, which countries have dotted road lines, which countries have green road signs, what fonts are on the signs, the construction materials used for the roads, the length of the roads, the license plates and types of cars on the road, and the types of vegetation on both sides of the road... All seemingly ordinary and tiny details are clues to the game.

In addition to the content of the pictures, the image quality of Google Street View also varies . As the smallest republic in the world, San Marino is a country within a country surrounded by Italy. The street view here looks very similar to that of Italy. However, the image quality of the two is different - San Marino's is worse.

In addition, you will sometimes see a small part of the Google Street View car. If you see black tape on the top of the Street View car, it means the location of the picture is Ghana, a country in West Africa.

Players also share clues, but with more than 220 billion Google Street View images, it's hard to remember them all, and master players often rely on intuition.

Although AI does not have the sixth sense of human players, it can not only see the obvious features that humans can see, but also pay attention to tiny details that are overlooked by humans.

AI beats human players

Last year, a group of graduate students at Stanford University were studying the artificial intelligence course "Deep Multitask and Meta Learning". Three students who all loved playing GeoGuessr needed a project, so they decided to start from their common hobby and try to create an artificial intelligence player that is better than human players.

The name of their project is "Predicting Image Geolocations", abbreviated to PIGEON according to the first letter of the letter. They used the CLIP neural network made by OpenAI (also the developer of ChatGPT) to understand and analyze images by reading text. After that, they used Google Street View images to train the system. The training data set contains about 500,000 Google Street View images.

AI divides the image into very small blocks for analysis and is very good at selecting special blocks.

For AI, a picture is not just an image, but also has corresponding text information. By integrating visual information such as buildings, street layouts, vegetation and landmarks in the image, as well as other auxiliary information such as weather conditions, seasons and climate, AI can also quickly predict the location of the picture like top players.

Despite the relatively small size of the data set, the AI ​​system performed very well in the end, with high accuracy and smaller errors than human guesses, and could locate within about 40 kilometers of the actual location. In the game with Ryanbolt, the AI ​​easily won many rounds.

"We're not the first AI to fight Ryanbolt, we're the first AI to defeat Ryanbolt," said Michal Skreta, one of the developers of the AI ​​system.

They believe that this technology has a variety of potential applications, such as identifying roads or power lines that need to be repaired and quickly discovering invasive plants . In future studies, they will further improve the model to improve positioning accuracy and expand the geolocation data set to cover more geographic areas and environments to improve the model's generalization ability (referring to the model's performance ability when faced with unseen data or new situations).

Possible risks

The program already seems to be able to geolocate photos outside of Street View. "I gave it some photos from a road trip more than a decade ago, and it figured out most of the places. It guessed a campground in Yellowstone Park, about 55 kilometers away. Another photo was taken on a street in San Francisco, and it guessed a location a few blocks away. This has caused some concern among experts," said Geoff Brumfiel, science editor at NPR.

In the face of increasingly powerful and unregulated AI, it seems increasingly difficult to protect personal privacy. From a privacy perspective, our location may be a very sensitive set of information. In the past, people could remove the GPS location tags of photos, but this may not work now.

A low-budget student project has the potential to make it easier for businesses and larger organizations to spy on ordinary people, and for bad actors to stalk and harm others.

Since you can’t beat the AI, just enjoy the game. Guess where this photo comes from?

References

[1] Haas, L., Alberti, S., & Skreta, M. (2023). PIGEON: Predicting Image Geolocations. ResearchGate. Retrieved from https://www.researchgate.net/publication/372313510_PIGEON_Predicting_Image_Geolocations

[2] Brumfiel, G. (2023). Artificial intelligence can find your location in photos, worrying privacy experts. NPR. Retrieved from https://www.npr.org/transcripts/1219984002

[3] Jones, R. (2023). If you were given a Google Street View image of anywhere on Earth, could you identify the location? the Guardian. Retrieved from https://www.theguardian.com/lifeandstyle/2023/dec/23/geoguessr-world-championships-2023-inside-story

[4] Rainbolt. (2023, May 11). world's best ai vs geoguessr pro. Youtube. Retrieved from https://www.youtube.com/watch?v=ts5lPDV--cU

[5] Wired. (2022, August 31). Every Trick a Pro GeoGuessr Player Uses to Win (ft. RAINBOLT) | WIRED. Youtube. Retrieved from https://www.youtube.com/watch?v=0p5Eb4OSZCs

[6] Lloyd, A. (2022). Rainbolt: GeoGuessr TikToker on going viral for location-tracking. Insider. Retrieved from https://www.insider.com/trevor-rainbolt-geoguessr-tiktoker-location-tracking-interview-2022-11

Planning and production

Source: Bringing Science Home (id: steamforkids)

Author | Editor of Cloud Magazine

Editor: Wang Mengru