LLMs vs. Geolocation: GPT-5 performs worse than different AI fashions

In June, Bellingcat ran 500 geolocation exams, evaluating LLMs from varied corporations in opposition to one another, in addition to Google Lens – a staple software for locating the placement of photographs.

On the time, ChatGPT o4-mini-high emerged because the clear winner, with Google Lens outperforming most different fashions. Simply two months later, with new variations of those AI instruments accessible, we re-ran the trial – this time together with Google “AI Mode,” GPT-5, GPT-5 Considering, and Grok 4 into the combo.

The authentic take a look at used 25 of Bellingcat’s personal vacation photographs. From cities to distant countryside, the pictures included scenes each with and with out recognisable options – akin to roads, signage, mountains, or structure. Photos have been sourced from each continent.

For the up to date trial, 5 take a look at photographs have been excluded, as they’d appeared in a earlier article, thus compromising the integrity of the outcomes.

All 24 fashions’ responses have been ranked on a scale from 0 to 10, with 10 indicating an correct and particular identification (akin to a neighbourhood, path, or landmark) and 0 indicating no try and determine the placement in any respect.

Google AI Mode was proven to be probably the most succesful geolocation software general. 

Grok 4 gave each higher and worse solutions in comparison with Grok 3 however, on common, scored marginally larger. Nonetheless, it was nonetheless much less correct than older variations of Gemini and GPT. 

GPT-5, even in ‘Considering’ and ‘Professional’ modes, was a substantial downgrade when put next with the capabilities demonstrated by GPT o4-mini-high. In a single instance, of a metropolis road with skyscrapers within the background, o4-mini-high accurately recognized the road, whereas GPT-5 in Considering mode pointed to the flawed nation. 

Regardless of delivering sooner solutions, GPT-5 appeared to sacrifice accuracy. A shocking variety of errors and a normal sense of disappointment within the new mannequin have additionally been reported by different customers.

Bellingcat examined GPT-5 and its ‘Considering’ mode by way of the Plus subscription, which prices roughly the identical as entry to 04-mini-high previous to its retirement. 5 of probably the most troublesome take a look at pictures have been additionally run by GPT-5 Professional. However even Professional, with a premium price ticket of €200 monthly, didn’t geolocate the photographs any extra precisely than GPT 04-mini-high.

A Seaside, a Resort and a Ferris Wheel

The disparity between Google and the GPT fashions grew to become much more obvious in Check 25 – a photograph of a shoreline resort in Noordwijk, the Netherlands, with a Ferris wheel rising simply past the dunes.

Within the earlier trial, most older fashions – together with these from GPT, Claude, Gemini and Grok – precisely recognized the nation because the Netherlands however didn’t find the city. Many latched onto the Ferris wheel however pointed as a substitute to the seaside city of Scheveningen, which additionally has a Ferris wheel, although located on a pier, not among the many sand dunes.

Nonetheless, the latest fashions, GPT-5 Professional and Considering, have been even much less correct, figuring out a seaside in France – a completely completely different nation. 

Sadly for open supply researchers, following the discharge of GPT-5, OpenAI eliminated the choice to pick out older fashions akin to o4-mini-high. After a wave of detrimental suggestions, OpenAI reinstated GPT-4o because the default mannequin for paid subscribers. Nonetheless, probably the most succesful geolocation fashions recognized in Bellingcat’s testing stay inaccessible.

Google AI Mode, however, was the primary, and solely mannequin thus far, to accurately determine Noordwijk as the placement in Check 25.  

Although AI Mode is powered by a model of Gemini 2.5, it outperformed Gemini 2.5 Professional Deep Analysis in these exams. Described by Google as its “strongest AI search, with extra superior reasoning and multimodality,” AI Mode geolocated take a look at pictures with larger accuracy than any GPT fashions, together with our earlier winner, o4-mini-high.

AI Mode is presently solely accessible in India, United Kingdom and the USA.

Nearly all of fashions, in some unspecified time in the future, returned a hallucination. Customers mustn’t rely solely on the solutions offered by LLMs. Even one of the best choices, together with Google AI Mode, nonetheless, at occasions, confidently level to the flawed location. 

The distinction in fashions’ capabilities in contrast with simply two months in the past exhibits how shortly this discipline is evolving. Nonetheless, OpenAI’s latest modifications additionally counsel that progress is just not assured, and that AI’s capability to geolocate could plateau and even worsen over time. As new fashions emerge, Bellingcat will proceed to check them.

Because of Nathan Patin for contributing to the unique benchmark exams.

Because of Nathan Patin for contributing to the unique benchmark exams.Bellingcat is a non-profit and the flexibility to hold out our work relies on the sort assist of particular person donors. If you need to assist our work, you are able to do so right here. You can too subscribe to our Patreon channel right here. Subscribe to our E-newsletter and observe us on Bluesky right here and Instagram right here.