90% right is 100% wrong
It doesn't matter how close you get, if you don't make your destination, you're wrong.
Winter mountaineering in Scotland is a cruel master. Having finished a route in the Northeastern Corries of Lochnagar, we were faced with a wind that stripped all heat from our bodies and built up a carapace of ice on every surface. There are typically two routes off the hill. One along the top of the corrie, which is shorter but more exposed to the storm, and another down to Glas-Alt-Sheil and Loch Muick, which would allow a quick exit from the screaming winter banshees, but a longer walk home. It is so often true in the storms of the Cairngorm Mountains that there is virtually no visibility. So, even if you could look up, there was nothing to see but white, or in this case, more of a grey, as the pale light shifted to the Northern winter darkness. There is something particularly fearsome about the cold dark.
A small intrepid team, we were very familiar with darkness and with these conditions. So we set off skiting the edge of the corrie, confident in our direction-finding skills, having been up here numerous times before. Within 20 minutes or so, we came across another set of footprints. These buoyed us further. We had not seen any other parties on the mountain, but in these conditions, they could have been immediately beside us, and we’d have missed them.
Within a further 10 minutes, we knew we were following our own tracks.
The stark realization that, in a place you thought you knew well, it was possible to walk in a complete circle was both frustrating and has stayed with me for the rest of my life. This was almost thirty years ago. We found our way back to the summit of Lochnagar, a place we could confirm, and set a bearing, committing to the long walk home.
We lost 30 minutes, and I learned a 30-year lesson. One that is becoming more true every day.
In geography, if you are 90% right, you are 100% wrong. Last-mile delivery has learned this lesson, as did Apple. The secondary lesson is to leave a trail of breadcrumbs, as it were, of confirmed locations.
GeoAI Benchmarks
I have become increasingly concerned about the lack of geographic competence in AI. There are now numerous excellent geographic technologies, such as Google, Apple, and Mapbox, offering very high-performance geocoding and routing. Yet, the more general AI Frontier models seem to rely on an implicit rather than an explicit understanding of geography. Indeed, when we consider the collision between geography and generative AI, we see a willingness to bend the laws of geography to meet the model's needs.
But attributes like “place” should not be treated with so little regard.
Being present in a location is the difference between arrival and loss, guilt and innocence, even between life and death. As we see increasing conflict across our planet, the confirmed knowledge of place is a determinant of humanity itself.
We can benchmark AI models. We also have a deep understanding of our changing Planet through Earth observation, geographic data, and geodesy. So, I believe we should start benchmarking frontier models against simple things like place names and locations, then we can add their algorithmic knowledge of geography, and finally, as we move towards a better understanding of the shape of our Planet, our knowledge of our geoid.
When I think about benchmarking, I think about questions like: ”Count all the football fields in Georgia.” Having confidence around context, and an AI’s confidence to ask clarifying questions: which kind of football leading to which field dimensions, is it a pitch or a field, and which Georgia? Modern generative tech tends to feel comfortable bending all rules evenly. Time and space are rules that should probably be harder to bend than most, as they tend to be for us humans. Yet those rules seem to be largely missing.
It’s actually not just about some fuzzy notion of accuracy, but knowledge of place is a feature of sovereignty in general. I talked about SovereignAI in this note. Knowing what things were/are called in certain places at certain times is a matter of knowing and accepting our history. This reminds me of Joshua Campbell’s work on SpaceTimeIDs, which he presented at the Free and Open Source Software for Geospatial (North America) conference in 2023. Changing geographic boundaries is more important today than it has been in the last 25 years.
History’s back, baby, and if you’re only 90% right, then your 100% wrong.