Is What I'm Seeing Real?
the affirmative half of “data is hostile until proven otherwise”
the first time it really landed i was looking at a steepness raster of my family’s land in southern Vermont, and a line came up out of the terrain that i did not have to interpret. a bench road, cut into the side of a steep ravine, running from the valley floor all the way up to the ridgeline. i knew that road. i had driven that road for years, on four wheelers up and down the mountain for fun, alone and with family, and later on a skidder working alongside my stepdad. one early spring i took a four wheeler up it while there were still patches of ice sitting on the shaded pitches, slipped off the outside edge, jumped off, and sat there watching the machine go end over end down the ravine wall until it landed in the brook at the bottom. i had eaten lunch at the turnarounds where the road widened out into working areas. i had done clearing at half the flat spots on that mountain.
and none of that was in my head when the line first appeared on the screen. the recognition arrived about a half second behind the seeing, the way a smell finds the memory before you can name it. i was not reading information off a map. i was looking at a road my own hands already knew, rendered out of a point cloud by an instrument that had never been anywhere near that mountain.
that half-second gap, between seeing the line and recognizing it as a road i had personally driven, is where this essay starts. because the question sitting underneath it is the one i put to every dataset i have ever touched: is what i’m seeing real?
calibration, not a trophy
it would be easy to tell this as a brag. the tool found a real road, look how good the data is, look how sharp the instrument is. that is not what it is. it is calibration, and the difference matters more than anything else i am going to say here.
here is the actual problem. a remote-sensing instrument lets you see ground you will never physically walk. not “have not walked yet,” will never walk: hundreds of thousands of parcels, whole counties, terrain measured in square miles you could not set foot on in ten lifetimes. so how do you earn the right to trust what it tells you about all of that? you cannot go check it. you can only ever walk some of it. and the some you should walk is not a random sample, it is the ground you already know, because known ground is the only place on earth where you can hold the instrument’s answer up against a truth you own independently of the instrument.
that is what a steepness view of my family’s mountain actually is. it is a calibration set. i already know that mountain in my body, every grade and turnaround and the exact place a four wheeler will get away from you in April. so when i run the instrument over it and the bench road comes back out, i have not learned anything new about the road. i have learned something about the instrument: it reproduces ground i can verify, which is the only reason i am allowed to believe it about ground i cannot.
the corpus of known ground
that one mountain is not the whole calibration set, it is just the clearest entry in it. the roads i cut for a logging job on a parcel up in northern Massachusetts, one of the tracts my stepdad wrote the management plan for and logged himself, showed up in the lidar the same way years later. various other properties i grew up moving through, ground i knew from spending my time in the woods rather than in front of a screen, check out the same way whenever i think to look.
every one of those is the identical test. known ground goes in, and i watch whether the instrument gives it back. and the corpus is the point, not any single entry in it. one road coming back is an anecdote, and an anecdote could just be luck, one good return on one hillside. the whole corpus coming back, different mountains, different jobs, different years, is a method, and a method is something you can actually measure your trust in.
and the method includes the misses. in fact the miss came first: the steepness raster that started this essay only exists because an earlier revision of the tooling failed this exact test. my mom’s property is a small parcel with a tall vertical retaining wall at its lower end, ground i have stood on plenty. that revision, built on ten-meter elevation data, gave no indication the wall existed at all. a wall taller than me, invisible to the instrument. that was not acceptable, and it is the reason everything got rewritten around one-meter lidar: when known ground goes in and does not come back, the answer is not to shrug and trust the instrument anyway, it is to find out exactly what the instrument cannot see, and then go get an instrument that can.
the loop running forward
that same loop, known ground in, does the instrument give it back, eventually gets pointed the other way. a feature surfaced in a one-meter dem in southwestern Vermont, ground i have never physically stood on, and i trusted it enough to send it to someone who reads terrain for a living, because the instrument that surfaced it had already proven itself to me on countless roads i had driven.
and even so, this summer i am going to go stand on it. on foot. which is the part that matters most, and the part most people skip.
the discipline that survives calibration
a calibrated instrument is trustworthy. it still does not get to decide.
this is the affirmative half of everything i believe about data, and it is the half that gets missed, because the skeptical half is louder and easier to say. the skeptical half is data is hostile until proven otherwise, the model does not get to judge, distrust it until it earns otherwise. all true. but pure skepticism ships nothing. at some point you have to actually use the data, and if all you have is distrust you are stuck. calibration is the bridge from distrust to use that does not pass through naivety on the way. you extend exactly as much trust as the instrument has earned against ground you can check, and not one inch past that.
so the calibrated dem is not the answer. it is a very good hypothesis, the best one i have, good enough to act on. the ground is still the authority. that is why i will go walk that feature even though i already believe it is there.
the same reflex is why i build the way i build. TALON, the terrain analysis platform i’m building around one-meter lidar for fourteen counties of western North Carolina, does its enrichment down at the data layer, so the ground truth is already sitting in the data before any judgment runs on top of it. and every validation gate in it is a small calibration check: data whose right answer i already know goes in, and the gate checks that it comes back out intact. known-ground-in, does-it-hold-up, asked in code. trust the instrument as far as it has earned, and check the rest against the actual ground.
close
the question was is what i’m seeing real, and the honest answer turned out to be a lifetime of known ground: roads i cut, grades i drove, turnarounds i ate lunch at, one ravine i watched a four wheeler tumble into. i trust the instrument because i spent years, without ever calling it that, checking it against terrain my body had already recorded. that is what the half-second gap at the start of this essay really was, trust arriving instantly because it had already been paid for. and i am still going to go stand on that Vermont hillside, because the instrument can only tell me what it sees, and the only way to know whether it is right is to go look. checking what the data says against what is actually there is most of what i actually do, in QA at the day job and in TALON on my own time.