I shipped an application this year that I cannot read.
Not the architecture. That part is mine, every decision of it. I chose how the pieces fit, what talks to what, where the data rests and where it moves, what the thing is even for. I can stand at a whiteboard and redraw the whole system from memory, explain why each part sits where it does, tell you what will happen when you poke it here. I know how it runs the way you know your own house in the dark.
But it's written in a language I'd never written before. If you sat me down, put a cursor on some line in the middle of it, and asked me to explain what that line does (or worse, to change it), I'm fairly sure I couldn't. The lines are not mine. The system is entirely mine. I've been turning that pair of sentences over for weeks, because I can't work out what it says about me.
The notebook test
The cleanest tool I have for the question is twenty-seven years old. In 1998 Andy Clark and David Chalmers published a short paper called "The Extended Mind" and asked you to imagine Otto, who has Alzheimer's and writes everything he needs to remember in a notebook. When Otto wants the address of the museum, he consults the notebook the way you'd consult your memory. Their claim (the one that made the paper famous) is that there's no principled reason to draw the boundary of Otto's mind at his skull. The notebook does the job memory does; by parity, it is part of his mind. If it would count as cognition were it happening in your head, it counts when it happens on the page.
Your editor passes this test without trying. I haven't held a full API signature in my head for years. The editor has them; I summon them with a keystroke; the knowledge is reliably there and I use it without friction. Kept in my skull we'd call that "knowing the library." Kept three inches away on the screen, it is the same knowledge in a different coat. The skin was never really the boundary.
The part nobody had to name
Otto's notebook had a property so obvious that the original paper didn't dwell on it: Otto wrote it. Every line in it was something he had once known, judged, and set down himself. The notebook could only ever hand back what Otto had already endorsed. It extended his memory because it was his memory: externalized, alphabetized, waiting.
The thing in my editor did not wait. It wrote. I described the system and it produced the lines, which I then read, ran, tested, and shipped, but did not author and, in the older sense of the word, do not know. My extension stopped storing my beliefs and started generating them. Otto's notebook never disagreed with him. Mine finishes my sentences before I've finished thinking them, and is usually right, which is the harder version of the problem, not the easier one.
So assess me
This is where it stops being a curiosity and becomes a question I can't answer. Last year José Hernández-Orallo published a paper arguing that the extended mind wrecks our entire apparatus for enhancement and assessment. We test people as though cognition stopped at the skin: the closed-book exam, the no-Google interview, the whiteboard with the marker and the watching eyes. The whole ritual is engineered to measure the unaided person. But the unaided person is increasingly a fiction. What we actually put into the world is the person plus their tools, and the pair behaves like neither half alone.
So assess me. Test me the old way: here is your codebase, explain this line, refactor that function, and I fail, flatly; I would fail a junior screen on the language my own production app is written in. Test me the new way: here is a problem, ship something that solves it, and I pass, because the system I built runs and serves real people. The same person, the same week, is both incompetent and competent, and the only thing that changed was where you drew the line around the word "me." The score measures the centaur and we keep writing it down next to the rider's name.
The same person, the same week, is incompetent and competent, depending only on where you draw the line around "me."
We don't have an honest test for this yet. We have tests that pretend the horse isn't there, and tests that can't tell the horse from the rider.
What I'm not sure about
I want to be careful here, because there's a flattering version of this essay that lets me off too lightly. Maybe "I designed it but can't read it" isn't a new kind of competence at all. Maybe it's the old kind, hollowed out. One recent paper makes that case directly, arguing that foundational knowledge stays indispensable: that you cannot truly direct what you could not, in principle, do yourself, and that the warm feeling of understanding-at-the-architecture-level is exactly the kind of confidence that shatters the first time the tool is wrong in a way you're not equipped to notice. I can't rule that out. Some night the app will break on a line I can't read, and I'll find out which essay I was living in.
What I'm left with is smaller than a thesis and harder to put down. The edge of a mind was always a convenience: a place we agreed to stop measuring. For Otto it sat just outside the skull, around a notebook he had filled himself. Mine now sits around a thing that fills itself, and I honestly can't tell whether I've extended my mind or quietly started renting someone else's. I shipped the app either way. That's the part that keeps me up: it works, and I'm not certain it's mine.
Sources
- The Extended Mind, Andy Clark and David Chalmers, Analysis, 1998 (Otto's notebook; the parity principle). Overview
- Enhancement and assessment in the AI age: An extended mind perspective, José Hernández-Orallo, Journal of Pacific Rim Psychology, Vol. 19, 2025. journals.sagepub.com
- The extended hollowed mind: why foundational knowledge is indispensable in the age of AI, Frontiers in Artificial Intelligence, 2025. frontiersin.org
Thanks for reading.If this resonated, the easiest way to support the project is to forward it to one person who'd like it, or to subscribe to the letter.