TL;DR
- The illusion of explanatory depth is a documented bias where people feel confident they understand something, a zip or a bicycle or a piece of legislation, until asked to walk through it step by step.
- It applies to AI-generated work: a presenter fumbled through a deck generated for the talk rather than by him, because the first test of his understanding happened live in front of people.
- The habit to build is interrogating anything a model hands you before accepting it, including pointing a model at your own reasoning and asking it to find the unchecked assumption or the skipped step.
- Nobody in the room could have said the deck’s content was wrong; what failed was the presenter’s ability to account for it unscripted.
In plain English
The illusion of explanatory depth was first studied on mechanical objects, and the gap it exposes appears in real time, surprising the person who has it. It applies to anything you did not build yourself and have never had to account for out loud, which now covers a large share of what appears on a screen in front of us.
The counter-habit borrows from red-teaming, which means deliberately attacking your own work to find where it breaks. Here that means asking a model to take apart your reasoning rather than support it. The test is explaining your work unscripted to someone who will ask a follow-up you did not prepare for, and watching for the point where you begin hand-waving, which marks where you grasped the shape of something without the substance.
There’s a documented bias called the illusion of explanatory depth: people are confident they understand how something works — a zip, a bicycle, a piece of legislation — right up until you ask them to walk through it step by step, at which point the gaps show up in real time and surprise the person who has them (The Decision Lab). It was first studied on mechanical objects. It applies just as well to anything you didn’t build yourself and have never had to account for out loud, which increasingly describes most of what shows up on a screen in front of us.
I watched a live version of this recently — a young presenter working through a deck that had clearly been generated for the talk rather than by the person giving it, and it showed. He fumbled his way through slide after slide because he hadn’t sat with the material closely enough to know what was coming next, and the deck had no patience for that — it just kept advancing whether he was ready or not. It wasn’t that the content was wrong. Nobody in the room could have told you that from the outside. It was that the first time his understanding of his own talk got tested was live, in front of people, and it failed the test in real time.
I try to build that test into my own work before anyone else gets to see it. When a model hands me something (a plan, a script, an analysis) my first move is to go find out what it actually is. Increasingly I’ll deliberately point a model at my own reasoning and ask it to take it apart: where’s the assumption I haven’t checked, where’s the step I’ve skipped past because it felt obvious. It’s an uncomfortable habit on purpose, and the discomfort is how I find the thin patches in my own understanding before a client does.
That’s the actual test for anything you build for someone else, AI-assisted or not: can you explain it, unscripted, to a person who’s going to ask a follow-up question you didn’t prepare for. Not “I could explain this if pressed” — actually doing it out loud, and noticing exactly where you start hand-waving. Those are the spots where you understood the shape of the thing but not the substance. Better to find that out at your own desk, with nobody watching, than on a deck you haven’t rehearsed.