When Inference Isn't Good Enough
AI infers. The more context you give it, the closer it gets. The more you push back on its answers, the more you can guide it to something useful. What it cannot do is tell you how confident to be in the answer it just gave you.
The Tony Stark problem
Tony Stark tested the Iron Man suit in every scenario he could construct, including the one where he pushed it into the upper atmosphere and experienced a catastrophic shutdown. When he later fought the copycat pilot, someone who had the hardware but not the field work, Tony lured him up high. Just before the suit died and sent the copycat tumbling back to earth, Tony asked: did you work out that one bug?
Knowing where a system fails is a structural advantage over someone who only knows what it can do.
Who’s auditing the inputs
The standard worry about non-experts using AI is that they can’t recognize a bad answer. That’s real. But there’s a prior problem: non-experts also can’t construct good context.
Ask the wrong question with the wrong framing and you get a confidently wrong answer at both ends of the pipeline. The model is doing its job on the prompt given. The prompt was just the wrong one. The output looks plausible because it’s fluent and it’s answering something. Just not the thing that actually needed answering.
Someone has to audit the inputs, not just the outputs. In practice, that person needs domain knowledge.
The confidence gap
Humans make inferences constantly. What we carry alongside those inferences is a running score: some rough internal probability that scales with experience, adjusts when something smells off, and influences when we slow down or stop. It’s not always right, but it persists and it updates.
LLMs have token-level confidence at generation time. That signal doesn’t survive once the token is committed, and it doesn’t aggregate into anything that tracks the claim the tokens assembled. A model can be statistically certain about each word in a statement that is conceptually wrong. There’s no persistent thread of “I’m uncertain here” running underneath the paragraph.
The human in the loop isn’t a formality. They’re tracking multiple sources and carrying the running score. AI can’t replicate this yet. The challenges are layered: breadth of knowledge, access to the right data at the right time, judgment about what’s relevant, memory that persists across contexts, and the ability to recognize when something needs a second look.
Stakes and reversibility
Inference errors in subjective work are cheap. An AI-generated draft you reject costs time and tokens. An AI-assisted opinion piece that misses the mark gets revised. And artwork is mostly subjective, except for too many fingers or something obviously wrong.
The math changes when the decision is critical and one-directional. Courts have sanctioned attorneys for filing briefs that cited AI-hallucinated cases that don’t exist. Patients have received incorrect dosage information from AI-assisted clinical tools. Healthcare, legal, and financial regulators are still working out who’s liable when AI-assisted decisions fail. The early signal is that “the model said so” is not going to hold as a defense.
The Iron Man analogy lands here too. The copycat’s confidence was unfounded because he hadn’t done the work of finding the edge cases. Worse, he didn’t even know to check. High-stakes domains have edges. Someone needs to know where they are.
The sandboxable exception
Coding is the clearest exception. When the model writes code, you can run it. The environment validates, errors surface, the model iterates. The sandbox is built in.
This is why AI looks so capable in software contexts. Not because code is simpler than medicine. The failure mode is mostly containable and observable before it matters. But the same local confidence that makes individual tokens unreliable applies to code: the model can be certain about each function or block and still lose coherence across the whole architecture. Left unchecked, that produces spaghetti code. Providing guidelines helps, and the failure mode is recoverable through refactoring and testing, but the model has no internal process for checking whether the pieces fit together.
Most high-stakes domains don’t have this. You can’t sandbox a surgical decision. You can’t iterate a filed legal brief after the ruling. World models are one attempt to bridge the gap. They try to predict the outcome of a decision before it’s made, acting as a kind of synthetic sandbox. When trained on the right data, they can produce real and useful predictions. But they still can’t map which factors cause which outcomes, or account for variables the model itself doesn’t know are missing.
How We See It
Using AI on anything iterative and low-stakes can be helpful. Let it draft, generate options, surface angles you hadn’t considered. Push back and guide it. The more you probe, the more it converges toward something useful. Test its assumptions, run separate verification searches on anything critical, and learn to spot the gotchas. Pay particular attention to vocabulary and the assumptions the AI is making when it draws inferences. Where possible, create ways to test before committing.
On anything critical, one-directional, or regulated: the model is a tool and the human expert is the requirement. The gap between those two roles is not a workflow inefficiency. It is the mandatory piece that turns something that seems to work on the surface into something that deeply considers all ramifications. An LLM’s attention mechanism does not cross context boundaries unless you manually feed it, and even then context is limited. Until that changes, it is the expert’s job to pull together the separated knowledge and make the full assessment.