The Debate That Goes Nowhere
Every few weeks someone sends me a thread arguing about whether AI “really” reasons. One side says these models think now — bar exam, olympiad math, reflecting on their own answers like a careful student. The other side says it’s autocomplete with a longer runway, a stochastic parrot in a nicer suit. Both sides are certain, both cite papers, and neither has helped anyone decide what to actually do with the thing on their desk.
I work with these models every day, and this argument has calcified into a binary that produces no useful guidance. Believe the model reasons, and you over-trust it and ship a confidently-wrong answer into a decision that mattered. Believe it just predicts tokens, and you under-invest and miss the work it does genuinely well. Same false dichotomy, two failure modes, and the question stays exactly where it was.
There’s a third description, and it’s sharper than either pole. The model doesn’t reason in the classical sense, and it doesn’t merely predict text. It simulates what reasoning looks like — well enough, often enough, to produce real reasoning behavior. It is a simulator of thought.
Once you hold that, the philosophy argument stops being interesting and a design problem walks in to take its chair.
What a Simulator of Thought Does
Classical reasoning systems manipulate explicit rules, variables, and states. You can read the logic and prove the output follows from the premises. Language models do nothing of the kind. They generate token by token, using learned representations and heuristics shaped by training [1] — no rule applied, no variable bound, just a trajectory traversed.
And yet the outputs increasingly behave as if reasoning has occurred. The cleanest way I’ve found to reconcile that is to see the model as a simulator: it has watched enough examples of how valid reasoning unfolds that it can reproduce those trajectories generatively [1]. A flight simulator reproduces aerodynamics well enough to train a pilot who never leaves the ground. It is not flight. It is a faithful enough model of flight that the difference stops mattering — right up until it does.
Reasoning tokens make the simulation visible. Models like o1 and o3 generate thousands of intermediate “thinking” steps before answering, and performance scales with how much of this you let them do [3]. That only makes sense if the model is searching a learned reasoning landscape, not running a fixed procedure — you don’t make a calculator more accurate by giving it time to think.
It gets stranger. Early work points the same way: one 2026 study of more than 8,000 reasoning problems reports that models like DeepSeek-R1 generate distinct viewpoints that argue with each other inside a single chain of thought, with dialogue-trained models reaching 38% accuracy where monologue-trained ones stalled at 28% [13]. The model is, in effect, simulating a small committee — one that has never held a meeting, because it has no members.
87.5% Human, 12.5% Revealing
ARC-AGI is built to resist pattern matching. Each problem demands a novel rule inferred from a handful of examples, the kind of abstraction that memorization shouldn’t be able to fake. The human reference score is 85%. OpenAI’s o3 scored 87.5% [6]: a simulator of reasoning beat the human average on a test meant to be hard for anything that only matches patterns.
The impressive number isn’t the story. The 12.5% is. When a simulator of reasoning fails, it doesn’t fail like software. It produces near-sense, not nonsense — confident, well-structured, plausible answers that happen to be wrong [7]. A traditional bug throws an error; a simulation failure hands you a beautifully argued conclusion built on a trajectory that went sideways three steps back and never flinched.
Consider Zalando, one of Europe’s largest fashion platforms. They deployed LLM reasoning for root cause analysis across their microservices — read the logs, correlate the events, name the culprit. The system was very good at writing root cause analyses, and wrong about ten percent of the time, in a specific and slightly comic way: if a log mentioned Kafka next to a database timeout, the model blamed Kafka — because Kafka appeared in the text, not because it caused anything [9]. It had learned the pattern of causal analysis without ever performing the analysis. The reports read perfectly. That was the problem.
Zalando’s fix is the whole thesis in miniature. They kept the model parsing logs — what it simulates well — and handed causal attribution to a deterministic dependency graph that traces the actual service calls [9]. The simulator did the reading. A formal system did the part that had to be right.
There’s a test you can run yourself, and it costs you one sentence. Chain-of-thought reasoning is fragile: add a semantically irrelevant line — a distractor a formal reasoner would ignore without blinking — and accuracy can drop 20 to 40% [5]. The logic is unchanged; only the surface noise moved. If irrelevant information derails the reasoning, the reasoning was following pattern trajectories, not logical rules. That isn’t a gotcha. It’s a stethoscope.
Why Scaling Makes the Simulation Better, Not Different
The obvious objection is that this is temporary — models keep improving, surely the gap closes. It does close, and that’s exactly the point: it closes by improving the simulation, not by converting it into something else. As models scale, the reasoning landscape gets richer and smoother, so the simulator lands on a correct trajectory more often [2]; capability density is even doubling roughly every 3.5 months [8]. But the thing that tells you the nature hasn’t changed is the failure signature. The errors keep their simulator shape no matter how much smoother the landscape gets — cheaper, more faithful, still a simulation.
There’s a tempting parallel here, and it’s worth sitting with rather than swatting away. Predictive-processing theories in neuroscience argue that human cognition is also a kind of simulation — the brain generates predictions and corrects them against sensory evidence. If that’s true, then calling these models “simulators” doesn’t separate them from us; it links us. Which is a slightly uncomfortable thing to notice about yourself mid-sentence.
I think the parallel is right, and it strengthens the operational case rather than weakening it. But the difference is real, and it isn’t flattering to the machine. Human simulation is wrapped in embodied feedback and metacognitive monitoring — the doubt, the surprise that tells you you’ve gone wrong, the small itch that makes you recheck. Models don’t reliably have that. A reasoning model can be maximally confident and maximally wrong in the same breath, with no internal alarm going off. We even have a number for how little the visible reasoning tells us: chain-of-thought traces reflected a model’s actual internal process only 25% of the time for one frontier system, and 39% for another [11]. The “thinking” you read is itself partly a simulation of thinking.
So the new category isn’t “simulation as reasoning” — nature got there first. It’s simulation as reasoning at machine speed, at organizational scale, without a human’s self-correcting instincts. That is the thing we haven’t had before, and it’s worth more than a Twitter argument about whether it counts.
The Objection That Almost Wins
One objection challenges whether the framing explains anything; another asks whether improving models will make it obsolete.
First, the falsifiability objection. Some say the framing is unfalsifiable — reason well and it “simulated well,” fail and it “simulated poorly,” a claim that explains everything and therefore explains nothing. But it makes testable predictions: failures look like trajectory errors, performance rises with simulation steps, formal guarantees stay absent regardless of scale. Each is checkable. So far, each has held.
The objection that almost wins is convergence. If understanding is emergent from prediction at scale, and o3 already matches humans on a pattern-resistant test, maybe “simulation” is a distinction with a shelf life of months — quaint by the time you finish reading this.
I take it seriously. But “almost” is carrying real weight in that sentence. Apple’s controlled-puzzle research found that frontier reasoning models suffer a complete accuracy collapse beyond certain complexity thresholds — and, stranger, that they reduce their reasoning effort as problems get harder, even with token budget to spare [12]. That is not a system smoothly approaching general reasoning. It’s a simulation hitting the edge of its training distribution. The same models that overthink the easy problems — one study clocked 1,953% more tokens than conventional models to reach the same answer [14] — quietly give up on the genuinely hard ones. Formal reasoners grind. These models perform grinding, then stop.
Convergence doesn’t dissolve the distinction, because the failure signature persists. As long as the errors are confident-but-subtly-wrong rather than provably-incorrect, the simulator framing drives different risk management and a different verification architecture [4]. The label doesn’t need to survive forever. It needs to be useful during the convergence period — which, given that 12.5% gap and the shape of those collapse curves, could last a decade.
Designing Around a Simulator
Accept the framing and the question becomes architectural. The most effective production systems already combine LLMs, traditional ML, and deterministic rules, each doing the job it’s good at [10]. And yes — the obvious response is to just wrap the simulator in a formal verifier and watch the problem evaporate. It does evaporate, for bounded domains: code correctness, math proofs, schema validation. It does not evaporate for the open-ended reasoning where these models actually earn their keep — strategy, analysis, the judgment calls — where there is no formal checker to wrap around the answer in the first place. A verifier helps only where the required answer can be checked against explicit rules. Open-ended work still needs someone to judge whether the reasoning fits the situation. Five moves make the rest deliberate.
Map the reasoning landscape. For every task where you expect the AI to reason, ask one question: if this output is subtly wrong but convincing, what does it cost? Sort into three tiers — simulation sufficient (drafting, summarizing), simulation with verification, and formal reasoning required (causal analysis, compliance, financial calculations). Most teams skip this and govern an email draft and a financial analysis with the exact same trust. Then a near-sense error surfaces somewhere it mattered, and the reflex is to distrust AI entirely — when the real mistake was miscategorizing the task.
Instrument the boundary. Run perturbation tests and watch for instability. The trap here is trusting the model’s confidence as your quality signal: a simulator of reasoning also simulates confidence, so it can be perfectly certain and perfectly wrong at the same time.
Design the verification architecture. For the higher tiers, have the LLM generate while a formal system validates — never the reverse. “Human in the loop” fails the moment it becomes a checkbox: the reviewer approves confident, well-structured output precisely because the simulation is persuasive enough to disarm the scrutiny it needed.
Calibrate, then evolve. Keep a simulation failure log and review it for patterns, not incidents. The boundary moves as models improve, so re-run the perturbation tests quarterly and downgrade a tier only on sustained evidence — not on a launch announcement. In regulated domains, or greenfield deployments with no error history to learn from, start at maximum verification and earn your way down.
What This Leaves You With
We have machines that can simulate thought well enough to matter, but not well enough to stand alone. That sentence is the whole operating manual.
For the work on Monday, the operational question is where a convincing error would matter. Map your tasks by the cost of a confident error instead. Run the perturbation test on your highest-stakes workflow this week — add one irrelevant sentence and watch the answer move.
What we have is stranger, and more useful, than either side of the old binary: a new category of intelligence that performs reasoning convincingly enough to do real work, and fails convincingly enough that you have to design for it. Treat it like a brilliant, tireless colleague who is never quite sure when they’re wrong. You wouldn’t fire that colleague. You also wouldn’t let them sign off on the financials alone.
References
[1] Shanahan, M., McDonell, K., & Reynolds, L. (2023). Role-Play with Large Language Models. Nature, Vol. 623; Janus (2022). Simulators. AI Alignment Forum. [2] Lizarraga et al. (2025). From Stochastic Parrots to Digital Intelligence. WIREs Computational Statistics. [3] OpenAI (2024). Learning to Reason with LLMs. [4] Formal Reasoning Meets LLMs (2025). Communications of the ACM. [5] Large Language Model Reasoning Failures (2026). arXiv. [6] OpenAI (2024). Learning to Reason with LLMs; ARC Prize, o3 Breakthrough. [7] Large Language Model Reasoning Failures (2026). arXiv; Reliability for Unreliable LLMs (2025). Stack Overflow Blog. [8] Densing Law of LLMs (2025). Nature Machine Intelligence. [9] ZenML, What 1,200 Production Deployments Reveal About LLMOps in 2025; Stack Overflow Blog (2025). [10] Reliability for Unreliable LLMs (2025). Stack Overflow Blog. [11] Scrutinizing LLM Reasoning Models (2025). Communications of the ACM. [12] Apple Machine Learning Research (2025). The Illusion of Thinking. [13] Reasoning Models Generate Societies of Thought (2026). Google / University of Chicago / Santa Fe Institute. arXiv. [14] IBM (2025). What Is a Reasoning Model?, citing a Tencent study.