What the explanation leaves out
Every week, someone tells me modern AI is “just pattern matching.” The phrase is often treated as a complete explanation.
The phrase is technically defensible. These systems train on huge corpora, learn statistical structure, and generate outputs by navigating probabilities rather than applying symbolic rules. That describes an important part of their construction.
To decide what to trust it with, we also need to look at the behavior that training produces.
Geoffrey Hinton has made this sharper than I can. Traditional autocomplete works on n-gram statistics — the probability that one specific token follows another in a window. That is not how a modern language model works. It takes words, converts them to features, lets the features interact in a high-dimensional space, and uses the derived features to predict the next word. The metaphor we keep reaching for to dismiss these systems is a mechanism they do not use.
Calling them “reasoning” is no better. They do not manipulate explicit variables. They do not maintain formal state. They provide no guarantees of logical completeness. A formal reasoner that gets a problem wrong runs out of time. A modern AI model can get it wrong while sounding confident — sometimes more confident than when it gets it right.
So we are stuck between two bad framings, and both produce predictable deployment mistakes. Underestimate, and you miss where the system is useful. Overestimate, and you deploy it where it does not belong. Most production failures I see stem from the wrong category, not the wrong prompt.
What Emerges When Pattern Learning Gets Rich Enough
One example comes from a model trained on a board game.
A team at Harvard trained a GPT-style transformer on nothing but sequences of Othello moves. Raw tokens. No board representation. No game rules. No spatial encoding. The model only ever saw which moves followed which, and was asked to predict the next legal move.
Pattern matching predicts you would learn move co-occurrence statistics. What researchers found, by probing the model’s hidden states, was an internal representation of the actual board. Nobody told it what a board was. It built one anyway. The representation was linearly decodable and manipulable with vector arithmetic — flip a cell in the model’s internal map, and its next move shifted accordingly. The board model was causally active, not decorative. The result replicated across GPT-2, Mistral, LLaMA-2, and Qwen — different architectures from different labs converging on the same emergent structure.
The internal board representation goes beyond move co-occurrence statistics.
A second example tests combinations of language skills. Princeton’s SkillMix test asks a model to produce coherent text combining 5 or 6 randomly selected language skills at once — argument, irony, metaphor, a specific topic, a specific tone. Probability calculations showed many of GPT-4’s correct compositions involved skill combinations almost certainly not present in its training data.
A lookup table cannot interpolate to a combination it has never indexed. The follow-up sharpened the point: training on simple two- or three-skill combinations improved performance on four- and five-skill combinations the model had never seen. That result demonstrates transfer beyond the combinations used in training.
A third example comes from mathematics. By 2025, reasoning-tuned LLMs from OpenAI, Google, and DeepSeek had reached gold-medal performance on International Mathematical Olympiad problems — designed each year to be novel and to require applying learned techniques to unseen structures. The IMO is not a benchmark you memorize. It is one you generalize on.
These results challenge an account based only on memorizing and retrieving examples.
Where the Approximation Shows Its Seams
Other results show why their performance should not be treated as formal reasoning with correctness guarantees.
A 2025 study asked frontier LLMs — GPT-4o, o1, Gemini 2.0 flash thinking — to handle physical concepts in two formats. Verbal description: easy, fluent, accurate. Grid-based spatial reasoning about the same concept: roughly 40 points behind humans.
The split is the signal. A pure pattern matcher would fail on novel verbal problems too. A formal reasoner would not show modality-dependent performance on the same concept. What we observe is a system strong where its compressed representations are rich (language patterns abundant in training data) and weak where they are sparse (grid-based physical simulation).
Apple’s 2025 work added a different seam. Tested across complexity bands, frontier reasoning systems showed three regimes: standard LLMs beat reasoning models on easy tasks; reasoning models pulled ahead in the middle band; both collapsed on hard tasks. Reasoning effort rose with complexity up to a point and then declined, even when the model had inference budget left. That is what a bounded approximator does when the search space exceeds its representation.
Nature put the boundary in plainer language: current AI systems lack the ability to engage in abductive reasoning, grasp analogies and metaphors the way a human does, or interpret sparse data.
A further limitation concerns generalization and hallucination. OpenAI proved a formal generalization-hallucination tradeoff: any model that generalizes beyond its training data will either hallucinate invalid outputs or suffer mode collapse on the full range of valid ones. The mechanism that lets these systems compose novel skills also lets them confabulate with confidence. Standard evaluation makes this worse by rewarding confident guessing over calibrated uncertainty.
This is not a bug. It is a property of the architecture. Generalization and hallucination are mathematically coupled.
So we have a system that generalizes in ways memorization cannot explain, fails in ways formal reasoning would not produce, and is provably confidently wrong some fraction of the time.
A New Cognitive Category, Not a Middle Ground
Three results, three signatures of approximation. Representation-dependent strength. Formally coupled hallucination. Bounded deliberation that declines instead of finishing. None of these is what a pattern matcher does. None is what a formal reasoner does. The category is sitting in the shape of the failures.
These models do not know rules explicitly. They internalize patterns of how good solutions unfold, then reproduce those trajectories generatively.
By “learned approximations of reasoning over compressed world representations,” I mean that the model uses patterns learned about the world to generate a path through a problem. This account has specific consequences: performance depends on how the problem is represented, new skill combinations become possible as models scale, and generalization comes with confident errors.
A 2025 WIREs Computational Statistics survey traced the field’s evolution from “stochastic parrots” to systems exhibiting cognitive capabilities and argued this is a qualitative shift requiring new frameworks — neither parrot nor AGI captures what these systems are. Work from DeepMind and others pushes the same idea from the compression side: language modeling is itself compression, and better compression of real-world data tracks deeper world knowledge. The internal representation is not a pile of statistics. It is a compressed approximation of the world the data describes.
This framing treats the models as a distinct kind of system, with capabilities and failure modes that need to be tested together.
Where Does This Break?
An engineering objection is that the distinction may not change deployment decisions.
If a system consistently solves novel problems — unseen math competitions, working code for new specs, analysis of unfamiliar business cases — arguing about whether the internal process is “real” reasoning or “just” pattern matching is philosophy. Pattern matching at scale, the argument goes, is functionally indistinguishable from reasoning. Unless you can name a task formal reasoning handles and scaled pattern matching cannot, the “learned approximation” label adds overhead without changing how I deploy.
For many business applications, this is correct. If you are using an LLM to draft marketing copy or summarize meeting transcripts, the category debate does not affect what you do next.
It does affect what you do when things go wrong. If these systems were “reasoning at scale,” their failures would be resource-bounded — timeouts, exhausted context, clean abandonment. What the physical-reasoning study shows is the opposite: a split profile, strong on the same concept in one format and ~40 points behind humans in another. Pure reasoners do not get half the problem right and half spectacularly wrong while sounding equally sure about both. The framing becomes useful when you ask: why did it fail this way?
A second objection: this is the stochastic parrot critique with extra steps. The distinction is empirical, not rhetorical. The parrot thesis predicts performance plateauing on combinations outside the training distribution. SkillMix shows the opposite. The hallucination tradeoff goes further: a system that generalizes beyond its training set must be doing more than recombining memorized patterns, because mode collapse on novel inputs would be the alternative.
A third objection: this is a snapshot dressed up as a framework. Chain-of-thought, tool use, agentic harnesses, and bigger models are closing the gaps right now. The 40% deficit might be 10% by the time you read this.
Granted on the benchmark. The framing is not about accuracy levels but about the category of process. The generalization-hallucination tradeoff is a formal mathematical result, not an engineering target. Unless we change the underlying mechanism — next-token prediction over compressed representations — improved scores do not change what kind of cognitive system this is. Benchmarks tell you how good the approximation is getting. They do not tell you it has stopped being an approximation.
What This Means for How You Deploy
In my daily work with these systems, I use five checks before deployment.
Probe. Find out what the system has learned internally. For frontier models you cannot open up, behavioral probes — tasks that require internal structure to solve. For open systems, probing classifiers and causal interventions like the ones that exposed Othello-GPT’s board. A memorizer and a generalizer look identical until the input distribution shifts.
Test for generalization, not recall. Build compositional tests for your domain. Present known elements in statistically novel combinations and watch performance degrade. SkillMix is the template. Most teams test inside the training distribution and conclude their system “understands” things. It does not, in any sense that survives a distribution shift.
Map how it fails. Where does it break? Formal reasoners fail at resource limits. Pattern matchers fail at distribution edges. Approximate reasoners fail in representation-dependent ways — strong where the compressed signal is rich, weak where it is sparse. Each failure is not a bug to patch. It is a signal about what kind of system you have.
Bound the missing guarantees. Approximate reasoners can be capable. They still lack formal guarantees of correctness, completeness, and consistency. For each task: how reversible is failure? How costly? How verifiable? For each gap, a specific mitigation — human review, verification layer, confidence threshold, deterministic check. (Trust is not an instruction — it is a property of the system around the model.)
Match task to cognitive mode. Most production systems work best as hybrids — AI for interpretation, synthesis, and generative breadth; deterministic software for exactness and verification. Treat the model as a formal reasoner and you get hallucinations in production. Treat it as glorified search and you leave value on the table.
HBR captured the organizational version in 2026: executives experience AI as strategic advantage; managers confront its flaws inside real workflows, without the time or support to compensate. The boardroom sees clean outputs and assumes formal reasoning. The middle layer sees confidently wrong outputs and assumes the technology is broken. Both are wrong about the same system — an approximate reasoner doing exactly what the architecture predicts.
The Question Has Changed
The debate I keep getting pulled into is whether these systems “really” think. For a deployment decision, I would start with a more concrete question: how do they get this job wrong, and what would that mean for the person relying on the result?
Map where your system breaks. The shape of its failures will tell you what it is. Use those results to set checks and limits around deployment.
The interesting debate is no longer whether these are pattern matchers. It is what kind of intelligence emerges when pattern learning becomes rich enough to simulate reasoning at useful scale.
Someone will have to act on the answer, catch the error, or live with its consequences. Understanding how the system fails gives that person a better basis for deciding when to rely on it.
References
- Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2023). “Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task.” ICLR 2023. arxiv.org/abs/2210.13382
- Arora, S., et al. (2023). “Are Language Models Mere Stochastic Parrots? The SkillMix Test Says No.” Princeton Language and Intelligence. pli.princeton.edu
- Kalai, A. T., Nachum, O., Vempala, S., & Zhang, Y. (2025). “Why Language Models Hallucinate.” OpenAI Research. arxiv.org/abs/2509.04664
- Hinton, G., quoted in Bing Qian (2026). “Geoffrey Hinton: What Language Models Really Understand.” Medium. medium.com/@bingqian
- “The Stochastic Parrot on LLM’s Shoulder: A Summative Assessment of Physical Concept Understanding” (2025). arXiv / OpenReview. arxiv.org/abs/2502.08946
- Arora, S., et al. (2024). SkillMix transfer learning follow-up. arxiv.org/abs/2409.19808
- Raschka, S. (2025). “State of LLMs 2025” — summarizing IMO gold-medal performance across OpenAI, Google, and DeepSeek reasoning-model announcements. magazine.sebastianraschka.com
- OpenAI (2025). “Why Language Models Hallucinate” — evaluation procedures rewarding confident guessing. openai.com/index/why-language-models-hallucinate
- Nanda, N. (2023). “Emergent Linear Representations in World Models of Self-Supervised Sequence Models.” BlackboxNLP at EMNLP 2023. arxiv.org/abs/2309.00941
- Lizarraga et al. (2025). “From Stochastic Parrots to Digital Intelligence: The Evolution of Language Models and Their Cognitive Capabilities.” WIREs Computational Statistics (Wiley). wires.onlinelibrary.wiley.com/doi/10.1002/wics.70035
- Apple Machine Learning Research (2025). “Understanding the Strengths and Limitations of Reasoning Models” (“The Illusion of Thinking”). machinelearning.apple.com/research/illusion-of-thinking
- “Artificial Intelligence and the limits of reason: a framework for responsible use in public and private sectors” (2025). Nature Humanities and Social Sciences Communications. nature.com/articles/s41599-025-05749-0
- Morris, J. (2024). “All AI Models Might Be The Same” (summarizing DeepMind “Language Modeling Is Compression”). blog.jxmo.io/p/there-is-only-one-model
- “Managers and Executives Disagree on AI — and It’s Costing Companies” (2026). Harvard Business Review. hbr.org/2026/04/managers-and-executives-disagree-on-ai-and-its-costing-companies