A familiar response to a proposed AI automation is: don’t automate a broken process. Harvard Business Review put it cleanly — automating one only amplifies its dysfunction at software speed. That leaves a practical question unanswered.
Before automating, how do you find out, quickly, what a better version of the process looks like — and where does the knowledge to build it come from? HBR also notes that the field lacks a disciplined method for judging whether a process is even good enough to automate. So we improvise. We Google. We ask the model. We quietly assume it picked up enough in training to cover whatever domain we happen to be in.
That leaves the automation dependent on training material you did not choose, inspect or make available for audit.
I would start by choosing the reference material. Collect the current documentation for the domain, organize it so individual claims can be found and checked, and give the model explicit access to it. The team can then see which knowledge the process uses, where it came from, and what needs updating. That makes reference material part of the infrastructure you own.
Why the Model Does Not Already Know This
A model’s broad training does not guarantee that it can reliably recall the specialist knowledge a workflow requires.
LLMs memorize unevenly. Carlini and colleagues found that models reliably reproduce frequent public text but recall specialized, rare, or recent content poorly — even when it sat in the training set. The model may well have “seen” a 2023 oncology protocol or some niche export-control rule. But seeing a thing once, in a firehose of trillions of tokens, is not the same as reciting it correctly on demand — the way skimming a textbook is not the same as passing the exam years later.
Nor is the gap a glitch that the next, bigger model quietly fixes. Stanford’s 2024 AI Index found frontier knowledge cutoffs lag deployment by six to eighteen months, and domain benchmarks in law, medicine, and finance stay stubbornly behind even as the general-purpose ones saturate. The distance between general capability and current, jurisdiction-specific knowledge does not close on its own. It is structural, not a phase.
If the model cannot reliably recall the knowledge the job needs, the team has to supply a dependable reference. A 2024 survey of retrieval-augmented systems concluded that retrieved knowledge sets the ceiling on output quality regardless of the underlying model — and that chunking, metadata, and plain index hygiene move quality more than a bigger model does. Even the arrangement matters: models use long contexts unevenly, dropping performance when the relevant passage is buried in the middle. Anthropic’s guidance also emphasizes the quality and arrangement of the supplied documents.
The natural objection is that all this is redundant. The model trained on the public corpus, so curating canonical references just re-shelves what it already knows. Partly true, for general topics. But recall of specialized public text is unreliable, and curation goes beyond aggregation. It means selecting material, separating it into checkable claims, cross-linking, and indexing — the work that turns “text the model maybe saw” into “reference it is provably grounded in.” And in regulated work you must show what the system was reasoning over, which training can never give you. There’s a reason the field renamed the discipline: prompt engineering became context engineering.
Where the Stakes Are Highest, This Already Works
Legal, medical and compliance workflows offer examples of systems built around curated reference material.
Start with law. Between 2022 and 2024, dozens of startups built legal assistants on frontier models, and most died in pilots when the models cheerfully fabricated case citations and missed jurisdictional nuance. (One such hallucination landed in a real court filing. It did not go well for the lawyer.) Harvey AI took a different bet: it built — and keeps expanding — a curated, indexed corpus of legal authority, tagged by jurisdiction, practice area, recency, and authority weight, holding the model constant as a swappable engine. By late 2024 Harvey had raised over $200M and served firms like Allen & Overy, attributing its reliability to that curated authority and reporting far fewer fabricated citations when answers are grounded in vetted sources rather than generated cold. The leading legal AI vendor is, underneath, a curation company.
Medicine built the template decades earlier. Mayo Clinic’s clinical pathways are a wiki-style library: atomized, evidence-anchored, version-controlled care plans, each atom dated, citation-backed, cross-linked, and reviewed on a cadence by a named owner. When a new trial changes practice, the relevant atoms update centrally and propagate everywhere; the work is staffed continuously, not project-funded. That gave Mayo an existing reference system to draw on when introducing generative AI.
The money follows the same logic. In BCG’s 2024 fintech survey, the compliance teams that captured real gains had built a structured, atomic, owned regulation wiki first, spending 60–70% of their effort on curation and 30–40% on integration — the exact inverse of how most firms scope these projects. McKinsey’s 2024 State of AI is the macro echo: the organizations capturing value were twice as likely to have invested in domain-specific knowledge bases than the peers who saw no return. Across these examples, the curated reference material contributes to the value of the system. This is the practical bridge between treating knowledge as infrastructure inside the organization and the observation that automating a broken process amplifies its dysfunction: curation is how you upgrade the process before AI scales it.
A Five-Step Audit Before You Automate
These steps make choosing and maintaining references someone’s actual job. Skip it for generic, low-stakes tasks where the model’s prior is genuinely good enough; it earns its keep in specialized, regulated, or audit-bound work, where a wrong answer is expensive and someone eventually asks how the system got there.
1. Identify the domain gap. Name the precise distance between your current process and what state-of-the-art looks like elsewhere — and have someone who has actually done the work outside your walls validate it. If you cannot articulate the gap, you are not ready to automate — better to learn that now than in month six.
2. Source canonical references. Assemble academic literature, regulator guidance, analyst reports, vendor playbooks, and the high-quality internal precedent currently siloed in someone’s inbox — each with provenance: date, authority weight, jurisdiction. Record those details alongside each source so its relevance and authority can be checked.
3. Atomize and cross-link wiki-style. Turn documents into atomic notes — one decision-relevant idea each, with citation, scope, effective date, owner, and links to related atoms. This makes individual claims easier to retrieve and update. Pairing the model with an owned, non-parametric index — reference material stored outside its learned weights — is, after all, the design that beat reliance on priors in the first place.
4. Wire the index to the model as explicit reference context. RAG, system-prompt anchoring, agent tool access — whatever the tool supports. The non-negotiable: every output must trace to the specific atoms behind it.
5. Operationalize audit and update as a continuous function. Named owners, a review SLA, event-driven updates, periodic external audit. Someone needs time and responsibility to keep the references useful after the project launches.
A hypothetical composite — call it GlobalChem, a chemicals manufacturer drawn from BCG’s observed patterns — shows the stakes. The team budgeted 70% to engineering, 30% to content. Six months in, the system ran fine technically, and yet the compliance director rejected over half the drafts: superseded regulations in the index, jurisdictions mis-tagged, internal precedent missing entirely. The failures came from the reference material rather than the software. They inverted the budget, rebuilt the wiki with dated, owner-assigned, cross-linked atoms, and the rejection rate fell from over half to under one in ten.
Doesn’t off-the-shelf RAG already do all this? It does the plumbing — chunkers, embedders, vector stores, a working system over a weekend. What it does not do is the selection, atomization, and currency of what enters the index. Those decisions need domain judgment, and that you cannot buy in a weekend, or at all.
Maintenance, cost and adoption
Won’t frontier models just absorb the curation cost? It’s tempting: spend nothing, write good prompts, let the next release make your wiki redundant. The risk is real — a heavy, static build genuinely can depreciate as models improve, and few teams want to own a museum. For general tasks, that worry usually wins. For specialized domains it doesn’t, because the lag is structural: Stanford’s six-to-eighteen-month domain gap persists even as the general benchmarks saturate. The answer isn’t to skip curation; it’s to design it as a living artifact — atomic, versioned, swappable — so it compounds rather than rots. When the process is the product, reproducibility and auditability aren’t optional.
Won’t the wiki go stale faster than anyone can maintain it? Yes — if you fund it as a project. Stale curation grounding a confident model is genuinely worse than no curation; it gives a hallucination the institutional weight of “our wiki said so.” Maintenance needs assigned owners and a funded review process. Mayo’s pathways survive on named owners, dated entries, and a review cadence; BCG’s value-capturers budgeted most of their effort on exactly this maintenance. Without an owner, the reference material will lose its reliability.
Isn’t adoption the real constraint? Often, yes — first. Most AI initiatives die on change management long before they reach a curation ceiling. But the two are sequential, not competing: McKinsey’s own data shows curated context is the differentiator among the organizations that already cleared the adoption gate. Adoption is gate one. Curation is the next, not a rival for the same budget line.
Before approving the project
Before you greenlight it, write down two things: the specific domain gap it depends on closing, and the canonical references that would close it. If those two artifacts don’t exist yet, the project needs more preparation before automation begins.
Then flip the budget. If your initiative is 70% engineering and 30% content, the proportions are upside down. Build for audit from day one: if you cannot trace an output back to the atoms it reasoned over, you do not have a defensible automation. And name the curator — whoever owns the index a year from now is quietly the most important hire on the project, whatever their title.
The person relying on an answer should be able to trace it to a relevant source. Make sure someone has the time and responsibility to keep that source current.
References
- Lewis et al. (2020). “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS. arxiv.org/abs/2005.11401
- Carlini et al. (2022). “How much do language models memorize?” Google Research. arxiv.org/abs/2202.07646
- Liu et al. (2023). “Lost in the Middle: How Language Models Use Long Contexts.” TACL. arxiv.org/abs/2307.03172
- Wang et al. (2024). “Best Practices for Retrieval-Augmented Generation: A Survey.” arxiv.org/abs/2407.01219
- Stanford HAI (2024). “AI Index Report 2024.” aiindex.stanford.edu/report
- McKinsey (2024). “The State of AI in 2024.” mckinsey.com
- Davenport & Mittal (2024). “Don’t Automate Broken Processes.” Harvard Business Review. hbr.org
- Harvey AI (2024). “Engineering Blog.” harvey.ai/blog
- Mayo Clinic (2023). “Clinical Pathways and Evidence-Based Order Sets.” mayoclinic.org
- Anthropic (2024). “Long Context Prompting and Reference Documents.” anthropic.com
- Boston Consulting Group (2024). “Scaling GenAI in Financial Services.” bcg.com
- Latent Space (swyx) (2024). “Context Engineering.” latent.space