Graepel is right: serious knowledge work needs more than a chat window

The short version

The body of this piece is written in the third person and authored by Claude with human guidance. Blocks headed Bursera’s Take are written by Hugh McCutchen, in his own voice (with some edits from Claude). Blocks headed Graepel are direct quotations from his op-ed. The block headed Claude Says is Claude’s own closing view, written in the third person; there, “the Model” means Claude.

Thore Graepel’s headline states that large language models (LLMs), the models behind AI chat tools, do not reason [S1]. This piece does not dispute that.

Graepel · Sophisticated prediction, not reasoning

“A large language model picks the next token, over and over. That amounts to system 1 in action—fast, associative, and surprisingly good pattern completion across almost every subject people write about.”

— Thore Graepel, MIT Technology Review [S1]

System 1 is the psychologist Daniel Kahneman’s term for fast, intuitive thinking; system 2 is slow and checks its own work [S4]. What an LLM predicts depends on the patterns it learned across billions of pages of human writing and on the context it is given in the conversation. That context is where a prediction becomes useful or sensible. When a model explains its reasoning, the explanation is another prediction: a plausible account of how an answer like this would be reached, produced by the same process that produced the answer. It is not a replay of the steps taken. People do much the same when asked afterward to explain a decision. They give a sensible account, though they rarely followed a method laid out in advance. Anthropic planted hints in prompts and checked whether the model’s written reasoning admitted using them; Claude 3.7 Sonnet mentioned the hint 25% of the time [S5]. Graepel concedes that step-by-step prompting improves results in mathematics and code. His point is that it adds no separate reasoning mechanism [S1].

Graepel · Bigger isn't reasoning

“I do not think we reach trustworthy machine intelligence by making system 1 bigger.”

— Thore Graepel, MIT Technology Review [S1]

The practical conclusion: no matter how good the model is, serious knowledge work needs more than a chat window on its own. It needs the model inside a larger system that keeps a record, checks the work, and is led by a person.

Graepel’s own answer is a different design. It has no single established name; the builders of such systems describe each one differently, and Graepel says his direction “draws on AlphaGo’s architecture” [S1]. A plain description is propose-and-verify: a model proposes each step, an independent part of the system tests it against evidence, and the system keeps an explicit record of what is settled, what is doubted, and what is still open [S1]. Where an automatic evaluator can score each answer, systems that pair a model with an automatic evaluator have produced new results in mathematics and computer science, such as DeepMind’s FunSearch and AlphaEvolve [S2, S3]. One AlphaEvolve result is a scheduling rule that Google reports recovers, on average, 0.7% of its worldwide compute. Every candidate rule could be scored automatically, which is why the method fits [S3]. Even there, people frame the problem: DeepMind’s AlphaProof had its competition problems “manually translated” into formal mathematical language [S10].

Most business questions have no automatic evaluator. There, the model should not be used on its own. When done well, it runs inside a framework: instructions and sources the model reads, tools it can use, and a record kept outside the chat. The framework stores what the model needs and puts it in front of the model each time. A person sets the goal, leads the work, and is accountable for the result. The check that an automatic evaluator would make falls to the person, the record, and a fresh session or a different model. In this piece, the system means all three together: the model, the framework, and the person.

1 · The model is one part of a working system

In real work, three parts surround the model:

1. Instructions and sources the model reads before it starts: the goal, the rules, and the material to work from.
2. Tools it can use, such as web search, running code, and running tests.
3. A record kept outside the chat: decisions made, questions still open, and what each check found. The model reads it at the start of each turn. Reviewed results go into it, and a machine-written trace of commands run and checks passed is added automatically. The record is the system’s memory, and it sits outside the model.

A person leads the work. They set the goal, decide what to ask next, and accept, revise, or reject each result. The questions come from the person, and some of them change the goal. Each request starts one turn. A turn produces a response or a draft, the person reviews it, and that review decides what goes into the record and starts the next turn.

Figure 1 shows the system as three loops at different speeds, one for each question in the table above: reasoning within a turn, remembering across a session, and learning through standing rules.

Because the record is written as the work goes, through updates and changes in direction, it shows what failed and what changed. What can be inspected is the record, not any single answer: what happened, what decisions the human leader made, and what methods and checks were used.

Graepel names three things the model lacks (rows 1 to 3 below) and asks for two more (rows 4 and 5): an independent test of each step, and learning from what was checked [S1]. The system described here offers a partial substitute for each: strongest on the record, weakest on independent checking.

Graepel · No record, no reasoning

“There is no open ledger that lays out the hypotheses a model is considering, the confidence it has in various explanations, the evidence it’s weighing, and the unresolved questions it’s holding onto—all things that should be systematically revised as new information arrives.”

— Thore Graepel, MIT Technology Review [S1]

Graepel · An independent check

“…an independent part of the system must evaluate each move by how much it actually resolves uncertainty, updating beliefs only when the change is backed by evidence.”

— Thore Graepel, MIT Technology Review [S1]

2 · Knowledge work gains unevenly from LLMs

Producing a possible answer is now cheap. Refining it into something worth using takes time and human leadership. In code and mathematics, a machine can check the answer or the human can prove whether it works. Most business work has no black and white answer.

Three studies measured AI on ordinary knowledge work:

• Consulting. 758 BCG consultants used GPT-4 on realistic consulting tasks. On tasks the AI handled well, they completed 12.2% more tasks, 25.1% faster, at more than 40% higher quality. On one task chosen to be outside what the AI could do, consultants using AI were 19 percentage points less likely to reach a correct answer than consultants without it [S6].
• Professional writing. 453 college-educated professionals did writing tasks from their own occupations. With ChatGPT, time fell 40% and quality rose 18% [S7].
• Customer support. At one company, an AI assistant raised issues resolved per hour by 14% on average across 5,179 agents, and by 34% for novice and low-skilled agents [S8].

These studies measure gains. Only the consulting study tested work outside what the AI does well, and on that task consultants using AI did worse than those without it. The study’s title names the problem: a “jagged technological frontier,” where AI is strong on some tasks and weak on others that look similarly difficult [S6]. Knowing which is which takes experience, and the gap is hard to define in advance or to recognize in the moment. Nothing in the chat response directly signals the problem, which is why the check has to sit outside the model: in the record, in a second look, and in the person leading. Knowledge workers using these tools need to build that judgment.

3 · Where knowledge work often fumbles

Human-led knowledge work has several limits:

1. A simple question against a murky topic. Some topics, like how a programming language works, are well documented, clear-cut, and testable. Others, such as whether a specific policy meets a security standard, are nuanced. Some are still being argued over. Asking an LLM whether something like AI data centers is good or bad can surface a lot of contradictory material.
2. Poorly defined context. Even the most straightforward black and white question can give an unhelpful answer with poor context. Asking about an event that just happened, when a similar event happened a year ago, tends to pull first from the model’s training (the earlier event). The answer isn’t “wrong”, but it isn’t what you wanted.
3. A model judging the ideas it produced. When the model checks its own work, the check is weak. In a study with more than 100 researchers, AI-generated research ideas were rated more novel than experts’ ideas but slightly less feasible, and the AI was an unreliable judge of the ideas it generated [S9]. The study tested the model judging its own ideas; a fresh session or a different model as the checker is this piece’s proposal, not a finding.
4. People screen out odd ideas. Like the models, people bring their own lens and context. A person reviewing output tends to keep what looks sensible to them, based on what they know. An unusual idea that would have worked, or even worked best, can be cut out by the human leader.
5. It is easy to be lazy. Models take shortcuts and miss instructions. The people leading the work can be lazy too: it is easy to take an early draft or response and run with it. Output that will be relied on needs a close read, and output headed to a wider audience needs more than one set of human eyes.

4 · Best practices for working with LLM systems

• Maintain the record outside the chat. Decisions, sources, and open questions go in a persistent document, not in a conversation that scrolls away.
• What happened is more important than why. “The test failed” is a fact. “It failed because of X” is a guess until it is checked.
• Results are checked. Numbers are recalculated, sources are opened, code is run.
• A second review comes from outside the work. A fresh session or a different model checks the work before anyone relies on it.
• A person makes the decision. The model proposes. A person reviews, accepts, revises, or rejects, and that decision goes into the record.
• Stay curious and be willing to change direction. Ask what a term means, what the evidence actually says, whether a sentence is true. Bring in outside reviewers early. Drop the position you started with when the work shows it is wrong, and let the record show why.
• Transparency. The work is described honestly: how it was done, how rigorous it was, and who stands behind it.

Final take

Sources