LLMs Reward Experts. Who Trains the Next Ones?

Sean Goedecke's "LLMs reward expertise" makes a sharp claim: the best prompting skill is not a bag of magic phrases. It is knowing enough about the subject to notice what looks wrong, suggest a better direction, and ignore the model's plausible detours.

I buy that. I also think it creates a problem that is easy to miss. If AI makes experienced people more productive while letting beginners skip the work that creates experience, then a team can improve today's throughput while quietly weakening tomorrow's judgment.

Answer Snapshot

Question	My read
What is the claim?	Domain experts get more from the same model because they can frame, steer, and verify the work.
What supports it?	A 2026 Anthropic analysis of roughly 400,000 Claude Code sessions found higher success rates and more agent work per instruction as apparent task expertise increased.
What does it not prove?	That every expert becomes faster, or that transcript-level success becomes durable production value.
My thesis	Expertise is becoming more valuable, not less. Teams now need to protect the learning loops that produce it.

Expertise is the steering wheel

Goedecke uses Terence Tao's conversation with ChatGPT about the Jacobian Conjecture as his main example. The striking part is not a secret prompt template. Tao asks short, specific questions, rejects directions that look unhelpful, and supplies mathematical leaps of his own. Copying his sentence length would not copy the knowledge behind those moves.

The same pattern appears in software. Someone with a working theory of a codebase can tell an agent, "this should reuse the existing path," or spot a solution that violates an implicit constraint. A newcomer may accept the first coherent patch because coherence is the only signal available to them.

Anthropic's June 2026 Claude Code study found this pattern in a large observational dataset. It analyzed roughly 400,000 interactive sessions from about 235,000 people between October 2025 and April 2026. Sessions rated novice reached the study's strictest "verified success" measure 15% of the time; intermediate or higher sessions reached 28% to 33%. Expert sessions also triggered about 12 agent actions per prompt, versus about five for novice sessions.

Reward does not always mean speed

I would not turn those numbers into a clean productivity slogan. Anthropic had Claude infer expertise from the transcript, using signals such as precise instructions, verification requests, and corrections. Its success measures also came from transcripts and supporting signals such as tests or commits. The researchers explicitly say they could not observe whether the resulting code was later used, discarded, or economically valuable.

A narrower study produced the opposite productivity result. METR's randomized study assigned 246 real issues from large open-source projects to 16 experienced maintainers, with or without early-2025 AI tools. Developers took 19% longer when AI was allowed, even though they believed afterward that it had sped them up by 20%.

That result does not refute Goedecke. METR studied maintainers working in repositories they already knew, on tasks averaging about two hours, with models and tools that have since changed. It does separate two ideas that are often bundled together: expertise can improve the quality of steering, while the cost of prompting, waiting, and reviewing can still erase the time saved.

The expertise pipeline is the risk

This leaves a staffing question: where do tomorrow's experts come from? Expertise grows through slow contact with mistakes. You read unfamiliar code, make a bad assumption, trace the failure, and revise your mental model. An agent can remove exactly that friction. Sometimes that is the point. Remove too much of it, though, and a junior engineer can finish the ticket without learning why the patch works.

A separate Anthropic randomized trial tested 52 mostly junior engineers learning an unfamiliar Python library. The AI group finished slightly faster, but the difference was not statistically significant. On a quiz immediately afterward, it scored 50% on average, compared with 67% for the hand-coding group. Participants who used AI to ask conceptual questions and check their understanding did better than those who simply delegated the code.

The study was small and measured immediate mastery, not career development. Still, its warning fits the larger picture. If expert judgment is the scarce input that makes agents useful, treating learning time as waste is a poor bargain.

What I would change on a team

I would stop teaching "prompt engineering" as a standalone craft. The useful skill is closer to supervised delegation: define the outcome, expose the relevant constraints, demand evidence, and know when the answer feels wrong. Those habits come from domain knowledge and from owning consequences.

That means experts should use agents aggressively where verification is cheap, but not absorb every difficult task merely because they can now finish it faster. Some work needs to remain legible to learners. Junior engineers need chances to predict an approach before asking the model, explain generated code before merging it, debug failures without immediately delegating the diagnosis, and see the production consequences of their decisions.

IBM researchers found a related concern in a 2025 enterprise case study: perceived productivity gains were uneven, and generated code raised questions about ownership and responsibility. For me, ownership is the useful test. The point is not to preserve manual coding as a ritual. It is to preserve enough understanding that somebody can challenge the machine and carry the result.

My takeaway

LLMs do reward expertise. The evidence is strongest when "expertise" means a working grasp of the specific problem, not status, seniority, or clever wording. A capable person can extract more useful work, recover from errors, and reject polished nonsense.

But that advantage has a maintenance cost. If companies use AI to concentrate more work around the people who already know the system, they may get a short-term gain and a thinner bench. I would measure both: what the agent helped ship, and who became more capable while shipping it. Otherwise we are spending expertise as if it were renewable on its own.