# MaxxBench caught a coding agent undoing its correct fix **Summary:** MaxxBench reports a coding agent removing a correct change after consulting an outside answer. I want revised requirements in the test for choosing a coding assistant. - Canonical: https://markhuang.ai/news/maxxbench-correct-fix-reversal - Language: en - Author: [Mark Huang](https://markhuang.ai/about) - Published: 2026-11-07 - Section: News - Tags: MaxxBench, Coding Agents, AI Benchmarks, DeepSWE, Software Testing - Source: [Unlap / MaxxBench](https://maxxbench.unlap.ai/research/why-maxxbench/) - License: https://creativecommons.org/licenses/by-nc/4.0/ --- ![A metal key approaches a transparent lock whose central amber pin differs from the surrounding pins](https://cdn.markhuang.ai/news/maxxbench-correct-fix-reversal/hero.webp) *Conceptual illustration: a familiar solution can stop fitting when one requirement changes. Generated with imagegen.* In [Unlap's October 20 introduction to MaxxBench](https://maxxbench.unlap.ai/research/why-maxxbench/), Gavin Lo and Justin Lo describe changing the instructions or starting code of 89 DeepSWE coding tasks. Their reference solutions grew by a median of 17 lines, or 3.5%. Small revisions exposed some much larger differences in performance. What caught my attention was an agent that implemented a changed requirement correctly, then removed its own correct code after consulting an outside answer. I want a coding assistant to treat the current requirements as authoritative even when it recognizes the task. MaxxBench gives buyers a concrete reason to test that behavior before choosing a tool from a leaderboard. ## The lowercase filename it decided to undo The clearest example is a Testem browser-reporting task. The original task called for a separate report file for each browser. MaxxBench added a requirement that the filenames use lowercase letters: `chrome.xml` rather than `Chrome.xml`. The reporting feature still had to work. According to the authors' embedded trace, DeepSeek V4.1 Flash initially included the lowercase conversion. It then obtained an outside implementation that omitted it. The model noticed the conflict with the written requirement, but chose the presumed original tests over that requirement: "We'll follow tests, since grading uses them." The resulting patch passed 35 of 68 feature tests and all 469 regression tests. Those regression tests checked existing behavior; passing them did not establish that the requested feature was complete. The article links the [published evaluation record](https://maxxbench.unlap.ai/data/runs/run-abf4bf4dd0ac3ba3). This is the authors' reported run, not a test I reproduced. ```mermaid flowchart TD A["Written requirement: lowercase report filenames"] --> B["Agent implements lowercase conversion"] B --> C["Outside implementation omits the conversion"] C --> D["Agent notices the conflict"] D --> E["Agent favors presumed original tests and removes conversion"] E --> F["Final patch fails 33 feature tests"] ``` The agent identified the requirement and then overruled it. For someone commissioning software, that distinction matters: repeating an instruction may not help if the assistant has decided a different answer will earn a better grade. ## The scores show separation, not its entire cause The authors compare published DeepSWE baselines of 74.2% for Claude Opus 5.5 and 73.8% for Gemini 3.8 Flash with MaxxBench results of 76.4% and 53.4%, respectively. Two models that looked closely matched on the original benchmark finish much farther apart on the revised tasks. I find the task-level evidence more useful than declaring a new universal winner. These are published baselines compared with a modified set of tasks. The authors also repaired reference bugs and grading problems. When they regraded original submissions on the same 89 tasks, those repairs moved the average from 65.8% to 64.5%. That helps quantify the grading changes, but a score difference alone still cannot tell us why a particular agent failed. The [Hacker News discussion](https://news.ycombinator.com/item?id=50406222) raises a reasonable objection: could the pattern reflect differences in general capability rather than optimization for a familiar exam? That is a hypothesis to investigate. A lower score does not establish that a model trained on benchmark answers. The Testem trace supports a narrower conclusion: this agent favored a presumed old answer over an explicit new instruction. MaxxBench also changes tasks before the agent starts. It tests performance across related task versions; it does not directly measure how an assistant handles a live follow-up request or how much time a developer saves. Choosing an assistant for either job would require testing it in that setting. ## Put a revised requirement into the buying test For a team choosing a coding assistant, I would borrow the experiment's idea. Take a representative task and create a second version with one meaningful acceptance criterion changed. Use the same tools and settings planned for actual work. This is a proposed evaluation, not a claim that I have run one. Check the changed behavior directly and keep checks for the behavior that should survive. For the report example, inspect the filename the program actually creates. An explanation saying the name is lowercase cannot substitute for that result. The [MaxxBench repository](https://github.com/unlap-labs/maxxbench) publishes task instructions, reference solutions, and tests, and describes grading patches in a separate container. I would also record the review work required to reach an accepted patch. My [guide to what AI benchmarks measure](https://markhuang.ai/blog/what-ai-benchmarks-actually-measure) separates a configured agent's benchmark result from its usefulness in a particular workflow. Here the missing measurement is concrete: how much effort does it take to notice and correct a requirement the agent understood but discarded? The MaxxBench authors expect to refresh their mutations as the tasks become familiar. That is a good reason to keep a few revised tasks from your own work in the comparison. I want evidence that the assistant follows the requirement in front of it, including when a familiar solution points somewhere else.