# A Stronger Coding Agent May Be Better Off With Bash **Summary:** Across 176 settings, context management, planning, and tools changed value with the model and task. I would profile the model-harness pair before adding another layer. - Canonical: https://markhuang.ai/news/stronger-coding-agents-fewer-tools - Language: en - Author: [Mark Huang](https://markhuang.ai/about) - Published: 2026-09-18 - Section: News - Tags: Coding Agents, Agent Harnesses, Context Management, Developer Tools, AI Evaluation - Source: [arXiv](https://arxiv.org/abs/2609.20804) - License: https://creativecommons.org/licenses/by-nc/4.0/ --- ![A luminous coding model core connected to interchangeable planning, memory, and tool modules on an engineering test bench](https://cdn.markhuang.ai/news/stronger-coding-agents-fewer-tools/hero.webp) *A coding model does not work alone. The machinery around it can rescue a run, waste tokens, or get in the way.* The new paper [An Empirical Study of Harness Design for Coding Agents](https://arxiv.org/abs/2609.20804) tests 176 settings across four models, two coding benchmarks, five context-management strategies, and context budgets from 32K to 128K tokens. No single configuration wins. I think that messy answer is the useful one. Teams should stop treating plans, memory, and specialized tools as a checklist. Each component has to solve a failure the model is actually having. A weaker model may need more structure to reach its first edit. A shell-capable model may do better with one broad interface than a rack of narrow tools. ## Context management mostly buys survival The cleanest result concerns context. The researchers compared no management with four managed strategies that elide old tool output, make it recoverable, summarize earlier history, or combine those mechanisms. On SWE-Bench Verified, the average success-rate advantage of managed context over no management fell from 35.7 percentage points at 32K tokens to 2.7 points at 128K. On Terminal-Bench 2.1, it fell from 9.5 points to 2.8. The reason was less romantic than better memory. Without management, runs often hit the window and stopped. At 32K, the model-averaged overflow rate was 78.7% on SWE-Bench and 61.0% on Terminal-Bench. Every managed tier had zero overflow failures at every tested budget. That is weak evidence for the idea that summarization makes an agent smarter. In this experiment, context management mostly kept the agent alive long enough to edit and verify. As the window grew, unmanaged runs survived more often and the accuracy gain nearly disappeared. The combined strategy had the best overall efficiency: first remove bulky old tool output, then summarize only if the history still crosses a higher threshold. It posted the lowest mean cost in seven of eight model-benchmark panels. Yet the recovery tool added for elided content was rarely used and did not improve accuracy over elision alone. A feature can sound reassuring and still fail to earn its place. ## A stronger model may prefer a smaller toolbelt The action-space result is the one I would bring into a product review. At the paper's 128K baseline, the 30B Nemotron model scored 25.2% on SWE-Bench with predefined file and search tools, but only 10.2% with bash alone. The 550B model moved the other way: 65.8% with predefined tools versus 69.4% with bash, while mean cost dropped from $2.33 to $1.11 per task. The larger model was not simply "better at tools." Bash let it combine operations into coarser actions. On Terminal-Bench, its median trajectory fell from 47 actions to 31, and a larger share of those actions wrote code. The narrower interface reduced the number of round trips. That does not make bash-only a universal recommendation. Mistral-Medium-3.5-128B dropped from 68.6% to 45.4% on SWE-Bench when the predefined tools were removed, even though its Terminal-Bench result moved in the opposite direction. Model size is a poor shortcut for shell proficiency, and the task still matters. > **Info:** > > I would start with the smallest harness that lets a specific model complete a representative task reliably. Add a component when the trajectory shows the failure it is meant to fix, then measure accuracy and cost again. ## A plan can be a stopping policy Planning also changed jobs as capability increased. For the 30B model on SWE-Bench, removing the persistent plan cut median trajectory length from 40 turns to 5. With no plan, 68.6% of runs ended without an edit. Planning raised the success rate from 13.6% to 25.2%, but mean task cost rose from $0.02 to $0.09. For stronger models, the plan did not materially improve accuracy in the same way. It mostly reduced repeated verification after an edit. Median SWE-Bench trajectories fell from 108 to 74 turns for the 550B model and from 68 to 53 for Mistral when planning was enabled. I usually think of a plan as a map for beginning work. This paper makes me think about the other end of the run. A persistent plan can tell a hesitant model to keep going, or tell an overactive one that the promised work is already done. In both cases, its value appears in where the trajectory stops. ## Where the evidence stops The paper is unusually candid about its limits. Each setting ran once per task. Terminal-Bench contained 89 tasks, so many comparisons there were not individually significant. SWE-Bench Verified contained 500 Python issues, and planning plus action-space ablations were tested only with the combined context strategy at 128K. The tool comparison also bundled several changes. The predefined interface included file-state tracking, read-before-write checks, and automatic diagnostics, while bash-only removed that whole package. The experiment tells me which complete interface worked in each condition. It cannot isolate whether the difference came from tool count, prompts, diagnostics, or action granularity. Other research points in the same direction without settling the recipe. [Same Model, Different Harness](https://arxiv.org/abs/2608.26218) found that mechanically shortening old tool results changed outcomes for unchanged models under tight context. A separate [full-factorial study of agent scaffolding](https://arxiv.org/abs/2605.05716) found that an all-components configuration was suboptimal on its question-answering and math tasks. The details differ, but both challenge the habit of equating more scaffolding with a more capable system. ## I would version the pair This study sharpens the point I made when [Astra's ARC score moved 45.1 points with a harness change](https://markhuang.ai/news/astra-arc-score-45-1-point-harness-gap): the product includes much more than a model name. Here, the researchers make that idea practical by varying individual parts and inspecting where trajectories change. For a buying decision, I would record the model, harness version, context policy, plan protocol, tool interface, token budget, and task type as one tested unit. Then I would keep a small evaluation set that exposes early abandonment, context overflow, shell mistakes, and excessive verification. Those failure labels tell me which component deserves another experiment. I am not stripping every strong coding agent down to bash after one paper. I am asking every layer to justify itself. If a plan saves tokens, keep it. If compaction only prevents an artificial 32K ceiling that production never hits, price that benefit honestly. If specialized tools slow down a model that already speaks shell fluently, remove them. I want the amount of machinery that matches the failures in front of me.