Which Parts of a Coding-Agent Harness Really Help?

What the paper studies

A coding agent is more than a language model. The model runs inside a "harness": a software layer with a control loop, a tool interface, and rules for which parts of the interaction history the model keeps. The paper "An Empirical Study of Harness Design for Coding Agents" (arXiv 2609.20804, submitted 17 September 2026) asks which parts of that layer do real work. Nine authors from UMass Amherst, Emory University and UNC Charlotte wrote it. The paper notes that part of the work was done during internships at Zoom Video Communications.

The authors say that earlier studies usually compare complete harnesses. They cite a cross-harness evaluation in which Claude-Opus-4.5 performed best with OpenHands, while Claude-Sonnet-4.5 performed best with SWE-Agent. Such a result does not show whether a gain comes from planning, tool design, context management or the way they interact with the model. So the authors built a lightweight harness from scratch. Its ReAct loop stays fixed. They vary only three components: planning, the action space, and context management. Permission handling, post-edit diagnostics and stuck detection stay fixed in every run.

The setup in brief

The harness is built on LangGraph, and the benchmarks run through Harbor. Each task gets at most 300 steps. The paper tests four models: three sizes of Nemotron-3 (30B, 120B and 550B) and Mistral-Medium-3.5-128B. It uses two benchmarks. SWE-Bench Verified has 500 human-verified GitHub issues. Terminal-Bench 2.1 has 89 command-line tasks. The paper reports success rate and mean cost per task, priced at OpenRouter rates.

Context management has five tiers. T0 does nothing and stops with an error when the window overflows. T1 replaces stale tool outputs with short stubs (elision). T2 adds external storage and a recall_event tool, so elided content can be read back. T3 uses LLM summarization only. T4 stages the three: elision first, and summarization only if the history is still too long. The soft and hard thresholds are 0.6 and 0.85 of the usable window. The authors run all five tiers at 32k, 64k, 96k and 128k tokens. Planning and the action space (predefined tools versus bash only) are ablated only at T4 with a 128k window. In total this gives 176 matched settings. The authors compare success rates with two-sided exact McNemar tests and control the false discovery rate at 0.05.

Four findings, with the paper's numbers

1. Context management matters most when the window is tight. Averaged over the models, the gap between managed tiers (T1 to T4) and T0 falls from 35.7 to 15.9, 5.5 and 2.7 percentage points on SWE-Bench as the window grows from 32k to 128k. On Terminal-Bench it falls from 9.5 to 7.5, 4.8 and 2.8. The T0 overflow rate falls from 78.7% to 8.7% on SWE-Bench and from 61.0% to 12.1% on Terminal-Bench. All managed tiers have zero overflow failures at every budget. One example from Table 3: at 32k, Nemotron-3 550B scores 6.40% on SWE-Bench with T0 and 55.60% with T4. The paper concludes that most of the benefit comes from preventing premature truncation. 2. Staged elision plus summarization (T4) is the most efficient; recall adds little. T4 reaches success rates similar to T1 to T3. It has the lowest cost in seven of eight model-benchmark panels, and the lowest mean cost per task at every window budget. It also has the lowest peak-context ratio at all four budgets. At 32k, T1 and T2 still reach about the full window. The recall mechanism is a different story. Across 32 matched comparisons, T2 beat T1 in 15 settings, lost in 14 and tied in three. The mean difference is minus 0.36 points. In 36 of 64 settings (56.3%) the model never called recall_event. The mean number of recall calls per task fell from 0.540 at 32k to 0.007 at 128k.

3. Planning shifts from an accuracy scaffold to a cost saver. For Nemotron-3 30B, planning raised success by 11.6 points on SWE-Bench and 4.5 points on Terminal-Bench, at higher cost. For the 120B model there was no consistent gain. For Nemotron-3 550B and Mistral-Medium-3.5-128B, planning cut SWE-Bench cost by about 30% and 32%, while success fell by 2.0 and 0.4 points. 4. The best action space depends on the model. For Nemotron-3 30B, the predefined tool set raised success by 15.0 points on SWE-Bench and 10.1 points on Terminal-Bench. For Nemotron-3 550B, bash only raised success by 3.6 and 5.6 points and cut cost by 53% and 30%. Mistral is mixed: the full tool set is better on SWE-Bench (plus 23.2 points), but bash only adds 6.7 points on Terminal-Bench.

What the trajectories show

The authors also labeled each turn of the agent runs with a workflow phase, using an LLM judge. Their reading explains the four findings. Context management mainly makes runs longer. At 32k without management, median SWE-Bench trajectories last 20 to 30 turns, and most runs stop while still localizing the bug. With management, median lengths rise to about 50 to 180 turns, and runs reach verification. Planning works differently for weak and strong models. For Nemotron-3 30B on SWE-Bench, turning planning off cut the median trajectory from 40 to 5 turns. Without planning, 68.6% of runs ended without any edit, against 27.8% with planning. For the two strongest models, planning shortened the median run from 108 to 74 turns (550B) and from 68 to 53 turns (Mistral). The authors attribute most of that to less post-edit verification.

The action space changes how code gets written. For Nemotron-3 30B on Terminal-Bench, 66% of bash-only runs ended after the model emitted calls to tools that the bash-only registry does not have. The average run shrank from 71 to 15 turns. For the 550B model on Terminal-Bench, bash only cut the median trajectory from 47 to 31 actions and raised the share of write-code actions from 16% to 27%. Re-patches of already edited files fell for all four models, for example from 3.3 to 0.4 for 30B.

Our analysis

Our reading is that the paper's main message is about conditions, not about winners. Every finding has an "if": if the window is tight, if the model is weak, if the model is fluent in bash. A single default harness cannot fit all of these. Anyone who reports "harness A beats harness B" should also name the model, the budget and the task type.

A second theme is that extra machinery is not free. The recall tool added machinery, and models rarely used it. Predefined tools helped a weak model but seemed to add overhead for a strong one, in the authors' words "action-selection and interaction overhead". At the same time, the results do not say "simpler is always better". Bash only nearly halved cost for the 550B model, but it hurt the 30B model badly, and it hurt Mistral on SWE-Bench.

A third point concerns cost. The dollar figures depend on OpenRouter prices from August 2026 for these specific models. The direction of the effect, such as fewer turns after planning, is more portable than the dollar values.

Limits and open questions

The authors state several limits. The results cover the specific components they built, not a universally best harness. Planning and the action space are ablated only at T4 with a 128k window, so a full factorial study would be needed to test other combinations. Each setting is run once per task. Terminal-Bench has only 89 tasks, so many contrasts there are not significant under the McNemar test, and the authors rely on consistent directions across models and budgets. SWE-Bench Verified is Python only. Model size is an imperfect proxy for capability, and the authors say the crossover points should be validated before use on other model families, harnesses or tasks. They also note that the action-space change bundles several things: tool availability, prompts, file-state tracking and automatic diagnostics.

We add two limits from reading only the paper text. We did not run the code or re-check the tables. And the study uses one open-weight family plus one other model. It says nothing about the closed frontier models used by many commercial coding tools.

Practical takeaways

For teams that build coding agents, the paper suggests a few habits. Measure how often long runs hit the context limit before you add clever memory features, because the paper finds most of the benefit comes from avoiding overflow. Try cheap rule-based elision before you pay for LLM summarization.

Do not assume a recall tool will be used: check the call logs. Test planning separately for each model, since it can raise cost for a weak model and cut cost for a strong one. Test a bash-only interface for strong models, but keep structured tools for models with weak shell skills. Finally, re-run these checks when you change the model or the context budget.