Harness Debt in AI Agents: Remove Old Assumptions Before Adding Orchestration

Agent scores are not decided by model weights alone. Adding a harness—the loop, tools, context management, verification, and permissions—does not raise results uniformly across tasks. Stale assumptions that “the model cannot do this” suppress what the current model can already achieve.

📑Table of Contents

1. The harness is the runtime outside the model
2. Adding more orchestration does not raise scores uniformly
3. Harness debt is a stale “cannot do” assumption
4. How to separate scaffolding to cut from boundaries to keep
5. Author Perspective
6. FAQ
7. Takeaway

On the same GPT-5 weights, Terminal-Bench 2.1 is 35.2% inside Terminus 2 and 49.6% inside Codex CLI, about a 14-point gap. The figure comes from the preprint arXiv:2609.01437. Before you add orchestration, separate scaffolding you can delete from consequence boundaries you should keep.

Cut stale assumptions before buying more tooling

Treat harness debt as stale capability assumptions, not as a budget request for more tooling. Keep a change only when the same model shows a better task success rate and the extra tokens have a clear explanation.

---

The harness is the runtime outside the model

Agent = Model + Harness

A harness is not the model’s internal reasoning. It is the surrounding runtime for tool use, memory, state, execution environments, and feedback. Wikipedia’s Agent harness page writes the relationship as Agent = Model + Harness. Because the wrapped component is non-deterministic, recovery from fabrication and early stopping is part of the design.

---

The study uses the same formula for the outer runtime

The independent study arXiv:2609.00006 (Barbaste et al., submitted 2026-07-15) uses the same formula. There the harness is the outer runtime: a control loop, tool surface, context handling, safety checks, orchestration, and extension points that connect the model to files, shells, and other systems.

---

Eleven production implementations

The anatomy covers eleven production coding harnesses—Claude Code, Codex CLI, Gemini CLI, OpenHands, and others—plus a meta-harness contrast, mapped onto seven canonical subsystems.

The cross-cutting observations are concrete.

• The surveyed Python, TypeScript, and Rust codebases total about four million lines, and none of those runtimes pull in a general-purpose agent framework.
• Code search is deterministic, not embedding retrieval.
• SKILL.md appears in 9 of 11 systems, MCP in 8, ACP in 6.

The authors argue that in early 2026 the harness shifted from tool to platform.

---

Encyclopedia taxonomy, not a product SDK

That taxonomy is encyclopedia and preprint structure, not a product-SDK standard. Treat it as a wider operational layer than prompt or context engineering. Stopping failure modes is closer to ハーネス設計でエージェント失敗を止める手順.

---

Adding more orchestration does not raise scores uniformly

Adding a harness looks like a technology investment. Holding weights fixed and changing only the harness can still move scores by a large margin, and not only upward.

Same-weight GPT-5 still splits about 14 points

HarnessDev (arXiv:2609.01437) reports the GPT-5 Terminal-Bench 2.1 split: 35.2% in Terminus 2 versus 49.6% in Codex CLI.

The Creation track scale is:

• six creator LLMs
• four domains
• 2,207 downstream instances

Generated harnesses lag mature human-engineered references on code and on search or research, while matching or exceeding the selected references on writing and ML experimentation. Execution tokens vary widely; high token spend does not guarantee high scores.

---

Evolution is not a uniform upgrade

Evolution is not a uniform upgrade either. Gains are unstable, transfer only partly to held-out tasks, and depend strongly on the executing model. An automatic evolution loop is not an adoption argument if the only rise occurs on the development set.

---

Task fit is not the same as volume

Fit is a different operation from volume. JIT-Agent (arXiv:2608.25593) reports that a task-adaptive harness lets DeepSeek-V4-Flash beat GPT-5.6 by +9.1 on DeepSearchQA and +4.3 on OdysseyBench, with GLM-5.2 gaining up to +20.2. Stacking a fixed scaffold and synthesizing a task-fit harness are not the same change.

Observation	Number	How to read it
Same GPT-5 / Terminal-Bench 2.1	Terminus 2 35.2% / Codex CLI 49.6%	About 14 points with weights held fixed
HarnessDev Creation	Behind human refs on code/search; match/exceed on writing/ML	A generated add-on harness is not always stronger

---

Evolution, adaptive generation, and experimental scaffolds

Observation	Number	How to read it
HarnessDev Evolution	Unstable gains; partial held-out transfer; model-dependent	An evolution loop is not uniform improvement
JIT-Agent adaptive generation	DeepSearchQA +9.1 / OdysseyBench +4.3 / GLM-5.2 up to +20.2	Task fit beats extra fixed scaffold
Anthropic experimental scaffold (secondary)	More than 2× the standard scaffold on the same model	The wrong orientation can also cut results in half

---

Preprint limits

Sources: arXiv:2609.01437, arXiv HTML, arXiv:2608.25593, AI Heroes (as of September 2026).

Keep the limits visible. HarnessDev and JIT-Agent are preprints. They evaluate generated harnesses; they do not prove that every human-built harness fails. The “more than 2×” experimental-scaffold figure is a secondary citation of an Anthropic report.

---

Always-on cost is a different axis

Always-on cost is a separate axis from harness volume. 常時稼働AIは起動・停止・上限・観測を1枚に書いてからGPUを予約する covers writing start, stop, caps, and observability before you reserve GPUs.

---

Harness debt is a stale “cannot do” assumption

A snapshot of yesterday’s model limits

Harness debt is a snapshot of yesterday’s model limits.

• tool wrappers
• orchestration
• oversized system prompts

AI Heroes (Marco Lobo) defines it as stacked assumptions about work the model can now do itself. Adding orchestration is often the opposite of the fix.

---

Official notes encode the same stale assumption

Official engineering language points the same way. Anthropic’s Managed Agents write-up says harnesses encode assumptions about what Claude cannot do on its own. Those assumptions go stale as models improve. If you do not question them, scaffolding becomes dead weight.

---

Context reset becomes dead weight after a model change

Context reset is the concrete example. Resets added for Sonnet 4.5 context anxiety became dead weight when the same harness moved to Opus 4.5.

Resets stop the anxiety, but they add overhead.

• orchestration
• tokens
• latency

Anthropic’s long-running harness-design note treats reset as more on-target than compaction for the core issue, and not free. Self-evaluation is lenient; a separate skeptical evaluator is the tractable lever.

---

Which layers moved

Secondary numbers split the layers that moved from the layers that did not. AI Heroes cites Opus 4.6 versus 4.5 as:

• Terminal-Bench 2.0 59.8% → 65.4%
• OSWorld 66.3% → 72.7%
• SWE-bench Verified 80.9% → 80.8%

The over-designed layer overlaps the layer the model actually improved.

• Prompt-only changes move SWE-bench Verified 80.84% → 81.4%.
• BrowseComp moves 45.3% → 61.6% when tool output is filtered in code.

Treat those benches as secondary citations of Anthropic reporting.

On the operations side, Anthropic reports that decoupling brain from hands dropped latency and orchestration cost.

• p50 TTFT by roughly 60%
• p95 by over 90%

---

Keep credentials out of the execution sandbox

Do not put credentials in the execution sandbox. Git tokens are injected at init; MCP goes through a proxy and a vault. Freezing internal procedure is closer to skills and evals: 社内AIエージェントのSkillsとEval設計｜合格基準を作る方法.

---

How to separate scaffolding to cut from boundaries to keep

Before you add anything, label each part in one line: capability gap-fill, or consequence boundary. Gap-fill is a retest candidate. Boundaries that change irreversible outcomes stay. On every model release, pick three parts with the strongest “cannot do” assumption and rerun them on the new model. Two weeks of deletion plus two weeks of live traffic finds rot faster than a 90-day pilot.

Use this seven-step check.

Cut candidates: retest the “cannot do” parts

1. Write one line per part: capability gap-fill or consequence boundary. Retest the first; keep the second as a candidate.
2. On each model release, pick three parts with strong “cannot do” assumptions and rerun them.
3. List cut candidates: a specialist search tool that bash plus an editor already covers, raw tool results dumped back into context, handwritten orchestration, and an oversized “just in case” system prompt.

---

Keep candidates: gates, tokens, and held-out checks

4. List keep candidates: confirmation gates for irreversible or external state changes, freshness checks before overwrite, structured logs, permission boundaries, and human approval for customer-facing or money-moving actions.
5. Keep credentials out of the execution sandbox. Inject Git tokens at init; put MCP behind a proxy and a vault.
6. If you add a harness, compare before/after task success and execution tokens on the same model. Do not adopt on token growth alone.
7. If you evolve a harness automatically, check held-out tasks and a different executing model. A rise on the development set is not enough.

---

Adoption rule

Sources: Anthropic Managed Agents, AI Heroes, arXiv:2609.01437 (as of September 2026).

The adoption rule is narrow. Keep the change when success rises on the same model and the extra tokens have an account. Unexplained token growth is a stale-assumption residue until proven otherwise.

---

Author Perspective

A proposal to add a harness looks like a feature. The real fork is capability gap-fill versus consequence boundary. The first rots when the model updates. The second is the asymmetric-cost surface: money movement, customer-facing actions, irreversible external writes.

Do not treat preprint points as a final product-selection score. What transfers is the direction: scores move when the harness changes and weights do not. Take before-and-after measurements on the same task set, and put tokens next to the score. Resets and split evaluators come with latency and orchestration cost; budget them as such.

---

FAQ

Q1. Is harness engineering just a longer prompt?

No. It is the design of the outer loop, tools, context, verification, and permissions. Prompts and context are only part of that runtime.

---

Q2. Does adding a harness always make the same model stronger?

No. The same GPT-5 still splits 35.2% versus 49.6% on Terminal-Bench 2.1. Generated harnesses can lag human references on code and search.

---

Q3. What should you cut, and what should you keep?

Retest and cut capability gap-fill: specialist tools, raw result dumps, oversized prompts. Keep confirmation and permissions for irreversible actions.

---

Q4. Is a context reset always required?

No. The Sonnet 4.5 reset became dead weight on Opus 4.5. When the model changes, question the premise.

---

Q5. Can you adopt the Dentsu Soken taxonomy as an industry standard?

No. That post is a discovery write-up, not a standard. Prefer Anthropic’s official engineering notes and the arXiv papers for definitions and numbers.

---

Takeaway

A harness is the runtime that connects a model to real work. Stacked stale “cannot do” assumptions suppress scores. Cut before you add.

The next action is specific. Retest three parts with strong “cannot do” assumptions, delete capability scaffolding, and keep irreversible-action boundaries. Adopt only on same-model success rate plus tokens.

Related articles:

• 13594
• 13574
• 13584