How to Stop Repeat Agent Failures With Harness Design

Repeated agent stalls and repeated mistakes usually come from a missing harness, not a weak model. A harness is the loop, tools, checks, and permissions that surround the model.

📑Table of Contents

1. A harness is the runtime around the model
2. The layer after prompts and context
3. Turn failures into Guides and Sensors
4. The same model can move when the harness changes
5. Checklist: add one control first
6. FAQ
7. Takeaway

LangChain reports Terminal Bench 2.0 moving from 52.8% to 66.5% (+13.7 points) on the same gpt-5.2-codex weights after harness-only changes. Natural-language rules still break probabilistically.

In an AI coding workflow, write down what the current runtime allows and what it verifies before you add another prompt. Talks and slide decks often stop at the model; the failure that repeats in production sits in the AI runtime around it.

After you read this

After this article, do the following.

• Pick one recent failure
• Classify it as a missing Guide (before work) or a missing Sensor (after work)
• Add one control

---

Same layer as loop harness design

Splitting instructions, guards, and verification is the same layer as harness design for safe agent loops.

---

A harness is the runtime around the model

An agent is not a model alone. The model supplies inference; surrounding software finishes work in a real repository. Agentic Dev defines a harness as the following.

• Loop
• Tools
• Context
• Memory
• Recovery
• Guardrails

Reliability is whether the agent finishes the correct work without supervision. Claude Code and Codex CLI are harness examples.

What a Copilot Studio harness decides

In Microsoft Learn for Copilot Studio, a harness is the runtime between the builder-designed agent and the model. It decides the following.

• When to call the model
• Which components to send
• How to interpret returns
• Which tools to invoke

GitHub Copilot, its standard version, and Copilot Chat harnesses differ in skills, memory, file support, recovery, and billing. The chat harness publishes to internal teams. The same word “agent” does not imply the same permissions.

---

List permissions before swapping models

Before swapping models, list what your current runtime allows and what it verifies. If you cannot list it, do not blame the weights. Changing to a new AI tool without that list simply moves the same stall to a new vendor.

---

The layer after prompts and context

A prompt is a one-turn instruction. Context is what the agent can see. Both sit inside the harness.

In a theater metaphor, they are the lines and the set; the building and fire system are the harness. Polishing instructions will not stop a dangerous command or send a failing test back if those controls are missing.

Nested layers and always-on rules

HP Japan treats prompt, context, and harness as nested layers, not rivals. AGENTS.md and CLAUDE.md are always-on project rules.

OpenAI keeps the following operating shape.

• Keep AGENTS.md near 100 lines as a table of contents
• Put durable knowledge in versioned docs/
• Treat giant manuals as something that will rot

The reported cut of a 20,000-line CLAUDE.md down to about 200 pointer lines follows the same rule.

---

Mechanical stops beat another prompt

Natural-language rules break probabilistically. Linters, tests, and blocks on rm -rf or DROP TABLE belong in the harness. Add a short always-on pointer file and a mechanical stop before you add another prompt.

---

Turn failures into Guides and Sensors

You need Guides before work and Sensors after work.

• Guides alone never check whether the rule held
• Sensors alone let the same mistake recur

Split computational and inferential controls

Birgitta Böckeler on martinfowler.com (2026-04-02) splits controls into two kinds.

• Computational: lint, types, tests; milliseconds to seconds; deterministic
• Inferential: LLM review; slower, costlier, non-deterministic

Misdiagnosis, over-building, and misunderstood instructions are not reliably caught. If the human never specified the wanted behavior, no sensor can certify correctness.

---

Ratchet observed failures, not guessed rules

GIGAZINE (2026-08-28) says not to close a repeated miss with a one-off chat fix; improve the harness.

Addy Osmani on O’Reilly Radar (2026-05-15) ratchets observed failures into the following.

• AGENTS.md lines
• Precommit hooks

Do not add speculative rules. Every line in a good AGENTS.md should trace to a specific failure.

---

Cross-session harness for long-running work

Long-running work needs a cross-session harness. Anthropic uses this sequence.

1. An initializer writes feature_list.json (passes: false), init.sh, and claude-progress.txt
2. Later sessions implement one feature at a time
3. Browser end-to-end checks run at session start

Puppeteer MCP cannot see native browser alerts, so those features stayed buggier. If a sensor cannot prove a feature is done, do not mark the feature done.

---

When sensors can be gamed

When the agent also writes tests, sensors can be gamed. That failure mode overlaps separating CI verification from agent-written tests. Pick one repeated failure, decide Guide versus Sensor, and add only one control.

---

The same model can move when the harness changes

There are official reports where rank moved after harness changes with weights held fixed. Waiting for a new AI model is not the first move. The numbers are conditional. Replace them with your own failure log.

How to read the published numbers

Report	Held fixed	Changed	Number	How to read it
LangChain deepagents-cli	gpt-5.2-codex	Prompt, tools, middleware	Terminal Bench 2.0 52.8%→66.5% (+13.7pt); outside Top 30 → Top 5	Official; 89 tasks. xhigh-only scored 53.9% on timeouts. They did not rerun the same loop for Claude
HumanLayer / Osmani cite	Claude Opus 4.6	Harness	Claude Code ~33rd → 5th on another harness	Secondary cite; rank depends on the leaderboard snapshot
BSWEN via HP	Same weights	Scaffold	SWE-Bench Pro 38%→60% (22pt)	Blog synthesis, not a paper

---

Source links for the published numbers

Source: LangChain, Improving Deep Agents with harness engineering, O’Reilly Radar, Agent Harness Engineering, HP Japan Tech&Device (as of September 2026)

---

LangChain’s published loop

LangChain’s published loop is short:

1. Aggregate failures from traces
2. Insert verification before completion (PreCompletionChecklistMiddleware)
3. Inject an environment map at start (LocalContextMiddleware)
4. Force a rethink after N edits to the same file (LoopDetectionMiddleware)

A common miss is writing a solution, rereading it, and stopping without tests. The intended path is the following.

1. Plan
2. Build with tests
3. Verify against the task spec
4. Fix

Running only xhigh dropped the score to 53.9% on timeouts; high scored 63.6%. Reasoning intensity is a harness variable.

---

OpenAI’s internal scale is not a template

OpenAI’s internal Codex experiment is a scale reference, not a template:

• 0 handwritten application lines
• about 1 million lines
• about 1,500 merged PRs
• three engineers at 3.5 PRs/engineer/day at the start
• a period when Fridays (20% of the week) went to slop cleanup

Do not generalize those numbers without similar environment investment.

---

Checklist: add one control first

Do not build the whole harness at once. Add the control that can stop an observed failure mechanically. Differentiation sits above a mature harness: patterns, skills, review, and gates. How to write pass criteria overlaps Skills and eval design for internal agents.

Observe and classify failures

1. Write down ten failures from the last month (ignored instructions, lost goal, repeated error, fake completion, dangerous command)
2. Label each as missing Guide, missing Sensor, or missing human approval

---

Add the mechanical control that matches the miss

3. Sensor candidates: block completion if tests fail; block on lint; refuse rm -rf and DROP TABLE in the harness
4. Guide candidates: cut AGENTS.md / CLAUDE.md to a short pointer file (about 100–200 lines) and move detail into docs/
5. Human approval before production deploys, customer email, high-value payments, and shared-file deletion
6. For long tasks, add a feature list, a progress file, and session-start end-to-end checks
7. When a new failure appears, ratchet one line. Do not add a rule with no observed failure

---

Copilot Studio choice and next-week review

If you use Copilot Studio, pick the harness by failure type.

• Multi-step work and file edits fit GitHub Copilot
• Predictable FAQ-style talk fits standard
• Microsoft 365 Chat extensions fit the chat harness

Billing and publish scope change with that choice. Next week’s review item is whether the Sensor you added actually fired. Zero fires means detection is still missing, not that the agent became reliable.

---

FAQ

Q1. If I write a longer prompt, do I still need a harness?

Yes. Natural-language rules break probabilistically. Lint and tests are the completion gate.

---

Q2. Can I put everything in AGENTS.md?

No. OpenAI recommends about 100 lines as a table of contents. A file that grew to about 20,000 lines was cut to about 200 pointer lines.

---

Q3. Can my team treat +13.7 points as the expected lift?

No. That figure is Terminal Bench 2.0, gpt-5.2-codex, and deepagents-cli. Start from your own failure classes.

---

Q4. Can sensors prove the user wanted this feature?

No. Böckeler puts Behaviour on spec plus tests plus a human check. Intent sits outside sensors.

---

Q5. Should we write a harness from scratch?

Most teams should adopt a mature harness first. Agentic Dev takes that position. Differentiate in patterns, skills, review, and gates.

---

Q6. Which Copilot Studio harness should we pick?

Long reasoning and file work: GitHub Copilot. Predictable rule-based chat: standard. Internal Microsoft 365 Chat extensions: chat. Billing differs.

---

Takeaway

A harness is the runtime around the model, the layer after prompts and context. Pair Guides and Sensors, and ratchet only real failures into rules and hooks. Official benches can move on a fixed model, but those numbers are conditional. Classify ten of your own failures first.

Next action: pick one repeated miss, add one Guide or one Sensor, and check next week whether that sensor fired. If it did not fire, fix detection, not the rule text.

Related articles:

• 13572
• 13574