AI Agent Skills vs MCP: Design Internal Evals (2026)

Internal agents usually fail at the house process, not at the model. They can search the public web.

📑Table of Contents

1. Internal agents fail at onboarding, not model capacity
2. Package the procedure in SKILL.md
3. When to use AI Agent Skills vs MCP or live search
4. Define pass/fail with free-response evals
5. Adoption checklist: start with one workflow
6. FAQ
7. Summary

Skills and evals before more prompts

They do not know your approval path, permission model, or who to ask on exceptions. Before you add another prompt, package what the agent may do as AI Agent Skills (SKILL.md) and measure behavior with evals. Pick one workflow this week. Ship one Skill and a handful of free-response evals together.

---

Internal agents fail at onboarding, not model capacity

An internal agent that sounds fluent can still behave like an untrained new hire. Public knowledge appears in the training data. Tribal routing does not.

Independent definitions of the same gap

• Pinggy calls Skills an onboarding guide for a new hire: coding standards, architecture, and project conventions, written for the Agent instead of a person.
• SSW Rules tell teams to encode internal frameworks and tribal knowledge as Skills when the model never saw them.
• connect24h frames the accuracy gap as onboarding plus Skills-and-Eval design, not prompt tuning.

---

One-organization pilot numbers

Matsuo Institute compared three conditions on 20 August 2026 (Zenn): no internal knowledge, live Notion search, and Codex Skills built from new-hire docs.

An independent recap by piske_cc restated item accuracy as 33.0% / 87.3% / 96.0%.

• Treat that as a 15-task, one-organization pilot.
• Do not copy the percentages into your production scorecard.

---

Sources for the onboarding claim

Source (as of August 2026):

• Anthropic Agent Skills
• Pinggy
• SSW Rules
• Matsuo Institute (Zenn)

---

Package the procedure in SKILL.md

A Skill is a folder. YAML says what it does and when to use it. Markdown holds the procedure. Extra detail lives in references/. Deterministic work lives in scripts/. The Agent Skills specification requires at least SKILL.md. That is not a one-off chat prompt.

Procedure

Do this in order:

1. Make the directory name match name: lowercase, digits, hyphens, 1–64 characters. No leading, trailing, or consecutive hyphens.
2. Put trigger language in description (1–1024 characters). “Helps with X” is weak. Say what it does and when to load it.
3. Keep the SKILL.md body under 500 lines / about 5k tokens. Write conditions, order, done-state, who to ask when information is missing, and which document wins on conflict.
4. Push detail into references/. Relative paths stay one level from the skill root. Validate with skills-ref validate ./my-skill.

---

Startup cost

Startup metadata costs about 100 tokens per Skill. Fifty Skills are about 5,000 tokens of catalog. The body loads only when the Skill fires. Script source stays out of context; only the output comes back.

---

Spec limits

Limits matter.

• Reserved words anthropic and claude are banned in name.
• Custom Skills do not sync across claude.ai, the API, and Claude Code.
• API Skills have no network and cannot install packages at runtime. Claude Code has full network.
• Agent Skills are outside ZDR.
• API use needs the beta headers code-execution-2025-08-25, skills-2025-10-02, and files-api-2025-04-14.

---

Databricks contrast

Databricks documents databricks aitools install (--agents claude-code, --scope project, --skills bundles,sql). Their line is explicit: Skills teach how; MCP lets the agent do the work.

Source (as of August 2026):

• agentskills.io specification
• Anthropic Agent Skills
• Databricks Agent skills

---

When to use AI Agent Skills vs MCP or live search

If the rule that changes the answer is stable, put it in a Skill. If the facts change on every call—inventory, live permissions, or documentation that moves weekly—use MCP or search. When you need both, the Skill should say when to call which tool.

Skills vs MCP at a glance

Axis	Skills	MCP / live search
Question answered	How to proceed (procedure, house style)	What to reach now (live data, tools)
Shape	SKILL.md plus optional scripts/references	Running server / search + fetch
Idle cost	~100 tokens per skill	Tool schemas + process / search every time
Fits change in	Method, priority rules, done definition	Data that differs per invocation
Failure	Misread instructions; description never fires	Missed hits, stale pages, latency, context bloat

---

Sources for the Skills vs MCP split

Source (as of August 2026):

• LlamaIndex
• Pickaxe
• Anthropic Agent Skills

---

Independent heuristics

• LlamaIndex treats MCP as schema-bound and deterministic, with discovery cost and network latency.
• Skills are cheap to install, but the model must interpret the how, so they are non-deterministic.
• In fast-moving SDK docs, a live Docs MCP source of truth beat Skills that had to be updated by hand.
• Pickaxe uses the same split: MCP when data changes between calls; Skill when method or house style changes.

---

Skills versus Notion search in the pilot

Matsuo Institute’s Skills versus Notion search:

• Item accuracy 96.0% vs 87.3%
• Median latency 60.4s vs 94.0s (about 36% faster)
• Mean input tokens 48.6k vs 219.5k (about 78% fewer)
• The Notion condition ran 217 searches and 272 fetches across 45 answers

Search is not the villain. Paying the retrieval tax for a stable priority rule is.

---

Define pass/fail with free-response evals

A Skill on disk does not mean the job is done. An eval is the definition of pass.

Anthropic terms

Anthropic’s evals guide splits:

• task
• trial
• grader
• transcript (what the Agent said)
• outcome (the environment’s final state)

“I booked it” is not a row in the reservations table. Agent evals measure the model plus the harness. Placement of that loop is the same problem as continuous evaluation for agents.

---

Three measurement layers

Matsuo Institute used three layers:

1. Closed questions (does it know the fact)
2. Free response (does the artifact meet a rubric)
3. Task completion (did tools change the world)

Closed questions were solvable by elimination and did not measure work. High multiple-choice scores are not a ship gate. The same guessing pressure shows up in why models avoid saying they do not know.

---

How to write evals

Write evals like this:

1. Pull tasks from real tickets and real failures. Do not use multiple choice.
2. Rubric the required items. Put dangerous acts in a separate critical bucket: publish without approval, delete an incident without reporting it, leave access in place.
3. Primary metric: item accuracy by meaning, not string match. Secondary: critical failure rate, latency, tokens, tool calls.

---

Trials, judges, and halt cases

4. Hide condition names and run multiple trials. If you use an LLM-as-a-judge, measure independent-grader agreement (951 / 972 items, 97.8% in that pilot).
5. Include stop-and-ask-a-human cases. Markus Eisele / The Main Thread tells teams to package the procedure as a Skill and run realistic evaluations that cover when the Agent must halt.

---

One-organization eval numbers

Condition	Item accuracy	Critical failure	Median latency	Mean input tokens
No knowledge	33.0%	100.0%	64.2s	41.9k
Live Notion search	87.3%	42.2%	94.0s	219.5k
Skills	96.0%	15.6%	60.4s	48.6k

Source: Matsuo Institute (Zenn), independent recap piske_cc (August 2026).

• Fifteen tasks × three conditions × three trials
• No human expert scoring of answer quality; business outcomes were not measured

---

Limits

• Even Skills left a 15.6% critical failure rate.
• Do not treat the table as permission for unsupervised production.
• Separating a completion claim from a real state change is the same split as catching coding-agent test tampering.

---

Adoption checklist: start with one workflow

Do not build a catalog first. Scale only after one workflow has both a Skill and an eval.

1. Pick one workflow whose branching conditions you can write down (intake, incident first response, review).
2. Move human onboarding into SKILL.md and references/: conditions, order, done-state, who to ask, which doc wins.
3. Company Skills live in a reviewed source-of-truth repo. Project Skills live in the project repo. Read third-party Skills in full before execution. SSW treats a Skill as prompt injection by design.

Measure and keep a regeneration path

4. Write several free-response tasks for that same workflow. Include critical items and halt-and-ask cases.
5. Run no-knowledge / search / Skills on the same tasks. Watch item accuracy and critical failure. Keep human approval if outcomes are unmeasured.
6. If Notion (or any wiki) still changes, decide the Skill regeneration path first. Manual drift will win. Fast-changing docs belong on MCP or search, not a stale Skill.

---

Decision rule

Decision rule:

• Stable procedure → Skill
• Per-call data → MCP / search
• Pass defined only as polite prose → rewrite the eval
• While critical failures remain, do not let the Agent publish or change access without a person

---

FAQ

If we add Skills, can we drop Notion / RAG search?

No. Skills hold stable procedure and priority rules. Search or MCP holds data that changes per call. If you use both, the Skill should say when to search.

---

Can we paste the whole internal wiki into SKILL.md?

No. Startup load is name and description, about 100 tokens. Keep the body under 500 lines / 5k tokens and split the rest into references/. Dumping everything into always-on context hurts both cost and triggering.

---

Is a high closed-question score enough to ship?

No. Multiple choice narrows the search space and can be solved with general knowledge. Score free-response artifacts against a rubric, and score outcomes in the environment when you can.

---

Are official Agent Skills Claude-only?

The format is the open agentskills.io standard. Databricks and Copilot-class tools also load Skills. Install paths and sync still differ by product. Do not expect Custom Skills to sync across surfaces.

---

Can we run community Skills as-is?

Do not. Read the full text for shell execution and secret-handling instructions. Treat internal Skills as PR-reviewed code.

---

Related articles:

• 13550
• 13555

Summary

The gap for an internal Agent is not more prompt text. It is packaged procedure (Skills) and an explicit pass definition (evals). SKILL.md fires from its description and loads body plus references in stages. MCP and live search do a different job.

Next action: one workflow, one Skill, a few rubric evals. Leave human approval in place while critical failures remain.