How to Delegate First-Pass Incident Triage to AI Safely: Evidence and Read-Only Gates

Whether AI should run first-pass production-incident triage is not a latency question. In the public case, a Sunday 02:33 PDT availability alarm produced an evidence-backed diagnosis on the ticket 13 minutes 35 seconds later.

📑Table of Contents

1. What the on-call actually did after a 13-minute diagnosis
2. Keep unattended investigation ReadOnly and leave resolution to humans
3. Compare official cases against an evaluation bar before you adopt
4. Checklist: decide whether your team can put this in production
5. FAQ
6. Summary

The on-call engineer typed one sentence: escalate, or treat it as a production bug.

The adoption gate for an AI SRE agent is not demo speed. It is the permission boundary: unattended sessions must stay read-only, and only humans may resolve tickets, change severity, or approve a deploy.

What to judge before you adopt

Judge adoption on the following criteria, not demo quality.

• evidence-backed root-cause analysis
• gated actions
• a secrets deny list

---

What the on-call actually did after a 13-minute diagnosis

First-pass investigation can be automated. The remaining human job is to read a finished brief and decide.

Official timeline

The Kiro post and the Japanese publication on the AWS Japan blog publish the same timeline.

1. A frontier-model availability alarm fired at 02:33 PDT on a Sunday. Monitoring opened a ticket.
2. At 02:46 PDT—13 minutes 35 seconds later—the ticket held a diagnosis: the stream had gone silent, customers were affected, competing hypotheses were ruled out, and a next action was named.
3. The on-call contribution was one decision.

---

Public investigation pipeline

The pipeline is also public.

1. An alarm opens a ticket.
2. A resident dispatcher starts one headless CLI session per ticket.
3. Scoped read-only connections search logs, tickets, pipelines, and code review. Claims link to the generated queries.
4. The visible thread gets a short post; the work log holds the detail.
5. The agent keeps an index of 107 skills and reads only the one matching playbook. Stuffing runbooks into a giant system prompt breaks around the tenth document, the official write-up says.

---

Volume numbers and the role change

The volume numbers are similarly specific.

• 96.9% of tool calls are reads.
• In another incident, 96.5% of errors concentrated in two delivery cells; the rest were zero.
• A representative month ran about 250 unattended investigations at a median of 13.6 minutes.
• Monthly spend was described as roughly nine of the vendor’s top subscriptions.
• Cost control is structural: session timeouts, stuck-run detection, concurrency caps, and cheaper models for fan-out—not self-restraint.

---

Role change in the independent digest

Antoine Buteau’s 2026-08-21 digest restates the same case as a shift from investigator to reviewer of AI-gathered evidence.

Use the digest for that role change; confirm the primary numbers on the official pages.

On-call rotation does not disappear. What changes is the 02:46 brief, not the 02:33 page.

---

Snapshot of the public figures

Lens	Figure or rule	Source
Alarm to diagnosis	13 minutes 35 seconds (02:33→02:46 PDT)	Kiro
Unattended volume	~250/month, median 13.6 minutes	AWS Japan
Default permission	Unattended sessions ReadOnly; Admin requests error	Same
Human-only actions	Resolve ticket, change severity, approve deploy	Kiro / Azure SRE Agent
Adoption bar	Evidence-backed RCA, visible integration failure, deny list	One2N

---

Sources for the public figures

Sources (as of August 2026):

• Kiro
• AWS Japan
• Microsoft Learn
• One2N

---

Keep unattended investigation ReadOnly and leave resolution to humans

Trust does not come from prompt polish. It comes from an allowlist and an explicit list of actions the agent must not take.

Kiro default permissions

In the Kiro design, unattended sessions are read-only by role allowlist.

• Admin requests error out.
• Credentials are scoped to one session and a declared task.
• Only humans resolve tickets or change severity.
• Autonomy is not a single slider. Grants and inhibitions are written per action in a steering file.

That is the same problem as keeping standing admin out of human on-call, which we covered for JIT access and TEAM on AWS. Decide the human elevation path before you hand the same blast radius to an agent.

---

Azure SRE Agent permission gate

Cloud SRE products repeat the split. The Azure SRE Agent overview on Microsoft Learn (updated 2026-07-30) evaluates tool calls with a permission gate before execution.

• The agent proposes mitigations and will not deploy a change without human sign-off.
• In the 02:47 payments-service example, it correlated a memory trend from 40 minutes before the alert with a GitHub deploy from two hours earlier, then pre-filled two options: restart pods, or adjust HPA.
• One investigation thread closed in seven minutes, without a war room or tab-hopping.

---

Limits that remain

• The chat UI is English-only.
• Availability depends on region and tenant settings.
• Billing is consumption-based.
• Wrong conclusions or unfit mitigations are expected. Review before approve.

---

SOC alert triage as a parallel

A SOC parallel is useful as a boundary check, not as the same product. IT Leaders (Keizo Hikawa, 2026-08-19) reports that Macnica began selling Prophet Agentic AI SOC Platform.

---

What the SOC report actually automates

• Agents investigate alerts from 200-plus SIEM, EDR, identity, mail, and SaaS sources, then notify via Slack, Teams, or webhook.
• Endpoint isolation can run automatically.
• New or tuned detection rules apply only after human approval.
• Macnica also offers 24/7 human severity judgment with escalation inside 30 minutes.

The domain is security-alert triage, not availability on-call. The shared pattern is automated investigation plus a human gate on lasting control-plane changes.

Excessive agency is also an early item on an AI security learning path.

---

Public failure modes

Failure modes are public too.

• One stale documentation line propagates across tickets at machine speed.
• A learning pipeline once ingested raw comments from interrupted sessions into the lesson store. The fix was schema validation before write, plus a nightly cleanup.
• A confident half-answer, issued before competing hypotheses are ruled out, is costlier than a wrong answer.

Do not let fluent prose replace query evidence.

---

Compare official cases against an evaluation bar before you adopt

Do not adopt on a 13-minute demo. The left two columns are investigation automation. The SOC column is a parallel in security operations. The right column is your pass line.

Trigger and default permission

Item	Production on-call first triage (public case)	Cloud SRE agent	SOC alert triage (sales report)	Pre-production evaluation bar
Trigger	Availability alarm → auto ticket	Monitor / PagerDuty / ServiceNow	200+ SIEM / EDR sources	Alertmanager etc.; singles and bursts
Default permission	ReadOnly; no Admin	Pre-exec permission gate	Investigate all; isolation may auto-run	Gated in prod; Secrets / DB denied

---

Human judgment and knowledge

Item	Production on-call first triage (public case)	Cloud SRE agent	SOC alert triage (sales report)	Pre-production evaluation bar
Human-owned judgment	Resolve and severity; one-sentence decision	Approve mitigation; no unsigned deploy	Detection-rule apply; 24h severity (30 min)	Admit missing evidence, or approve an action
Knowledge	107-skill index plus archive	Investigations persist as org knowledge	Rules visualized on MITRE	Fail if runbooks / history cannot be retrieved

---

Sources for each column

Item	Production on-call first triage (public case)	Cloud SRE agent	SOC alert triage (sales report)	Pre-production evaluation bar
Source	Kiro	Azure SRE Agent	IT Leaders	One2N

Sources: URLs in the table (as of August 2026).

---

Pre-production evaluation bar

Hemant Kumar (Staff SRE at One2N, 2026-08-11) sets the bar as whether you can trust the agent during incident triage given the evidence it can actually reach—not demo quality.

The five axes are:

• actionability and control safety
• RCA quality
• observability correlation
• incident memory
• security

---

OpenSRE v0.1 score

An OpenSRE v0.1 proof of concept, run about six hours on Claude Opus 4.8, scored as follows: defer, not reject.

• Actionability 0
• RCA -1
• Integrations -1
• Knowledge -1
• Security 0
• total -3

Fail reasons:

• It proposed a Helm rollback without evidence.
• It kept answering after an integration failed.
• It could not verify runbooks or history.

An “Inferred” label does not pass RCA without query evidence.

Read the matrix this way. The public case shows first triage can move forward when read-only access and hypothesis rule-out are in place.

The evaluation bar says defer when your integrations and incident memory are not there. Using the speed figure as a substitute for that design is the failure mode.

---

Checklist: decide whether your team can put this in production

Run the agent once in your environment, then choose defer, limited rollout, or production qualification. Popularity and republished headlines are not evidence.

1. Test a single alert and a multi-alert burst. Do not stop at a single-alert demo.
2. Break one integration. See whether the agent names the gap or invents a cause.
3. Confirm context outside the Helm happy path (kubectl / Kustomize).
4. Put a deny list on secrets, databases, and production writes before any write grant. Treat credentials with the same priority as GitHub Actions privilege-escalation controls.

Evidence, steering, and learning-loop checks

5. Require query evidence while hypotheses form. If evidence is missing, the agent must write that, not a guess as a conclusion.
6. Enumerate per-action bans in steering (“resolve and severity are human-only”). Do not use one autonomy slider.
7. Turn documentation fixes into lessons. Agents amplify doc bugs at machine speed.
8. Do not ingest unverified text into the learning loop. Schema-validate writes first, then run a nightly cleanup.

---

Azure and cost checks

Score the five axes. If the score is short, defer.

For Azure, add these checks.

• English-only UI
• regional availability
• consumption billing
• human review of proposals

For cost, the public reference is ~250 runs per month at a 13.6-minute median, with caps in timeouts, concurrency, and cheaper models—not willpower.

---

FAQ

Q1. If diagnosis lands in 13 minutes, can we drop on-call?

No. The public case keeps a rotation. What changes is reading the 02:46 brief, not starting investigation at 02:33.

---

Q2. Is default ReadOnly enough for production?

It is necessary, not sufficient. Keep production qualification closed until you have a secrets/DB deny list, masking checks, and a failure mode that does not stay silent when an integration breaks.

---

Q3. Does a stronger model stop skipped hypotheses?

The official write-up credits steering that rules out competing hypotheses before a conclusion, plus automatic checks on arithmetic, alarm state, and duplicates. A model bump is not a substitute.

---

Q4. Is SOC auto-isolation the same as SRE first triage?

No. SOC targets SIEM/EDR alerts. SRE targets availability, latency, and deploy correlation. The shared pattern is automated investigation with humans retaining rule application, severity, and deploy.

---

Q5. Can the learning loop run unconditionally?

No. Unverified text in the lesson store compounds. Validate schema before write.

---

Related articles:

• 12301
• 12300
• 12872

Summary

First-pass investigation is already running on public figures: 13 minutes 35 seconds, about 250 unattended cases per month. The differentiator for an AI SRE agent in 2026 is not speed.

It is ReadOnly investigation, hypothesis rule-out, and human-only resolve, severity, and deploy approval.

Next: run one single-alert case and one burst in your stack, then use the checklist to choose defer, limited rollout, or production qualification.