SRE Agent host setup and usage
The plugin supplies investigation instructions, generated investigator roles, local CLI/MCP tools, and Claude investigator hooks. The agent host follows the workflow and presents approval decisions; the Go executable collects evidence and maintains local records. It does not run an autonomous repair service. Start with the operator walkthrough, or consult the architecture for the implementation boundaries.
Investigation workflow
The skill starts at Phase 0: load the project memo and check its freshness. Phases 1 through 6 then scope the incident, discover the environment, collect evidence, analyze hypotheses and expected behavior, propose options for approval, and apply/validate the approved option. Failed validation returns to evidence collection. Each option names its target, preview steps, risk, impact, and rollback plan.
Seven playbooks cover Kubernetes, metrics, logs, changes, traces, EKS, and GKE. Symptom-based triage selects an initial wave and an escalation wave from the applicable investigators. Hosts with supported subagents can dispatch them; otherwise the skill executes their playbooks inline. State snapshots around waves compare selected fingerprints, not all possible mutations. Unreachable clusters or missing tools make that check inconclusive and must be recorded.
Installation
The host integration follows Claude Code plugin packaging and the Agent Skills format.
For Claude Code, run these commands in the host:
/plugin marketplace add glapsfun/sre-agent
/plugin install sre-agent@sre-agent
For noninteractive setup, the corresponding commands are
claude plugin marketplace add glapsfun/sre-agent and
claude plugin install sre-agent@sre-agent. A local marketplace installation
uses the absolute checkout path containing .claude-plugin/marketplace.json.
Update with /plugin marketplace update sre-agent.
The Claude plugin provides /sre-agent, investigator agents, MCP registration,
and hooks. Skills-only installation copies the skill and leaves host
registration to you:
npx skills add glapsfun/sre-agent --skill sre-agent --agent codex --global -y
npx skills add glapsfun/sre-agent --skill sre-agent --agent gemini-cli --global -y
npx skills add glapsfun/sre-agent --skill sre-agent --agent copilot --global -y
Choose the command for your host. Omit --global for a project installation.
npx skills does not install plugin-level hooks, slash commands, MCP
registration, or Claude agents. It can run the investigation playbooks inline.
For Codex subagents, locate the installed skill and run its bundled
scripts/install-codex-agents.sh. The default target is
${CODEX_HOME:-~/.codex}/agents/; with --project, run it from the intended
project because the target resolves against the current directory. Re-run it
after updating the skill to keep the generated roles synchronized. Configure
the parent-session MCP filter below before dispatching investigators.
Claude and Codex local fixtures were exercised. The Codex portable MCP fixture used version 0.167.1 and a locally built binary. Gemini was unavailable for runtime checks; Copilot CLI and VS Code packaging were not runtime-tested. Their configuration recipes are setup guidance, not a claim of verified host behavior. JSON validation alone does not prove host startup or permission filtering. Bash fallbacks still need their tools, network access, and a read-only identity.
MCP tools
The source-built sre mcp command serves stdio only. It exposes exactly
discover, logs_fold, prom_query, timeline, recall, run_resume,
ledger_add, and report. It has no HTTP listener, resources, or authentication
layer. stdout carries only MCP messages; diagnostics go to stderr.
The launcher pins v0.18.0. If that release or its mcp command is unavailable,
use a verified source build or the CLI and script fallbacks below.
Prefer available typed read tools for covered operations. Supply the exact
nonempty pinned cluster context to discover, logs_fold, prom_query, and
timeline, including inline log folding, an explicit Prometheus URL, and automatic endpoint discovery.
Preserve the random external-data boundaries in returned text. run_resume
reads an explicit run without binding a session. report returns content and
has no output-file argument. Use the guarded Bash fallback for uncovered
operations or unavailable registration/server, and record the limitation.
The recommended Claude read permissions are individually named:
mcp__plugin_sre-agent_sre__discover
mcp__plugin_sre-agent_sre__logs_fold
mcp__plugin_sre-agent_sre__prom_query
mcp__plugin_sre-agent_sre__timeline
mcp__plugin_sre-agent_sre__recall
mcp__plugin_sre-agent_sre__run_resume
mcp__plugin_sre-agent_sre__report
Claude investigators receive only the subset relevant to their role: discovery for
all seven; logs and timeline for Kubernetes/EKS/GKE; logs for the logs
investigator; Prometheus queries for metrics; timeline for changes. Trace
queries retain their guarded Bash recipes. Recall, resume, and report are
available to the orchestrator. ledger_add is the sole write tool and requires
a separate orchestrator permission decision. Never grant it to investigators.
A server-wide wildcard such as mcp__plugin_sre-agent_sre__* includes ledger
writes and is not the recommended read-only setup. readOnlyHint annotations
describe tools; they do not enforce permissions or replace the guard.
Semantic reads such as resume and recall can create or migrate the
repository-local database when it opens. They do not append ledger entries.
The report tool instead requires an existing compatible database and opens it read-only.
The final encoded MCP result is limited to 8000 bytes, including SDK-added
fields. Oversized reads return result_too_large with a CLI route instead of
silently truncating the result. A committed ledger write can return a compact
receipt with stable run/entry IDs and a gap; retry an uncertain append only
with the same entry ID and payload.
Claude loads plugin-root .mcp.json, whose command is
${CLAUDE_PLUGIN_ROOT}/skills/sre-agent/scripts/sre with the separate argument
mcp. Codex uses the portable plugin.json and
mcp.json: bash receives
${PLUGIN_ROOT}/skills/sre-agent/scripts/sre and mcp as separate arguments.
That portable layout started and called discover in an installed Codex
0.167.1 fixture. The compatibility layout was recognized too, but needed an
absolute launcher path because its root placeholders stayed literal in that
runtime. These fixtures used a locally built binary, not a published release.
Gemini's gemini-extension.json declares server sre
with ${extensionPath}/skills/sre-agent/scripts/sre and the separate mcp
argument, following the extension reference.
Gemini runtime expansion has not been tested locally because Gemini is absent.
Keep the placeholders literal in manifest files; the named host must expand
them. Parsing JSON alone does not verify startup or tool execution.
Required Codex investigator permissions
Before starting a Codex investigation, install the seven-read allowlist in
its parent session. Custom agent TOML files inherit parent MCP servers;
their instructions cannot override that policy. Keep ledger writes in a
separate orchestrator-only session that never spawns investigators. Do not
add ledger_add to an investigation session or resume investigators from a
write-enabled parent.
For the bundled portable plugin, add this policy to your Codex config.toml.
Replace sre-agent@sre-agent with the exact installed plugin ID, including
its marketplace suffix, shown by /plugins:
[plugins."sre-agent@sre-agent".mcp_servers.sre]
enabled_tools = ["discover", "logs_fold", "prom_query", "timeline", "recall", "run_resume", "report"]
For manual registration, locate the installed skill's executable scripts/sre
or a verified source-built sre, then use this complete configuration:
[mcp_servers.sre]
command = "/absolute/path/to/skills/sre-agent/scripts/sre"
args = ["mcp"]
enabled_tools = ["discover", "logs_fold", "prom_query", "timeline", "recall", "run_resume", "report"]
codex mcp add sre -- /absolute/path/to/skills/sre-agent/scripts/sre mcp
registers the transport only. Add the allowlist before using investigators.
Choose one registration path; disable duplicate SRE registrations that could
expose unfiltered tools. These policies use the supported
Codex MCP tool allowlist;
subagents inherit parent MCP configuration.
Start a fresh investigation session after setup. Before dispatching any
investigator, inspect /mcp and verify the effective SRE tool list contains
exactly the seven reads above and excludes ledger_add. Call discover with
the pinned context to check startup. If the effective filter cannot be
verified, disable that MCP registration for the investigation and use the
guarded CLI fallback, recording the limitation. A role's prose restriction
or a tool's read-only annotation is not proof of enforcement.
In the separate orchestrator-only session, grant ledger_add only after the
write-permission decision. Keep investigators in the filtered session and
return their evidence to the orchestrator for recording. Retain the CLI
ledger workflow when MCP write permission is unavailable.
For Copilot CLI,
use /mcp add, name the server sre, and enter the absolute launcher path
followed by mcp in Command. Select only the seven read tools above (using the
host's displayed tool names), with ledger writes decided separately.
For VS Code,
open Chat: Open Customizations, select MCP Servers, and add a stdio server
named sre with that executable and argument. Copilot's native configuration
uses mcpServers; older editor-managed .vscode/mcp.json uses servers.
Neither client was runtime-tested for this packaging change.
npx skills installs the skill without registering these plugin manifests.
Use manual registration or keep the CLI and bundled scripts. When MCP is
unavailable, locate the launcher and use sre discover --context <ctx>,
sre logs fold --context <ctx> --namespace <ns> --pod <pod>,
sre prom query '<PromQL>' --context <ctx>, or
sre timeline --context <ctx> --namespace <ns> with the operation's remaining
arguments. CLI stdin folding uses sre logs fold < app.log without cluster
flags. Local reads use sre recall, sre run resume <run-id>, and
sre report <run-id>. If the launcher or command is unavailable too, retain
the existing discovery/evidence scripts and report the gap. Investigator Bash
fallback remains subject to the read-only guard where installed and the
read-only identity on every host; a denied call is not a reason to bypass it.
CLI launcher
Run skills/sre-agent/scripts/sre version from this plugin directory. The
launcher downloads the pinned v0.18.0 release on first use for Linux or macOS,
on amd64 or arm64. It requires Bash, curl, and a SHA-256 tool: OpenSSL,
sha256sum, or shasum.
It prefers OpenSSL when available and verifies all binary bytes on every launch.
The binary prints one JSON object with ok, data, gaps, and errors.
Use --format text for a readable text view. The CLI provides version,
discover, logs fold, prom query, timeline, run, ledger,
incident add, recall, report, and process-log log; the
existing investigation scripts remain available.
If the installed binary lacks a command, use a source build or the existing
scripts for that operation.
sre logs fold reads stdin and returns redacted patterns, counts, first/last
timestamps, levels, and samples. For example:
sre logs fold --max-patterns 20 < app.log
sre logs fold --context prod --namespace payments --pod checkout-abc \
--container app --previous --since 1h --format text
Errors sort first, then warnings, then other levels; rarer patterns sort first
within each group. The default limit is 20 patterns. omitted_patterns and
omitted_lines account for everything outside the limit. Samples and templates
carry fresh external-data markers. Pod collection streams a finite snapshot,
enables timestamps, and uses a 30-second deadline. It selects explicit context,
then SRE_KUBE_CONTEXT, then the current context without changing it.
See log folding details
for masking, timestamp handling, and limits.
sre timeline returns every matching retained Kubernetes event, ReplicaSet
revision, Flux/Argo revision, local git commit and currently firing Alertmanager
alert in one sorted list. Namespace is required; workload accepts an exact name
or KIND/NAME. For example:
sre timeline --context prod --namespace payments --workload deployment/checkout \
--since 2h --repo ./checkout --alertmanager-url http://localhost:9093 \
--timeout 30s --format json
The default window is the last two hours, captured once in UTC. Context follows
--context, SRE_KUBE_CONTEXT, then one current-context lookup. Setup and all
collectors share the timeout. Alertmanager uses --alertmanager-url, then
SRE_ALERTMANAGER_URL; supply an already reachable HTTP(S) endpoint. The command
does not start a port-forward or use discovery hints as endpoints. Missing repo
or endpoint produces a gap. A failed source sets ok=false and exits 1 while
retaining independent successes; absence and historical-coverage gaps alone
exit 0. An unresolved requested workload never falls back to namespace scope.
JSON keeps the complete typed list at data.entries, plus all seven source
summaries, undated revision context and structured gaps. One fresh external-data
boundary marks the report; text presents the same records and coverage. There
is no entry limit or pagination. Events and ReplicaSets cover retained objects,
not deleted history; aggregate event counts do not date each occurrence.
ReplicaSet creation does not prove rollout or revision application time. Safe
pod-template comparisons omit literal environment values, annotations, commands
and other excluded fields while preserving references such as Secret names and
keys. Git covers local HEAD ancestry by committer time, not deployment time.
GitOps ownership must be verifiable; ambiguous remote associations and unknown
timestamps remain gaps or undated context. Alertmanager covers current firing
alerts, including suppressed alerts, not resolved alert history. Cloud audits,
GitHub activity, Helm history and mesh evidence still use the existing helpers.
The cache lives under CLAUDE_PLUGIN_DATA/sre-agent when set, otherwise
XDG_CACHE_HOME/sre-agent, otherwise HOME/.cache/sre-agent. Each version and
platform has its own directory. Downloads use HTTPS, bounded timeouts, and a
release SHA256SUMS manifest. The launcher verifies the checksum before
publishing a cache entry and before every execution. Checksums detect changed
downloads and cached binaries; they trust the release manifest and local cache.
See release operations for version alignment, publication, and recovery. The launcher does not silently execute fallback scripts.
A valid cached release works offline. With no cache, a failed download exits
nonzero and points to the existing scripts, such as sre-env-discovery.sh and
sre-evidence.sh, including when the pinned release is unavailable. A checksum
mismatch refuses execution; remove the affected version cache and retry online.
A concurrent download exits with a retry message. After an interrupted download,
remove its stale .lock directory only after confirming no launcher is running.
Release tags build and run the version command on all four native platforms
before publishing binaries and SHA256SUMS. Updating the launcher pin is a
separate source change when adopting a later release.
Local database
The CLI storage package creates .sre-agent/sre.db at the repository root on
first use. Linked Git worktrees get their own database. Keep .sre-agent/
out of version control in repositories where you use the agent. The directory
and database are private to the current user.
The store uses a pure Go SQLite driver, WAL, and a five-second busy timeout for concurrent local processes. Use a local filesystem; SQLite WAL does not support sharing a database between machines over a network filesystem. Schema upgrades run in a transaction when the database opens. A database from a newer binary is refused. After all users close it, a deleted database is recreated empty on next use.
The investigation ledger stores append-only typed updates and redacts secret values before each committed write. Existing incident Markdown and project memos stay in place.
sre run start "checkout errors" --id checkout-run --repo ./checkout
sre ledger add evidence --run checkout-run --id evidence-1 --file evidence.json --repo ./checkout
sre run resume checkout-run --repo ./checkout
sre run show checkout-run --offset 1 --repo ./checkout --format text
For example, evidence.json contains one concise observation:
{"fact":"checkout returned HTTP 503","source":"provided incident observation"}
The accepted kinds are metadata, evidence, hypothesis, expected-behavior,
option, approval, action, validation, and safety-violation. Use --file -
for stdin. Input is a single JSON object of at most 64 KiB with no unknown fields
or trailing JSON. Every add requires a stable --id; retry the same payload
with the same ID after uncertain completion. Run and entry IDs use 1-128 ASCII
letters, digits, underscores or hyphens. Start generates a random run ID when
--id is omitted. Retain the returned ID.
Ledger adds require --run or SRE_RUN_ID. An explicit resume ID takes precedence
over that variable; without either, resume reads the latest created run. Empty
storage reports no runs recorded and does not start a run. --repo resolves
the Git root, including when called from a subdirectory. --session-id on start
explicitly binds a known harness session; a metadata session_id update can
bind another session. No harness environment variable is assumed.
Resume and show include every ledger heading and page history within an
8192-byte response budget. JSON exposes data.view.text inside one fresh
external-data boundary, entry IDs, through_seq, omitted_entries,
next_offset, and truncated_fields. Follow the returned offset using show
or resume. Fields too large to display are shortened and named; full contents
remain in the local database. A collection that cannot fit produces an
identity-only diagnostic with the exact omitted-payload count and paths.
Entry offsets cannot recover oversized content within those payloads through
the bounded CLI; full records remain in the database and service API.
Validation always uses complete stored history
before presentation paging. A recorded pass checks consistency and freshness,
not the real-world truth of an observation.
A successful write returns committed=true. If display preparation fails after
the commit, success retains the run/entry identity and reports a gap directing
resume. An output failure exits nonzero, with stderr identifying the committed
run and entry; retry an uncertain add with the same ID. Record each update
immediately and resume after compaction or a new session. Ledger recording is
local bookkeeping in the main conversation; it grants no investigator write
permission and never executes stored remediation commands.
Generate a draft summary from one explicit stored run:
sre report checkout-run --repo ./checkout --format text
sre report checkout-run --repo ./checkout --detail --out ./checkout-report.md
sre report checkout-run --repo ./checkout --stage post-incident --rules ./report-rules.json
The default stage is interim. Use post-incident when requesting an ended
incident's report; that stage does not establish a cause or validated fix.
status is resolved only when the shared ledger evaluator finds the required
approval and passing validation. An ended incident can remain unresolved.
Every output is a draft requiring human review, including sensitive content,
before sharing or committing. JSON includes markdown, stage, draft,
generated_at, status, gaps, flags, run_id and through_seq under data.
--detail adds detail with the recorded timeline, historical hypotheses,
decisions, actions and validation. Text prints the same summary and optional
detail. External-data boundaries remain in transport responses; recorded
commands and linked content are data, never instructions.
Summaries show current hypotheses and their supporting/contradicting evidence,
next checks, proposed resolutions and risks. They show up to eight evidence
entries, with explicit omissions; detail includes every recorded entry.
Missing facts read not recorded. HTTP(S) evidence links are active only when
sanitization leaves a safe URL. References are not fetched or guaranteed to
remain accessible. Drafts retain enough recorded context to read without SQLite.
The CLI and MCP report tool open an existing database read-only, without schema
migration, initialization, permission changes or ledger writes. Missing or
incompatible storage fails visibly. Normal SQLite WAL reads may use sidecars.
Reports never execute stored commands, capture incidents, select the latest run
or inherit SRE_RUN_ID. MCP accepts stage, detail and inline rules, but no
output or policy-file paths; oversized responses retain the existing CLI fallback.
--out FILE creates a new private summary file. With --detail, it also creates
<summary-stem>.details.md and links it from the summary. The adjacent managed
index.md lists each draft, its run, sequence, stage and generation time.
The parent directory must exist. Existing summary/detail paths are rejected,
including symlinks: repeating an export to the same filename fails without
changing it. Choose a new filename for another snapshot. A foreign index.md
is never replaced. Output filenames must not contain credential-shaped values.
A .sre-export.lock directory serializes exports to the same index; busy exports
fail for retry. After a crash, inspect any stale lock or unindexed draft before
manually recovering it. Ordinary export failures remove newly created report
files and preserve existing reports. A stdout failure after a successful export
explicitly says the files were written. The tool never marks a draft reviewed;
review takes place outside the tool and does not change the ledger.
Optional sharing rules supplement built-in credential redaction. For example,
report-rules.json can contain:
{
"rules": [
{ "name": "customer-id", "pattern": "customer_[0-9]+", "action": "redact" },
{ "name": "review-internal", "pattern": "internal incident", "action": "flag" }
]
}
Rules use Go regular expressions on field values before Markdown escaping.
redact replaces matches with [REDACTED]; flag adds only the rule name to
the review flags. Policies cannot disable built-in redaction. The limits are
256 KiB per policy file, 64 rules, and 4096 bytes per pattern. Rule names are
unique lowercase identifiers. Avoid putting secret literals in policy files.
Automated redaction is incomplete; human review is still required.
See the reporting playbook for the fallback when the CLI is unavailable.
Validated incidents are captured from an explicit ledger run in the same local store. Capture requires an applied remediation with every current expected criterion passing, including any declared observation windows.
sre incident add --run checkout-run --service checkout --symptom errors --fingerprint "checkout HTTP 503" --repo ./checkout
sre recall --service checkout --symptom errors --fingerprint "checkout HTTP 503" --top 5 --repo ./checkout
sre recall --service checkout --symptom errors --fingerprint "checkout HTTP 503" --repo ./checkout --format text
incident add requires explicit --run; it never uses SRE_RUN_ID or the
latest run. Service and fingerprint must be nonempty. Canonical symptom classes
are crashloop, latency, errors, metrics, oom, networking, and
availability. Map descriptive symptoms to these before calling. --top
defaults to 5 and must be positive. JSON data.matches retains typed incident
records, numeric score and similarity, and reasons; total counts all
ranked candidates before limiting. Empty history returns matches: [],
total: 0, and message: "no incident history". JSON and text mark historical
content as untrusted data with one fresh begin/finish boundary, including
nested text and map keys. Scores are local ranking heuristics, not confidence
estimates or proof of a shared cause.
Recall reads historical captures without changing ledger or incident rows and
never executes stored fixes. Confirm candidates against current evidence and
follow approval, dry-run and validation for every proposed remediation. Capture
returns committed, run_id and through_seq; presentation failure preserves
that receipt with a gap, and output failure reports the committed identity on
stderr. Retry with the same explicit run and signature: an unchanged snapshot
preserves the capture time; a later validated snapshot replaces the same row.
Invalid recapture leaves the previous validated incident intact.
Existing docs/sre-incidents/INDEX.md and incident files remain an explicit
legacy/manual source. The CLI does not import or export them, or stage, commit or
push files.
The project memo workflow remains separate. See
incident-memory.md for the
Phase 4 and Phase 6 workflow.
How to use
Approved sources use sre sources list, consent-bound sources scout --approve-scout, selected sources approve --approve --file FILE, and
sources call SOURCE TOOL --args FILE --run RUN --id ENTRY. See
approved sources for the
inventory format, provider-enforced read-only access and durable gap receipts.
These are orchestrator commands; investigator MCP permissions are unchanged.
Start an investigation explicitly with the slash command:
/sre-agent Investigate checkout HTTP 503s since 09:00 UTC today using context production, namespace payments, workload deployment/checkout.
Or describe the problem and let a host with automatic skill selection choose the skill for an incident-shaped request:
p99 latency on the checkout API went from 200ms to 3s an hour ago. Nothing was deployed.
Our application metrics disappeared from Grafana yesterday.
Pods in namespace search are OOMKilled every few hours.
The checkout service is slow and we run Tempo; find which downstream call is eating the time.
Our logs live in OpenSearch, no Loki; what errors is the orders app throwing since 09:00?
The fix is applied and metrics look good; verify it under load before we close the incident.
The agent then walks the loop above, keeping a visible investigation ledger (environment, evidence with the command that produced each fact, hypotheses with confidence, actions, validation results) updated at every phase. When it reaches the approval gate it presents the options and stops - pick one, ask for more evidence, or reject them all.
After resolution you get a final incident report (summary, impact, timeline, root cause, actions, validation, rollback info, follow-ups) that you can save to your repo.
What's inside
| Component | Purpose |
|---|---|
skills/sre-agent/SKILL.md | The orchestrator: loop, phase gates, safety rules, investigation ledger |
skills/sre-agent/references/investigators/ | The seven investigator playbooks (k8s, metrics, logs, changes, traces, eks, gke); single source of truth; executed inline when subagents are unavailable |
agents/; sre-k8s-investigator, sre-metrics-analyst, sre-logs-investigator, sre-change-historian, sre-trace-analyst, sre-eks-investigator, sre-gke-investigator | Read-only Claude Code subagents dispatched in parallel for evidence collection; generated from references/investigators/ by scripts/gen-sre-agent-artifacts.sh |
skills/sre-agent/agents/codex/ | The same seven subagents in Codex TOML format (generated); installed by install-codex-agents.sh |
skills/sre-agent/references/ | Deep knowledge loaded on demand: discovery, PromQL (golden signals, kube-state, burn rates), LogQL and error taxonomy, Elasticsearch/OpenSearch query DSL, TraceQL/Jaeger tracing, deep Kubernetes evidence (nodes, NetworkPolicy, DNS, storage), service mesh (Istio/Linkerd), Grafana API discovery, root-cause analysis method, remediation templates, Terraform remediation (locating IaC-managed resources, safe plan reading, plan-classified apply/rollback), Pulumi remediation (locating IaC-managed resources, safe preview reading, preview-classified apply/rollback), Crossplane remediation (ownership detection, Ready/Synced condition and trace interpretation, status-check-verified apply/rollback), validation checklist and report format, k6 load validation, GitHub investigation via gh (repo discovery, change timeline, prior-incident and code search), sibling-skill deference (argocd/fluxcd/helm/bash-scripting/kubernetes-operator/karpenter/gh-guru/glab-guru), pinned official sources |
skills/sre-agent/scripts/ | Evidence, preview, and setup helpers: sre-env-discovery.sh (tools, cluster, GitOps, cloud), sre-obs-discovery.sh (Prometheus/Alertmanager/Grafana/Loki/Mimir/Tempo/Jaeger/Elasticsearch endpoints, service-mesh and k6 detection), sre-evidence.sh <ns> <workload> (one-shot evidence pack), sre-snapshot.sh (Phase 3 mutation verification: snapshot/diff spec-hash fingerprints around each wave's dispatch), sre-gh-discovery.sh (read-only GitHub discovery/search via gh: repo/timeline/incidents/code subcommands), terraform-plan-check.sh <tf-dir> (Phase 5/6 Terraform plan classification: no-op/create/update/delete/replace counts, flags any destroy/replace), pulumi-preview-check.sh <stack-dir> (Phase 5/6 Pulumi preview classification: same/create/update/delete/replace counts, flags any destroy/replace), crossplane-status-check.sh [TYPE[.VERSION][.GROUP][/NAME]] (Phase 2-6 Crossplane evidence and verification: classifies Provider/Managed/Composite Resource Installed/Healthy/Ready/Synced conditions), install-codex-agents.sh (Codex subagent install) |
commands/sre-agent.md | The /sre-agent <problem> entry point (Claude Code) |
hooks/ | The read-only guard for the seven investigator subagents (Claude Code) |
Safety model
- Read-only until approval: the skill instructs investigators to collect evidence only. The main conversation may apply changes only after you select an option. These instructions depend on host permissions and credentials.
- Dry-run first:
kubectl diff/--dry-run=server,helm --dry-run,flux diff,terraform plan,pulumi previewbefore every apply. - Secrets: metadata only: names, ages, revisions; never values.
- GitOps- and IaC-aware: fixes to Flux/Argo-managed workloads, and to Terraform/Pulumi/CDK/eksctl-managed EKS infrastructure (node groups, IRSA roles, ALB config), are directed to the source repository/IaC, not hand-edited directly.
- Gentlest effective action: restart over delete, scale over replace; blast radius stated before anything destructive.
- Load generation is mutation-class: the optional k6 validation step has its own approval gate, states target environment/RPS/duration/blast radius up front, and never targets production unless you explicitly say so.
Read-only enforcement
In a Claude Code plugin install, a hook (hooks/readonly-guard.sh) checks matching tool calls
from the seven configured investigators and allows only known read operations. Anything it doesn't recognize is blocked and
recorded as a gap. It also blocks reads of common credential files.
It stops mistakes and simple prompt injection, but it's not a sandbox. It can't see inside a script it allows, a GET request can still carry data out, and it doesn't cover the main conversation, which is where approved fixes run. The packaged Go checker runs through the checksum-verified launcher. If it cannot start, investigator calls fail closed; the main conversation and other agents remain outside the guard.
The pre-hook records each recognized investigator call in the repository's .sre-agent/sre.db.
Post success/failure hooks reconcile the same (session_id, tool_use_id) row. guard=allowed only
means the guard allowed the request; outcome=unknown does not establish execution or success.
Logging errors produce a nonblocking diagnostic and never change the guard's decision.
Bind the harness session when starting a run, or set SRE_RUN_ID to an existing run before launching
the harness. An explicit run ID takes precedence over session binding; hooks never select the latest run.
sre run start 'Investigate API latency' --id incident-1 --session-id SESSION_ID
sre log incident-1 --format text
sre log incident-1 --slowest 10 --by-investigator
sre log incident-1 --blocked --repo /path/to/repository
log defaults to JSON. Filters combine; investigator counts describe the selected rows after limiting.
Measured durations sort before unknown durations, with stable observation/session/call ordering for ties.
The timestamp records the first observation, not a tool start time. Absent duration and numeric exit
status remain null. The hook adapter does not infer process exit status from tool output or failure text.
Metadata contains sanitized command/path text, never complete inputs, responses, or failure messages.
Credential patterns, sensitive shell assignments, and user credential flags, including curl clusters,
are redacted before writing. Ambiguous credential arguments cause the
entire command text to be stored as [REDACTED]; simple values retain sanitized command context.
Identity fields that require redaction are rejected to avoid identity collisions. Do not place arbitrary
secret values in commands or identifiers.
The first observation pins the validated run and call metadata. Contradictory events or metadata produce
an absorbing conflict outcome and clear numeric metrics; a run-binding conflict leaves the original row
unchanged and emits a diagnostic. Repeated delivery cannot create another row for the same call.
The matchers cover every declared investigator tool: Bash, Read, Grep, Glob and WebFetch, plus
mcp__plugin_sre-agent_sre__.*. Glob records only its optional sanitized path; WebFetch records its
sanitized URL in the existing path field. Glob patterns, WebFetch prompts and responses are omitted.
Only the seven planned MCP read operations are allowed: discover, logs_fold, prom_query,
timeline, recall, run_resume, and report. ledger_add and unknown operations in this namespace are blocked for investigators.
Calls with no delivered hook cannot be logged. Permission denial can omit a post event, and schema
rejection can omit both events. Legacy pre events with all of session_id, tool_use_id, and cwd
absent remain guard-only; partially supplied or invalid metadata emits a logging diagnostic. Live MCP
and harness acceptance remain separate from the local hook-shaped integration tests.
The hard limit is a read-only identity. Have an administrator provision Kubernetes RBAC and AWS/GCP read permissions appropriate to the evidence you need, then investigate using that identity. Unreadable resources become gaps. Each remediation option names its target context and needs separate approval and credentials that permit the selected change.
The other documented install paths do not register this Claude investigator hook. Codex documents that custom agent instructions don't create a read-only
sandbox and that subagents inherit the session's sandbox policy, and npx skills installs only the skill. On
those paths, the read-only identity is the only hard limit.
Degraded environments
Missing tooling is the normal case, not an error. The playbooks describe fallbacks for common missing capabilities: no Prometheus → kubectl top + events; no Loki →
Elasticsearch/OpenSearch, else kubectl logs; no trace backend → latency
RCA proceeds metrics/logs-only; mesh present but no istioctl/linkerd
CLI → kubectl-only mesh evidence; no k6 → passive validation plus a script
you can run yourself; no cluster access at all → static analysis of repo
manifests with the limitation stated. Every gap is recorded in the ledger;
the skill requires live evidence before claiming a fix is verified. Permission
denials remain gaps and must never be bypassed through a fallback.
Works best with
Install alongside the cloud-native plugins from the
cnative-skills marketplace; the
orchestrator defers to them for deep work when they are available, and falls
back to its own playbooks when they are not:
kubernetes-operator, helm,
fluxcd, argocd, karpenter.