Skip to content

talkops-ai/talkops-alertmanager

v0.2.0Apache-2.0

Operational skills and MCP execution server for Prometheus Alertmanager triage, notification diagnostics, silence lifecycle management, routing validation, and governance.


Architecture: Skill Layer vs. Execution Layer

talkops-alertmanager-mcp-server strictly separates procedural SRE playbooks (Agent Skills) from deterministic API execution (FastMCP server):

src/alertmanager-mcp-server/
├── plugin.json                              # Agent Plugins v1.0.0 manifest (talkops-alertmanager v0.2.0)
├── mcp.json                                 # Default stdio MCP server wiring (read-only by default)
├── skills/                                  # 5 Progressive-Disclosure Agent Skills (4 scripts, 11 references)
│   ├── triaging-alertmanager-alerts/
│   ├── diagnosing-missed-notifications/
│   ├── managing-alert-silences/
│   ├── validating-alertmanager-routing/
│   └── auditing-alertmanager-governance/
└── alertmanager_mcp_server/                 # FastMCP Execution Plane
    ├── domain/                              # Matcher grammar, recursive route tree, inhibition algebra, SP001-SP010
    ├── services/                            # Multi-backend Alertmanager v2 HTTP client & /metrics exposition parser
    ├── resources/                           # 10 live am:// URI resources (deeply redacted)
    └── tools/                               # 14 tools across 4 selectable toolsets (all <= 8 params, 0 prompts)

5 Bundled Agent Skills (skills/)

Skill DirectoryPurposeOffline Scripts (scripts/)Reference Guides (references/)
triaging-alertmanager-alertsStructured on-call alert triage, storm correlation, inhibition awareness, and shift handovers(uses am_summarize_oncall)severity-response-matrix.md, handover-template.md
diagnosing-missed-notifications6-gate missed-notification decision tree across Prometheus firing state, route tree, mute intervals, inhibition, silences, and receiver /metrics(uses am_explain_routing & am_check_notification_health)notification-pipeline-decision-tree.md, integration-failure-catalog.md
managing-alert-silencesSafe maintenance window planning, blast-radius preview, SP001–SP010 policy enforcement, atomic in-place updates, and controlled expirationplan_maintenance_window.py, validate_silence_request.pysilence-policy.md, matcher-cookbook.md
validating-alertmanager-routingStatic Alertmanager YAML linting (LINT001–LINT010), recursive continue: true route simulation, and time-interval testinglint_alertmanager_config.py, test_routes.pyconfig-lint-rules.md, routing-best-practices.md
auditing-alertmanager-governanceFleet governance audit across default-route leakage, missing summary/runbook_url annotations, Watchdog health, and silence debt(uses am_audit_alert_hygiene & am_audit_silences)alert-hygiene-standards.md, governance-checklist.md, governance-report-template.md

14 MCP Tools Across 4 Selectable Toolsets

Control exposed toolsets via --toolsets or MCP_TOOLSETS (all or comma-separated triage,routing,silences,governance). Mutating tools are unregistered when MCP_ALLOW_WRITE=false (the default).

ToolsetToolAccessMax ParamsDescription
triageam_list_alertsRead-only8Query alerts from Alertmanager with full matcher grammar (=, !=, =~, !~), receiver/state filters, and pagination
triageam_list_alert_groupsRead-only6List Alertmanager dispatch groups from /api/v2/alerts/groups grouped by route group_by labels
triageam_summarize_oncallRead-only5Generate a structured on-call triage summary of active and suppressed alerts filtered by env, service, team, or severity
routingam_explain_routingRead-only5Simulate Alertmanager's nested routing tree (continue, inherited group_by/timers, mute_time_intervals) and live inhibition rules for labels or a live fingerprint
routingam_check_notification_healthRead-only2Inspect Alertmanager readiness, config reload status, HA gossip cluster peers, and /metrics receiver delivery failure counters
routingam_push_test_alertWrite (confirm=True)6Push a time-bounded synthetic test alert tagged with mcp_test=true to verify end-to-end receiver delivery
silencesam_list_silencesRead-only7List silences by state (active, pending, expired), matcher filter, created_by, or comment substring
silencesam_preview_silenceRead-only7Dry-run a silence against active alerts, affected receivers, overlapping silences, and SilencePolicy (SP001–SP010)
silencesam_create_silenceWrite8Create a silence after enforcing SilencePolicy (SP001–SP010), overlapping-silence deduplication, and blast-radius confirmation
silencesam_silence_alertWrite8Derive scoped matchers (instance or service) from a firing alert's fingerprint or labels and create a policy-validated silence
silencesam_update_silenceWrite7Atomically update/extend an active silence in-place (POST /api/v2/silences with id), enforcing the total lifetime cap (SP007)
silencesam_expire_silenceDestructive (confirm=True)5Expire an active silence after previewing how many suppressed alerts will immediately resume paging
governanceam_audit_alert_hygieneRead-only5Audit active alerts for default-route leakage, missing summary/runbook_url annotations, invalid severities, chronic long-firing alerts, and Watchdog presence
governanceam_audit_silencesRead-only3Audit silence history, active silence policy compliance (SP001–SP009), stale 0-match silences, and silences expiring soon

10 Live am:// Resources

URIDescription
am://backendsConfigured Alertmanager backends with live health, version, and default flag
am://backends/{backend_id}Detailed status, readiness, version, uptime, and HA gossip cluster peers for {backend_id}
am://backends/{backend_id}/configDeeply redacted Alertmanager YAML configuration, nested routing tree, inhibition rules, and time intervals
am://backends/{backend_id}/receiversConfigured receivers (Slack, PagerDuty, email, webhook, OpsGenie) with redacted integration credentials
am://backends/{backend_id}/alerts/activeBounded live snapshot of active firing alerts on {backend_id}
am://backends/{backend_id}/alerts/groupsLive snapshot of Alertmanager dispatch groups on {backend_id}
am://backends/{backend_id}/alerts/{fingerprint}Full labels, annotations, receivers, and suppression status for alert {fingerprint}
am://backends/{backend_id}/silences/activeLive snapshot of active silences on {backend_id}
am://backends/{backend_id}/silences/{silence_id}Full matcher details, author, comment, and status for {silence_id}
am://audit-logRecent MCP-initiated operations (create/update/expire silence, push test alert, policy denials)

Quick Start & Configuration

CLI Usage

# Default safe mode: Read-only over stdio
uv run alertmanager-mcp-server --transport stdio --alertmanager-url http://localhost:9093

# Enable mutating tools (silence creation/updates & synthetic test alerts)
uv run alertmanager-mcp-server --allow-write --alertmanager-url http://localhost:9093

# Restrict to specific toolsets
uv run alertmanager-mcp-server --read-only --toolsets triage,routing,governance

# Run over Streamable HTTP on port 8769
uv run alertmanager-mcp-server --transport http --host 0.0.0.0 --port 8769

Environment Variables

VariableDefaultDescription
ALERTMANAGER_BASE_URL / ALERTMANAGER_URLhttp://localhost:9093Single-backend Alertmanager base URL
ALERTMANAGER_BACKENDS""JSON array of multi-backend configs (id, base_url, auth_header, headers, path_prefix, is_default)
MCP_ALLOW_WRITEfalseSet true (or pass --allow-write) to register mutating silence and test-alert tools
MCP_READ_ONLY(unset)When true, forces read-only mode (overrides MCP_ALLOW_WRITE=true)
MCP_TOOLSETSallComma-separated toolsets (triage,routing,silences,governance or all)
MCP_TRANSPORTstdioTransport protocol (stdio or http)
MCP_PORT8769HTTP bind port when MCP_TRANSPORT=http
AM_MAX_SILENCE_MINUTES1440Maximum allowed silence lifetime in minutes (SP007)
AM_SILENCE_CONFIRM_THRESHOLD50Matched-alert blast-radius threshold requiring explicit confirmation (SP010)
AM_AUDIT_LOG_PATH""Optional JSONL file path for durable write audit logging

Development & Verification

# Run full unit, tool, regression, and skill/manifest test suite
uv run pytest -v

# Run static type checking
uv run pyright .

# Build wheel and sdist
uv build