Architecture: Strict Skill & Execution Separation
Following the TalkOps Skill & Execution Separation Architecture (docs/skill-and-execution-separation.md), this repository separates how to think (Agent Skills) from how to act (MCP Execution Layer):
src/prometheus-mcp-server/
├── plugin.json # Agent Plugins v1.0.0 manifest
├── mcp.json # Portable MCP launcher config
├── skills/ # 6 Progressive-Disclosure Agent Skills
│ ├── triaging-prometheus-incidents-and-slos/ # RED/USE triage, multi-window SLOs, exemplars
│ ├── authoring-promql-and-analyzing-metrics/ # PromQL 3.0, Native Histograms, UTF-8 syntax
│ ├── authoring-and-testing-prometheus-rules/ # Alerting/recording rules & promtool testing
│ ├── optimizing-tsdb-cardinality-and-finops/ # Cardinality FinOps, relabeling, remote-write
│ ├── configuring-prometheus-scraping-and-discovery/ # ServiceMonitor/PodMonitor/Probe/ScrapeConfig/file_sd
│ └── onboarding-exporters-and-instrumentation/ # OpenMetrics snippets & 19-exporter K8s catalog
└── prometheus_mcp_server/ # FastMCP Execution Server (19 Tools, 11 Resources)
├── tools/ # 5 Toolsets (diagnostics, query, rules, scrape, exporters)
├── resources/ # 11 Dynamic prom:// Resources
├── services/ # Prometheus HTTP API v1 + Kubernetes CRD merge services
└── utils/ # Redaction, JSONL audit trail, PromQL 3.0 token-boundary AST
Clear Bounded Context (Zero Overlap with Sibling MCP Servers)
prometheus-mcp-server strictly owns Prometheus signal generation, PromQL querying (including Native Histograms and UTF-8 names), Exemplar extraction (/api/v1/query_exemplars), TSDB cardinality & health (/api/v1/status/tsdb, /api/v1/status/walreplay), scrape targets (ServiceMonitor, PodMonitor, Probe, ScrapeConfig, file_sd), promtool validation/unit testing, and Prometheus's own /api/v1/rules & /api/v1/alerts rule evaluation state.
alertmanager-mcp-server (src/alertmanager-mcp-server) exclusively owns Alertmanager v2 APIs (/api/v2/alerts, /api/v2/silences, routing trees, receivers, and inhibition rules).
tempo-mcp-server (src/tempo-mcp-server) owns distributed trace retrieval using trace_id values extracted via prom_query_exemplars.
6 Bundled Agent Skills (skills/)
Skill Directory Purpose Local Deterministic Script (scripts/) triaging-prometheus-incidents-and-slosIncident triage using RED/USE, Google SRE multi-window multi-burn-rate SLOs, and Exemplar-to-Trace handoffs scripts/generate_slo_rules.pyauthoring-promql-and-analyzing-metricsSafe PromQL 3.0 query construction, Native Histograms, and anomaly period comparisons scripts/lint_promql.pyauthoring-and-testing-prometheus-rulesAlerting & recording rule lifecycle, promtool unit tests, and safe multi-group CRD merges scripts/build_promtool_test_suite.pyoptimizing-tsdb-cardinality-and-finopsTSDB cardinality investigation, metric_relabel_configs, and remote-write FinOps scripts/generate_finops_config.pyconfiguring-prometheus-scraping-and-discoveryKubernetes Operator CRDs (ServiceMonitor, PodMonitor, Probe, ScrapeConfig) and VM file_sd scripts/validate_scrape_manifest.pyonboarding-exporters-and-instrumentationLanguage SDK instrumentation (Go/Python/Java/Node) and 19-exporter K8s lifecycle scripts/render_exporter_bundle.py
19 MCP Tools Across 5 Toolsets
Every tool has $\le 8$ parameters and explicit MCP ToolAnnotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint).
1. diagnostics Toolset (2 Read-Only Tools)
Tool Description prom_get_diagnosticsConsolidated control-plane & TSDB diagnostics (overview, runtime_config, tsdb, wal_replay, scrape_pools). prom_analyze_cardinalityLive TSDB cardinality analysis combining /api/v1/status/tsdb with per-metric label fanout via /api/v1/series.
2. query Toolset (7 Read-Only Tools)
Tool Description prom_explore_metricsUnified discovery of metric names, metadata (TYPE/HELP/UNIT), series matches, or label values. prom_query_instantExecute point-in-time PromQL query (or validate_only=True syntax/counter preflight check). prom_query_rangeExecute time-range PromQL query with automatic downsampling and summary/full/chart response modes. prom_query_exemplarsQuery /api/v1/query_exemplars to extract OpenTelemetry trace_id/span_id exemplars for tempo-mcp-server. prom_compare_periodsCompare current range query against a baseline offset (e.g. 1d, 7d) with delta% and z-score shift. prom_analyze_service_redCompound RED (Rate, Error ratio, Latency p50/p95/p99) analysis supporting classic and Native Histograms. prom_evaluate_slo_burnCompound Google SRE multi-window multi-burn-rate SLO evaluation (1h/5m, 6h/30m, 24h/2h, 72h/6h).
3. rules Toolset (3 Read-Only, 1 Write, 1 Destructive)
Tool Description prom_get_firing_alertsInspect active (pending/firing) alerts from Prometheus /api/v1/alerts & /api/v1/rules. prom_validate_rulesValidate rule group YAML via promtool check rules and optional promtool test rules unit tests. prom_simulate_rule_firingBacktest an alert expression against historical range data (mode="historical") or synthetic series (mode="synthetic"). prom_upsert_rule_groupCreate/update a rule group via yaml_only, local_file, k8s_crd (non-destructive group merge), or http_ruler. prom_delete_rule_groupDelete a rule group (preserves sibling groups in PrometheusRule CRDs; deletes CRD only when last group removed).
4. scrape Toolset (1 Read-Only, 1 Write, 1 Destructive)
Tool Description prom_verify_metrics_endpointScrape and validate a /metrics endpoint + optional Prometheus up{} series verification. prom_apply_scrape_configApply servicemonitor, podmonitor, probe, scrapeconfig, or file_sd target configuration. prom_delete_scrape_configDelete a Kubernetes scrape CRD or remove a file_sd job entry.
5. exporters Toolset (1 Write, 1 Destructive)
Tool Description prom_install_exporterDeploy one of 19 supported exporters to Kubernetes (dry_run=True by default). prom_uninstall_exporterUninstall a previously deployed exporter from Kubernetes (confirm=True required).
11 Dynamic prom:// Resources
Resource URI Description prom://system/backendsAll configured Prometheus backends with health status prom://system/backends/{backend_id}Detailed capabilities, version, and build info for a single backend prom://config/runtimeSecret-redacted runtime configuration, flags, and TSDB summary prom://topology/targetsConsolidated service topology and failed/down scrape targets prom://metadata/catalogLive metric catalog (TYPE, HELP, UNIT) prom://tsdb/cardinalityLive TSDB cardinality overview from /api/v1/status/tsdb prom://rules/groupsSummary of all alerting and recording rule groups prom://rules/{backend_id}/groups/{group_name}Full YAML/JSON definition and evaluation health of a specific rule group prom://kubernetes/scrape-crdsInventory of ServiceMonitor, PodMonitor, Probe, ScrapeConfig, and PrometheusRule CRDs prom://kubernetes/manifests/{kind}/{namespace}/{name}Secret-redacted Kubernetes CRD manifest prom://audit/recentRecent mutating action audit log entries (ring buffer + optional JSONL persistence)
CLI Flags & Environment Configuration
# Default (Read-Only safe mode: mutating tools are hidden unless --allow-write is set)
uv run prometheus-mcp-server --prometheus-url http://localhost:9090
# Enable write tools and restrict to specific toolsets
uv run prometheus-mcp-server \
--prometheus-url http://localhost:9090 \
--allow-write \
--toolsets diagnostics,query,rules \
--tenant-id team-payments \
--audit-log-path ./audit.jsonl
Environment Variable CLI Flag Default Description PROMETHEUS_BASE_URL--prometheus-urlhttp://localhost:9090Prometheus HTTP API base URL PROMETHEUS_ALLOW_WRITE--allow-write / --read-onlyfalseRegister mutating/destructive tools (false = read-only) PROMETHEUS_TOOLSETS--toolsetsallComma-separated toolsets (diagnostics,query,rules,scrape,exporters) PROMETHEUS_TENANT_ID--tenant-id(none) Default X-Scope-OrgID header for Mimir/Cortex/Thanos PROMETHEUS_RULER_URL— (none) Optional Mimir/Thanos Ruler base URL PROMETHEUS_AUDIT_LOG_PATH--audit-log-path(none) Optional JSONL file path for persistent mutation audit trail
Running Tests
The repository includes a 3-tier verification suite (115 tests across unit, integration, and skill/plugin compliance):
cd src/prometheus-mcp-server
uv run pytest -v
License
Apache 2.0 — see LICENSE .