Skip to content

k8s-servant/k8s-skills

v0.5.0Apache-2.0

Kubernetes troubleshooting skills for any AI agent: read first, ask before changing anything

k8s-skills

Kubernetes troubleshooting skills for any AI agent: read first, ask before changing anything.

CI Release License

A real run of the oom-killed scenario on a kind cluster, with a view-only kubeconfig. The commands and output are shortened, and the answer is the agent's own words.

k8s-skills is a set of Agent Skills that teach an AI agent to investigate Kubernetes problems the way an experienced SRE does: what to look at, in which order, what each finding means, and what to propose. The skills work in Claude Code, Codex, Gemini CLI, GitHub Copilot and any other agent that reads SKILL.md files and can run kubectl.

How it works, how each skill is tested, and what breaking a kind cluster 76 ways showed about the docs: k8s-skills: Teaching AI Agents to Troubleshoot Kubernetes Without Breaking It.

Quick start

npx skills add k8s-servant/k8s-skills -a claude-code   # or -a codex, ...
gemini extensions install https://github.com/k8s-servant/k8s-skills   # Gemini CLI

Then describe the problem to your agent, for example "my-app in my-namespace keeps restarting. Find out why." For an agent that can only look, give it a read-only kubeconfig (Your agent, your decision). More ways to install are under Install.

Why you can trust it

  • 26 skills, 79 scenarios. Each scenario breaks something on purpose in a kind cluster. A fresh agent with only the skill and a view-only kubeconfig has to name the cause, point at the evidence and propose an acceptable fix, without changing anything.
  • Quoted from a real cluster. Every error message and log line a skill quotes was reproduced on kind, at a pinned version. Where the cluster said something different from the docs, the skill quotes the cluster.
  • Checked on every change. CI checks each command against its tier, requires an # Undo: line for every change, and rejects any command that reads Secrets.

Read first, ask before changing

Every command in a skill falls into one of three tiers. CI checks each one against the allow-lists in tools/skillcheck/allowlist.yaml.

TierWhatWhat the agent does
Run freelyRead-only commands: get, describe, logs, events, top and a short list of others, plus read-only az ... show/list for the AKS skillRuns them while it investigates
Ask firstReversible changes (set image, set resources, rollout undo, scale, patch, cordon, uncordon and a few more) and one read-only program run in a container with exec (nslookup, curl, cat, ...)Shows the command, what it changes and how to undo it, under > Ask first: the agent runs this only after the user approves it., and runs it only after you say yes
Never runAnything destructive or hard to undo (delete, drain, replace, apply, ...)Proposes it under > Proposed fix: review before running. The agent does not run this., with an # Undo: line, for you to review and run

Secrets are never read, in any tier. Every change, in either tier, comes with an # Undo: line.

Skills

SkillUse it when
k8s-triageSomething is wrong in the cluster and it is not clear where: a ranked first look at nodes, pods, events and resource use, with a first fix and the skill that goes deeper
k8s-workload-troubleshootingA pod or workload is unhealthy: Pending, CrashLoopBackOff, image pull errors, OOMKilled, failing probes, stuck rollouts, failing Jobs
k8s-loki-troubleshootingLoki rejects, drops or cannot query logs: pushes refused with 400, 401 or 429 (old timestamps, missing tenant, rate and stream limits), crashes on schema errors, storage flushes, rings, query limits
k8s-network-troubleshootingTraffic does not reach running pods: Services without endpoints, wrong selectors or ports, DNS, NetworkPolicies, Ingress and Gateway routes
k8s-coredns-troubleshootingCluster DNS itself fails: CoreDNS crashing on Corefile errors or loops, reloads that fail, upstream timeouts and SERVFAIL, stub domains, rewrites and hosts, NodeLocal DNSCache, the AKS, EKS and GKE DNS ConfigMaps
k8s-storage-troubleshootingStorage blocks a workload: PVCs stuck Pending, missing StorageClasses or unsupported access modes, attach and mount errors, full volumes
k8s-autoscaling-troubleshootingAutoscaling does not scale: HPA targets <unknown> (missing requests, no metrics), stuck at maxReplicas, metrics-server and kubectl top, KEDA ScaledObjects not Ready or without an HPA, the VPA
k8s-cert-manager-troubleshootingcert-manager does not issue or renew: Certificates not Ready, issuers not Ready, a wrong issuerRef, ACME challenges stuck, the cert-manager webhook unreachable
k8s-flux-troubleshootingFlux does not deploy: sources not Ready (missing branch, credentials, chart version), Kustomizations with a wrong path or failed build, HelmReleases stalled, suspended objects
k8s-argocd-troubleshootingArgo CD does not deploy: comparison errors (wrong path or revision), destinations not allowed by the project, no automated sync, failed syncs, drift, health
k8s-helm-troubleshootingHelm installs or upgrades fail or a release is stuck: --wait timeouts, ownership conflicts, another operation is in progress, hooks, immutable fields; helm commands are left to the user
k8s-kyverno-troubleshootingKyverno denies or fails to police resources: Enforce denials, Audit-only policies and PolicyReports, ClusterPolicy and CEL ValidatingPolicy, mutate and generate rules, webhooks down
k8s-fluent-bit-troubleshootingFluent Bit does not deliver logs: config errors that crash the DaemonSet, outputs that cannot connect or drop chunks, records without Kubernetes metadata, tail paths and CRI parsers, backpressure, the Fluent Operator
k8s-gatekeeper-troubleshootingOPA Gatekeeper denies or fails to police resources: constraint denials and actions, templates that do not compile, a webhook that fails open, exemptions, sync, mutation
k8s-external-secrets-troubleshootingExternal Secrets does not create or update a Secret: sync errors from the controller log, stores not Ready, targets owned by another ExternalSecret, refresh; never reads the Secrets
k8s-opentelemetry-collector-troubleshootingTelemetry does not get through the OpenTelemetry Collector: config errors that crash it, receivers on localhost or outside every pipeline, exporters that cannot reach the backend, queues, memory_limiter, k8s_attributes RBAC, the Operator and auto-instrumentation
k8s-prometheus-troubleshootingPrometheus does not scrape or alert: ServiceMonitors not selected, wrong port names, targets down, rules not loaded, alerts not reaching Alertmanager, memory and disk
k8s-grafana-troubleshootingGrafana fails: provisioning errors that crash the pod, dashboards that do not load, datasources pointing at the wrong Service, database locks, login, grafana-operator selectors
k8s-cilium-troubleshootingTraffic fails on a Cilium cluster: CiliumNetworkPolicy drops and default deny, DNS blocked by egress policies, L7 403s, Hubble flows, agent, endpoint and IPAM problems
k8s-istio-troubleshootingRequests fail in an Istio mesh: Envoy access logs and response flags (UH, NR, filter_chain_not_found), AuthorizationPolicy 403s, subsets without pods, STRICT mTLS, injection, ambient
k8s-envoy-troubleshootingAn Envoy proxy fails requests: 503 no healthy upstream or upstream connect error, 404 with an empty body, 504, response flags (UH, UF, NR, UT, UO), crashes on bootstrap errors, read-only admin endpoints; standalone or under Contour, Emissary, Gloo
k8s-envoy-gateway-troubleshootingEnvoy Gateway does not route: Gateways not Programmed, HTTPRoutes not Accepted (hostnames, allowedRoutes), unresolved backends and missing ReferenceGrants, 404/500/503, policies
k8s-rbac-troubleshootingA request is refused: Forbidden, cannot list resource ... in the namespace or at the cluster scope, bindings in the wrong namespace or for the wrong subject, ClusterRoles bound with RoleBindings, 401 versus 403, escalation and roleRef errors
k8s-api-troubleshootingThe API server refuses a request: failed calling webhook and ReplicaSets with no pods, webhook and ValidatingAdmissionPolicy denials, APIServices not Available, namespaces stuck Terminating, no matches for kind, unknown fields, 401, 429, etcd and size limits
k8s-node-troubleshootingA node is NotReady or under pressure, or pods will not go onto it: taints, cordoned nodes, requests that do not fit, an autoscaler that does not scale
k8s-aks-troubleshootingA problem is specific to AKS: a cluster or node pool Failed, upgrades blocked by PDBs, quota and VM capacity, LoadBalancer Services <pending> (SyncLoadBalancerFailed), Azure Disk and Files PVCs, workload identity AADSTS errors, coredns-custom, error dialing backend, Azure Policy, API server access and Azure RBAC; uses read-only az commands

Planned: EKS and GKE skills, and more ecosystem skills.

How each skill is verified

A skill is merged only after four checks:

  1. Official sources. Facts come from the project's own documentation and, where the docs are silent, its source code at the release named below. Each skill lists the pages and files in its Sources section.
  2. A real cluster. Every status message, log line and response a skill quotes was reproduced on a throwaway kind cluster, with the tool installed at a pinned version whose checksum is verified first (tests/versions.env). Where the cluster printed something different from the docs, the skill quotes the cluster.
  3. Scenarios run by an agent. Each scenario in tests/<skill>/<case>/ breaks something on purpose, and its expected.md names the cause, the evidence, the acceptable fixes and the wrong answers. A fresh agent gets the skill, the prompt and a view-only kubeconfig, and passes only if it names the cause, points at the evidence, proposes an acceptable fix and changes nothing. The runs are recorded in the skill's pull request.
  4. CI on every change. skillcheck checks each command against its tier, the ask-first and fix blocks, the # Undo: lines and that no command reads Secrets; tests, linters and a link check run as well.
SkillVerified withScenarios
k8s-aks-troubleshootingaks, kubernetes 1.33azure-disk-too-large, internal-lb-subnet-missing, workload-identity-unknown-client-id
k8s-api-troubleshootingkubernetes 1.37apiservice-unavailable, crd-version-not-served, webhook-backend-down
k8s-argocd-troubleshootingkubernetes 1.37, argo-cd 3.5app-path-missing, auto-sync-off, project-destination-denied
k8s-autoscaling-troubleshootingkubernetes 1.37, metrics-server 0.9, keda 2.21keda-target-wrong, max-replicas-reached, missing-cpu-request
k8s-cert-manager-troubleshootingkubernetes 1.37, cert-manager 1.21ca-issuer-missing-key-pair, cluster-issuer-kind-missing, webhook-down
k8s-cilium-troubleshootingkubernetes 1.37, cilium 1.20egress-policy-blocks-dns, ingress-policy-wrong-label, l7-rule-denies-path
k8s-coredns-troubleshootingkubernetes 1.37, coredns 1.14corefile-typo, forward-unreachable, rewrite-target-wrong
k8s-envoy-troubleshootingkubernetes 1.37, envoy 1.40bootstrap-typo, dns-name-wrong, upstream-port-wrong
k8s-envoy-gateway-troubleshootingkubernetes 1.37, envoy-gateway 1.9, gateway-api 1.6backend-not-found, cross-namespace-no-referencegrant, hostname-mismatch
k8s-external-secrets-troubleshootingkubernetes 1.37, external-secrets 2.14provider-key-missing, store-name-wrong, target-owned-by-other
k8s-flux-troubleshootingkubernetes 1.37, flux 2.9git-branch-missing, helm-chart-version-missing, kustomization-path-missing
k8s-fluent-bit-troubleshootingkubernetes 1.37, fluent-bit 5.1config-plugin-typo, kubernetes-filter-rbac, output-host-wrong
k8s-gatekeeper-troubleshootingkubernetes 1.37, gatekeeper 3.23constraint-denies-deployment, template-rego-error, webhook-down-fails-open
k8s-grafana-troubleshootingkubernetes 1.37, grafana 13.2dashboard-json-invalid, datasource-two-defaults, datasource-url-wrong
k8s-helm-troubleshootingkubernetes 1.37, helm 4.3ownership-conflict, release-stuck-pending, upgrade-timeout-bad-image
k8s-istio-troubleshootingkubernetes 1.37, istio 1.31authorizationpolicy-denies, strict-mtls-plaintext-client, subset-without-pods
k8s-kyverno-troubleshootingkubernetes 1.37, kyverno 1.19audit-does-not-block, enforce-blocks-deployment, validatingpolicy-denies
k8s-loki-troubleshootingkubernetes 1.37, loki 3.7missing-tenant, old-timestamps, schema-tsdb-period
k8s-network-troubleshootingkubernetes 1.37, gateway-api 1.6networkpolicy-blocks-namespace, selector-mismatch, wrong-target-port
k8s-node-troubleshootingkubernetes 1.37node-not-ready, requests-full, untolerated-taint
k8s-opentelemetry-collector-troubleshootingkubernetes 1.37, opentelemetry-collector 0.163exporter-endpoint-wrong, pipeline-receiver-undefined, receiver-localhost
k8s-prometheus-troubleshootingkubernetes 1.37, prometheus-operator 0.94, prometheus 3.14servicemonitor-not-selected, servicemonitor-port-name-wrong, target-path-404
k8s-rbac-troubleshootingkubernetes 1.37clusterrole-bound-in-namespace, role-missing-verb, subject-namespace-wrong
k8s-storage-troubleshootingkubernetes 1.37missing-storageclass, unsupported-access-mode, waiting-for-first-consumer
k8s-triagekubernetes 1.37crashloop-somewhere, healthy-cluster, namespace-only-access, node-not-ready
k8s-workload-troubleshootingkubernetes 1.37image-pull-wrong-tag, oom-killed, readiness-probe-failing

verified-with and verified-on in each SKILL.md record the same versions and the date. Other versions usually behave the same, but messages and defaults can change between releases. kind cannot reproduce cloud-only behaviour (load balancers, cloud storage, managed node pools). The cloud skills' scenarios run on a real managed test cluster instead, in one namespace and only when you name its context (see Cloud scenarios); the rest of their content is checked against the provider's docs. To run a scenario yourself, see Scenarios and the test kit.

Install

Install with the skills CLI, or copy the folders yourself. Both work for one skill or all of them.

All skills or one? We recommend installing all of them. The skills hand off to each other by name (k8s-triage to the workload, network or node skill, a tool's skill to k8s-network-troubleshooting or k8s-cert-manager-troubleshooting), and a hand-off works best when the target is installed. It costs little: agents load only each skill's name and description at startup and read the rest when a skill is used (progressive disclosure); a description is at most 1,024 characters, so the whole set costs a few thousand tokens. Install one skill when you only run that tool, or when your agent limits how many skills it loads.

# All skills, for Claude Code (use -a codex, ... for other agents)
npx skills add k8s-servant/k8s-skills -a claude-code

# One skill
npx skills add k8s-servant/k8s-skills -s k8s-triage -a claude-code

The CLI sends anonymous usage telemetry; set DISABLE_TELEMETRY=1 to turn it off.

Or copy from a clone of this repository. Into your home directory, the skills are available in every project:

git clone https://github.com/k8s-servant/k8s-skills.git ~/k8s-skills
mkdir -p ~/.claude/skills && cp -R ~/k8s-skills/skills/* ~/.claude/skills/

For one project only, copy into the project instead, from its root folder:

mkdir -p .claude/skills && cp -R ~/k8s-skills/skills/* .claude/skills/

In a project, other agents read .agents/skills (Codex, Gemini CLI, GitHub Copilot) or .github/skills (GitHub Copilot).

Other ways

  • Claude Code plugin: /plugin marketplace add k8s-servant/k8s-skills, then /plugin install k8s-skills@k8s-skills for all skills or /plugin install k8s-triage@k8s-skills for one.
  • GitHub CLI (preview): gh skill install k8s-servant/k8s-skills --all, or one skill by name, pinned with @vX.Y.Z.
  • Gemini CLI: gemini extensions install https://github.com/k8s-servant/k8s-skills for all skills. Gemini reads project skills only in a folder you have trusted.

Skills that hand over to each other still work alone: a hand-off to a skill that is not installed tells the agent to continue with the steps it has and name the skill that would help.

Your agent, your decision

How you run your agent, and with which permissions, is your decision and your risk. The skills are written and checked to keep to these tiers, but that is not a 100% guarantee: "ask first" is an instruction the agent follows, not a lock, and an agent in an auto-approve mode may not stop to ask. What actually limits an agent is the Kubernetes RBAC of the credentials you give it. If you want it to only investigate, give it a read-only identity, for example one bound to the built-in view ClusterRole. view cannot read cluster-scoped objects, so for k8s-triage on a whole cluster and for the node and storage skills also allow get, list and watch on nodes, persistentvolumes, and in storage.k8s.io on storageclasses, csidrivers, csinodes and volumeattachments. view cannot read RBAC objects either, so for k8s-rbac-troubleshooting allow get and list on roles, rolebindings, clusterroles and clusterrolebindings, and for k8s-api-troubleshooting on validatingwebhookconfigurations, mutatingwebhookconfigurations, validatingadmissionpolicies, apiservices, customresourcedefinitions, flowschemas and prioritylevelconfigurations. k8s-aks-troubleshooting also runs read-only az commands, as whatever Azure login the agent's shell has: give that identity the Reader role on the cluster's resource groups, nothing more. Give it more only if you want it to apply the changes you approve (see Using RBAC Authorization).

Contributing

New skills and fixes are welcome. Read CONTRIBUTING.md first. What runs on each pull request is described in docs/ci.md. Report security problems as described in SECURITY.md.

License

Apache-2.0