kubectl-investigator

Name: kubectl-investigator
Author: anyshift-io

16stars0forks

How to install kubectl-investigator?

anyshift-io/sre-skills/kubectl-investigator

$npx -y skills add anyshift-io/sre-skills --skill kubectl-investigator

Installs into the current project.

Summary

Investigate a live or recent incident in a Kubernetes cluster. Anchor the window, bisect the change surface (rollouts, ConfigMaps/Secrets, RBAC, HPA/cluster changes, CronJobs), classify against four reference failure paths (OOM, DNS, cascading-failure, deploy-correlator), confirm

Files · 1

View on GitHub

SKILL.md

1	# kubectl-investigator
2
3	Methodology skill for investigating a live or recent incident on Kubernetes. Produces a timeline, a ranked set of hypotheses, a blast-radius estimate, and a recommended mitigation. Hands off cleanly to `postmortem-author` once the incident is mitigated.
4
5	Scope: workloads running on Kubernetes (Deployments, StatefulSets, DaemonSets, Jobs/CronJobs) and the cluster primitives around them (Services, Ingress, CoreDNS, ConfigMaps/Secrets, RBAC, HPA, nodes). External dependencies (third-party APIs, partner TLS endpoints, managed databases) are in scope only as seen from a Kubernetes workload — the methodology investigates the cluster-side symptom and the in-cluster change surface.
6
7	## When to invoke
8
9	- A `PrometheusRule` / Alertmanager alert just fired on a workload and the agent needs to triage before paging a human.
10	- A user asks "what is breaking in the cluster right now" or "why did Deployment X just page".
11	- A `kubectl rollout` / Helm release / Argo CD sync went out in the last hour and a metric moved; need to know whether they are linked.
12	- Pods are crash-looping, `OOMKilled`, or `Pending`, or customer impact is reported with no alert yet; need to find the failing surface.
13
14	## The methodology, in order
15
16	The order matters. Skipping a step produces confident wrong answers.
17
18	### 1. Anchor the window
19
20	Lock two timestamps before doing anything else:
21
22	- T0: the trigger timestamp. Apply this order:
23	1. If an alert is provided as the trigger, T0 = alert fire time. Use this verbatim. Do not substitute an earlier "first error in logs / first `OOMKilled` event" timestamp just because one exists; the alert fire time is the agreed-upon coordination point for the incident.
24	2. If a customer report is the trigger, T0 = report timestamp.
25	3. If neither exists (operator-initiated investigation, "pods slow all morning"), T0 = earliest unambiguous signal in the available telemetry (first `OOMKilled` event, first `SERVFAIL`, first error-rate inflection), and mark T0 as ambiguous (see below).
26	- Tnow: current time, or the timestamp the investigation was triggered.
27
28	Every later signal is filtered to `[T0 - 15min, Tnow]`. The 15-minute lead-in catches changes that landed just before the symptom surfaced (a rollout's pods take time to roll, an HPA scale-down takes time to bite).
29
30	If T0 is ambiguous (operator-triggered with no alert, or "slow all morning"-class reports), the methodology's recommended mitigation in step 6 must begin with "re-run the investigation with a widened window" before any irreversible action. The change identified within the original narrow window is likely incomplete; the actual causal change may sit outside it. Do not silently round and do not skip the re-run step.
31
32	### 2. Bisect the change surface
33
34	Pull every change event that overlaps the window. On Kubernetes the change surface is:
35
36	- Workload rollouts: `kubectl rollout` / `kubectl apply` / `kubectl set image`, new image tags, new ReplicaSets, Helm releases, Argo CD / Flux syncs.
37	- Cluster / capacity changes: node-pool scaling, node cordon/drain, resource `requests`/`limits` edits, HPA / VPA changes, PodDisruptionBudget edits, PV/PVC/StorageClass changes.
38	- RBAC / ServiceAccount changes: Role / ClusterRole / RoleBinding / ClusterRoleBinding edits, ServiceAccount or its token/permissions changed (these break Secret reads, API access, admission).
39	- Config / feature-flag changes: ConfigMap / Secret edits, CoreDNS `Corefile` ConfigMap edits, Ingress/NetworkPolicy changes, feature-flag flips.
40	- Admission / operator changes: Validating/MutatingWebhookConfiguration edits, CRD or controller upgrades.
41	- CronJobs / Jobs that ran in the window (batch, data migrations, cluster maintenance jobs).
42
43	If the window has zero change events, treat it as a strong signal in itself: the failure is likely external (upstream provider, certificate expiry, DNS, capacity drift from organic growth) rather than a self-inflicted regression.
44
45	### 3. Classify against the four reference paths
46
47	Match the failure shape to one of these four canonical paths first. They cover the majority of Kubernetes incidents; only branch out once

People who liked this also viewed

6 picks

Type

kubectl-investigator

16stars0forks

View on GitHub

How to install kubectl-investigator?

anyshift-io/sre-skills/kubectl-investigator

$npx -y skills add anyshift-io/sre-skills --skill kubectl-investigator

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# kubectl-investigator
2
3	Methodology skill for investigating a live or recent incident on Kubernetes. Produces a timeline, a ranked set of hypotheses, a blast-radius estimate, and a recommended mitigation. Hands off cleanly to `postmortem-author` once the incident is mitigated.
4
5	Scope: workloads running on Kubernetes (Deployments, StatefulSets, DaemonSets, Jobs/CronJobs) and the cluster primitives around them (Services, Ingress, CoreDNS, ConfigMaps/Secrets, RBAC, HPA, nodes). External dependencies (third-party APIs, partner TLS endpoints, managed databases) are in scope only as seen from a Kubernetes workload — the methodology investigates the cluster-side symptom and the in-cluster change surface.
6
7	## When to invoke
8
9	- A `PrometheusRule` / Alertmanager alert just fired on a workload and the agent needs to triage before paging a human.
10	- A user asks "what is breaking in the cluster right now" or "why did Deployment X just page".
11	- A `kubectl rollout` / Helm release / Argo CD sync went out in the last hour and a metric moved; need to know whether they are linked.
12	- Pods are crash-looping, `OOMKilled`, or `Pending`, or customer impact is reported with no alert yet; need to find the failing surface.
13
14	## The methodology, in order
15
16	The order matters. Skipping a step produces confident wrong answers.
17
18	### 1. Anchor the window
19
20	Lock two timestamps before doing anything else:
21
22	- T0: the trigger timestamp. Apply this order:
23	1. If an alert is provided as the trigger, T0 = alert fire time. Use this verbatim. Do not substitute an earlier "first error in logs / first `OOMKilled` event" timestamp just because one exists; the alert fire time is the agreed-upon coordination point for the incident.
24	2. If a customer report is the trigger, T0 = report timestamp.
25	3. If neither exists (operator-initiated investigation, "pods slow all morning"), T0 = earliest unambiguous signal in the available telemetry (first `OOMKilled` event, first `SERVFAIL`, first error-rate inflection), and mark T0 as ambiguous (see below).
26	- Tnow: current time, or the timestamp the investigation was triggered.
27
28	Every later signal is filtered to `[T0 - 15min, Tnow]`. The 15-minute lead-in catches changes that landed just before the symptom surfaced (a rollout's pods take time to roll, an HPA scale-down takes time to bite).
29
30	If T0 is ambiguous (operator-triggered with no alert, or "slow all morning"-class reports), the methodology's recommended mitigation in step 6 must begin with "re-run the investigation with a widened window" before any irreversible action. The change identified within the original narrow window is likely incomplete; the actual causal change may sit outside it. Do not silently round and do not skip the re-run step.
31
32	### 2. Bisect the change surface
33
34	Pull every change event that overlaps the window. On Kubernetes the change surface is:
35
36	- Workload rollouts: `kubectl rollout` / `kubectl apply` / `kubectl set image`, new image tags, new ReplicaSets, Helm releases, Argo CD / Flux syncs.
37	- Cluster / capacity changes: node-pool scaling, node cordon/drain, resource `requests`/`limits` edits, HPA / VPA changes, PodDisruptionBudget edits, PV/PVC/StorageClass changes.
38	- RBAC / ServiceAccount changes: Role / ClusterRole / RoleBinding / ClusterRoleBinding edits, ServiceAccount or its token/permissions changed (these break Secret reads, API access, admission).
39	- Config / feature-flag changes: ConfigMap / Secret edits, CoreDNS `Corefile` ConfigMap edits, Ingress/NetworkPolicy changes, feature-flag flips.
40	- Admission / operator changes: Validating/MutatingWebhookConfiguration edits, CRD or controller upgrades.
41	- CronJobs / Jobs that ran in the window (batch, data migrations, cluster maintenance jobs).
42
43	If the window has zero change events, treat it as a strong signal in itself: the failure is likely external (upstream provider, certificate expiry, DNS, capacity drift from organic growth) rather than a self-inflicted regression.
44
45	### 3. Classify against the four reference paths
46
47	Match the failure shape to one of these four canonical paths first. They cover the majority of Kubernetes incidents; only branch out once

People who liked this also viewed

6 picks

Type