$npx -y skills add Rune-kit/rune --skill incidentStructured incident response. Use when user reports an outage, production error, or says 'incident', 'something is down', 'users are affected'. Triage severity, contain blast radius, root-cause, document timeline, generate postmortem.
| 1 | # incident |
| 2 | |
| 3 | ## Purpose |
| 4 | |
| 5 | Structured incident response for production issues. Follows a strict order: triage first, contain before investigating, root-cause after stable, postmortem last. Prevents the most common incident anti-pattern — developers debugging while the system is still on fire. Covers P1 outages, P2 degraded service, and P3 minor issues with appropriate urgency at each level. |
| 6 | |
| 7 | ## Triggers |
| 8 | |
| 9 | - `/rune incident "description of what's broken"` — direct user invocation |
| 10 | - Called by `launch` (L1): watchdog alerts during Phase 3 VERIFY |
| 11 | - Called by `deploy` (L2): health check fails post-deploy |
| 12 | - Signal: auto-triggers when `watchdog` emits `incident.detected` — no manual invocation needed |
| 13 | |
| 14 | ## Calls (outbound) |
| 15 | |
| 16 | - `watchdog` (L3): current system state — which endpoints are down, response times |
| 17 | - `autopsy` (L2): root cause analysis after containment |
| 18 | - `journal` (L3): record incident timeline and decisions |
| 19 | - `sentinel` (L2): check for security dimension (data exposure, unauthorized access) |
| 20 | - `neural-memory` (ext): after resolution — capture incident root cause + fix pattern cross-session so the same failure mode is never diagnosed twice |
| 21 | |
| 22 | ## Called By (inbound) |
| 23 | |
| 24 | - `launch` (L1): monitoring alert during production verification |
| 25 | - `deploy` (L2): post-deploy health check failure |
| 26 | - User: `/rune incident` direct invocation |
| 27 | |
| 28 | ## Executable Steps |
| 29 | |
| 30 | ### Step 1 — Triage |
| 31 | |
| 32 | Classify severity using this matrix: |
| 33 | |
| 34 | | Severity | Definition | Contain Within | |
| 35 | |----------|-----------|----------------| |
| 36 | | **P1** | Full outage — core feature unavailable for all users | 15 minutes | |
| 37 | | **P2** | Partial degradation — feature broken for subset of users or degraded for all | 1 hour | |
| 38 | | **P3** | Minor issue — cosmetic, edge case, or non-blocking degradation | 4 hours | |
| 39 | |
| 40 | P1 indicators: 5xx on root `/`, auth endpoint down, payment flow broken, data loss detected |
| 41 | P2 indicators: elevated error rate (>1%) on key flow, 1+ regions down, performance >5x baseline |
| 42 | P3 indicators: UI glitch, non-critical feature broken, low error rate (<0.1%) |
| 43 | |
| 44 | Emit: `TRIAGE: [P1|P2|P3] — [one-line impact description]` |
| 45 | |
| 46 | ### Step 2 — Contain |
| 47 | |
| 48 | <HARD-GATE> |
| 49 | During active incident (before CONTAINED status), DO NOT attempt code fixes or root cause analysis. |
| 50 | Contain first. Ship code during active P1/P2 without containment = turning P2s into P1s. |
| 51 | </HARD-GATE> |
| 52 | |
| 53 | Choose containment strategy based on what's available and severity: |
| 54 | |
| 55 | | Strategy | When to Use | |
| 56 | |----------|------------| |
| 57 | | **Rollback** | Last deploy caused regression (check git log vs incident start time) | |
| 58 | | **Feature flag off** | Feature-gated code — disable without deploy | |
| 59 | | **Traffic shift** | Multi-region: route away from affected region | |
| 60 | | **Scale up** | Resource exhaustion (CPU/memory/connection pool) | |
| 61 | | **Rate limit** | Abuse pattern or traffic spike | |
| 62 | | **Manual intervention** | DB locked record, stuck job, cache corruption | |
| 63 | |
| 64 | Execute containment action. Then invoke `watchdog` to verify system is stable before proceeding. |
| 65 | |
| 66 | Emit: `CONTAINED: [strategy used] — [timestamp]` or `CONTAINMENT_FAILED: [what was tried] — escalate` |
| 67 | |
| 68 | ### Step 3 — Verify Containment |
| 69 | |
| 70 | Invoke `watchdog` with current base_url and critical endpoints. |
| 71 | |
| 72 | Proceed to Step 4 only if watchdog returns `ALL_HEALTHY` or `DEGRADED` with upward trend. |
| 73 | If watchdog returns `DOWN` — return to Step 2 with a different containment strategy. |
| 74 | |
| 75 | ### Step 4 — Security Check |
| 76 | |
| 77 | Invoke `sentinel` to check if the incident has a security dimension: |
| 78 | - Data exposure (PII, credentials in logs/responses) |
| 79 | - Unauthorized access pattern in logs |
| 80 | - Injection attack vector triggered the incident |
| 81 | - Dependency with known CVE involved |
| 82 | |
| 83 | If `sentinel` returns `BLOCK`: escalate to security incident — different protocol (notify security team, preserve logs, document access chain). |
| 84 | If `sentinel` returns `PASS` or `WARN`: continue to root cause. |
| 85 | |
| 86 | ### Step 5 — Root Cause Analysis |
| 87 | |
| 88 | Invoke `autopsy` with context: |
| 89 | - Incident start timestamp |
| 90 | - Failing components identified in Step 2-3 |
| 91 | - Recent deploy info (commit hash, deploy timestamp, changed files) |
| 92 | |
| 93 | `autopsy` returns: root cause hypothesis with evidence, affected code paths, contributing factors. |
| 94 | |
| 95 | Do not attempt fixes — `incident` only investigates. Any code changes are a separate task. |
| 96 | |
| 97 | ### Step 6 — Timeline Construction |
| 98 | |
| 99 | Construct incident timeline using: |
| 100 | - Incident start time (when first detected) |
| 101 | - Triage time (when severity classified) |
| 102 | - Containment time (when system stabilized) |
| 103 | - RCA time (when root cause identified) |
| 104 | - Resolution time (when fully resolved) |
| 105 | |
| 106 | Format: |
| 107 | ``` |