$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill incident-managementImplement incident management processes and escalation procedures. Configure on-call schedules and post-incident reviews. Use when managing production incidents.
| 1 | # Incident Management |
| 2 | |
| 3 | Implement effective incident management processes including severity definitions, escalation matrices, war room procedures, and blameless post-mortem templates. |
| 4 | |
| 5 | ## When to Use |
| 6 | |
| 7 | - Establishing incident management processes for production systems |
| 8 | - Defining severity levels and escalation procedures |
| 9 | - Running war rooms and coordinating incident response |
| 10 | - Conducting blameless post-incident reviews |
| 11 | - Building on-call schedules and notification workflows |
| 12 | - Meeting compliance requirements for incident response (SOC 2, HIPAA, PCI DSS) |
| 13 | |
| 14 | ## Severity Levels |
| 15 | |
| 16 | ```yaml |
| 17 | severity_definitions: |
| 18 | SEV1_critical: |
| 19 | impact: "Complete service outage or data breach affecting all/most customers" |
| 20 | examples: |
| 21 | - Production site completely down |
| 22 | - Data breach confirmed or suspected |
| 23 | - Complete loss of a critical business function |
| 24 | - Security incident with active exploitation |
| 25 | response_time: "Immediate (within 5 minutes)" |
| 26 | update_frequency: "Every 15-30 minutes" |
| 27 | who_is_paged: "On-call engineer, engineering manager, incident commander, executive on-call" |
| 28 | communication: "Status page update, customer email, executive notification" |
| 29 | resolution_target: "< 1 hour to mitigate" |
| 30 | |
| 31 | SEV2_major: |
| 32 | impact: "Major feature broken or severe degradation affecting many customers" |
| 33 | examples: |
| 34 | - Key feature completely non-functional |
| 35 | - Significant performance degradation (>5x latency) |
| 36 | - Data processing pipeline completely stalled |
| 37 | - Partial outage affecting a region or segment |
| 38 | response_time: "Within 15 minutes" |
| 39 | update_frequency: "Every 30-60 minutes" |
| 40 | who_is_paged: "On-call engineer, engineering manager" |
| 41 | communication: "Status page update if customer-facing" |
| 42 | resolution_target: "< 4 hours to mitigate" |
| 43 | |
| 44 | SEV3_moderate: |
| 45 | impact: "Minor feature impaired or degradation affecting some customers" |
| 46 | examples: |
| 47 | - Non-critical feature broken |
| 48 | - Moderate performance degradation |
| 49 | - Elevated error rate (below threshold for SEV2) |
| 50 | - Single-customer impact on non-critical function |
| 51 | response_time: "Within 1 hour during business hours" |
| 52 | update_frequency: "Every 2-4 hours" |
| 53 | who_is_paged: "On-call engineer" |
| 54 | communication: "Internal only unless customer inquires" |
| 55 | resolution_target: "< 1 business day" |
| 56 | |
| 57 | SEV4_low: |
| 58 | impact: "Cosmetic issue, minor inconvenience, or non-customer-facing problem" |
| 59 | examples: |
| 60 | - UI cosmetic bug |
| 61 | - Non-critical monitoring gap |
| 62 | - Internal tool degradation |
| 63 | - Documentation inaccuracy in production |
| 64 | response_time: "Next business day" |
| 65 | update_frequency: "As needed" |
| 66 | who_is_paged: "None (ticket created)" |
| 67 | communication: "None" |
| 68 | resolution_target: "Within sprint planning cycle" |
| 69 | ``` |
| 70 | |
| 71 | ## Escalation Matrix |
| 72 | |
| 73 | ```yaml |
| 74 | escalation_matrix: |
| 75 | tier_1_on_call_engineer: |
| 76 | reached_via: "PagerDuty / OpsGenie alert" |
| 77 | responsibilities: |
| 78 | - Acknowledge alert within 5 minutes |
| 79 | - Assess severity and impact |
| 80 | - Begin troubleshooting |
| 81 | - Escalate to Tier 2 if unable to resolve within 30 minutes (SEV1/2) |
| 82 | escalation_trigger: "Cannot resolve, needs additional expertise, or severity upgrade" |
| 83 | |
| 84 | tier_2_team_lead_or_sme: |
| 85 | reached_via: "PagerDuty escalation or direct page" |
| 86 | responsibilities: |
| 87 | - Provide subject matter expertise |
| 88 | - Assist with diagnosis and resolution |
| 89 | - Coordinate with other teams if cross-service issue |
| 90 | - Escalate to Tier 3 if broader coordination needed |
| 91 | escalation_trigger: "Multi-service issue, needs executive decision, or customer-facing SEV1" |
| 92 | |
| 93 | tier_3_engineering_management: |
| 94 | reached_via: "PagerDuty escalation or direct call" |
| 95 | responsibilities: |
| 96 | - Assign incident commander (if not already) |
| 97 | - Allocate additional resources |
| 98 | - Make business decisions (feature disable, rollback, etc.) |
| 99 | - Coordinate external communication |
| 100 | escalation_trigger: "Business impact decision, extended outage, or PR/legal concern" |
| 101 | |
| 102 | tier_4_executive: |
| 103 | reached_via: "Direct phone call" |
| 104 | responsibilities: |
| 105 | - Authorize extraordinary measures |
| 106 | - Manage board/investor communication |
| 107 | - Approve public statements |
| 108 | - Engage external resources (vendors, consultants) |
| 109 | escalation_trigger: "Major breach, extended SEV1, regulatory or legal implication" |
| 110 | |
| 111 | time_based_escalation: |
| 112 | sev1: |
| 113 | "15 min no ack": "Re-page on-call + backup on-call" |
| 114 | "30 min unresolved": "Page team lead" |
| 115 | "1 hour unresolved": "Page engineering manager + executive on-call" |
| 116 | "2 hours unresolved": "All-hands engineering involvement" |
| 117 | sev2: |
| 118 | "30 min no ack": "Re-page on-call + backup on-call" |
| 119 | "1 hour unresolved": "Page team lead" |