ai-sre-incident-response

Name: ai-sre-incident-response
Author: BagelHole

42stars4forks

How to install ai-sre-incident-response?

bagelhole/devops-security-agent-skills/ai-sre-incident-response

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill ai-sre-incident-response

Installs into the current project.

Summary

Build AI-focused SRE incident response practices for LLM outages, degraded quality, runaway cost events, and safety regressions.

Files · 1

View on GitHub

SKILL.md

1	# AI SRE Incident Response
2
3	Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.
4
5	## When to Use This Skill
6
7	- An LLM endpoint begins returning degraded or hallucinated answers
8	- Token spend spikes beyond budget thresholds
9	- A model provider goes down and traffic must fail over
10	- Safety guardrails fire at abnormal rates
11	- A new model deployment causes latency or accuracy regression
12
13	## Prerequisites
14
15	- Prometheus and Alertmanager deployed with scrape targets for AI services
16	- Grafana dashboards for golden signals (latency, error rate, cost, quality)
17	- On-call rotation configured in PagerDuty, Opsgenie, or equivalent
18	- Runbook repository accessible to responders
19	- Rollback mechanism for model and prompt versions (GitOps or feature flags)
20
21	## AI Incident Classes
22
23	- Availability incident: model/provider unavailable, timeout storm.
24	- Quality incident: answer accuracy or tool success drops below SLO.
25	- Safety incident: harmful or policy-violating outputs increase.
26	- Cost incident: unexpected token or provider spend spike.
27
28	## Severity Framework
29
30	\| Severity \| Criteria \| Response Time \| Notification \|
31	\|----------\|----------\|---------------\|--------------\|
32	\| SEV1 \| User-facing outage, compliance risk, data leak \| 5 min \| Page on-call + incident commander \|
33	\| SEV2 \| Major degradation in key flows \| 15 min \| Page on-call \|
34	\| SEV3 \| Limited impact or internal-only issue \| 1 hour \| Slack alert \|
35	\| SEV4 \| Cosmetic or low-priority regression \| Next business day \| Ticket \|
36
37	## Golden Signals for AI Services
38
39	- Request success rate
40	- Latency (queue + generation + tool execution)
41	- Hallucination/groundedness proxy metrics
42	- Cost per minute and per tenant
43	- Guardrail violation rate
44
45	## Prometheus Alert Rules
46
47	```yaml
48	# prometheus-ai-alerts.yaml
49	groups:
50	- name: ai-service-alerts
51	rules:
52	- alert: ModelEndpointDown
53	expr: up{job="llm-inference"} == 0
54	for: 2m
55	labels:
56	severity: sev1
57	annotations:
58	summary: "LLM inference endpoint {{ $labels.instance }} is down"
59	runbook_url: "https://runbooks.internal/ai/model-outage"
60
61	- alert: HighHallucinationRate
62	expr: \|
63	rate(llm_hallucination_detected_total[10m])
64	/ rate(llm_requests_total[10m]) > 0.15
65	for: 5m
66	labels:
67	severity: sev2
68	annotations:
69	summary: "Hallucination rate above 15% for {{ $labels.model }}"
70	runbook_url: "https://runbooks.internal/ai/quality-regression"
71
72	- alert: TokenCostExplosion
73	expr: \|
74	sum(rate(llm_token_cost_dollars[5m])) by (tenant)
75	> 0.50
76	for: 3m
77	labels:
78	severity: sev2
79	annotations:
80	summary: "Token spend exceeds $0.50/min for tenant {{ $labels.tenant }}"
81	runbook_url: "https://runbooks.internal/ai/cost-spike"
82
83	- alert: LatencyP95Exceeded
84	expr: \|
85	histogram_quantile(0.95,
86	rate(llm_request_duration_seconds_bucket[5m])
87	) > 5
88	for: 5m
89	labels:
90	severity: sev2
91	annotations:
92	summary: "LLM p95 latency exceeds 5s for {{ $labels.service }}"
93
94	- alert: GuardrailViolationSpike
95	expr: \|
96	rate(llm_guardrail_violations_total[10m])
97	/ rate(llm_requests_total[10m]) > 0.05
98	for: 5m
99	labels:
100	severity: sev1
101	annotations:
102	summary: "Guardrail violations above 5% for {{ $labels.model }}"
103	runbook_url: "https://runbooks.internal/ai/safety-incident"
104
105	- alert: ModelQualityDrop
106	expr: \|
107	llm_eval_score{metric="groundedness"} < 0.70
108	for: 10m
109	labels:
110	severity: sev2
111	annotations:
112	summary: "Groundedness score dropped below 0.70 for {{ $labels.model }}"
113
114	- alert: ProviderErrorRateHigh
115	expr: \|
116	rate(llm_provider_errors_total[5m])
117	/ rate(llm_provider_requests_total[5m]) > 0.10
118	for: 3m
119	labels:
120	severity: sev2
121	annotations:
122	summary: "Provider {{ $labels.provider }} error rate above 10%"
123	```
124
125	## Response Playbooks
126
127	### Model Outage Runbook
128
129	```text
130	TRIGGER: ModelEndpointDown fires for > 2 minutes
131	RESPONDER: On-call AI platform engineer
132
133	1. Acknowledge alert in PagerDuty.
134	2. Check provider status page (e.g., status.openai.com).
135	3. Verify network connectivity:
136	curl -s -o /dev/null -w "%{http_code}" https://api.provider.com/health
137	4. If provider is down:
138	a. Enable fallback model route in gateway config.
139	b. kubectl set env deployment/llm-gateway FALLBACK_ENABLED=true
140	c. Verify fallback traffic is flowing via Grafana dashboard.
141	5. If se

People who liked this also viewed

6 picks

Type

ai-sre-incident-response

42stars4forks

View on GitHub

How to install ai-sre-incident-response?

bagelhole/devops-security-agent-skills/ai-sre-incident-response

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill ai-sre-incident-response

Installs into the current project.

Summary

Build AI-focused SRE incident response practices for LLM outages, degraded quality, runaway cost events, and safety regressions.

Files · 1

View on GitHub

SKILL.md

1	# AI SRE Incident Response
2
3	Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.
4
5	## When to Use This Skill
6
7	- An LLM endpoint begins returning degraded or hallucinated answers
8	- Token spend spikes beyond budget thresholds
9	- A model provider goes down and traffic must fail over
10	- Safety guardrails fire at abnormal rates
11	- A new model deployment causes latency or accuracy regression
12
13	## Prerequisites
14
15	- Prometheus and Alertmanager deployed with scrape targets for AI services
16	- Grafana dashboards for golden signals (latency, error rate, cost, quality)
17	- On-call rotation configured in PagerDuty, Opsgenie, or equivalent
18	- Runbook repository accessible to responders
19	- Rollback mechanism for model and prompt versions (GitOps or feature flags)
20
21	## AI Incident Classes
22
23	- Availability incident: model/provider unavailable, timeout storm.
24	- Quality incident: answer accuracy or tool success drops below SLO.
25	- Safety incident: harmful or policy-violating outputs increase.
26	- Cost incident: unexpected token or provider spend spike.
27
28	## Severity Framework
29
30	\| Severity \| Criteria \| Response Time \| Notification \|
31	\|----------\|----------\|---------------\|--------------\|
32	\| SEV1 \| User-facing outage, compliance risk, data leak \| 5 min \| Page on-call + incident commander \|
33	\| SEV2 \| Major degradation in key flows \| 15 min \| Page on-call \|
34	\| SEV3 \| Limited impact or internal-only issue \| 1 hour \| Slack alert \|
35	\| SEV4 \| Cosmetic or low-priority regression \| Next business day \| Ticket \|
36
37	## Golden Signals for AI Services
38
39	- Request success rate
40	- Latency (queue + generation + tool execution)
41	- Hallucination/groundedness proxy metrics
42	- Cost per minute and per tenant
43	- Guardrail violation rate
44
45	## Prometheus Alert Rules
46
47	```yaml
48	# prometheus-ai-alerts.yaml
49	groups:
50	- name: ai-service-alerts
51	rules:
52	- alert: ModelEndpointDown
53	expr: up{job="llm-inference"} == 0
54	for: 2m
55	labels:
56	severity: sev1
57	annotations:
58	summary: "LLM inference endpoint {{ $labels.instance }} is down"
59	runbook_url: "https://runbooks.internal/ai/model-outage"
60
61	- alert: HighHallucinationRate
62	expr: \|
63	rate(llm_hallucination_detected_total[10m])
64	/ rate(llm_requests_total[10m]) > 0.15
65	for: 5m
66	labels:
67	severity: sev2
68	annotations:
69	summary: "Hallucination rate above 15% for {{ $labels.model }}"
70	runbook_url: "https://runbooks.internal/ai/quality-regression"
71
72	- alert: TokenCostExplosion
73	expr: \|
74	sum(rate(llm_token_cost_dollars[5m])) by (tenant)
75	> 0.50
76	for: 3m
77	labels:
78	severity: sev2
79	annotations:
80	summary: "Token spend exceeds $0.50/min for tenant {{ $labels.tenant }}"
81	runbook_url: "https://runbooks.internal/ai/cost-spike"
82
83	- alert: LatencyP95Exceeded
84	expr: \|
85	histogram_quantile(0.95,
86	rate(llm_request_duration_seconds_bucket[5m])
87	) > 5
88	for: 5m
89	labels:
90	severity: sev2
91	annotations:
92	summary: "LLM p95 latency exceeds 5s for {{ $labels.service }}"
93
94	- alert: GuardrailViolationSpike
95	expr: \|
96	rate(llm_guardrail_violations_total[10m])
97	/ rate(llm_requests_total[10m]) > 0.05
98	for: 5m
99	labels:
100	severity: sev1
101	annotations:
102	summary: "Guardrail violations above 5% for {{ $labels.model }}"
103	runbook_url: "https://runbooks.internal/ai/safety-incident"
104
105	- alert: ModelQualityDrop
106	expr: \|
107	llm_eval_score{metric="groundedness"} < 0.70
108	for: 10m
109	labels:
110	severity: sev2
111	annotations:
112	summary: "Groundedness score dropped below 0.70 for {{ $labels.model }}"
113
114	- alert: ProviderErrorRateHigh
115	expr: \|
116	rate(llm_provider_errors_total[5m])
117	/ rate(llm_provider_requests_total[5m]) > 0.10
118	for: 3m
119	labels:
120	severity: sev2
121	annotations:
122	summary: "Provider {{ $labels.provider }} error rate above 10%"
123	```
124
125	## Response Playbooks
126
127	### Model Outage Runbook
128
129	```text
130	TRIGGER: ModelEndpointDown fires for > 2 minutes
131	RESPONDER: On-call AI platform engineer
132
133	1. Acknowledge alert in PagerDuty.
134	2. Check provider status page (e.g., status.openai.com).
135	3. Verify network connectivity:
136	curl -s -o /dev/null -w "%{http_code}" https://api.provider.com/health
137	4. If provider is down:
138	a. Enable fallback model route in gateway config.
139	b. kubectl set env deployment/llm-gateway FALLBACK_ENABLED=true
140	c. Verify fallback traffic is flowing via Grafana dashboard.
141	5. If se

People who liked this also viewed

6 picks

Type