operating-production-services

Name: operating-production-services
Author: mjunaidca

21installs11forks

How to install operating-production-services?

mjunaidca/mjs-agent-skills/operating-production-services

$npx -y skills add mjunaidca/mjs-agent-skills --skill operating-production-services

Installs into the current project.

Summary

SRE patterns for production service reliability: SLOs, error budgets, postmortems, and incident response. Use when defining reliability targets, writing postmortems, implementing SLO alerting, or establishing on-call practices. NOT for initial service development (use scaffolding

Files · 1

View on GitHub

SKILL.md

1	# Operating Production Services
2
3	Production reliability patterns: measure what matters, learn from failures, improve systematically.
4
5	## Quick Reference
6
7	\| Need \| Go To \|
8	\|------\|-------\|
9	\| Define reliability targets \| [SLOs & Error Budgets](#slos--error-budgets) \|
10	\| Write incident report \| [Postmortem Templates](#postmortem-templates) \|
11	\| Set up SLO alerting \| [references/slo-alerting.md](references/slo-alerting.md) \|
12
13	---
14
15	## SLOs & Error Budgets
16
17	### The Hierarchy
18
19	```
20	SLA (Contract) → SLO (Target) → SLI (Measurement)
21	```
22
23	### Common SLIs
24
25	```promql
26	# Availability: successful requests / total requests
27	sum(rate(http_requests_total{status!~"5.."}[28d]))
28	/
29	sum(rate(http_requests_total[28d]))
30
31	# Latency: requests below threshold / total requests
32	sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
33	/
34	sum(rate(http_request_duration_seconds_count[28d]))
35	```
36
37	### SLO Targets Reality Check
38
39	\| SLO % \| Downtime/Month \| Downtime/Year \|
40	\|-------\|----------------\|---------------\|
41	\| 99% \| 7.2 hours \| 3.65 days \|
42	\| 99.9% \| 43 minutes \| 8.76 hours \|
43	\| 99.95% \| 22 minutes \| 4.38 hours \|
44	\| 99.99% \| 4.3 minutes \| 52 minutes \|
45
46	Don't aim for 100%. Each nine costs exponentially more.
47
48	### Error Budget
49
50	```
51	Error Budget = 1 - SLO Target
52	```
53
54	Example: 99.9% SLO = 0.1% error budget = 43 minutes/month
55
56	Policy:
57	\| Budget Remaining \| Action \|
58	\|------------------\|--------\|
59	\| > 50% \| Normal velocity \|
60	\| 10-50% \| Postpone risky changes \|
61	\| < 10% \| Freeze non-critical changes \|
62	\| 0% \| Feature freeze, fix reliability \|
63
64	See [references/slo-alerting.md](references/slo-alerting.md) for Prometheus recording rules and multi-window burn rate alerts.
65
66	---
67
68	## Postmortem Templates
69
70	### The Blameless Principle
71
72	\| Blame-Focused \| Blameless \|
73	\|---------------\|-----------\|
74	\| "Who caused this?" \| "What conditions allowed this?" \|
75	\| Punish individuals \| Improve systems \|
76	\| Hide information \| Share learnings \|
77
78	### When to Write Postmortems
79
80	- SEV1/SEV2 incidents
81	- Customer-facing outages > 15 minutes
82	- Data loss or security incidents
83	- Near-misses that could have been severe
84	- Novel failure modes
85
86	### Standard Template
87
88	```markdown
89	# Postmortem: [Incident Title]
90
91	Date: YYYY-MM-DD \| Duration: X min \| Severity: SEVX
92
93	## Executive Summary
94	One paragraph: what happened, impact, root cause, resolution.
95
96	## Timeline (UTC)
97	\| Time \| Event \|
98	\|------\|-------\|
99	\| HH:MM \| First alert fired \|
100	\| HH:MM \| On-call acknowledged \|
101	\| HH:MM \| Root cause identified \|
102	\| HH:MM \| Fix deployed \|
103	\| HH:MM \| Service recovered \|
104
105	## Root Cause Analysis
106
107	### 5 Whys
108	1. Why did service fail? → [Answer]
109	2. Why did [1] happen? → [Answer]
110	3. Why did [2] happen? → [Answer]
111	4. Why did [3] happen? → [Answer]
112	5. Why did [4] happen? → [Root cause]
113
114	## Impact
115	- Customers affected: X
116	- Duration: X minutes
117	- Revenue impact: $X
118	- Support tickets: X
119
120	## Action Items
121	\| Priority \| Action \| Owner \| Due \| Ticket \|
122	\|----------\|--------\|-------\|-----\|--------\|
123	\| P0 \| [Immediate fix] \| @name \| Date \| XXX-123 \|
124	\| P1 \| [Prevent recurrence] \| @name \| Date \| XXX-124 \|
125	\| P2 \| [Improve detection] \| @name \| Date \| XXX-125 \|
126	```
127
128	### Quick Template (Minor Incidents)
129
130	```markdown
131	# Quick Postmortem: [Title]
132
133	Date: YYYY-MM-DD \| Duration: X min \| Severity: SEV3
134
135	## What Happened
136	One sentence description.
137
138	## Timeline
139	- HH:MM - Trigger
140	- HH:MM - Detection
141	- HH:MM - Resolution
142
143	## Root Cause
144	One sentence.
145
146	## Fix
147	- Immediate: [What was done]
148	- Long-term: [Ticket XXX-123]
149	```
150
151	---
152
153	## Postmortem Meeting Guide
154
155	### Structure (60 min)
156
157	1. Opening (5 min) - Remind: "We're here to learn, not blame"
158	2. Timeline (15 min) - Walk through events chronologically
159	3. Analysis (20 min) - What failed? Why? What allowed it?
160	4. Action Items (15 min) - Prioritize, assign owners, set dates
161	5. Closing (5 min) - Summarize learnings, confirm owners
162
163	### Facilitation Tips
164
165	- Redirect blame to systems: "What made this mistake possible?"
166	- Time-box tangents
167	- Document dissenting views
168	- Encourage quiet participants
169
170	---
171
172	## Anti-Patterns
173
174	\| Don't \| Do Instead \|
175	\|-------\|------------\|
176	\| Aim for 100% SLO \| Accept error budget exists \|
177	\| Skip small incidents \| Small incidents reveal patterns \|
178	\| Orphan action items \| Every item needs owner + date + ticket \|
179	\| Blame individuals \| Ask "what conditions allowed this?" \|
180	\| Create busywork actions \| Actions should prevent recurrence \|
181
182	---
183
184	## Verification
185
186	Run: `python scripts/verify.py`
187
188	## References
189
190	- [references/slo-alerting.md](references/slo-alerting.md) - Prometheus rules, burn rate alerts, Grafana dashboards

People who liked this also viewed

6 picks

Type

operating-production-services

21installs11forks

View on GitHub

How to install operating-production-services?

mjunaidca/mjs-agent-skills/operating-production-services

$npx -y skills add mjunaidca/mjs-agent-skills --skill operating-production-services

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Operating Production Services
2
3	Production reliability patterns: measure what matters, learn from failures, improve systematically.
4
5	## Quick Reference
6
7	\| Need \| Go To \|
8	\|------\|-------\|
9	\| Define reliability targets \| [SLOs & Error Budgets](#slos--error-budgets) \|
10	\| Write incident report \| [Postmortem Templates](#postmortem-templates) \|
11	\| Set up SLO alerting \| [references/slo-alerting.md](references/slo-alerting.md) \|
12
13	---
14
15	## SLOs & Error Budgets
16
17	### The Hierarchy
18
19	```
20	SLA (Contract) → SLO (Target) → SLI (Measurement)
21	```
22
23	### Common SLIs
24
25	```promql
26	# Availability: successful requests / total requests
27	sum(rate(http_requests_total{status!~"5.."}[28d]))
28	/
29	sum(rate(http_requests_total[28d]))
30
31	# Latency: requests below threshold / total requests
32	sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
33	/
34	sum(rate(http_request_duration_seconds_count[28d]))
35	```
36
37	### SLO Targets Reality Check
38
39	\| SLO % \| Downtime/Month \| Downtime/Year \|
40	\|-------\|----------------\|---------------\|
41	\| 99% \| 7.2 hours \| 3.65 days \|
42	\| 99.9% \| 43 minutes \| 8.76 hours \|
43	\| 99.95% \| 22 minutes \| 4.38 hours \|
44	\| 99.99% \| 4.3 minutes \| 52 minutes \|
45
46	Don't aim for 100%. Each nine costs exponentially more.
47
48	### Error Budget
49
50	```
51	Error Budget = 1 - SLO Target
52	```
53
54	Example: 99.9% SLO = 0.1% error budget = 43 minutes/month
55
56	Policy:
57	\| Budget Remaining \| Action \|
58	\|------------------\|--------\|
59	\| > 50% \| Normal velocity \|
60	\| 10-50% \| Postpone risky changes \|
61	\| < 10% \| Freeze non-critical changes \|
62	\| 0% \| Feature freeze, fix reliability \|
63
64	See [references/slo-alerting.md](references/slo-alerting.md) for Prometheus recording rules and multi-window burn rate alerts.
65
66	---
67
68	## Postmortem Templates
69
70	### The Blameless Principle
71
72	\| Blame-Focused \| Blameless \|
73	\|---------------\|-----------\|
74	\| "Who caused this?" \| "What conditions allowed this?" \|
75	\| Punish individuals \| Improve systems \|
76	\| Hide information \| Share learnings \|
77
78	### When to Write Postmortems
79
80	- SEV1/SEV2 incidents
81	- Customer-facing outages > 15 minutes
82	- Data loss or security incidents
83	- Near-misses that could have been severe
84	- Novel failure modes
85
86	### Standard Template
87
88	```markdown
89	# Postmortem: [Incident Title]
90
91	Date: YYYY-MM-DD \| Duration: X min \| Severity: SEVX
92
93	## Executive Summary
94	One paragraph: what happened, impact, root cause, resolution.
95
96	## Timeline (UTC)
97	\| Time \| Event \|
98	\|------\|-------\|
99	\| HH:MM \| First alert fired \|
100	\| HH:MM \| On-call acknowledged \|
101	\| HH:MM \| Root cause identified \|
102	\| HH:MM \| Fix deployed \|
103	\| HH:MM \| Service recovered \|
104
105	## Root Cause Analysis
106
107	### 5 Whys
108	1. Why did service fail? → [Answer]
109	2. Why did [1] happen? → [Answer]
110	3. Why did [2] happen? → [Answer]
111	4. Why did [3] happen? → [Answer]
112	5. Why did [4] happen? → [Root cause]
113
114	## Impact
115	- Customers affected: X
116	- Duration: X minutes
117	- Revenue impact: $X
118	- Support tickets: X
119
120	## Action Items
121	\| Priority \| Action \| Owner \| Due \| Ticket \|
122	\|----------\|--------\|-------\|-----\|--------\|
123	\| P0 \| [Immediate fix] \| @name \| Date \| XXX-123 \|
124	\| P1 \| [Prevent recurrence] \| @name \| Date \| XXX-124 \|
125	\| P2 \| [Improve detection] \| @name \| Date \| XXX-125 \|
126	```
127
128	### Quick Template (Minor Incidents)
129
130	```markdown
131	# Quick Postmortem: [Title]
132
133	Date: YYYY-MM-DD \| Duration: X min \| Severity: SEV3
134
135	## What Happened
136	One sentence description.
137
138	## Timeline
139	- HH:MM - Trigger
140	- HH:MM - Detection
141	- HH:MM - Resolution
142
143	## Root Cause
144	One sentence.
145
146	## Fix
147	- Immediate: [What was done]
148	- Long-term: [Ticket XXX-123]
149	```
150
151	---
152
153	## Postmortem Meeting Guide
154
155	### Structure (60 min)
156
157	1. Opening (5 min) - Remind: "We're here to learn, not blame"
158	2. Timeline (15 min) - Walk through events chronologically
159	3. Analysis (20 min) - What failed? Why? What allowed it?
160	4. Action Items (15 min) - Prioritize, assign owners, set dates
161	5. Closing (5 min) - Summarize learnings, confirm owners
162
163	### Facilitation Tips
164
165	- Redirect blame to systems: "What made this mistake possible?"
166	- Time-box tangents
167	- Document dissenting views
168	- Encourage quiet participants
169
170	---
171
172	## Anti-Patterns
173
174	\| Don't \| Do Instead \|
175	\|-------\|------------\|
176	\| Aim for 100% SLO \| Accept error budget exists \|
177	\| Skip small incidents \| Small incidents reveal patterns \|
178	\| Orphan action items \| Every item needs owner + date + ticket \|
179	\| Blame individuals \| Ask "what conditions allowed this?" \|
180	\| Create busywork actions \| Actions should prevent recurrence \|
181
182	---
183
184	## Verification
185
186	Run: `python scripts/verify.py`
187
188	## References
189
190	- [references/slo-alerting.md](references/slo-alerting.md) - Prometheus rules, burn rate alerts, Grafana dashboards

People who liked this also viewed

6 picks

Type