runbook-creation

Name: runbook-creation
Author: BagelHole

42stars4forks

How to install runbook-creation?

bagelhole/devops-security-agent-skills/runbook-creation

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill runbook-creation

Installs into the current project.

Summary

Create operational runbooks and standard operating procedures. Document troubleshooting guides and recovery procedures. Use when documenting operational knowledge.

Files · 1

View on GitHub

SKILL.md

1	# Runbook Creation
2
3	Create effective operational runbooks, standard operating procedures, and
4	troubleshooting guides that any on-call engineer can follow under pressure.
5
6	## Runbook Template — Full Structure
7
8	````markdown
9	# Runbook: [Service / Process Name]
10
11	Owner: [Team or individual]
12	Last Reviewed: YYYY-MM-DD
13	Version: X.Y
14	Severity if unavailable: SEV[1-4]
15
16	---
17
18	## Overview
19
20	Brief description of the service, why this runbook exists, and when to
21	use it.
22
23	## Prerequisites
24
25	- [ ] Required access / IAM role: [details]
26	- [ ] Tools installed: [kubectl, aws-cli, psql, etc.]
27	- [ ] VPN connected to [environment]
28	- [ ] Communication channel open: [Slack #channel]
29
30	## Procedure
31
32	### Step 1 — [Action Name]
33
34	[Explanation of what this step does and why.]
35
36	```bash
37	# command here
38	```
39
40	Expected output: [describe what success looks like]
41
42	### Step 2 — [Action Name]
43
44	```bash
45	# command here
46	```
47
48	Expected output: [description]
49
50	(Continue with numbered steps...)
51
52	## Verification
53
54	How to confirm the procedure succeeded:
55
56	- [ ] [Check 1 — e.g., health endpoint returns 200]
57	- [ ] [Check 2 — e.g., no errors in logs for 5 minutes]
58	- [ ] [Check 3 — e.g., metrics return to baseline]
59
60	## Rollback
61
62	If the procedure fails or causes unexpected issues:
63
64	### Rollback Step 1
65	```bash
66	# rollback command
67	```
68
69	### Rollback Step 2
70	```bash
71	# rollback command
72	```
73
74	## Troubleshooting
75
76	\| Symptom \| Likely Cause \| Resolution \|
77	\|---------\|-------------\|------------\|
78	\| [symptom 1] \| [cause] \| [fix] \|
79	\| [symptom 2] \| [cause] \| [fix] \|
80
81	## Escalation
82
83	If unresolved after [X] minutes:
84	- Primary: @[team-lead] — [phone/Slack]
85	- Secondary: @[manager] — [phone/Slack]
86
87	## Related Runbooks
88
89	- [Link to related runbook 1]
90	- [Link to related runbook 2]
91
92	## Change Log
93
94	\| Date \| Author \| Change \|
95	\|------\|--------\|--------\|
96	\| YYYY-MM-DD \| [Name] \| Initial version \|
97	````
98
99	## Example Runbook — Database Failover
100
101	````markdown
102	# Runbook: PostgreSQL Database Failover
103
104	Owner: Platform / DBA team
105	Last Reviewed: 2025-06-15
106	Version: 2.1
107	Severity if unavailable: SEV1
108
109	---
110
111	## Overview
112
113	Failover the primary PostgreSQL instance to the synchronous replica when
114	the primary is unreachable or degraded. This runbook covers both planned
115	(maintenance) and unplanned (emergency) failover.
116
117	## Prerequisites
118
119	- [ ] DBA or SRE-level access to primary and replica hosts
120	- [ ] `psql` client installed (v14+)
121	- [ ] VPN connected to production network
122	- [ ] Slack channel #db-ops open
123	- [ ] Confirm replica is in sync: replication lag < 1 MB
124
125	## Procedure
126
127	### Step 1 — Verify Replica Health
128
129	```bash
130	psql -h replica.db.internal -U dba -d postgres -c \
131	"SELECT pg_is_in_recovery(), pg_last_wal_replay_lsn();"
132	```
133
134	Expected output: `pg_is_in_recovery = t`, LSN advancing.
135
136	### Step 2 — Stop Application Writes
137
138	```bash
139	kubectl scale deployment api-server --replicas=0 -n production
140	kubectl scale deployment worker --replicas=0 -n production
141	```
142
143	Expected output: Deployments scaled to 0 pods.
144
145	### Step 3 — Confirm Write Quiesce
146
147	```bash
148	psql -h primary.db.internal -U dba -d postgres -c \
149	"SELECT count(*) FROM pg_stat_activity WHERE state = 'active' AND query !~ 'pg_stat';"
150	```
151
152	Expected output: Count = 0 (no active queries).
153
154	### Step 4 — Promote Replica
155
156	```bash
157	psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_promote();"
158	```
159
160	Wait up to 30 seconds, then confirm:
161
162	```bash
163	psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_is_in_recovery();"
164	```
165
166	Expected output: `pg_is_in_recovery = f` (no longer a replica).
167
168	### Step 5 — Update DNS
169
170	```bash
171	aws route53 change-resource-record-sets \
172	--hosted-zone-id Z1234567890 \
173	--change-batch '{
174	"Changes": [{
175	"Action": "UPSERT",
176	"ResourceRecordSet": {
177	"Name": "db.internal.example.com",
178	"Type": "CNAME",
179	"TTL": 60,
180	"ResourceRecords": [{"Value": "replica.db.internal"}]
181	}
182	}]
183	}'
184	```
185
186	### Step 6 — Restart Application
187
188	```bash
189	kubectl scale deployment api-server --replicas=6 -n production
190	kubectl scale deployment worker --replicas=4 -n production
191	```
192
193	## Verification
194
195	- [ ] `psql -h db.internal.example.com -c "SELECT 1;"` returns successfully
196	- [ ] Application logs show successful DB connections (no errors for 5 min)
197	- [ ] Transaction throughput returns to baseline on Grafana dashboard
198	- [ ] No replication-lag alerts firing
199
200	## Rollback
201
202	If the promoted replica has issues, restore from the most recent backup:
203
204	```bash
205	# Restore latest automated snapshot (RDS example)
206	aws rds restore-db-instance-from-db-snapshot \
207	--db-instance-identifier prod-db-restored \
208	--db-snapshot-identifier prod-db-latest-snapshot
209	```
210
211	## Escalation
212
213	If unresolved after 15 minutes:

People who liked this also viewed

6 picks

Type

runbook-creation

42stars4forks

View on GitHub

How to install runbook-creation?

bagelhole/devops-security-agent-skills/runbook-creation

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill runbook-creation

Installs into the current project.

Summary

Create operational runbooks and standard operating procedures. Document troubleshooting guides and recovery procedures. Use when documenting operational knowledge.

Files · 1

View on GitHub

SKILL.md

1	# Runbook Creation
2
3	Create effective operational runbooks, standard operating procedures, and
4	troubleshooting guides that any on-call engineer can follow under pressure.
5
6	## Runbook Template — Full Structure
7
8	````markdown
9	# Runbook: [Service / Process Name]
10
11	Owner: [Team or individual]
12	Last Reviewed: YYYY-MM-DD
13	Version: X.Y
14	Severity if unavailable: SEV[1-4]
15
16	---
17
18	## Overview
19
20	Brief description of the service, why this runbook exists, and when to
21	use it.
22
23	## Prerequisites
24
25	- [ ] Required access / IAM role: [details]
26	- [ ] Tools installed: [kubectl, aws-cli, psql, etc.]
27	- [ ] VPN connected to [environment]
28	- [ ] Communication channel open: [Slack #channel]
29
30	## Procedure
31
32	### Step 1 — [Action Name]
33
34	[Explanation of what this step does and why.]
35
36	```bash
37	# command here
38	```
39
40	Expected output: [describe what success looks like]
41
42	### Step 2 — [Action Name]
43
44	```bash
45	# command here
46	```
47
48	Expected output: [description]
49
50	(Continue with numbered steps...)
51
52	## Verification
53
54	How to confirm the procedure succeeded:
55
56	- [ ] [Check 1 — e.g., health endpoint returns 200]
57	- [ ] [Check 2 — e.g., no errors in logs for 5 minutes]
58	- [ ] [Check 3 — e.g., metrics return to baseline]
59
60	## Rollback
61
62	If the procedure fails or causes unexpected issues:
63
64	### Rollback Step 1
65	```bash
66	# rollback command
67	```
68
69	### Rollback Step 2
70	```bash
71	# rollback command
72	```
73
74	## Troubleshooting
75
76	\| Symptom \| Likely Cause \| Resolution \|
77	\|---------\|-------------\|------------\|
78	\| [symptom 1] \| [cause] \| [fix] \|
79	\| [symptom 2] \| [cause] \| [fix] \|
80
81	## Escalation
82
83	If unresolved after [X] minutes:
84	- Primary: @[team-lead] — [phone/Slack]
85	- Secondary: @[manager] — [phone/Slack]
86
87	## Related Runbooks
88
89	- [Link to related runbook 1]
90	- [Link to related runbook 2]
91
92	## Change Log
93
94	\| Date \| Author \| Change \|
95	\|------\|--------\|--------\|
96	\| YYYY-MM-DD \| [Name] \| Initial version \|
97	````
98
99	## Example Runbook — Database Failover
100
101	````markdown
102	# Runbook: PostgreSQL Database Failover
103
104	Owner: Platform / DBA team
105	Last Reviewed: 2025-06-15
106	Version: 2.1
107	Severity if unavailable: SEV1
108
109	---
110
111	## Overview
112
113	Failover the primary PostgreSQL instance to the synchronous replica when
114	the primary is unreachable or degraded. This runbook covers both planned
115	(maintenance) and unplanned (emergency) failover.
116
117	## Prerequisites
118
119	- [ ] DBA or SRE-level access to primary and replica hosts
120	- [ ] `psql` client installed (v14+)
121	- [ ] VPN connected to production network
122	- [ ] Slack channel #db-ops open
123	- [ ] Confirm replica is in sync: replication lag < 1 MB
124
125	## Procedure
126
127	### Step 1 — Verify Replica Health
128
129	```bash
130	psql -h replica.db.internal -U dba -d postgres -c \
131	"SELECT pg_is_in_recovery(), pg_last_wal_replay_lsn();"
132	```
133
134	Expected output: `pg_is_in_recovery = t`, LSN advancing.
135
136	### Step 2 — Stop Application Writes
137
138	```bash
139	kubectl scale deployment api-server --replicas=0 -n production
140	kubectl scale deployment worker --replicas=0 -n production
141	```
142
143	Expected output: Deployments scaled to 0 pods.
144
145	### Step 3 — Confirm Write Quiesce
146
147	```bash
148	psql -h primary.db.internal -U dba -d postgres -c \
149	"SELECT count(*) FROM pg_stat_activity WHERE state = 'active' AND query !~ 'pg_stat';"
150	```
151
152	Expected output: Count = 0 (no active queries).
153
154	### Step 4 — Promote Replica
155
156	```bash
157	psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_promote();"
158	```
159
160	Wait up to 30 seconds, then confirm:
161
162	```bash
163	psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_is_in_recovery();"
164	```
165
166	Expected output: `pg_is_in_recovery = f` (no longer a replica).
167
168	### Step 5 — Update DNS
169
170	```bash
171	aws route53 change-resource-record-sets \
172	--hosted-zone-id Z1234567890 \
173	--change-batch '{
174	"Changes": [{
175	"Action": "UPSERT",
176	"ResourceRecordSet": {
177	"Name": "db.internal.example.com",
178	"Type": "CNAME",
179	"TTL": 60,
180	"ResourceRecords": [{"Value": "replica.db.internal"}]
181	}
182	}]
183	}'
184	```
185
186	### Step 6 — Restart Application
187
188	```bash
189	kubectl scale deployment api-server --replicas=6 -n production
190	kubectl scale deployment worker --replicas=4 -n production
191	```
192
193	## Verification
194
195	- [ ] `psql -h db.internal.example.com -c "SELECT 1;"` returns successfully
196	- [ ] Application logs show successful DB connections (no errors for 5 min)
197	- [ ] Transaction throughput returns to baseline on Grafana dashboard
198	- [ ] No replication-lag alerts firing
199
200	## Rollback
201
202	If the promoted replica has issues, restore from the most recent backup:
203
204	```bash
205	# Restore latest automated snapshot (RDS example)
206	aws rds restore-db-instance-from-db-snapshot \
207	--db-instance-identifier prod-db-restored \
208	--db-snapshot-identifier prod-db-latest-snapshot
209	```
210
211	## Escalation
212
213	If unresolved after 15 minutes:

People who liked this also viewed

6 picks

Type