alerting-oncall

Name: alerting-oncall
Author: BagelHole

42stars4forks

How to install alerting-oncall?

bagelhole/devops-security-agent-skills/alerting-oncall

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill alerting-oncall

Installs into the current project.

Summary

Set up alerting rules, configure on-call rotations, and manage incident response workflows. Integrate with PagerDuty, Opsgenie, or Grafana OnCall for alert routing and escalation. Use when implementing alerting strategies and on-call management for production systems.

Files · 1

View on GitHub

SKILL.md

1	# Alerting & On-Call
2
3	Configure effective alerting and on-call management for production systems.
4
5	## When to Use This Skill
6
7	Use this skill when:
8	- Setting up alerting rules and thresholds
9	- Configuring on-call rotations and schedules
10	- Implementing alert routing and escalation
11	- Reducing alert fatigue
12	- Managing incident response workflows
13
14	## Prerequisites
15
16	- Monitoring system (Prometheus, Datadog, etc.)
17	- On-call platform (PagerDuty, Opsgenie, Grafana OnCall)
18	- Communication channels (Slack, email)
19
20	## Alerting Best Practices
21
22	### Alert Categories
23
24	```yaml
25	# Severity levels
26	critical:
27	- Service completely down
28	- Data loss imminent
29	- Security breach
30	response: Immediate page, wake people up
31
32	high:
33	- Service degraded significantly
34	- Error rate above SLO
35	- Capacity near limit
36	response: Page during business hours, notify after hours
37
38	medium:
39	- Performance degradation
40	- Non-critical component failure
41	- Warning thresholds exceeded
42	response: Notify via Slack, review next business day
43
44	low:
45	- Informational alerts
46	- Capacity planning triggers
47	- Routine maintenance needed
48	response: Email notification, weekly review
49	```
50
51	### Alert Design Principles
52
53	```yaml
54	# Good alert characteristics
55	alerts:
56	actionable:
57	- Every alert should require human action
58	- Include runbook links
59	- Clear remediation steps
60
61	relevant:
62	- Alert on symptoms, not causes
63	- Focus on user impact
64	- Avoid alerting on expected behavior
65
66	timely:
67	- Appropriate thresholds
68	- Suitable evaluation windows
69	- Account for normal variance
70
71	unique:
72	- No duplicate alerts
73	- Proper alert grouping
74	- Clear ownership
75	```
76
77	## Prometheus Alerting
78
79	### Alert Rules
80
81	```yaml
82	# prometheus/rules/alerts.yml
83	groups:
84	- name: service_alerts
85	rules:
86	# High-level service health
87	- alert: ServiceDown
88	expr: up{job="myapp"} == 0
89	for: 1m
90	labels:
91	severity: critical
92	annotations:
93	summary: "Service {{ $labels.instance }} is down"
94	description: "{{ $labels.job }} on {{ $labels.instance }} has been down for more than 1 minute."
95	runbook_url: "https://wiki.example.com/runbooks/service-down"
96
97	# Error rate alert
98	- alert: HighErrorRate
99	expr: \|
100	sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
101	/ sum(rate(http_requests_total[5m])) by (service) > 0.05
102	for: 5m
103	labels:
104	severity: critical
105	annotations:
106	summary: "High error rate for {{ $labels.service }}"
107	description: "Error rate is {{ $value \| humanizePercentage }} for the last 5 minutes"
108
109	# Latency alert (SLO-based)
110	- alert: HighLatency
111	expr: \|
112	histogram_quantile(0.95,
113	sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
114	) > 0.5
115	for: 5m
116	labels:
117	severity: high
118	annotations:
119	summary: "P95 latency above 500ms for {{ $labels.service }}"
120	```
121
122	### Alertmanager Configuration
123
124	```yaml
125	# alertmanager.yml
126	global:
127	resolve_timeout: 5m
128	slack_api_url: 'https://hooks.slack.com/services/xxx'
129	pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
130
131	templates:
132	- '/etc/alertmanager/templates/*.tmpl'
133
134	route:
135	receiver: 'default-receiver'
136	group_by: ['alertname', 'service']
137	group_wait: 30s
138	group_interval: 5m
139	repeat_interval: 4h
140
141	routes:
142	# Critical alerts go to PagerDuty
143	- match:
144	severity: critical
145	receiver: 'pagerduty-critical'
146	group_wait: 0s
147	repeat_interval: 1h
148
149	# High severity during business hours
150	- match:
151	severity: high
152	receiver: 'slack-high'
153	active_time_intervals:
154	- business-hours
155
156	# Route by team
157	- match_re:
158	team: platform.*
159	receiver: 'platform-team'
160
161	receivers:
162	- name: 'default-receiver'
163	slack_configs:
164	- channel: '#alerts'
165	send_resolved: true
166
167	- name: 'pagerduty-critical'
168	pagerduty_configs:
169	- service_key: 'xxx'
170	severity: critical
171	description: '{{ .CommonAnnotations.summary }}'
172	details:
173	firing: '{{ template "pagerduty.firing" . }}'
174
175	- name: 'slack-high'
176	slack_configs:
177	- channel: '#alerts-high'
178	title: '{{ .CommonAnnotations.summary }}'
179	text: '{{ .CommonAnnotations.description }}'
180	actions:
181	- type: button
182	text: 'Runbook'
183	url: '{{ .CommonAnnotations.runbook_url }}'
184	- type: button
185	text: 'Dashboard'
186	url: '{{ .CommonAnnotations.dashboard_url }

People who liked this also viewed

6 picks

Type

alerting-oncall

42stars4forks

View on GitHub

How to install alerting-oncall?

bagelhole/devops-security-agent-skills/alerting-oncall

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill alerting-oncall

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Alerting & On-Call
2
3	Configure effective alerting and on-call management for production systems.
4
5	## When to Use This Skill
6
7	Use this skill when:
8	- Setting up alerting rules and thresholds
9	- Configuring on-call rotations and schedules
10	- Implementing alert routing and escalation
11	- Reducing alert fatigue
12	- Managing incident response workflows
13
14	## Prerequisites
15
16	- Monitoring system (Prometheus, Datadog, etc.)
17	- On-call platform (PagerDuty, Opsgenie, Grafana OnCall)
18	- Communication channels (Slack, email)
19
20	## Alerting Best Practices
21
22	### Alert Categories
23
24	```yaml
25	# Severity levels
26	critical:
27	- Service completely down
28	- Data loss imminent
29	- Security breach
30	response: Immediate page, wake people up
31
32	high:
33	- Service degraded significantly
34	- Error rate above SLO
35	- Capacity near limit
36	response: Page during business hours, notify after hours
37
38	medium:
39	- Performance degradation
40	- Non-critical component failure
41	- Warning thresholds exceeded
42	response: Notify via Slack, review next business day
43
44	low:
45	- Informational alerts
46	- Capacity planning triggers
47	- Routine maintenance needed
48	response: Email notification, weekly review
49	```
50
51	### Alert Design Principles
52
53	```yaml
54	# Good alert characteristics
55	alerts:
56	actionable:
57	- Every alert should require human action
58	- Include runbook links
59	- Clear remediation steps
60
61	relevant:
62	- Alert on symptoms, not causes
63	- Focus on user impact
64	- Avoid alerting on expected behavior
65
66	timely:
67	- Appropriate thresholds
68	- Suitable evaluation windows
69	- Account for normal variance
70
71	unique:
72	- No duplicate alerts
73	- Proper alert grouping
74	- Clear ownership
75	```
76
77	## Prometheus Alerting
78
79	### Alert Rules
80
81	```yaml
82	# prometheus/rules/alerts.yml
83	groups:
84	- name: service_alerts
85	rules:
86	# High-level service health
87	- alert: ServiceDown
88	expr: up{job="myapp"} == 0
89	for: 1m
90	labels:
91	severity: critical
92	annotations:
93	summary: "Service {{ $labels.instance }} is down"
94	description: "{{ $labels.job }} on {{ $labels.instance }} has been down for more than 1 minute."
95	runbook_url: "https://wiki.example.com/runbooks/service-down"
96
97	# Error rate alert
98	- alert: HighErrorRate
99	expr: \|
100	sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
101	/ sum(rate(http_requests_total[5m])) by (service) > 0.05
102	for: 5m
103	labels:
104	severity: critical
105	annotations:
106	summary: "High error rate for {{ $labels.service }}"
107	description: "Error rate is {{ $value \| humanizePercentage }} for the last 5 minutes"
108
109	# Latency alert (SLO-based)
110	- alert: HighLatency
111	expr: \|
112	histogram_quantile(0.95,
113	sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
114	) > 0.5
115	for: 5m
116	labels:
117	severity: high
118	annotations:
119	summary: "P95 latency above 500ms for {{ $labels.service }}"
120	```
121
122	### Alertmanager Configuration
123
124	```yaml
125	# alertmanager.yml
126	global:
127	resolve_timeout: 5m
128	slack_api_url: 'https://hooks.slack.com/services/xxx'
129	pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
130
131	templates:
132	- '/etc/alertmanager/templates/*.tmpl'
133
134	route:
135	receiver: 'default-receiver'
136	group_by: ['alertname', 'service']
137	group_wait: 30s
138	group_interval: 5m
139	repeat_interval: 4h
140
141	routes:
142	# Critical alerts go to PagerDuty
143	- match:
144	severity: critical
145	receiver: 'pagerduty-critical'
146	group_wait: 0s
147	repeat_interval: 1h
148
149	# High severity during business hours
150	- match:
151	severity: high
152	receiver: 'slack-high'
153	active_time_intervals:
154	- business-hours
155
156	# Route by team
157	- match_re:
158	team: platform.*
159	receiver: 'platform-team'
160
161	receivers:
162	- name: 'default-receiver'
163	slack_configs:
164	- channel: '#alerts'
165	send_resolved: true
166
167	- name: 'pagerduty-critical'
168	pagerduty_configs:
169	- service_key: 'xxx'
170	severity: critical
171	description: '{{ .CommonAnnotations.summary }}'
172	details:
173	firing: '{{ template "pagerduty.firing" . }}'
174
175	- name: 'slack-high'
176	slack_configs:
177	- channel: '#alerts-high'
178	title: '{{ .CommonAnnotations.summary }}'
179	text: '{{ .CommonAnnotations.description }}'
180	actions:
181	- type: button
182	text: 'Runbook'
183	url: '{{ .CommonAnnotations.runbook_url }}'
184	- type: button
185	text: 'Dashboard'
186	url: '{{ .CommonAnnotations.dashboard_url }

People who liked this also viewed

6 picks

Type