aws-cloudwatch-investigation

Name: aws-cloudwatch-investigation
Author: github

37kstars4.6kforks

How to install aws-cloudwatch-investigation?

github/awesome-copilot/aws-cloudwatch-investigation

$npx -y skills add github/awesome-copilot --skill aws-cloudwatch-investigation

Installs into the current project.

Summary

Reusable investigation patterns for AWS CloudWatch: Logs Insights query templates, alarm-to-deployment correlation, blast-radius narrowing decision tree, and PromQL-style metric query patterns for structured incident triage.

Files · 1

View on GitHub

SKILL.md

1	# AWS CloudWatch Investigation Skill
2
3	Reusable patterns for investigating production incidents using CloudWatch Logs, Metrics, and Alarms. These patterns are designed to be composed together during incident triage.
4
5	---
6
7	## Pattern 1: Logs Insights Query Templates
8
9	### Error Spike Detection
10
11	Find the top errors in a time window, grouped by error type:
12
13	```
14	fields @timestamp, @message, @logStream
15	\| filter @message like /(?i)(error\|exception\|fatal\|critical)/
16	\| stats count(*) as errorCount by bin(5m), @logStream
17	\| sort errorCount desc
18	\| limit 20
19	```
20
21	### P99 Latency Breakdown by Operation
22
23	Identify which operations are driving latency spikes:
24
25	```
26	fields @timestamp, @duration, operation
27	\| filter ispresent(@duration)
28	\| stats avg(@duration) as avgMs,
29	pct(@duration, 50) as p50Ms,
30	pct(@duration, 95) as p95Ms,
31	pct(@duration, 99) as p99Ms,
32	count(*) as invocations
33	by operation
34	\| sort p99Ms desc
35	\| limit 15
36	```
37
38	### Lambda Cold Start Detection
39
40	Quantify cold start impact during an incident:
41
42	```
43	fields @timestamp, @duration, @initDuration, @memorySize, @maxMemoryUsed
44	\| filter ispresent(@initDuration)
45	\| stats count(*) as coldStarts,
46	avg(@initDuration) as avgInitMs,
47	max(@initDuration) as maxInitMs,
48	avg(@duration) as avgDurationMs
49	by bin(5m)
50	\| sort @timestamp desc
51	```
52
53	### Out-of-Memory (OOM) Detection
54
55	Find Lambda functions or containers killed by memory pressure:
56
57	```
58	fields @timestamp, @message, @logStream, @memorySize, @maxMemoryUsed
59	\| filter @message like /Runtime exited\|out of memory\|OOMKilled\|Cannot allocate memory\|MemoryError/
60	\| stats count(*) as oomEvents by @logStream, bin(10m)
61	\| sort oomEvents desc
62	\| limit 10
63	```
64
65	For memory utilization trending before OOM:
66
67	```
68	fields @timestamp, @maxMemoryUsed, @memorySize
69	\| filter ispresent(@maxMemoryUsed)
70	\| stats max(@maxMemoryUsed / @memorySize * 100) as peakMemPct,
71	avg(@maxMemoryUsed / @memorySize * 100) as avgMemPct
72	by bin(5m)
73	\| sort @timestamp desc
74	```
75
76	### Timeout Detection
77
78	Find invocations that hit the configured timeout:
79
80	```
81	fields @timestamp, @duration, @logStream, @requestId
82	\| filter @message like /Task timed out/ or @duration > 28000
83	\| stats count(*) as timeouts by @logStream, bin(5m)
84	\| sort timeouts desc
85	```
86
87	---
88
89	## Pattern 2: Alarm History to Deploy-Event Correlation
90
91	### Process
92
93	1. Get alarm transition time — note the exact timestamp when the alarm entered ALARM state.
94	2. Query CloudTrail for deployment-related events in a window of [alarm_time - 30min, alarm_time]:
95
96	```
97	# CloudTrail Lake query for deployment events
98	SELECT eventTime, eventName, userIdentity.arn, requestParameters
99	FROM <event-data-store-id>
100	WHERE eventTime > '<alarm_time_minus_30m>'
101	AND eventTime < '<alarm_time>'
102	AND eventName IN (
103	'UpdateFunctionCode', 'UpdateFunctionConfiguration',
104	'UpdateService', 'CreateDeployment', 'RegisterTaskDefinition',
105	'CreateChangeSet', 'ExecuteChangeSet',
106	'StartPipelineExecution', 'PutImage'
107	)
108	ORDER BY eventTime DESC
109	```
110
111	3. Correlation criteria — a deploy is "correlated" if:
112	- It targets the same service/resource as the alarm
113	- It completed within 15 minutes before the alarm transition
114	- The deployer identity matches a CI/CD role (not a human applying a hotfix)
115
116	4. Strengthening the correlation:
117	- Check if the same alarm was healthy in the previous deployment cycle
118	- Verify no other environmental changes (scaling events, config changes) in the same window
119	- Look for canary/synthetic monitor failures that started at the same time
120
121	### Output Format
122
123	```
124	Deploy Correlation:
125	Event: UpdateFunctionCode
126	Time: 2024-03-15T14:23:07Z (12 min before alarm)
127	Actor: arn:aws:sts::123456789012:assumed-role/github-actions-deploy/session
128	Resource: arn:aws:lambda:us-east-1:123456789012:function:payment-processor
129	Correlation: STRONG — same resource, CI/CD actor, alarm was OK prior cycle
130	```
131
132	---
133
134	## Pattern 3: Narrow the Blast Radius Decision Tree
135
136	Use this tree to systematically scope an incident from broadest to most specific:
137
138	```
139	START
140	\|
141	v
142	[1] ACCOUNT — Which account(s) show the alarm?
143	\| - Check: Are alarms firing in multiple accounts?
144	\| - If yes → suspect shared service (SSO, networking, shared deployment pipeline)
145	\| - If no → proceed to Region
146	v
147	[2] REGION — Which region(s) are affected?
148	\| - Check: Same alarm in other regions?
149	\| - If multi-region → suspect global service (IAM, Route53, S3 global)
150	\| - If single-region → proceed to Service
151	v
152	[3] SERVICE — Which service namespace shows degradation?
153	\| - Check CloudWatch namespace: AWS/Lambda, AWS/ECS, AWS/ApiGateway, etc.
154	\| - If multiple services → suspect shared dependency (VPC,

People who liked this also viewed

6 picks

Type

aws-cloudwatch-investigation

37kstars4.6kforks

View on GitHub

How to install aws-cloudwatch-investigation?

github/awesome-copilot/aws-cloudwatch-investigation

$npx -y skills add github/awesome-copilot --skill aws-cloudwatch-investigation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# AWS CloudWatch Investigation Skill
2
3	Reusable patterns for investigating production incidents using CloudWatch Logs, Metrics, and Alarms. These patterns are designed to be composed together during incident triage.
4
5	---
6
7	## Pattern 1: Logs Insights Query Templates
8
9	### Error Spike Detection
10
11	Find the top errors in a time window, grouped by error type:
12
13	```
14	fields @timestamp, @message, @logStream
15	\| filter @message like /(?i)(error\|exception\|fatal\|critical)/
16	\| stats count(*) as errorCount by bin(5m), @logStream
17	\| sort errorCount desc
18	\| limit 20
19	```
20
21	### P99 Latency Breakdown by Operation
22
23	Identify which operations are driving latency spikes:
24
25	```
26	fields @timestamp, @duration, operation
27	\| filter ispresent(@duration)
28	\| stats avg(@duration) as avgMs,
29	pct(@duration, 50) as p50Ms,
30	pct(@duration, 95) as p95Ms,
31	pct(@duration, 99) as p99Ms,
32	count(*) as invocations
33	by operation
34	\| sort p99Ms desc
35	\| limit 15
36	```
37
38	### Lambda Cold Start Detection
39
40	Quantify cold start impact during an incident:
41
42	```
43	fields @timestamp, @duration, @initDuration, @memorySize, @maxMemoryUsed
44	\| filter ispresent(@initDuration)
45	\| stats count(*) as coldStarts,
46	avg(@initDuration) as avgInitMs,
47	max(@initDuration) as maxInitMs,
48	avg(@duration) as avgDurationMs
49	by bin(5m)
50	\| sort @timestamp desc
51	```
52
53	### Out-of-Memory (OOM) Detection
54
55	Find Lambda functions or containers killed by memory pressure:
56
57	```
58	fields @timestamp, @message, @logStream, @memorySize, @maxMemoryUsed
59	\| filter @message like /Runtime exited\|out of memory\|OOMKilled\|Cannot allocate memory\|MemoryError/
60	\| stats count(*) as oomEvents by @logStream, bin(10m)
61	\| sort oomEvents desc
62	\| limit 10
63	```
64
65	For memory utilization trending before OOM:
66
67	```
68	fields @timestamp, @maxMemoryUsed, @memorySize
69	\| filter ispresent(@maxMemoryUsed)
70	\| stats max(@maxMemoryUsed / @memorySize * 100) as peakMemPct,
71	avg(@maxMemoryUsed / @memorySize * 100) as avgMemPct
72	by bin(5m)
73	\| sort @timestamp desc
74	```
75
76	### Timeout Detection
77
78	Find invocations that hit the configured timeout:
79
80	```
81	fields @timestamp, @duration, @logStream, @requestId
82	\| filter @message like /Task timed out/ or @duration > 28000
83	\| stats count(*) as timeouts by @logStream, bin(5m)
84	\| sort timeouts desc
85	```
86
87	---
88
89	## Pattern 2: Alarm History to Deploy-Event Correlation
90
91	### Process
92
93	1. Get alarm transition time — note the exact timestamp when the alarm entered ALARM state.
94	2. Query CloudTrail for deployment-related events in a window of [alarm_time - 30min, alarm_time]:
95
96	```
97	# CloudTrail Lake query for deployment events
98	SELECT eventTime, eventName, userIdentity.arn, requestParameters
99	FROM <event-data-store-id>
100	WHERE eventTime > '<alarm_time_minus_30m>'
101	AND eventTime < '<alarm_time>'
102	AND eventName IN (
103	'UpdateFunctionCode', 'UpdateFunctionConfiguration',
104	'UpdateService', 'CreateDeployment', 'RegisterTaskDefinition',
105	'CreateChangeSet', 'ExecuteChangeSet',
106	'StartPipelineExecution', 'PutImage'
107	)
108	ORDER BY eventTime DESC
109	```
110
111	3. Correlation criteria — a deploy is "correlated" if:
112	- It targets the same service/resource as the alarm
113	- It completed within 15 minutes before the alarm transition
114	- The deployer identity matches a CI/CD role (not a human applying a hotfix)
115
116	4. Strengthening the correlation:
117	- Check if the same alarm was healthy in the previous deployment cycle
118	- Verify no other environmental changes (scaling events, config changes) in the same window
119	- Look for canary/synthetic monitor failures that started at the same time
120
121	### Output Format
122
123	```
124	Deploy Correlation:
125	Event: UpdateFunctionCode
126	Time: 2024-03-15T14:23:07Z (12 min before alarm)
127	Actor: arn:aws:sts::123456789012:assumed-role/github-actions-deploy/session
128	Resource: arn:aws:lambda:us-east-1:123456789012:function:payment-processor
129	Correlation: STRONG — same resource, CI/CD actor, alarm was OK prior cycle
130	```
131
132	---
133
134	## Pattern 3: Narrow the Blast Radius Decision Tree
135
136	Use this tree to systematically scope an incident from broadest to most specific:
137
138	```
139	START
140	\|
141	v
142	[1] ACCOUNT — Which account(s) show the alarm?
143	\| - Check: Are alarms firing in multiple accounts?
144	\| - If yes → suspect shared service (SSO, networking, shared deployment pipeline)
145	\| - If no → proceed to Region
146	v
147	[2] REGION — Which region(s) are affected?
148	\| - Check: Same alarm in other regions?
149	\| - If multi-region → suspect global service (IAM, Route53, S3 global)
150	\| - If single-region → proceed to Service
151	v
152	[3] SERVICE — Which service namespace shows degradation?
153	\| - Check CloudWatch namespace: AWS/Lambda, AWS/ECS, AWS/ApiGateway, etc.
154	\| - If multiple services → suspect shared dependency (VPC,

People who liked this also viewed

6 picks

Type