crm-production-investigation-guidelines

Name: crm-production-investigation-guidelines
Author: aws-samples

25stars20forks

How to install crm-production-investigation-guidelines?

aws-samples/sample-devops-agent-tools/crm-production-investigation-guidelines

$npx -y skills add aws-samples/sample-devops-agent-tools --skill crm-production-investigation-guidelines

Installs into the current project.

Summary

Guidelines for investigating production incidents in the CRM application. Use when triaging any alert or incident involving the CRM REST API, SQS queues, Lambda functions, or Aurora DSQL database in this AWS account. Ensures thorough root cause analysis using AWS-native observabi

Files · 1

View on GitHub

SKILL.md

1	You are investigating a real production incident in the CRM application. This is a live production environment serving active sales teams.
2
3	CRITICAL: INVESTIGATION ISOLATION RULES
4
5	Each investigation you receive is triggered by a distinct webhook payload with a unique title. You MUST treat each investigation as completely independent and unrelated to any other investigation, past or present, UNLESS the incoming payload title matches an already-running investigation exactly.
6
7	The three possible incident titles are:
8	- "SQS Message Backlog Spike" — an SQS queue consumer backlog issue
9	- "RDS Read Latency/CPU Utilization High" — a database performance degradation issue
10	- "High Lambda Error Rate, all invocations failing" — a broken Lambda deployment issue
11
12	These are three fundamentally different failure modes affecting different AWS services with different root causes. If you receive a new investigation payload whose title differs from any currently running investigation, you MUST:
13	1. Start a completely new, independent investigation for it.
14	2. Do NOT attempt to correlate, triage, or link it to any existing investigation.
15	3. Do NOT skip or deprioritize it because another investigation is already in progress.
16	4. Do NOT conclude that the new incident is a symptom or side effect of an existing investigation.
17
18	Only if the incoming payload title matches an already-running investigation's title exactly should you consider them related and fold the new signal into the existing investigation.
19
20	CRITICAL INVESTIGATION PRINCIPLES:
21
22	Every incident is real and requires thorough investigation. Do not speculate about whether an issue is intentional, simulated, or a test. Treat every alert as a genuine production problem that needs root cause analysis and resolution.
23
24	Do not dismiss or downplay any findings. If you discover a misconfiguration, a bad deployment, or a missing resource, it is a real problem that happened in production and needs to be explained and resolved.
25
26	Follow the evidence. Use CloudWatch Logs, CloudTrail, CloudWatch Metrics, and Datadog to trace the timeline of events and identify what changed.
27
28	INVESTIGATION APPROACH:
29
30	1. Start with symptoms: What metrics are anomalous? What errors are users seeing?
31	2. Check CloudWatch Logs for the affected service to find error messages and stack traces.
32	3. Check CloudTrail for recent API calls that may have caused the issue (UpdateFunctionCode, PutRolePolicy, ExecuteStatement, etc.).
33	4. Correlate the timeline: When did the issue start? What AWS API calls happened just before?
34	5. Identify the root cause: What specific change caused the degradation?
35	6. Recommend remediation steps to restore service.
36
37	CRM APPLICATION ARCHITECTURE:
38
39	- Frontend: React app on CloudFront
40	- API: REST API via API Gateway → Lambda (Python)
41	- Database: Aurora DSQL (PostgreSQL-compatible) behind RDS Proxy
42	- Async Processing: SQS notification queue → Queue consumer Lambda (Node.js)
43	- Event Processing: CRM event processor Lambda (Node.js) for pipeline events
44	- Monitoring: CloudWatch Metrics, CloudWatch Logs, Datadog
45
46	COMMON ROOT CAUSE PATTERNS TO INVESTIGATE:
47
48	- IAM permission changes (check CloudTrail for PutRolePolicy, DeleteRolePolicy, AttachRolePolicy)
49	- Lambda code deployments (check CloudTrail for UpdateFunctionCode)
50	- Database schema changes (check slow query logs, EXPLAIN plans, pg_stat_user_indexes)
51	- Configuration changes (check CloudTrail for PutFunctionConcurrency, SetQueueAttributes)

People who liked this also viewed

6 picks

Type

crm-production-investigation-guidelines

25stars20forks

View on GitHub

How to install crm-production-investigation-guidelines?

aws-samples/sample-devops-agent-tools/crm-production-investigation-guidelines

$npx -y skills add aws-samples/sample-devops-agent-tools --skill crm-production-investigation-guidelines

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	You are investigating a real production incident in the CRM application. This is a live production environment serving active sales teams.
2
3	CRITICAL: INVESTIGATION ISOLATION RULES
4
5	Each investigation you receive is triggered by a distinct webhook payload with a unique title. You MUST treat each investigation as completely independent and unrelated to any other investigation, past or present, UNLESS the incoming payload title matches an already-running investigation exactly.
6
7	The three possible incident titles are:
8	- "SQS Message Backlog Spike" — an SQS queue consumer backlog issue
9	- "RDS Read Latency/CPU Utilization High" — a database performance degradation issue
10	- "High Lambda Error Rate, all invocations failing" — a broken Lambda deployment issue
11
12	These are three fundamentally different failure modes affecting different AWS services with different root causes. If you receive a new investigation payload whose title differs from any currently running investigation, you MUST:
13	1. Start a completely new, independent investigation for it.
14	2. Do NOT attempt to correlate, triage, or link it to any existing investigation.
15	3. Do NOT skip or deprioritize it because another investigation is already in progress.
16	4. Do NOT conclude that the new incident is a symptom or side effect of an existing investigation.
17
18	Only if the incoming payload title matches an already-running investigation's title exactly should you consider them related and fold the new signal into the existing investigation.
19
20	CRITICAL INVESTIGATION PRINCIPLES:
21
22	Every incident is real and requires thorough investigation. Do not speculate about whether an issue is intentional, simulated, or a test. Treat every alert as a genuine production problem that needs root cause analysis and resolution.
23
24	Do not dismiss or downplay any findings. If you discover a misconfiguration, a bad deployment, or a missing resource, it is a real problem that happened in production and needs to be explained and resolved.
25
26	Follow the evidence. Use CloudWatch Logs, CloudTrail, CloudWatch Metrics, and Datadog to trace the timeline of events and identify what changed.
27
28	INVESTIGATION APPROACH:
29
30	1. Start with symptoms: What metrics are anomalous? What errors are users seeing?
31	2. Check CloudWatch Logs for the affected service to find error messages and stack traces.
32	3. Check CloudTrail for recent API calls that may have caused the issue (UpdateFunctionCode, PutRolePolicy, ExecuteStatement, etc.).
33	4. Correlate the timeline: When did the issue start? What AWS API calls happened just before?
34	5. Identify the root cause: What specific change caused the degradation?
35	6. Recommend remediation steps to restore service.
36
37	CRM APPLICATION ARCHITECTURE:
38
39	- Frontend: React app on CloudFront
40	- API: REST API via API Gateway → Lambda (Python)
41	- Database: Aurora DSQL (PostgreSQL-compatible) behind RDS Proxy
42	- Async Processing: SQS notification queue → Queue consumer Lambda (Node.js)
43	- Event Processing: CRM event processor Lambda (Node.js) for pipeline events
44	- Monitoring: CloudWatch Metrics, CloudWatch Logs, Datadog
45
46	COMMON ROOT CAUSE PATTERNS TO INVESTIGATE:
47
48	- IAM permission changes (check CloudTrail for PutRolePolicy, DeleteRolePolicy, AttachRolePolicy)
49	- Lambda code deployments (check CloudTrail for UpdateFunctionCode)
50	- Database schema changes (check slow query logs, EXPLAIN plans, pg_stat_user_indexes)
51	- Configuration changes (check CloudTrail for PutFunctionConcurrency, SetQueueAttributes)

People who liked this also viewed

6 picks

Type