$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill sre-dashboardsDesign and operationalize SRE dashboards that surface reliability, latency, error, saturation, and capacity signals across services. Use when building observability views for SLOs, incident response, and executive reliability reporting.
| 1 | # SRE Dashboards |
| 2 | |
| 3 | Build dashboards that help teams detect, triage, and prevent reliability incidents. |
| 4 | |
| 5 | ## When to Use This Skill |
| 6 | |
| 7 | Use this skill when: |
| 8 | - Defining service-level dashboards for production systems |
| 9 | - Tracking SLO health and error-budget burn |
| 10 | - Creating incident command-center views |
| 11 | - Standardizing dashboard patterns across teams |
| 12 | |
| 13 | ## Prerequisites |
| 14 | |
| 15 | - Metrics pipeline (Prometheus, OpenTelemetry, or vendor equivalent) |
| 16 | - Logs/traces linked to services and environments |
| 17 | - Agreed service taxonomy (team, service, tier, environment) |
| 18 | |
| 19 | ## Dashboard Architecture |
| 20 | |
| 21 | Structure dashboards in layers: |
| 22 | |
| 23 | 1. **Executive Reliability View**: SLO attainment, incident counts, MTTR trends. |
| 24 | 2. **Service Health View**: RED/USE metrics, dependency health, release markers. |
| 25 | 3. **Deep-Dive View**: Per-endpoint latency, resource saturation, error categories. |
| 26 | |
| 27 | Keep each view answer-oriented: |
| 28 | - *Are customers impacted?* |
| 29 | - *What changed?* |
| 30 | - *Where is the bottleneck?* |
| 31 | |
| 32 | ## Core SRE Panels |
| 33 | |
| 34 | ### Golden Signals |
| 35 | |
| 36 | - **Latency**: p50/p95/p99 request duration by endpoint |
| 37 | - **Traffic**: request throughput and queue depth |
| 38 | - **Errors**: 5xx rate, failed jobs, timeout ratio |
| 39 | - **Saturation**: CPU, memory, disk I/O, thread/connection pool exhaustion |
| 40 | |
| 41 | ### SLO Panels |
| 42 | |
| 43 | - Current SLI value (rolling windows: 5m, 1h, 24h, 30d) |
| 44 | - Error-budget remaining (%) |
| 45 | - Burn-rate panels (fast and slow windows) |
| 46 | - Multi-window burn alert status |
| 47 | |
| 48 | ### Change Correlation |
| 49 | |
| 50 | - Deployment markers and config-change annotations |
| 51 | - Feature flag state overlays |
| 52 | - Upstream/downstream dependency error rates |
| 53 | |
| 54 | ## Example PromQL Snippets |
| 55 | |
| 56 | ```promql |
| 57 | # API error rate (%) |
| 58 | 100 * sum(rate(http_requests_total{status=~"5.."}[5m])) |
| 59 | / sum(rate(http_requests_total[5m])) |
| 60 | ``` |
| 61 | |
| 62 | ```promql |
| 63 | # p95 latency by route |
| 64 | histogram_quantile(0.95, |
| 65 | sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])) |
| 66 | ) |
| 67 | ``` |
| 68 | |
| 69 | ```promql |
| 70 | # Fast burn rate (5m / 1h) |
| 71 | ( |
| 72 | sum(rate(http_requests_total{status=~"5.."}[5m])) |
| 73 | / sum(rate(http_requests_total[5m])) |
| 74 | ) |
| 75 | / |
| 76 | ( |
| 77 | sum(rate(http_requests_total{status=~"5.."}[1h])) |
| 78 | / sum(rate(http_requests_total[1h])) |
| 79 | ) |
| 80 | ``` |
| 81 | |
| 82 | ## Operational Guidelines |
| 83 | |
| 84 | - Use consistent color semantics (green=healthy, yellow=degrading, red=breach) |
| 85 | - Label units explicitly (ms, req/s, %, cores) |
| 86 | - Default time windows to incident-friendly ranges (15m, 1h, 6h, 24h) |
| 87 | - Minimize panel count per dashboard to reduce cognitive load |
| 88 | - Add runbook links directly in panel descriptions |
| 89 | |
| 90 | ## Troubleshooting |
| 91 | |
| 92 | ### Panel appears flat or empty |
| 93 | |
| 94 | - Verify label cardinality and filters (`service`, `env`, `region`) |
| 95 | - Confirm scrape/ingest latency is within expected range |
| 96 | - Check metric rename regressions after instrumentation updates |
| 97 | |
| 98 | ### High cardinality slows dashboards |
| 99 | |
| 100 | - Aggregate by stable dimensions (`service`, `route_group`) instead of raw IDs |
| 101 | - Use recording rules for expensive percentile and ratio queries |
| 102 | - Split deep-dive dashboards from NOC summary dashboards |
| 103 | |
| 104 | ## Related Skills |
| 105 | |
| 106 | - [prometheus-grafana](../prometheus-grafana/) - Dashboard implementation and PromQL |
| 107 | - [opentelemetry](../opentelemetry/) - Standardized telemetry instrumentation |
| 108 | - [alerting-oncall](../alerting-oncall/) - Reliability alert routing and escalation |
| 109 | - [agent-observability](../../ai/agent-observability/) - AI workload reliability telemetry |