$npx -y skills add wshobson/agents --skill service-mesh-observabilityImplement comprehensive observability for service meshes including distributed tracing, metrics, and visualization. Use when setting up mesh monitoring, debugging latency issues, or implementing SLOs for service communication.
| 1 | # Service Mesh Observability |
| 2 | |
| 3 | Complete guide to observability patterns for Istio, Linkerd, and service mesh deployments. |
| 4 | |
| 5 | ## When to Use This Skill |
| 6 | |
| 7 | - Setting up distributed tracing across services |
| 8 | - Implementing service mesh metrics and dashboards |
| 9 | - Debugging latency and error issues |
| 10 | - Defining SLOs for service communication |
| 11 | - Visualizing service dependencies |
| 12 | - Troubleshooting mesh connectivity |
| 13 | |
| 14 | ## Core Concepts |
| 15 | |
| 16 | ### 1. Three Pillars of Observability |
| 17 | |
| 18 | ``` |
| 19 | ┌─────────────────────────────────────────────────────┐ |
| 20 | │ Observability │ |
| 21 | ├─────────────────┬─────────────────┬─────────────────┤ |
| 22 | │ Metrics │ Traces │ Logs │ |
| 23 | │ │ │ │ |
| 24 | │ • Request rate │ • Span context │ • Access logs │ |
| 25 | │ • Error rate │ • Latency │ • Error details │ |
| 26 | │ • Latency P50 │ • Dependencies │ • Debug info │ |
| 27 | │ • Saturation │ • Bottlenecks │ • Audit trail │ |
| 28 | └─────────────────┴─────────────────┴─────────────────┘ |
| 29 | ``` |
| 30 | |
| 31 | ### 2. Golden Signals for Mesh |
| 32 | |
| 33 | | Signal | Description | Alert Threshold | |
| 34 | | -------------- | ------------------------- | ----------------- | |
| 35 | | **Latency** | Request duration P50, P99 | P99 > 500ms | |
| 36 | | **Traffic** | Requests per second | Anomaly detection | |
| 37 | | **Errors** | 5xx error rate | > 1% | |
| 38 | | **Saturation** | Resource utilization | > 80% | |
| 39 | |
| 40 | ## Templates and detailed worked examples |
| 41 | |
| 42 | Full template library and detailed worked examples live in `references/details.md`. Read that file when you need the concrete templates. |
| 43 | |
| 44 | ## Best Practices |
| 45 | |
| 46 | ### Do's |
| 47 | |
| 48 | - **Sample appropriately** - 100% in dev, 1-10% in prod |
| 49 | - **Use trace context** - Propagate headers consistently |
| 50 | - **Set up alerts** - For golden signals |
| 51 | - **Correlate metrics/traces** - Use exemplars |
| 52 | - **Retain strategically** - Hot/cold storage tiers |
| 53 | |
| 54 | ### Don'ts |
| 55 | |
| 56 | - **Don't over-sample** - Storage costs add up |
| 57 | - **Don't ignore cardinality** - Limit label values |
| 58 | - **Don't skip dashboards** - Visualize dependencies |
| 59 | - **Don't forget costs** - Monitor observability costs |