$npx -y skills add Jeffallan/claude-skills --skill monitoring-expertConfigures monitoring systems, implements structured logging pipelines, creates Prometheus/Grafana dashboards, defines alerting rules, and instruments distributed tracing. Implements Prometheus/Grafana stacks, conducts load testing, performs application profiling, and plans infra
| 1 | # Monitoring Expert |
| 2 | |
| 3 | Observability and performance specialist implementing comprehensive monitoring, alerting, tracing, and performance testing systems. |
| 4 | |
| 5 | ## Core Workflow |
| 6 | |
| 7 | 1. **Assess** — Identify what needs monitoring (SLIs, critical paths, business metrics) |
| 8 | 2. **Instrument** — Add logging, metrics, and traces to the application (see examples below) |
| 9 | 3. **Collect** — Configure aggregation and storage (Prometheus scrape, log shipper, OTLP endpoint); verify data arrives before proceeding |
| 10 | 4. **Visualize** — Build dashboards using RED (Rate/Errors/Duration) or USE (Utilization/Saturation/Errors) methods |
| 11 | 5. **Alert** — Define threshold and anomaly alerts on critical paths; validate no false-positive flood before shipping |
| 12 | |
| 13 | ## Quick-Start Examples |
| 14 | |
| 15 | ### Structured Logging (Node.js / Pino) |
| 16 | ```js |
| 17 | import pino from 'pino'; |
| 18 | |
| 19 | const logger = pino({ level: 'info' }); |
| 20 | |
| 21 | // Good — structured fields, includes correlation ID |
| 22 | logger.info({ requestId: req.id, userId: req.user.id, durationMs: elapsed }, 'order.created'); |
| 23 | |
| 24 | // Bad — string interpolation, no correlation |
| 25 | console.log(`Order created for user ${userId}`); |
| 26 | ``` |
| 27 | |
| 28 | ### Prometheus Metrics (Node.js) |
| 29 | ```js |
| 30 | import { Counter, Histogram, register } from 'prom-client'; |
| 31 | |
| 32 | const httpRequests = new Counter({ |
| 33 | name: 'http_requests_total', |
| 34 | help: 'Total HTTP requests', |
| 35 | labelNames: ['method', 'route', 'status'], |
| 36 | }); |
| 37 | |
| 38 | const httpDuration = new Histogram({ |
| 39 | name: 'http_request_duration_seconds', |
| 40 | help: 'HTTP request latency', |
| 41 | labelNames: ['method', 'route'], |
| 42 | buckets: [0.05, 0.1, 0.3, 0.5, 1, 2, 5], |
| 43 | }); |
| 44 | |
| 45 | // Instrument a route |
| 46 | app.use((req, res, next) => { |
| 47 | const end = httpDuration.startTimer({ method: req.method, route: req.path }); |
| 48 | res.on('finish', () => { |
| 49 | httpRequests.inc({ method: req.method, route: req.path, status: res.statusCode }); |
| 50 | end(); |
| 51 | }); |
| 52 | next(); |
| 53 | }); |
| 54 | |
| 55 | // Expose scrape endpoint |
| 56 | app.get('/metrics', async (req, res) => { |
| 57 | res.set('Content-Type', register.contentType); |
| 58 | res.end(await register.metrics()); |
| 59 | }); |
| 60 | ``` |
| 61 | |
| 62 | ### OpenTelemetry Tracing (Node.js) |
| 63 | ```js |
| 64 | import { NodeSDK } from '@opentelemetry/sdk-node'; |
| 65 | import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'; |
| 66 | import { trace } from '@opentelemetry/api'; |
| 67 | |
| 68 | const sdk = new NodeSDK({ |
| 69 | traceExporter: new OTLPTraceExporter({ url: 'http://jaeger:4318/v1/traces' }), |
| 70 | }); |
| 71 | sdk.start(); |
| 72 | |
| 73 | // Manual span around a critical operation |
| 74 | const tracer = trace.getTracer('order-service'); |
| 75 | async function processOrder(orderId) { |
| 76 | const span = tracer.startSpan('order.process'); |
| 77 | span.setAttribute('order.id', orderId); |
| 78 | try { |
| 79 | const result = await db.saveOrder(orderId); |
| 80 | span.setStatus({ code: SpanStatusCode.OK }); |
| 81 | return result; |
| 82 | } catch (err) { |
| 83 | span.recordException(err); |
| 84 | span.setStatus({ code: SpanStatusCode.ERROR }); |
| 85 | throw err; |
| 86 | } finally { |
| 87 | span.end(); |
| 88 | } |
| 89 | } |
| 90 | ``` |
| 91 | |
| 92 | ### Prometheus Alerting Rule |
| 93 | ```yaml |
| 94 | groups: |
| 95 | - name: api.rules |
| 96 | rules: |
| 97 | - alert: HighErrorRate |
| 98 | expr: | |
| 99 | rate(http_requests_total{status=~"5.."}[5m]) |
| 100 | / rate(http_requests_total[5m]) > 0.05 |
| 101 | for: 2m |
| 102 | labels: |
| 103 | severity: critical |
| 104 | annotations: |
| 105 | summary: "Error rate above 5% on {{ $labels.route }}" |
| 106 | ``` |
| 107 | |
| 108 | ### k6 Load Test |
| 109 | ```js |
| 110 | import http from 'k6/http'; |
| 111 | import { check, sleep } from 'k6'; |
| 112 | |
| 113 | export const options = { |
| 114 | stages: [ |
| 115 | { duration: '1m', target: 50 }, // ramp up |
| 116 | { duration: '5m', target: 50 }, // sustained load |
| 117 | { duration: '1m', target: 0 }, // ramp down |
| 118 | ], |
| 119 | thresholds: { |
| 120 | http_req_duration: ['p(95)<500'], // 95th percentile < 500 ms |
| 121 | http_req_failed: ['rate<0.01'], // error rate < 1% |
| 122 | }, |
| 123 | }; |
| 124 | |
| 125 | export default function () { |
| 126 | const res = http.get('https://api.example.com/orders'); |
| 127 | check(res, { 'status is 200': (r) => r.status === 200 }); |
| 128 | sleep(1); |
| 129 | } |
| 130 | ``` |
| 131 | |
| 132 | ## Reference Guide |
| 133 | |
| 134 | Load detailed guidance based on context: |
| 135 | |
| 136 | | Topic | Reference | Load When | |
| 137 | |-------|-----------|-----------| |
| 138 | | Logging | `references/structured-logging.md |