$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill opentelemetryInstrument applications and infrastructure with OpenTelemetry for unified traces, metrics, and logs. Use when implementing distributed tracing, service-level troubleshooting, or vendor-neutral observability.
| 1 | # OpenTelemetry |
| 2 | |
| 3 | Adopt vendor-neutral telemetry with consistent instrumentation across services. |
| 4 | |
| 5 | ## When to Use This Skill |
| 6 | |
| 7 | - Debugging latency across microservices |
| 8 | - Standardizing observability data model and naming |
| 9 | - Sending telemetry to Prometheus, Grafana, Datadog, or OTLP backends |
| 10 | - Building SLO dashboards with trace-to-log correlation |
| 11 | - Instrumenting Python or Node.js applications with tracing and metrics |
| 12 | - Setting up auto-instrumentation for existing services without code changes |
| 13 | |
| 14 | ## Prerequisites |
| 15 | |
| 16 | - Application services running in containers or on VMs |
| 17 | - Backend for traces (Jaeger, Tempo, Datadog, or any OTLP receiver) |
| 18 | - Backend for metrics (Prometheus, Mimir, or OTLP receiver) |
| 19 | - Kubernetes cluster (for collector deployment) or VM with systemd |
| 20 | - Network access from services to collector, and collector to backends |
| 21 | |
| 22 | ## Core Workflow |
| 23 | |
| 24 | 1. Define semantic conventions for services, environments, and versions. |
| 25 | 2. Add SDK or auto-instrumentation in each service. |
| 26 | 3. Run an OpenTelemetry Collector to receive, transform, and export telemetry. |
| 27 | 4. Validate cardinality and sampling to control cost. |
| 28 | 5. Create golden signals dashboards and alerting from collected data. |
| 29 | |
| 30 | ## Collector Production Configuration |
| 31 | |
| 32 | ```yaml |
| 33 | # otel-collector-config.yaml |
| 34 | receivers: |
| 35 | otlp: |
| 36 | protocols: |
| 37 | grpc: |
| 38 | endpoint: 0.0.0.0:4317 |
| 39 | http: |
| 40 | endpoint: 0.0.0.0:4318 |
| 41 | |
| 42 | # Scrape Prometheus endpoints |
| 43 | prometheus: |
| 44 | config: |
| 45 | scrape_configs: |
| 46 | - job_name: "kubernetes-pods" |
| 47 | kubernetes_sd_configs: |
| 48 | - role: pod |
| 49 | relabel_configs: |
| 50 | - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] |
| 51 | action: keep |
| 52 | regex: "true" |
| 53 | - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port] |
| 54 | action: replace |
| 55 | target_label: __address__ |
| 56 | regex: (.+) |
| 57 | replacement: $$1 |
| 58 | |
| 59 | # Host metrics for infrastructure monitoring |
| 60 | hostmetrics: |
| 61 | collection_interval: 30s |
| 62 | scrapers: |
| 63 | cpu: {} |
| 64 | memory: {} |
| 65 | disk: {} |
| 66 | network: {} |
| 67 | load: {} |
| 68 | |
| 69 | processors: |
| 70 | batch: |
| 71 | send_batch_size: 1024 |
| 72 | timeout: 5s |
| 73 | |
| 74 | memory_limiter: |
| 75 | check_interval: 1s |
| 76 | limit_mib: 512 |
| 77 | spike_limit_mib: 128 |
| 78 | |
| 79 | attributes: |
| 80 | actions: |
| 81 | - key: deployment.environment |
| 82 | value: production |
| 83 | action: upsert |
| 84 | |
| 85 | # Drop high-cardinality attributes to control cost |
| 86 | filter/drop-debug: |
| 87 | traces: |
| 88 | span: |
| 89 | - 'attributes["http.request.header.x-debug"] == "true"' |
| 90 | |
| 91 | # Reduce cardinality on URL paths |
| 92 | transform/normalize-routes: |
| 93 | trace_statements: |
| 94 | - context: span |
| 95 | statements: |
| 96 | - replace_pattern(attributes["url.path"], "/users/[0-9]+", "/users/{id}") |
| 97 | - replace_pattern(attributes["url.path"], "/orders/[0-9]+", "/orders/{id}") |
| 98 | |
| 99 | # Resource detection for cloud environments |
| 100 | resourcedetection: |
| 101 | detectors: [env, system, gcp, aws, azure] |
| 102 | timeout: 5s |
| 103 | |
| 104 | exporters: |
| 105 | # Send traces to Tempo/Jaeger |
| 106 | otlp/traces: |
| 107 | endpoint: tempo:4317 |
| 108 | tls: |
| 109 | insecure: true |
| 110 | |
| 111 | # Send metrics to Prometheus via remote write |
| 112 | prometheusremotewrite: |
| 113 | endpoint: http://mimir:9009/api/v1/push |
| 114 | tls: |
| 115 | insecure: true |
| 116 | |
| 117 | # Send logs to Loki |
| 118 | otlp/logs: |
| 119 | endpoint: loki:4317 |
| 120 | tls: |
| 121 | insecure: true |
| 122 | |
| 123 | # Debug exporter for development |
| 124 | debug: |
| 125 | verbosity: basic |
| 126 | |
| 127 | service: |
| 128 | telemetry: |
| 129 | logs: |
| 130 | level: info |
| 131 | metrics: |
| 132 | address: 0.0.0.0:8888 |
| 133 | |
| 134 | pipelines: |
| 135 | traces: |
| 136 | receivers: [otlp] |
| 137 | processors: [memory_limiter, resourcedetection, transform/normalize-routes, batch, attributes] |
| 138 | exporters: [otlp/traces] |
| 139 | metrics: |
| 140 | receivers: [otlp, prometheus, hostmetrics] |
| 141 | processors: [memory_limiter, resourcedetection, batch, attributes] |
| 142 | exporters: [prometheusremotewrite] |
| 143 | logs: |
| 144 | receivers: [otlp] |
| 145 | processors: [memory_limiter, resourcedetection, batch, attributes] |
| 146 | exporters: [otlp/logs] |
| 147 | ``` |
| 148 | |
| 149 | ## Collector Kubernetes Deployment |
| 150 | |
| 151 | ```yaml |
| 152 | # otel-collector-deployment.yaml |
| 153 | apiVersion: apps/v1 |
| 154 | kind: Deployment |
| 155 | metadata: |
| 156 | name: otel-collector |
| 157 | namespace: observability |
| 158 | spec: |
| 159 | replicas: 2 |
| 160 | selector: |
| 161 | matchLabels: |
| 162 | app: otel-collector |
| 163 | template: |
| 164 | metadata: |
| 165 | labels: |
| 166 | app: otel-collector |
| 167 | spec: |
| 168 | containers: |
| 169 | - name: collector |
| 170 | image: otel/opentelemetry-collector-contrib:0.98.0 |
| 171 | args: ["--config=/etc/otel/config.yaml"] |
| 172 | ports: |
| 173 | - containerPort: 4317 |
| 174 | name: otlp-grpc |
| 175 | - containerPort: 4318 |
| 176 | name: otlp-http |
| 177 | - containerPort: 8888 |
| 178 | name: metrics |
| 179 | resou |