$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill prometheus-grafanaSet up metrics collection and visualization with Prometheus and Grafana. Configure scrape targets, create PromQL queries, build dashboards, and implement alerting. Use when implementing monitoring, metrics collection, or visualization for applications and infrastructure.
| 1 | # Prometheus & Grafana |
| 2 | |
| 3 | Collect metrics and visualize system performance with the Prometheus-Grafana stack. |
| 4 | |
| 5 | ## When to Use This Skill |
| 6 | |
| 7 | Use this skill when: |
| 8 | - Setting up metrics collection infrastructure |
| 9 | - Creating monitoring dashboards |
| 10 | - Writing PromQL queries for analysis |
| 11 | - Configuring alerting rules |
| 12 | - Monitoring Kubernetes clusters |
| 13 | |
| 14 | ## Prerequisites |
| 15 | |
| 16 | - Docker or Kubernetes for deployment |
| 17 | - Network access to monitored targets |
| 18 | - Basic understanding of metrics concepts |
| 19 | |
| 20 | ## Prometheus Setup |
| 21 | |
| 22 | ### Docker Deployment |
| 23 | |
| 24 | ```yaml |
| 25 | # docker-compose.yml |
| 26 | version: '3.8' |
| 27 | |
| 28 | services: |
| 29 | prometheus: |
| 30 | image: prom/prometheus:v2.48.0 |
| 31 | ports: |
| 32 | - "9090:9090" |
| 33 | volumes: |
| 34 | - ./prometheus.yml:/etc/prometheus/prometheus.yml |
| 35 | - ./rules:/etc/prometheus/rules |
| 36 | - prometheus-data:/prometheus |
| 37 | command: |
| 38 | - '--config.file=/etc/prometheus/prometheus.yml' |
| 39 | - '--storage.tsdb.path=/prometheus' |
| 40 | - '--storage.tsdb.retention.time=15d' |
| 41 | |
| 42 | grafana: |
| 43 | image: grafana/grafana:10.2.0 |
| 44 | ports: |
| 45 | - "3000:3000" |
| 46 | volumes: |
| 47 | - grafana-data:/var/lib/grafana |
| 48 | environment: |
| 49 | - GF_SECURITY_ADMIN_PASSWORD=admin |
| 50 | |
| 51 | volumes: |
| 52 | prometheus-data: |
| 53 | grafana-data: |
| 54 | ``` |
| 55 | |
| 56 | ### Configuration |
| 57 | |
| 58 | ```yaml |
| 59 | # prometheus.yml |
| 60 | global: |
| 61 | scrape_interval: 15s |
| 62 | evaluation_interval: 15s |
| 63 | |
| 64 | alerting: |
| 65 | alertmanagers: |
| 66 | - static_configs: |
| 67 | - targets: |
| 68 | - alertmanager:9093 |
| 69 | |
| 70 | rule_files: |
| 71 | - /etc/prometheus/rules/*.yml |
| 72 | |
| 73 | scrape_configs: |
| 74 | - job_name: 'prometheus' |
| 75 | static_configs: |
| 76 | - targets: ['localhost:9090'] |
| 77 | |
| 78 | - job_name: 'node' |
| 79 | static_configs: |
| 80 | - targets: |
| 81 | - 'node-exporter:9100' |
| 82 | |
| 83 | - job_name: 'applications' |
| 84 | static_configs: |
| 85 | - targets: |
| 86 | - 'app1:8080' |
| 87 | - 'app2:8080' |
| 88 | metrics_path: /metrics |
| 89 | ``` |
| 90 | |
| 91 | ## Kubernetes Deployment |
| 92 | |
| 93 | ### Using Helm |
| 94 | |
| 95 | ```bash |
| 96 | # Add Prometheus community Helm repo |
| 97 | helm repo add prometheus-community https://prometheus-community.github.io/helm-charts |
| 98 | |
| 99 | # Install kube-prometheus-stack |
| 100 | helm install prometheus prometheus-community/kube-prometheus-stack \ |
| 101 | --namespace monitoring \ |
| 102 | --create-namespace \ |
| 103 | --set grafana.adminPassword=admin |
| 104 | ``` |
| 105 | |
| 106 | ### ServiceMonitor |
| 107 | |
| 108 | ```yaml |
| 109 | apiVersion: monitoring.coreos.com/v1 |
| 110 | kind: ServiceMonitor |
| 111 | metadata: |
| 112 | name: myapp |
| 113 | namespace: monitoring |
| 114 | spec: |
| 115 | selector: |
| 116 | matchLabels: |
| 117 | app: myapp |
| 118 | endpoints: |
| 119 | - port: metrics |
| 120 | interval: 30s |
| 121 | path: /metrics |
| 122 | namespaceSelector: |
| 123 | matchNames: |
| 124 | - default |
| 125 | ``` |
| 126 | |
| 127 | ## PromQL Queries |
| 128 | |
| 129 | ### Basic Queries |
| 130 | |
| 131 | ```promql |
| 132 | # Current CPU usage |
| 133 | node_cpu_seconds_total{mode="idle"} |
| 134 | |
| 135 | # Rate of HTTP requests per second |
| 136 | rate(http_requests_total[5m]) |
| 137 | |
| 138 | # Average response time |
| 139 | avg(http_request_duration_seconds_sum / http_request_duration_seconds_count) |
| 140 | |
| 141 | # Memory usage percentage |
| 142 | (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 |
| 143 | ``` |
| 144 | |
| 145 | ### Aggregations |
| 146 | |
| 147 | ```promql |
| 148 | # Sum requests by status code |
| 149 | sum by (status_code) (rate(http_requests_total[5m])) |
| 150 | |
| 151 | # Average CPU by instance |
| 152 | avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])) |
| 153 | |
| 154 | # Top 5 endpoints by request count |
| 155 | topk(5, sum by (endpoint) (rate(http_requests_total[5m]))) |
| 156 | |
| 157 | # 95th percentile latency |
| 158 | histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) |
| 159 | ``` |
| 160 | |
| 161 | ### Time-Based Queries |
| 162 | |
| 163 | ```promql |
| 164 | # Compare to 1 hour ago |
| 165 | http_requests_total - http_requests_total offset 1h |
| 166 | |
| 167 | # Predict disk space in 4 hours |
| 168 | predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600) |
| 169 | |
| 170 | # Changes in last 5 minutes |
| 171 | changes(up[5m]) |
| 172 | |
| 173 | # Average over 24 hours |
| 174 | avg_over_time(http_requests_total[24h]) |
| 175 | ``` |
| 176 | |
| 177 | ## Alerting Rules |
| 178 | |
| 179 | ```yaml |
| 180 | # rules/alerts.yml |
| 181 | groups: |
| 182 | - name: application |
| 183 | rules: |
| 184 | - alert: HighErrorRate |
| 185 | expr: | |
| 186 | sum(rate(http_requests_total{status=~"5.."}[5m])) |
| 187 | / sum(rate(http_requests_total[5m])) > 0.05 |
| 188 | for: 5m |
| 189 | labels: |
| 190 | severity: critical |
| 191 | annotations: |
| 192 | summary: "High error rate detected" |
| 193 | description: "Error rate is {{ $value | humanizePercentage }}" |
| 194 | |
| 195 | - alert: ServiceDown |
| 196 | expr: up == 0 |
| 197 | for: 1m |
| 198 | labels: |
| 199 | severity: critical |
| 200 | annotations: |
| 201 | summary: "Service {{ $labels.instance }} is down" |
| 202 | |
| 203 | - alert: HighMemoryUsage |
| 204 | expr: | |
| 205 | (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.9 |
| 206 | for: 5m |
| 207 | labels: |
| 208 | severity: warning |
| 209 | annotations: |
| 210 | summary: "High memory usage on {{ $labels.instance }}" |
| 211 | description: "Memory usage is {{ $value | humanizePercentage }}" |
| 212 | |
| 213 | - alert: DiskSpaceLow |
| 214 | expr: | |
| 215 | (node_filesys |