elk-stack
starsforks
How to install elk-stack?
bagelhole/devops-security-agent-skills/elk-stack
$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill elk-stackDeploy and manage the ELK Stack (Elasticsearch, Logstash, Kibana) for log aggregation and analysis. Configure log pipelines, create visualizations, and implement log-based monitoring. Use when centralizing logs, implementing search functionality, or building log analytics platfor
SKILL.md
| 1 | # ELK Stack |
| 2 | |
| 3 | Centralize and analyze logs with Elasticsearch, Logstash, and Kibana. |
| 4 | |
| 5 | ## When to Use This Skill |
| 6 | |
| 7 | Use this skill when: |
| 8 | - Centralizing logs from multiple sources |
| 9 | - Building log search and analytics platforms |
| 10 | - Creating log-based dashboards and alerts |
| 11 | - Implementing full-text search for logs |
| 12 | - Processing and transforming log data |
| 13 | |
| 14 | ## Prerequisites |
| 15 | |
| 16 | - Docker or server infrastructure |
| 17 | - Sufficient disk space for log storage |
| 18 | - Network access from log sources |
| 19 | |
| 20 | ## Docker Deployment |
| 21 | |
| 22 | ```yaml |
| 23 | # docker-compose.yml |
| 24 | version: '3.8' |
| 25 | |
| 26 | services: |
| 27 | elasticsearch: |
| 28 | image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0 |
| 29 | environment: |
| 30 | - discovery.type=single-node |
| 31 | - xpack.security.enabled=false |
| 32 | - "ES_JAVA_OPTS=-Xms1g -Xmx1g" |
| 33 | ports: |
| 34 | - "9200:9200" |
| 35 | volumes: |
| 36 | - elasticsearch-data:/usr/share/elasticsearch/data |
| 37 | |
| 38 | logstash: |
| 39 | image: docker.elastic.co/logstash/logstash:8.11.0 |
| 40 | volumes: |
| 41 | - ./logstash/pipeline:/usr/share/logstash/pipeline |
| 42 | - ./logstash/config:/usr/share/logstash/config |
| 43 | ports: |
| 44 | - "5044:5044" |
| 45 | - "5000:5000" |
| 46 | depends_on: |
| 47 | - elasticsearch |
| 48 | |
| 49 | kibana: |
| 50 | image: docker.elastic.co/kibana/kibana:8.11.0 |
| 51 | ports: |
| 52 | - "5601:5601" |
| 53 | environment: |
| 54 | - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 |
| 55 | depends_on: |
| 56 | - elasticsearch |
| 57 | |
| 58 | filebeat: |
| 59 | image: docker.elastic.co/beats/filebeat:8.11.0 |
| 60 | user: root |
| 61 | volumes: |
| 62 | - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro |
| 63 | - /var/lib/docker/containers:/var/lib/docker/containers:ro |
| 64 | - /var/run/docker.sock:/var/run/docker.sock:ro |
| 65 | depends_on: |
| 66 | - logstash |
| 67 | |
| 68 | volumes: |
| 69 | elasticsearch-data: |
| 70 | ``` |
| 71 | |
| 72 | ## Elasticsearch Configuration |
| 73 | |
| 74 | ### Index Templates |
| 75 | |
| 76 | ```json |
| 77 | PUT _index_template/logs-template |
| 78 | { |
| 79 | "index_patterns": ["logs-*"], |
| 80 | "template": { |
| 81 | "settings": { |
| 82 | "number_of_shards": 1, |
| 83 | "number_of_replicas": 1, |
| 84 | "index.lifecycle.name": "logs-policy" |
| 85 | }, |
| 86 | "mappings": { |
| 87 | "properties": { |
| 88 | "@timestamp": { "type": "date" }, |
| 89 | "message": { "type": "text" }, |
| 90 | "level": { "type": "keyword" }, |
| 91 | "service": { "type": "keyword" }, |
| 92 | "host": { "type": "keyword" }, |
| 93 | "trace_id": { "type": "keyword" } |
| 94 | } |
| 95 | } |
| 96 | } |
| 97 | } |
| 98 | ``` |
| 99 | |
| 100 | ### Index Lifecycle Management |
| 101 | |
| 102 | ```json |
| 103 | PUT _ilm/policy/logs-policy |
| 104 | { |
| 105 | "policy": { |
| 106 | "phases": { |
| 107 | "hot": { |
| 108 | "min_age": "0ms", |
| 109 | "actions": { |
| 110 | "rollover": { |
| 111 | "max_size": "50GB", |
| 112 | "max_age": "1d" |
| 113 | } |
| 114 | } |
| 115 | }, |
| 116 | "warm": { |
| 117 | "min_age": "7d", |
| 118 | "actions": { |
| 119 | "shrink": { "number_of_shards": 1 }, |
| 120 | "forcemerge": { "max_num_segments": 1 } |
| 121 | } |
| 122 | }, |
| 123 | "cold": { |
| 124 | "min_age": "30d", |
| 125 | "actions": { |
| 126 | "freeze": {} |
| 127 | } |
| 128 | }, |
| 129 | "delete": { |
| 130 | "min_age": "90d", |
| 131 | "actions": { |
| 132 | "delete": {} |
| 133 | } |
| 134 | } |
| 135 | } |
| 136 | } |
| 137 | } |
| 138 | ``` |
| 139 | |
| 140 | ## Logstash Pipeline |
| 141 | |
| 142 | ### Basic Pipeline |
| 143 | |
| 144 | ```ruby |
| 145 | # logstash/pipeline/main.conf |
| 146 | input { |
| 147 | beats { |
| 148 | port => 5044 |
| 149 | } |
| 150 | |
| 151 | tcp { |
| 152 | port => 5000 |
| 153 | codec => json_lines |
| 154 | } |
| 155 | } |
| 156 | |
| 157 | filter { |
| 158 | # Parse JSON logs |
| 159 | if [message] =~ /^\{/ { |
| 160 | json { |
| 161 | source => "message" |
| 162 | } |
| 163 | } |
| 164 | |
| 165 | # Parse timestamp |
| 166 | date { |
| 167 | match => ["timestamp", "ISO8601", "yyyy-MM-dd HH:mm:ss"] |
| 168 | target => "@timestamp" |
| 169 | } |
| 170 | |
| 171 | # Add environment tag |
| 172 | mutate { |
| 173 | add_field => { "environment" => "production" } |
| 174 | } |
| 175 | |
| 176 | # Grok pattern for nginx logs |
| 177 | if [type] == "nginx" { |
| 178 | grok { |
| 179 | match => { |
| 180 | "message" => '%{IPORHOST:client_ip} - %{USER:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:bytes}' |
| 181 | } |
| 182 | } |
| 183 | } |
| 184 | } |
| 185 | |
| 186 | output { |
| 187 | elasticsearch { |
| 188 | hosts => ["elasticsearch:9200"] |
| 189 | index => "logs-%{+YYYY.MM.dd}" |
| 190 | } |
| 191 | } |
| 192 | ``` |
| 193 | |
| 194 | ### Advanced Filtering |
| 195 | |
| 196 | ```ruby |
| 197 | filter { |
| 198 | # Parse application logs |
| 199 | grok { |
| 200 | match => { |
| 201 | "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:service}\] %{GREEDYDATA:log_message}" |
| 202 | } |
| 203 | } |
| 204 | |
| 205 | # Extract trace ID from message |
| 206 | if [log_message] =~ /trace_id=/ { |
| 207 | grok { |
| 208 | match => { "log_message" => "trace_id=%{UUID:trace_id}" } |
| 209 | } |
| 210 | } |
| 211 | |
| 212 | # GeoIP lookup |
| 213 | if [client_ip] { |
| 214 | geoip { |
| 215 | source => "client_ip" |
| 216 | target => "geoip" |
| 217 | } |
| 218 | } |
| 219 | |
| 220 | # Drop debug logs in production |
| 221 | if [level] == "DEBUG" and [environment] == "production" { |
| 222 | drop {} |
| 223 | } |
| 224 | |
| 225 | # Enrich with lookup |
| 226 | translate { |
| 227 | field => "status" |
| 228 | destination => "status_description" |
| 229 | dictionary => { |
| 230 | "200" => "OK" |
| 231 | "404" => "N |
People who liked this also viewed
6 picksType
- microsoft-foundryDeploy, evaluate, fine-tune, and manage Foundry agents end-to-end with azd: hosted agent scaffold/run/deploy, prompt agent create, batch eval, continuous eval, prompt optimizer, Agent Optimizer scaffoSkillsJul 2026
- azure-deployExecute Azure deployments for ALREADY-PREPARED applications that have existing .azure/deployment-plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application SkillsJul 2026
- azure-preparePrepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wSkillsJul 2026
- azure-validate--- name: azure-validate description: "Pre-deployment validation for Azure readiness. Run deep checks on configuration, infrastructure (Bicep or Terraform), RBAC role assignments, managed identity pSkillsJul 2026
- azure-aigatewayConfigure Azure API Management as an AI Gateway for AI models, MCP tools, and agents. WHEN: semantic caching, token limit, content safety, load balancing, AI model governance, MCP rate limiting, jailbSkillsJul 2026
- azure-hosted-copilot-sdkOfficial agent plugin providing skills and MCP server configurations for Azure scenarios.SkillsJul 2026
