loki-logging

Name: loki-logging
Author: BagelHole

42stars4forks

How to install loki-logging?

bagelhole/devops-security-agent-skills/loki-logging

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill loki-logging

Installs into the current project.

Summary

Configure Grafana Loki for log aggregation and analysis. Set up Promtail for log collection, write LogQL queries, and integrate with Grafana for visualization. Use when implementing lightweight log aggregation, especially in Kubernetes environments.

Files · 1

View on GitHub

SKILL.md

1	# Grafana Loki
2
3	Aggregate and query logs with Grafana Loki, the Prometheus-inspired logging system.
4
5	## When to Use This Skill
6
7	Use this skill when:
8	- Implementing cost-effective log aggregation
9	- Building logging for Kubernetes environments
10	- Integrating logs with Grafana dashboards
11	- Querying logs with label-based filtering
12	- Preferring lighter-weight alternative to ELK
13
14	## Prerequisites
15
16	- Docker or Kubernetes
17	- Grafana for visualization
18	- Promtail or other log shipper
19
20	## Architecture Overview
21
22	```
23	┌─────────────┐ ┌──────────┐ ┌──────────┐
24	│ Application │────▶│ Promtail │────▶│ Loki │
25	└─────────────┘ └──────────┘ └──────────┘
26	│
27	▼
28	┌──────────┐
29	│ Grafana │
30	└──────────┘
31	```
32
33	## Docker Deployment
34
35	```yaml
36	# docker-compose.yml
37	version: '3.8'
38
39	services:
40	loki:
41	image: grafana/loki:2.9.0
42	ports:
43	- "3100:3100"
44	volumes:
45	- ./loki-config.yaml:/etc/loki/local-config.yaml
46	- loki-data:/loki
47	command: -config.file=/etc/loki/local-config.yaml
48
49	promtail:
50	image: grafana/promtail:2.9.0
51	volumes:
52	- ./promtail-config.yaml:/etc/promtail/config.yaml
53	- /var/log:/var/log:ro
54	- /var/lib/docker/containers:/var/lib/docker/containers:ro
55	command: -config.file=/etc/promtail/config.yaml
56
57	grafana:
58	image: grafana/grafana:10.2.0
59	ports:
60	- "3000:3000"
61	volumes:
62	- grafana-data:/var/lib/grafana
63	- ./grafana/provisioning:/etc/grafana/provisioning
64	environment:
65	- GF_AUTH_ANONYMOUS_ENABLED=true
66	- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
67
68	volumes:
69	loki-data:
70	grafana-data:
71	```
72
73	## Loki Configuration
74
75	```yaml
76	# loki-config.yaml
77	auth_enabled: false
78
79	server:
80	http_listen_port: 3100
81
82	common:
83	path_prefix: /loki
84	storage:
85	filesystem:
86	chunks_directory: /loki/chunks
87	rules_directory: /loki/rules
88	replication_factor: 1
89	ring:
90	kvstore:
91	store: inmemory
92
93	schema_config:
94	configs:
95	- from: 2020-10-24
96	store: boltdb-shipper
97	object_store: filesystem
98	schema: v11
99	index:
100	prefix: index_
101	period: 24h
102
103	storage_config:
104	boltdb_shipper:
105	active_index_directory: /loki/index
106	cache_location: /loki/cache
107	shared_store: filesystem
108
109	limits_config:
110	reject_old_samples: true
111	reject_old_samples_max_age: 168h
112	max_query_series: 5000
113	max_query_parallelism: 2
114
115	chunk_store_config:
116	max_look_back_period: 168h
117
118	table_manager:
119	retention_deletes_enabled: true
120	retention_period: 168h
121	```
122
123	## Promtail Configuration
124
125	```yaml
126	# promtail-config.yaml
127	server:
128	http_listen_port: 9080
129	grpc_listen_port: 0
130
131	positions:
132	filename: /tmp/positions.yaml
133
134	clients:
135	- url: http://loki:3100/loki/api/v1/push
136
137	scrape_configs:
138	# System logs
139	- job_name: system
140	static_configs:
141	- targets:
142	- localhost
143	labels:
144	job: varlogs
145	__path__: /var/log/*.log
146
147	# Docker container logs
148	- job_name: docker
149	docker_sd_configs:
150	- host: unix:///var/run/docker.sock
151	refresh_interval: 5s
152	relabel_configs:
153	- source_labels: ['__meta_docker_container_name']
154	regex: '/(.*)'
155	target_label: 'container'
156	- source_labels: ['__meta_docker_container_log_stream']
157	target_label: 'stream'
158
159	# Application logs with parsing
160	- job_name: application
161	static_configs:
162	- targets:
163	- localhost
164	labels:
165	job: application
166	__path__: /var/log/app/*.log
167	pipeline_stages:
168	- json:
169	expressions:
170	level: level
171	message: message
172	timestamp: timestamp
173	- labels:
174	level:
175	- timestamp:
176	source: timestamp
177	format: RFC3339
178	```
179
180	## Kubernetes Deployment
181
182	```bash
183	# Using Helm
184	helm repo add grafana https://grafana.github.io/helm-charts
185	helm install loki grafana/loki-stack \
186	--namespace monitoring \
187	--create-namespace \
188	--set grafana.enabled=true \
189	--set promtail.enabled=true
190	```
191
192	### Promtail DaemonSet
193
194	```yaml
195	apiVersion: apps/v1
196	kind: DaemonSet
197	metadata:
198	name: promtail
199	namespace: monitoring
200	spec:
201	selector:
202	matchLabels:
203	app: promtail
204	template:
205	metadata:
206	labels:
207	app: promtail
208	spec:
209	containers:
210	- name: promtail
211	image: grafana/promtail:2.9.0
212	args:
213	- -config.file=/etc/promtail/promtail.yaml
214	volumeMounts:
215	- name: config
216	mountPath: /etc/promtail

People who liked this also viewed

6 picks

Type

loki-logging

42stars4forks

View on GitHub

How to install loki-logging?

bagelhole/devops-security-agent-skills/loki-logging

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill loki-logging

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Grafana Loki
2
3	Aggregate and query logs with Grafana Loki, the Prometheus-inspired logging system.
4
5	## When to Use This Skill
6
7	Use this skill when:
8	- Implementing cost-effective log aggregation
9	- Building logging for Kubernetes environments
10	- Integrating logs with Grafana dashboards
11	- Querying logs with label-based filtering
12	- Preferring lighter-weight alternative to ELK
13
14	## Prerequisites
15
16	- Docker or Kubernetes
17	- Grafana for visualization
18	- Promtail or other log shipper
19
20	## Architecture Overview
21
22	```
23	┌─────────────┐ ┌──────────┐ ┌──────────┐
24	│ Application │────▶│ Promtail │────▶│ Loki │
25	└─────────────┘ └──────────┘ └──────────┘
26	│
27	▼
28	┌──────────┐
29	│ Grafana │
30	└──────────┘
31	```
32
33	## Docker Deployment
34
35	```yaml
36	# docker-compose.yml
37	version: '3.8'
38
39	services:
40	loki:
41	image: grafana/loki:2.9.0
42	ports:
43	- "3100:3100"
44	volumes:
45	- ./loki-config.yaml:/etc/loki/local-config.yaml
46	- loki-data:/loki
47	command: -config.file=/etc/loki/local-config.yaml
48
49	promtail:
50	image: grafana/promtail:2.9.0
51	volumes:
52	- ./promtail-config.yaml:/etc/promtail/config.yaml
53	- /var/log:/var/log:ro
54	- /var/lib/docker/containers:/var/lib/docker/containers:ro
55	command: -config.file=/etc/promtail/config.yaml
56
57	grafana:
58	image: grafana/grafana:10.2.0
59	ports:
60	- "3000:3000"
61	volumes:
62	- grafana-data:/var/lib/grafana
63	- ./grafana/provisioning:/etc/grafana/provisioning
64	environment:
65	- GF_AUTH_ANONYMOUS_ENABLED=true
66	- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
67
68	volumes:
69	loki-data:
70	grafana-data:
71	```
72
73	## Loki Configuration
74
75	```yaml
76	# loki-config.yaml
77	auth_enabled: false
78
79	server:
80	http_listen_port: 3100
81
82	common:
83	path_prefix: /loki
84	storage:
85	filesystem:
86	chunks_directory: /loki/chunks
87	rules_directory: /loki/rules
88	replication_factor: 1
89	ring:
90	kvstore:
91	store: inmemory
92
93	schema_config:
94	configs:
95	- from: 2020-10-24
96	store: boltdb-shipper
97	object_store: filesystem
98	schema: v11
99	index:
100	prefix: index_
101	period: 24h
102
103	storage_config:
104	boltdb_shipper:
105	active_index_directory: /loki/index
106	cache_location: /loki/cache
107	shared_store: filesystem
108
109	limits_config:
110	reject_old_samples: true
111	reject_old_samples_max_age: 168h
112	max_query_series: 5000
113	max_query_parallelism: 2
114
115	chunk_store_config:
116	max_look_back_period: 168h
117
118	table_manager:
119	retention_deletes_enabled: true
120	retention_period: 168h
121	```
122
123	## Promtail Configuration
124
125	```yaml
126	# promtail-config.yaml
127	server:
128	http_listen_port: 9080
129	grpc_listen_port: 0
130
131	positions:
132	filename: /tmp/positions.yaml
133
134	clients:
135	- url: http://loki:3100/loki/api/v1/push
136
137	scrape_configs:
138	# System logs
139	- job_name: system
140	static_configs:
141	- targets:
142	- localhost
143	labels:
144	job: varlogs
145	__path__: /var/log/*.log
146
147	# Docker container logs
148	- job_name: docker
149	docker_sd_configs:
150	- host: unix:///var/run/docker.sock
151	refresh_interval: 5s
152	relabel_configs:
153	- source_labels: ['__meta_docker_container_name']
154	regex: '/(.*)'
155	target_label: 'container'
156	- source_labels: ['__meta_docker_container_log_stream']
157	target_label: 'stream'
158
159	# Application logs with parsing
160	- job_name: application
161	static_configs:
162	- targets:
163	- localhost
164	labels:
165	job: application
166	__path__: /var/log/app/*.log
167	pipeline_stages:
168	- json:
169	expressions:
170	level: level
171	message: message
172	timestamp: timestamp
173	- labels:
174	level:
175	- timestamp:
176	source: timestamp
177	format: RFC3339
178	```
179
180	## Kubernetes Deployment
181
182	```bash
183	# Using Helm
184	helm repo add grafana https://grafana.github.io/helm-charts
185	helm install loki grafana/loki-stack \
186	--namespace monitoring \
187	--create-namespace \
188	--set grafana.enabled=true \
189	--set promtail.enabled=true
190	```
191
192	### Promtail DaemonSet
193
194	```yaml
195	apiVersion: apps/v1
196	kind: DaemonSet
197	metadata:
198	name: promtail
199	namespace: monitoring
200	spec:
201	selector:
202	matchLabels:
203	app: promtail
204	template:
205	metadata:
206	labels:
207	app: promtail
208	spec:
209	containers:
210	- name: promtail
211	image: grafana/promtail:2.9.0
212	args:
213	- -config.file=/etc/promtail/promtail.yaml
214	volumeMounts:
215	- name: config
216	mountPath: /etc/promtail

People who liked this also viewed

6 picks

Type