distributed-logging

Name: distributed-logging
Author: proyecto26

45installs1forks

How to install distributed-logging?

proyecto26/system-design-skills/distributed-logging

$npx -y skills add proyecto26/system-design-skills --skill distributed-logging

Installs into the current project.

Summary

This skill should be used when the user designs "distributed logging", "log aggregation", "centralized logs", an "ELK" or "EFK" stack, "log shipping", "structured logging", a "correlation ID" or "trace ID" in logs, "log retention", or "high-volume log ingest". It gives the collec

Files · 1

View on GitHub

SKILL.md

1	# Distributed logging
2
3	Move logs from thousands of processes into one searchable place, fast enough to
4	debug a live incident and cheap enough to keep for months. Getting it wrong is a
5	classic "ignore failure" miss: the logging pipeline is itself a distributed system
6	that buckles under the exact traffic spike you most need it during, and a naive
7	design either drops the evidence or takes down the app it instruments.
8
9	## When to reach for this
10	More than one process emits logs and someone needs to search them together; an
11	incident requires correlating a request across services; log volume has outgrown
12	`grep` on a box; or compliance demands retention. The pipeline buys central search,
13	cross-service correlation, and a durable record decoupled from any single host.
14
15	## When NOT to
16	A single service on one host where `journald` + log rotation is enough — a full
17	pipeline is pure operational overhead (YAGNI). Numeric time-series questions ("what
18	is p99 latency", "is error rate up") belong to metrics, not log scans — that is
19	`observability`'s job; logs answer "what exactly happened to this request". Don't
20	ship every debug line at full volume before a number shows the volume justifies the
21	cost; sample first.
22
23	## Clarify first
24	- Volume and peak — lines/sec and bytes/sec, average and peak (→ `back-of-the-envelope`). This sizes every stage.
25	- Structured or free-text — can producers emit JSON now, or is there legacy text to parse?
26	- Query latency need — interactive search in seconds (hot index) vs. occasional forensic/audit reads (cold archive)?
27	- Retention + compliance — how long hot, how long cold, any legal hold or PII redaction requirement?
28	- Loss tolerance — may logs be dropped/sampled under overload, or is every line evidence (audit/financial)?
29
30	## The options
31
32	Collection (agent on the host)
33	- Sidecar/node agent (Fluentd, Fluent Bit, Vector, Filebeat): tails files or
34	reads stdout, adds metadata, ships out. Use when apps log to files/stdout and you
35	want app code untouched — the default.
36	- Direct-to-bus SDK: the app writes structured events straight to a transport.
37	Use when you control the code and want exact structure, accepting tighter coupling.
38
39	Transport / buffer (the shock absorber)
40	- Agent-side buffer + direct ship to indexer: simplest; agent disk-buffers and
41	retries. Use at low-to-moderate volume with one consumer.
42	- Durable log bus (`messaging-streaming`, e.g. Kafka): producers write to a
43	partitioned bus; indexers consume at their own pace. Use at high volume or when
44	multiple sinks (search, archive, analytics) read the same stream.
45
46	Index / store (the search backend)
47	- Full-text index (Elasticsearch/OpenSearch — the "E" in ELK/EFK): rich queries,
48	expensive RAM/disk. Use when interactive field search matters.
49	- Label-indexed store (Loki): indexes only labels, stores log bodies compressed;
50	much cheaper, grep-style queries. Use for high volume where you mostly filter by
51	service/label then scan.
52
53	Retention / tiering
54	- Hot index → cold object store (`blob-store`): keep days of searchable data
55	hot, roll older data to compressed objects. Use whenever retention exceeds the
56	hot window economically (almost always).
57
58	## Trade-offs
59
60	\| Option \| What it solves \| What it worsens \| Change it when \|
61	\|---\|---\|---\|---\|
62	\| Node agent (Fluent Bit/Vector) \| No app changes; central metadata + routing \| One more daemon per host; parsing CPU; agent can lag \| You need exact structure → emit structured events from the app \|
63	\| Direct-to-bus SDK \| Clean structured events, no file parse \| Couples app to transport; app blocks/loses logs if bus is down \| Coupling/availability hurts → go back to agent + local buffer \|
64	\| Agent buffer → direct indexer \| Fewest moving parts \| Indexer backpressure hits producers; no replay; one sink \| Volume spikes or you need >1 sink → add a log bus \|
65	\| Durable log bus (Kafka) \| Absorbs spikes, decouples, replay, fan-out \| Extra system to run; ordering only per-partition; cost \| Volume is low and single-sink → drop the bus \|
66	\| Full-text index (ES/OpenSearch) \| Fast rich field search \| RAM/disk hungry; mapping explosions; costly at scale \| Cost dominates and queries are label-filtered → Loki \|
67	\| Label-indexed (Loki) \| Cheap storage at high volume \| Weak full-text; slow on high-cardinality scans \| You truly need arbitrary field search → full-text index \|
68	\| Hot index + cold archi

People who liked this also viewed

6 picks

Type

distributed-logging

45installs1forks

View on GitHub

How to install distributed-logging?

proyecto26/system-design-skills/distributed-logging

$npx -y skills add proyecto26/system-design-skills --skill distributed-logging

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Distributed logging
2
3	Move logs from thousands of processes into one searchable place, fast enough to
4	debug a live incident and cheap enough to keep for months. Getting it wrong is a
5	classic "ignore failure" miss: the logging pipeline is itself a distributed system
6	that buckles under the exact traffic spike you most need it during, and a naive
7	design either drops the evidence or takes down the app it instruments.
8
9	## When to reach for this
10	More than one process emits logs and someone needs to search them together; an
11	incident requires correlating a request across services; log volume has outgrown
12	`grep` on a box; or compliance demands retention. The pipeline buys central search,
13	cross-service correlation, and a durable record decoupled from any single host.
14
15	## When NOT to
16	A single service on one host where `journald` + log rotation is enough — a full
17	pipeline is pure operational overhead (YAGNI). Numeric time-series questions ("what
18	is p99 latency", "is error rate up") belong to metrics, not log scans — that is
19	`observability`'s job; logs answer "what exactly happened to this request". Don't
20	ship every debug line at full volume before a number shows the volume justifies the
21	cost; sample first.
22
23	## Clarify first
24	- Volume and peak — lines/sec and bytes/sec, average and peak (→ `back-of-the-envelope`). This sizes every stage.
25	- Structured or free-text — can producers emit JSON now, or is there legacy text to parse?
26	- Query latency need — interactive search in seconds (hot index) vs. occasional forensic/audit reads (cold archive)?
27	- Retention + compliance — how long hot, how long cold, any legal hold or PII redaction requirement?
28	- Loss tolerance — may logs be dropped/sampled under overload, or is every line evidence (audit/financial)?
29
30	## The options
31
32	Collection (agent on the host)
33	- Sidecar/node agent (Fluentd, Fluent Bit, Vector, Filebeat): tails files or
34	reads stdout, adds metadata, ships out. Use when apps log to files/stdout and you
35	want app code untouched — the default.
36	- Direct-to-bus SDK: the app writes structured events straight to a transport.
37	Use when you control the code and want exact structure, accepting tighter coupling.
38
39	Transport / buffer (the shock absorber)
40	- Agent-side buffer + direct ship to indexer: simplest; agent disk-buffers and
41	retries. Use at low-to-moderate volume with one consumer.
42	- Durable log bus (`messaging-streaming`, e.g. Kafka): producers write to a
43	partitioned bus; indexers consume at their own pace. Use at high volume or when
44	multiple sinks (search, archive, analytics) read the same stream.
45
46	Index / store (the search backend)
47	- Full-text index (Elasticsearch/OpenSearch — the "E" in ELK/EFK): rich queries,
48	expensive RAM/disk. Use when interactive field search matters.
49	- Label-indexed store (Loki): indexes only labels, stores log bodies compressed;
50	much cheaper, grep-style queries. Use for high volume where you mostly filter by
51	service/label then scan.
52
53	Retention / tiering
54	- Hot index → cold object store (`blob-store`): keep days of searchable data
55	hot, roll older data to compressed objects. Use whenever retention exceeds the
56	hot window economically (almost always).
57
58	## Trade-offs
59
60	\| Option \| What it solves \| What it worsens \| Change it when \|
61	\|---\|---\|---\|---\|
62	\| Node agent (Fluent Bit/Vector) \| No app changes; central metadata + routing \| One more daemon per host; parsing CPU; agent can lag \| You need exact structure → emit structured events from the app \|
63	\| Direct-to-bus SDK \| Clean structured events, no file parse \| Couples app to transport; app blocks/loses logs if bus is down \| Coupling/availability hurts → go back to agent + local buffer \|
64	\| Agent buffer → direct indexer \| Fewest moving parts \| Indexer backpressure hits producers; no replay; one sink \| Volume spikes or you need >1 sink → add a log bus \|
65	\| Durable log bus (Kafka) \| Absorbs spikes, decouples, replay, fan-out \| Extra system to run; ordering only per-partition; cost \| Volume is low and single-sink → drop the bus \|
66	\| Full-text index (ES/OpenSearch) \| Fast rich field search \| RAM/disk hungry; mapping explosions; costly at scale \| Cost dominates and queries are label-filtered → Loki \|
67	\| Label-indexed (Loki) \| Cheap storage at high volume \| Weak full-text; slow on high-cardinality scans \| You truly need arbitrary field search → full-text index \|
68	\| Hot index + cold archi

People who liked this also viewed

6 picks

Type