messaging-streaming

Name: messaging-streaming
Author: proyecto26

46installs1forks

How to install messaging-streaming?

proyecto26/system-design-skills/messaging-streaming

$npx -y skills add proyecto26/system-design-skills --skill messaging-streaming

Installs into the current project.

Summary

This skill should be used when the user designs a "message queue", reaches for "Kafka", "RabbitMQ", "SQS", "Kinesis", "pub/sub", or "event-driven" architecture, asks about "async processing", "background jobs", "stream processing", or wrestles with "exactly-once vs at-least-once"

Files · 1

View on GitHub

SKILL.md

1	# Messaging & Streaming
2
3	Move work off the synchronous request path and decouple producers from
4	consumers, so a slow, spiky, or failure-prone operation doesn't block the
5	caller. Getting this wrong is subtle: a queue silently changes the delivery and
6	ordering guarantees, and under load it can absorb a spike gracefully or become
7	the thing that hides a meltdown until the backlog is unrecoverable.
8
9	## When to reach for this
10	A step is too slow to do inline (image transcode, fan-out, third-party call); the
11	write path is spiky and needs a buffer to smooth bursts (→ `back-of-the-envelope`
12	for the spike factor); two services must be decoupled so one can fail or deploy
13	independently; or many consumers need the same event stream. The async hand-off
14	buys responsiveness, isolation, and elasticity (scale producers and consumers
15	separately).
16
17	## When NOT to
18	The caller needs the result now to proceed (a synchronous read, a balance check
19	before confirming) — a queue only adds latency and a place for work to get lost.
20	Strong read-after-write within one request. Trivial in-process work that a
21	function call handles. Don't add a broker before a number or a coupling problem
22	justifies it (YAGNI): it's a new stateful system to operate, monitor, and reason
23	about under failure. "We'll need Kafka eventually" is name-dropping, not a
24	requirement.
25
26	## Clarify first
27	- Sync or async? Does the caller need the result inline, or is fire-and-react
28	acceptable? This decides whether a queue belongs here at all.
29	- Delivery guarantee needed — is a dropped message acceptable (at-most-once),
30	or must every message be processed (at-least-once + idempotent consumers)?
31	- Ordering — must messages be processed in order, globally or per-key (per
32	user, per account)? Global ordering is expensive; per-key usually suffices.
33	- Throughput and retention — messages/sec at peak, and how long must they be
34	replayable? (→ `back-of-the-envelope`.) One-shot work vs. a replayable log.
35	- Consumer count and pattern — one worker pool draining a job, or many
36	independent subscribers each reading every event?
37	- Failure handling — what happens to a message that keeps failing? Where does
38	it go, and who looks at it?
39
40	## The options
41
42	Sync vs. async — settle this before picking a tool. Stay synchronous when the
43	caller needs the result to continue and the call is fast and reliable; a direct
44	request is simpler to build, trace, and reason about. Go async when the work is
45	slow, spiky, fan-out-heavy, or the caller can react to the result later — this
46	trades immediate consistency and an easy stack trace for responsiveness and
47	isolation. Only after choosing async do the options below apply. Building
48	request/reply over a queue to fake a synchronous answer is a smell — a direct
49	call is the better design.
50
51	Queue (work/task queue) — one logical consumer group competes to drain
52	messages; a message is delivered to one worker and removed when acked. Use when
53	there are background jobs or commands to process exactly once-ish, and workers
54	should scale to drain a backlog.
55
56	Pub/sub (fan-out) — each subscriber gets its own copy of every message;
57	producers don't know subscribers. Use when multiple independent consumers react
58	to the same event (notify, index, audit) and loose coupling matters.
59
60	Stream (durable, replayable log) — an append-only, partitioned, retained log;
61	consumers track their own offset and can replay history. Use when the design
62	needs ordering per partition, multiple consumers at different positions, event
63	sourcing, or reprocessing (→ `data-storage` for event sourcing/outbox).
64
65	Durable workflow (orchestration engine) — code that survives process crashes;
66	the engine persists each step and resumes where it left off, with built-in
67	retries, timers, and compensation. Use when a multi-step process with retries,
68	human delays, and rollback (a saga) would otherwise become a fragile hand-rolled
69	mesh of queues, state flags, and cron jobs.
70
71	Delivery semantics cut across all of these: at-most-once (fire and forget,
72	may drop), at-least-once (retries until acked, may duplicate — the practical
73	default), exactly-once (no loss, no dup). True end-to-end exactly-once is
74	impractical: a broker's "EOS" (e.g. Kafka) is intra-cluster only, so across
75	systems you always implement it as **at-least-once + id

People who liked this also viewed

6 picks

Type

messaging-streaming

46installs1forks

View on GitHub

How to install messaging-streaming?

proyecto26/system-design-skills/messaging-streaming

$npx -y skills add proyecto26/system-design-skills --skill messaging-streaming

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Messaging & Streaming
2
3	Move work off the synchronous request path and decouple producers from
4	consumers, so a slow, spiky, or failure-prone operation doesn't block the
5	caller. Getting this wrong is subtle: a queue silently changes the delivery and
6	ordering guarantees, and under load it can absorb a spike gracefully or become
7	the thing that hides a meltdown until the backlog is unrecoverable.
8
9	## When to reach for this
10	A step is too slow to do inline (image transcode, fan-out, third-party call); the
11	write path is spiky and needs a buffer to smooth bursts (→ `back-of-the-envelope`
12	for the spike factor); two services must be decoupled so one can fail or deploy
13	independently; or many consumers need the same event stream. The async hand-off
14	buys responsiveness, isolation, and elasticity (scale producers and consumers
15	separately).
16
17	## When NOT to
18	The caller needs the result now to proceed (a synchronous read, a balance check
19	before confirming) — a queue only adds latency and a place for work to get lost.
20	Strong read-after-write within one request. Trivial in-process work that a
21	function call handles. Don't add a broker before a number or a coupling problem
22	justifies it (YAGNI): it's a new stateful system to operate, monitor, and reason
23	about under failure. "We'll need Kafka eventually" is name-dropping, not a
24	requirement.
25
26	## Clarify first
27	- Sync or async? Does the caller need the result inline, or is fire-and-react
28	acceptable? This decides whether a queue belongs here at all.
29	- Delivery guarantee needed — is a dropped message acceptable (at-most-once),
30	or must every message be processed (at-least-once + idempotent consumers)?
31	- Ordering — must messages be processed in order, globally or per-key (per
32	user, per account)? Global ordering is expensive; per-key usually suffices.
33	- Throughput and retention — messages/sec at peak, and how long must they be
34	replayable? (→ `back-of-the-envelope`.) One-shot work vs. a replayable log.
35	- Consumer count and pattern — one worker pool draining a job, or many
36	independent subscribers each reading every event?
37	- Failure handling — what happens to a message that keeps failing? Where does
38	it go, and who looks at it?
39
40	## The options
41
42	Sync vs. async — settle this before picking a tool. Stay synchronous when the
43	caller needs the result to continue and the call is fast and reliable; a direct
44	request is simpler to build, trace, and reason about. Go async when the work is
45	slow, spiky, fan-out-heavy, or the caller can react to the result later — this
46	trades immediate consistency and an easy stack trace for responsiveness and
47	isolation. Only after choosing async do the options below apply. Building
48	request/reply over a queue to fake a synchronous answer is a smell — a direct
49	call is the better design.
50
51	Queue (work/task queue) — one logical consumer group competes to drain
52	messages; a message is delivered to one worker and removed when acked. Use when
53	there are background jobs or commands to process exactly once-ish, and workers
54	should scale to drain a backlog.
55
56	Pub/sub (fan-out) — each subscriber gets its own copy of every message;
57	producers don't know subscribers. Use when multiple independent consumers react
58	to the same event (notify, index, audit) and loose coupling matters.
59
60	Stream (durable, replayable log) — an append-only, partitioned, retained log;
61	consumers track their own offset and can replay history. Use when the design
62	needs ordering per partition, multiple consumers at different positions, event
63	sourcing, or reprocessing (→ `data-storage` for event sourcing/outbox).
64
65	Durable workflow (orchestration engine) — code that survives process crashes;
66	the engine persists each step and resumes where it left off, with built-in
67	retries, timers, and compensation. Use when a multi-step process with retries,
68	human delays, and rollback (a saga) would otherwise become a fragile hand-rolled
69	mesh of queues, state flags, and cron jobs.
70
71	Delivery semantics cut across all of these: at-most-once (fire and forget,
72	may drop), at-least-once (retries until acked, may duplicate — the practical
73	default), exactly-once (no loss, no dup). True end-to-end exactly-once is
74	impractical: a broker's "EOS" (e.g. Kafka) is intra-cluster only, so across
75	systems you always implement it as **at-least-once + id

People who liked this also viewed

6 picks

Type