kafka-resilience-and-schema-evolution

Name: kafka-resilience-and-schema-evolution
Author: vaquarkhan

19stars4forks

How to install kafka-resilience-and-schema-evolution?

vaquarkhan/data-engineering-agent-skills/kafka-resilience-and-schema-evolution

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill kafka-resilience-and-schema-evolution

Installs into the current project.

Summary

Enforces production Kafka guardrails including non-breaking schema evolution, dead-letter queues for poison messages, and acks=all producer durability. Use when designing or changing Kafka topics, producers, consumers, schema registry policies, or streaming recovery paths.

Files · 1

View on GitHub

SKILL.md

1	# Kafka Resilience And Schema Evolution
2
3	## Overview
4
5	Generic streaming guidance is not enough for production Kafka. Agents routinely introduce breaking schema changes, under-provisioned durability settings, and missing poison-message isolation. This skill mandates enforceable broker, producer, consumer, and registry guardrails before any production change ships.
6
7	## When to Use
8
9	- creating or modifying `Kafka` topics, producers, or consumers
10	- setting or changing schema registry compatibility policies
11	- designing dead-letter queue (DLQ) routing for poison pill messages
12	- hardening producer durability (`acks`, retries, idempotence)
13	- reviewing consumer lag, replay, or failover behavior on Kafka-backed pipelines
14
15	Pair with `streaming-and-messaging-systems` for broader event design. Pair with `avro-protobuf-json-schema-registry` when registry subjects and compatibility CI are in scope.
16
17	## Workflow
18
19	1. Define the production contract before broker changes.
20	Document:
21	- topic key strategy and partition count rationale
22	- retention, compaction, and replay policy
23	- schema format and registry subject naming
24	- consumer groups and downstream sinks
25	- delivery semantics target (at-least-once with idempotent sinks, or stricter)
26
27	2. Enforce producer durability defaults.
28	Require unless explicitly waived with owner approval:
29	- `acks=all` (or `acks=-1`)
30	- `enable.idempotence=true` when ordering and deduplication matter
31	- bounded `retries` with `delivery.timeout.ms` aligned to SLA
32	- `max.in.flight.requests.per.connection=1` when strict ordering is required
33	- TLS/SASL configuration documented for non-development clusters
34
35	3. Block breaking schema evolution.
36	Before any schema change:
37	- set compatibility policy per subject (`BACKWARD`, `FORWARD`, or `FULL` — not `NONE` in production)
38	- run compatibility checks in CI against registered schemas
39	- document producer-then-consumer or consumer-then-producer rollout order
40	- reject field removals, renames, or type changes without migration plan
41	- load `references/kafka-production-guardrails.md` for DLQ and evolution patterns
42
43	4. Mandate dead-letter and poison pill isolation.
44	Every production consumer that parses external payloads must define:
45	- DLQ topic or sink with retention and access controls
46	- classification rules (deserialization failure, schema mismatch, business rule violation)
47	- alert routing when DLQ rate exceeds threshold
48	- replay procedure with deduplication keys
49	- no silent drop of unparseable records
50
51	5. Make lag and recovery observable.
52	Plan for:
53	- consumer group lag alerts with owner routing
54	- offset reset policy documented and restricted
55	- replay runbook that does not bypass DLQ classification
56	- broker disk and retention monitoring for high-throughput topics
57
58	6. Load MCP observability when diagnosing live lag.
59	Use `mcp-data-observability-integration` with `mcp/kafka.mcp.json` to inspect consumer group lag and topic metadata before changing consumer code or partition counts.
60
61	## Common Rationalizations
62
63	\| Rationalization \| Reality \|
64	\| --- \| --- \|
65	\| "acks=1 is fine because Kafka is durable." \| Leader acknowledgment without full ISR acknowledgment loses events under failure scenarios. \|
66	\| "We can fix schema breaks by redeploying consumers quickly." \| Breaking changes propagate to many consumers and batch sinks before redeploy completes. \|
67	\| "DLQs add too much operational overhead." \| Poison pills without DLQs stall partitions, inflate lag, and hide data loss as consumer retries. \|
68	\| "Schema compatibility NONE is okay for internal topics." \| Internal topics still feed warehouses, stream processors, and audit systems. \|
69
70	## Red Flags
71
72	- production subjects use `NONE` compatibility
73	- producers use `acks=0` or `acks=1` without documented waiver
74	- consumers have no DLQ path for deserialization failures
75	- schema changes ship without CI compatibility validation
76	- consumer group lag has no alert owner
77	- replay procedures reset offsets without reconciliation or publish pause
78
79	## Verification
80
81	- [ ] Producer durability settings meet `acks=all` and idempotence requirements
82	- [ ] Schema compatibility policy is set and CI-validated for production subjects
83	- [ ] DLQ routing, alerts, and replay procedure are documented
84	- [ ] Consumer lag and retention monitoring exist with named owners
85	- [ ] Rollout order for schema changes is explicit and tested in non-production

People who liked this also viewed

6 picks

Type

kafka-resilience-and-schema-evolution

19stars4forks

View on GitHub

How to install kafka-resilience-and-schema-evolution?

vaquarkhan/data-engineering-agent-skills/kafka-resilience-and-schema-evolution

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill kafka-resilience-and-schema-evolution

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Kafka Resilience And Schema Evolution
2
3	## Overview
4
5	Generic streaming guidance is not enough for production Kafka. Agents routinely introduce breaking schema changes, under-provisioned durability settings, and missing poison-message isolation. This skill mandates enforceable broker, producer, consumer, and registry guardrails before any production change ships.
6
7	## When to Use
8
9	- creating or modifying `Kafka` topics, producers, or consumers
10	- setting or changing schema registry compatibility policies
11	- designing dead-letter queue (DLQ) routing for poison pill messages
12	- hardening producer durability (`acks`, retries, idempotence)
13	- reviewing consumer lag, replay, or failover behavior on Kafka-backed pipelines
14
15	Pair with `streaming-and-messaging-systems` for broader event design. Pair with `avro-protobuf-json-schema-registry` when registry subjects and compatibility CI are in scope.
16
17	## Workflow
18
19	1. Define the production contract before broker changes.
20	Document:
21	- topic key strategy and partition count rationale
22	- retention, compaction, and replay policy
23	- schema format and registry subject naming
24	- consumer groups and downstream sinks
25	- delivery semantics target (at-least-once with idempotent sinks, or stricter)
26
27	2. Enforce producer durability defaults.
28	Require unless explicitly waived with owner approval:
29	- `acks=all` (or `acks=-1`)
30	- `enable.idempotence=true` when ordering and deduplication matter
31	- bounded `retries` with `delivery.timeout.ms` aligned to SLA
32	- `max.in.flight.requests.per.connection=1` when strict ordering is required
33	- TLS/SASL configuration documented for non-development clusters
34
35	3. Block breaking schema evolution.
36	Before any schema change:
37	- set compatibility policy per subject (`BACKWARD`, `FORWARD`, or `FULL` — not `NONE` in production)
38	- run compatibility checks in CI against registered schemas
39	- document producer-then-consumer or consumer-then-producer rollout order
40	- reject field removals, renames, or type changes without migration plan
41	- load `references/kafka-production-guardrails.md` for DLQ and evolution patterns
42
43	4. Mandate dead-letter and poison pill isolation.
44	Every production consumer that parses external payloads must define:
45	- DLQ topic or sink with retention and access controls
46	- classification rules (deserialization failure, schema mismatch, business rule violation)
47	- alert routing when DLQ rate exceeds threshold
48	- replay procedure with deduplication keys
49	- no silent drop of unparseable records
50
51	5. Make lag and recovery observable.
52	Plan for:
53	- consumer group lag alerts with owner routing
54	- offset reset policy documented and restricted
55	- replay runbook that does not bypass DLQ classification
56	- broker disk and retention monitoring for high-throughput topics
57
58	6. Load MCP observability when diagnosing live lag.
59	Use `mcp-data-observability-integration` with `mcp/kafka.mcp.json` to inspect consumer group lag and topic metadata before changing consumer code or partition counts.
60
61	## Common Rationalizations
62
63	\| Rationalization \| Reality \|
64	\| --- \| --- \|
65	\| "acks=1 is fine because Kafka is durable." \| Leader acknowledgment without full ISR acknowledgment loses events under failure scenarios. \|
66	\| "We can fix schema breaks by redeploying consumers quickly." \| Breaking changes propagate to many consumers and batch sinks before redeploy completes. \|
67	\| "DLQs add too much operational overhead." \| Poison pills without DLQs stall partitions, inflate lag, and hide data loss as consumer retries. \|
68	\| "Schema compatibility NONE is okay for internal topics." \| Internal topics still feed warehouses, stream processors, and audit systems. \|
69
70	## Red Flags
71
72	- production subjects use `NONE` compatibility
73	- producers use `acks=0` or `acks=1` without documented waiver
74	- consumers have no DLQ path for deserialization failures
75	- schema changes ship without CI compatibility validation
76	- consumer group lag has no alert owner
77	- replay procedures reset offsets without reconciliation or publish pause
78
79	## Verification
80
81	- [ ] Producer durability settings meet `acks=all` and idempotence requirements
82	- [ ] Schema compatibility policy is set and CI-validated for production subjects
83	- [ ] DLQ routing, alerts, and replay procedure are documented
84	- [ ] Consumer lag and retention monitoring exist with named owners
85	- [ ] Rollout order for schema changes is explicit and tested in non-production

People who liked this also viewed

6 picks

Type