debezium-and-kafka-connect-cdc

Name: debezium-and-kafka-connect-cdc
Author: vaquarkhan

19stars4forks

How to install debezium-and-kafka-connect-cdc?

vaquarkhan/data-engineering-agent-skills/debezium-and-kafka-connect-cdc

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill debezium-and-kafka-connect-cdc

Installs into the current project.

Summary

Guides agents through Debezium and Kafka Connect CDC workflows. Use when streaming database changes into Kafka topics, managing connectors, snapshots, schema evolution, or downstream CDC consumers.

Files · 1

View on GitHub

SKILL.md

1	# Debezium And Kafka Connect CDC
2
3	## Overview
4
5	Use this skill when database changes must be captured and delivered through `Debezium` and `Kafka Connect`. It helps agents define connector safety, snapshot behavior, schema handling, and downstream CDC contracts.
6
7	## When to Use
8
9	- setting up `Debezium` connectors for `PostgreSQL`, `MySQL`, `MongoDB`, `SQL Server`, or `Oracle`
10	- designing Kafka-based CDC from transactional databases
11	- handling initial snapshots, incremental streaming, and connector recovery
12	- feeding downstream stream processors, lakehouse sinks, or search indexes
13	- managing schema evolution when source tables change
14
15	Do not use this when changes can be captured through application-level events or batch extracts are sufficient for freshness requirements.
16
17	## Workflow
18
19	1. Define source tables, keys, and change event semantics.
20	Include:
21	- which tables to capture and which to exclude
22	- primary key and unique key strategy for each table
23	- expected change operations: inserts, updates, deletes, truncates
24	- whether tombstone records are needed for compacted topics
25	- expected change volume and peak throughput
26
27	2. Plan snapshot behavior explicitly.
28	- initial snapshot: full table scan on first connector start
29	- define snapshot mode: `initial`, `schema_only`, `never`, or `when_needed`
30	- understand lock behavior during snapshot (especially for `PostgreSQL` and `MySQL`)
31	- plan for snapshot duration on large tables — can take hours
32	- document what happens if a snapshot is interrupted
33
34	3. Configure connector for operational resilience.
35	- set appropriate `max.batch.size` and `poll.interval.ms`
36	- configure heartbeat intervals to prevent WAL/binlog retention issues
37	- define slot or binlog retention policies on the source database
38	- plan for connector task failures and automatic restarts
39	- monitor connector lag and offset position
40
41	4. Define topic contracts and downstream expectations.
42	- topic naming convention: `{prefix}.{schema}.{table}`
43	- event envelope format: include before/after, operation type, source metadata
44	- schema registry integration for event contracts
45	- retention and compaction policy per topic
46	- document which consumers depend on each topic
47
48	5. Handle schema evolution safely.
49	- source DDL changes (column adds, renames, type changes) propagate through CDC
50	- define compatibility policy in schema registry (backward, forward, full)
51	- test that downstream consumers handle schema changes without failure
52	- plan for breaking changes: connector restart, re-snapshot, or topic migration
53
54	6. Plan recovery, replay, and operational safety.
55	- define what happens when the connector falls too far behind (slot overflow, binlog expiry)
56	- document re-snapshot procedure for recovery
57	- plan for database failover: does the connector reconnect automatically?
58	- define monitoring alerts: lag, errors, rebalances, and dead-letter routing
59
60	## Common Rationalizations
61
62	\| Rationalization \| Reality \|
63	\| --- \| --- \|
64	\| "CDC is just a connector — set it and forget it." \| Connectors require active monitoring, offset management, and schema evolution handling. Neglected connectors silently lose data. \|
65	\| "We don't need to worry about snapshots after the first one." \| Database failovers, slot loss, and connector resets can trigger re-snapshots. The team must plan for snapshot impact on source load. \|
66	\| "Schema changes in the source don't affect CDC." \| Every DDL change propagates through the CDC stream. Without compatibility policies, downstream consumers break silently. \|
67	\| "Kafka handles deduplication for us." \| Debezium provides at-least-once delivery. Consumers must handle duplicates or use idempotent processing patterns. \|
68
69	## Red Flags
70
71	- no monitoring of connector lag or WAL/binlog retention
72	- snapshot mode is undefined or set to `always` without understanding impact
73	- no schema registry integration for CDC topics
74	- downstream consumers assume exactly-once delivery without deduplication logic
75	- heartbeat intervals are not configured, risking WAL bloat on the source
76	- no documented recovery procedure for slot loss or binlog expiry
77	- topic retention is unlimited with no compaction policy
78	- connector runs with a single task on high-volume multi-table sources
79
80	## Verification
81
82	- [ ] Source tables, keys, and change semantics are explicitly documented
83	- [ ] Snapshot behavior and impact on source database are understood and planned
84	- [ ] Topic contracts include naming, envelope format, retention, and schema compatibility
85	- [ ] Schema evolution paths are tested for both additive and breaking changes
86	- [ ] Connector monitoring covers lag, errors, rebalances, and offset position
87	- [ ] Recovery procedu

People who liked this also viewed

6 picks

Type

debezium-and-kafka-connect-cdc

19stars4forks

View on GitHub

How to install debezium-and-kafka-connect-cdc?

vaquarkhan/data-engineering-agent-skills/debezium-and-kafka-connect-cdc

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill debezium-and-kafka-connect-cdc

Installs into the current project.

Summary

Guides agents through Debezium and Kafka Connect CDC workflows. Use when streaming database changes into Kafka topics, managing connectors, snapshots, schema evolution, or downstream CDC consumers.

Files · 1

View on GitHub

SKILL.md

1	# Debezium And Kafka Connect CDC
2
3	## Overview
4
5	Use this skill when database changes must be captured and delivered through `Debezium` and `Kafka Connect`. It helps agents define connector safety, snapshot behavior, schema handling, and downstream CDC contracts.
6
7	## When to Use
8
9	- setting up `Debezium` connectors for `PostgreSQL`, `MySQL`, `MongoDB`, `SQL Server`, or `Oracle`
10	- designing Kafka-based CDC from transactional databases
11	- handling initial snapshots, incremental streaming, and connector recovery
12	- feeding downstream stream processors, lakehouse sinks, or search indexes
13	- managing schema evolution when source tables change
14
15	Do not use this when changes can be captured through application-level events or batch extracts are sufficient for freshness requirements.
16
17	## Workflow
18
19	1. Define source tables, keys, and change event semantics.
20	Include:
21	- which tables to capture and which to exclude
22	- primary key and unique key strategy for each table
23	- expected change operations: inserts, updates, deletes, truncates
24	- whether tombstone records are needed for compacted topics
25	- expected change volume and peak throughput
26
27	2. Plan snapshot behavior explicitly.
28	- initial snapshot: full table scan on first connector start
29	- define snapshot mode: `initial`, `schema_only`, `never`, or `when_needed`
30	- understand lock behavior during snapshot (especially for `PostgreSQL` and `MySQL`)
31	- plan for snapshot duration on large tables — can take hours
32	- document what happens if a snapshot is interrupted
33
34	3. Configure connector for operational resilience.
35	- set appropriate `max.batch.size` and `poll.interval.ms`
36	- configure heartbeat intervals to prevent WAL/binlog retention issues
37	- define slot or binlog retention policies on the source database
38	- plan for connector task failures and automatic restarts
39	- monitor connector lag and offset position
40
41	4. Define topic contracts and downstream expectations.
42	- topic naming convention: `{prefix}.{schema}.{table}`
43	- event envelope format: include before/after, operation type, source metadata
44	- schema registry integration for event contracts
45	- retention and compaction policy per topic
46	- document which consumers depend on each topic
47
48	5. Handle schema evolution safely.
49	- source DDL changes (column adds, renames, type changes) propagate through CDC
50	- define compatibility policy in schema registry (backward, forward, full)
51	- test that downstream consumers handle schema changes without failure
52	- plan for breaking changes: connector restart, re-snapshot, or topic migration
53
54	6. Plan recovery, replay, and operational safety.
55	- define what happens when the connector falls too far behind (slot overflow, binlog expiry)
56	- document re-snapshot procedure for recovery
57	- plan for database failover: does the connector reconnect automatically?
58	- define monitoring alerts: lag, errors, rebalances, and dead-letter routing
59
60	## Common Rationalizations
61
62	\| Rationalization \| Reality \|
63	\| --- \| --- \|
64	\| "CDC is just a connector — set it and forget it." \| Connectors require active monitoring, offset management, and schema evolution handling. Neglected connectors silently lose data. \|
65	\| "We don't need to worry about snapshots after the first one." \| Database failovers, slot loss, and connector resets can trigger re-snapshots. The team must plan for snapshot impact on source load. \|
66	\| "Schema changes in the source don't affect CDC." \| Every DDL change propagates through the CDC stream. Without compatibility policies, downstream consumers break silently. \|
67	\| "Kafka handles deduplication for us." \| Debezium provides at-least-once delivery. Consumers must handle duplicates or use idempotent processing patterns. \|
68
69	## Red Flags
70
71	- no monitoring of connector lag or WAL/binlog retention
72	- snapshot mode is undefined or set to `always` without understanding impact
73	- no schema registry integration for CDC topics
74	- downstream consumers assume exactly-once delivery without deduplication logic
75	- heartbeat intervals are not configured, risking WAL bloat on the source
76	- no documented recovery procedure for slot loss or binlog expiry
77	- topic retention is unlimited with no compaction policy
78	- connector runs with a single task on high-volume multi-table sources
79
80	## Verification
81
82	- [ ] Source tables, keys, and change semantics are explicitly documented
83	- [ ] Snapshot behavior and impact on source database are understood and planned
84	- [ ] Topic contracts include naming, envelope format, retention, and schema compatibility
85	- [ ] Schema evolution paths are tested for both additive and breaking changes
86	- [ ] Connector monitoring covers lag, errors, rebalances, and offset position
87	- [ ] Recovery procedu

People who liked this also viewed

6 picks

Type