cdc-and-incremental-loading

Name: cdc-and-incremental-loading
Author: vaquarkhan

19stars4forks

How to install cdc-and-incremental-loading?

vaquarkhan/data-engineering-agent-skills/cdc-and-incremental-loading

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill cdc-and-incremental-loading

Installs into the current project.

Summary

Guides agents through change data capture and incremental load design. Use when building or modifying watermark-based loads, upserts, deduplication, merge logic, late data handling, or replayable incremental pipelines.

Files · 1

View on GitHub

SKILL.md

1	# CDC And Incremental Loading
2
3	## Overview
4
5	Use this skill when the system changes data over time and full reloads are not the right answer. It helps agents design safe incremental behavior with clear watermarks, merge rules, deduplication, and replay semantics.
6
7	## When to Use
8
9	- implementing CDC ingestion
10	- designing watermark-based or timestamp-based incremental loads
11	- merging inserts, updates, and deletes
12	- handling late-arriving data
13	- fixing duplicate or missing incremental records
14
15	Do not use this when a bounded full refresh is simpler and operationally safer.
16
17	## Workflow
18
19	1. Define the change contract.
20	Include:
21	- business key
22	- change key or sequence
23	- delete behavior
24	- update semantics
25	- source ordering guarantees
26
27	2. Choose the incremental strategy.
28	Typical options:
29	- append-only
30	- merge/upsert
31	- snapshot replacement
32	- log-based CDC
33
34	3. Make watermark logic explicit.
35	Record:
36	- watermark column
37	- lag tolerance
38	- reprocessing window
39	- retry and replay rules
40
41	4. Guard against duplicates and missed changes.
42	Deduplication and late-data handling should be first-class logic.
43
44	5. Prove reruns are safe.
45	Incremental pipelines that cannot be replayed safely are incomplete.
46
47	## Common Rationalizations
48
49	\| Rationalization \| Reality \|
50	\| --- \| --- \|
51	\| "We can just use updated_at as the watermark." \| Timestamps alone often miss late changes, clock skew, or delete semantics. \|
52	\| "Duplicates are rare enough." \| Rare duplicates are still expensive when they affect published metrics. \|
53	\| "We can rebuild everything if something goes wrong." \| Large backfills are often slow, expensive, or operationally risky. \|
54
55	## Red Flags
56
57	- no documented business key
58	- delete handling is absent
59	- watermark logic depends on assumptions no one wrote down
60	- replay requires manual surgery in production tables
61
62	## Verification
63
64	- [ ] The incremental contract covers inserts, updates, and deletes where relevant
65	- [ ] Watermark, lag, deduplication, and replay rules are explicit
66	- [ ] Late data behavior is defined
67	- [ ] Reruns and backfills are operationally safe

People who liked this also viewed

6 picks

Type

cdc-and-incremental-loading

19stars4forks

View on GitHub

How to install cdc-and-incremental-loading?

vaquarkhan/data-engineering-agent-skills/cdc-and-incremental-loading

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill cdc-and-incremental-loading

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# CDC And Incremental Loading
2
3	## Overview
4
5	Use this skill when the system changes data over time and full reloads are not the right answer. It helps agents design safe incremental behavior with clear watermarks, merge rules, deduplication, and replay semantics.
6
7	## When to Use
8
9	- implementing CDC ingestion
10	- designing watermark-based or timestamp-based incremental loads
11	- merging inserts, updates, and deletes
12	- handling late-arriving data
13	- fixing duplicate or missing incremental records
14
15	Do not use this when a bounded full refresh is simpler and operationally safer.
16
17	## Workflow
18
19	1. Define the change contract.
20	Include:
21	- business key
22	- change key or sequence
23	- delete behavior
24	- update semantics
25	- source ordering guarantees
26
27	2. Choose the incremental strategy.
28	Typical options:
29	- append-only
30	- merge/upsert
31	- snapshot replacement
32	- log-based CDC
33
34	3. Make watermark logic explicit.
35	Record:
36	- watermark column
37	- lag tolerance
38	- reprocessing window
39	- retry and replay rules
40
41	4. Guard against duplicates and missed changes.
42	Deduplication and late-data handling should be first-class logic.
43
44	5. Prove reruns are safe.
45	Incremental pipelines that cannot be replayed safely are incomplete.
46
47	## Common Rationalizations
48
49	\| Rationalization \| Reality \|
50	\| --- \| --- \|
51	\| "We can just use updated_at as the watermark." \| Timestamps alone often miss late changes, clock skew, or delete semantics. \|
52	\| "Duplicates are rare enough." \| Rare duplicates are still expensive when they affect published metrics. \|
53	\| "We can rebuild everything if something goes wrong." \| Large backfills are often slow, expensive, or operationally risky. \|
54
55	## Red Flags
56
57	- no documented business key
58	- delete handling is absent
59	- watermark logic depends on assumptions no one wrote down
60	- replay requires manual surgery in production tables
61
62	## Verification
63
64	- [ ] The incremental contract covers inserts, updates, and deletes where relevant
65	- [ ] Watermark, lag, deduplication, and replay rules are explicit
66	- [ ] Late data behavior is defined
67	- [ ] Reruns and backfills are operationally safe

People who liked this also viewed

6 picks

Type