etl-elt-and-modernization-strategy

Name: etl-elt-and-modernization-strategy
Author: vaquarkhan

19stars4forks

How to install etl-elt-and-modernization-strategy?

vaquarkhan/data-engineering-agent-skills/etl-elt-and-modernization-strategy

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill etl-elt-and-modernization-strategy

Installs into the current project.

Summary

Guides agents through ETL, ELT, and transformation-modernization decisions. Use when choosing execution boundaries, redesigning transformation layers, or moving from legacy ETL estates to warehouse- or lakehouse-centered ELT patterns.

Files · 1

View on GitHub

SKILL.md

1	# ETL ELT And Modernization Strategy
2
3	## Overview
4
5	Use this skill when the hard part is not a single job, but deciding where transformations should run and how a data estate should modernize over time. It helps agents reason about `ETL` versus `ELT`, pushdown versus external compute, orchestration boundaries, migration sequencing, and proof of parity during modernization.
6
7	## When to Use
8
9	- choosing between `ETL`, `ELT`, or hybrid transformation patterns
10	- moving from legacy ETL tools into warehouse, dbt, Spark, or lakehouse execution
11	- redesigning ingestion and transformation boundaries across raw, curated, and publish layers
12	- reducing operational sprawl caused by duplicate transformation logic
13	- modernizing batch-first estates without breaking existing delivery expectations
14
15	Do not assume `ELT` is always better just because the warehouse is powerful.
16
17	## Workflow
18
19	1. Define the transformation problem clearly.
20	Clarify:
21	- source latency and volume
22	- data quality expectations
23	- transformation complexity
24	- cost sensitivity
25	- publish or consumption latency
26
27	2. Map the current execution estate.
28	Include:
29	- where extraction happens
30	- where transformations happen today
31	- what logic is duplicated across tools
32	- where lineage or observability breaks
33	- what jobs are hardest to change safely
34
35	3. Choose the right execution boundary.
36	Consider:
37	- `ETL` when data must be reshaped or protected before landing
38	- `ELT` when warehouse or lakehouse pushdown improves maintainability and scaling
39	- hybrid patterns when extraction, privacy controls, or heavy preprocessing must happen before durable load
40
41	4. Plan the modernization path.
42	Decide:
43	- what stays temporarily on the old path
44	- what moves first
45	- how parity will be measured
46	- how cutover and rollback will work
47
48	5. Prove the new shape operationally.
49	Require:
50	- reconciliation evidence
51	- cost and performance review
52	- lineage continuity
53	- ownership and support readiness
54
55	## Common Rationalizations
56
57	\| Rationalization \| Reality \|
58	\| --- \| --- \|
59	\| "Everything should become ELT." \| Some workloads still need pre-load shaping, masking, or protocol-specific extraction controls. \|
60	\| "The ETL tool is the problem." \| The real issue may be unclear ownership, poor contracts, or duplicated logic across layers. \|
61	\| "We can rewrite all transformations at once." \| Big-bang modernization usually breaks parity, runbooks, and downstream trust. \|
62
63	## Red Flags
64
65	- the same business logic exists in extraction jobs, Spark, and warehouse SQL
66	- ETL versus ELT is chosen by tool preference instead of workload needs
67	- modernization plans skip parity, cutover, or rollback
68	- sensitive fields are moved into ELT layers without revisiting controls
69
70	## Verification
71
72	- [ ] The transformation boundary matches the real workload constraints
73	- [ ] ETL, ELT, and hybrid choices are explicit rather than assumed
74	- [ ] Modernization sequencing, parity proof, and rollback are defined
75	- [ ] Cost, lineage, controls, and support ownership were considered together

People who liked this also viewed

6 picks

Type

etl-elt-and-modernization-strategy

19stars4forks

View on GitHub

How to install etl-elt-and-modernization-strategy?

vaquarkhan/data-engineering-agent-skills/etl-elt-and-modernization-strategy

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill etl-elt-and-modernization-strategy

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# ETL ELT And Modernization Strategy
2
3	## Overview
4
5	Use this skill when the hard part is not a single job, but deciding where transformations should run and how a data estate should modernize over time. It helps agents reason about `ETL` versus `ELT`, pushdown versus external compute, orchestration boundaries, migration sequencing, and proof of parity during modernization.
6
7	## When to Use
8
9	- choosing between `ETL`, `ELT`, or hybrid transformation patterns
10	- moving from legacy ETL tools into warehouse, dbt, Spark, or lakehouse execution
11	- redesigning ingestion and transformation boundaries across raw, curated, and publish layers
12	- reducing operational sprawl caused by duplicate transformation logic
13	- modernizing batch-first estates without breaking existing delivery expectations
14
15	Do not assume `ELT` is always better just because the warehouse is powerful.
16
17	## Workflow
18
19	1. Define the transformation problem clearly.
20	Clarify:
21	- source latency and volume
22	- data quality expectations
23	- transformation complexity
24	- cost sensitivity
25	- publish or consumption latency
26
27	2. Map the current execution estate.
28	Include:
29	- where extraction happens
30	- where transformations happen today
31	- what logic is duplicated across tools
32	- where lineage or observability breaks
33	- what jobs are hardest to change safely
34
35	3. Choose the right execution boundary.
36	Consider:
37	- `ETL` when data must be reshaped or protected before landing
38	- `ELT` when warehouse or lakehouse pushdown improves maintainability and scaling
39	- hybrid patterns when extraction, privacy controls, or heavy preprocessing must happen before durable load
40
41	4. Plan the modernization path.
42	Decide:
43	- what stays temporarily on the old path
44	- what moves first
45	- how parity will be measured
46	- how cutover and rollback will work
47
48	5. Prove the new shape operationally.
49	Require:
50	- reconciliation evidence
51	- cost and performance review
52	- lineage continuity
53	- ownership and support readiness
54
55	## Common Rationalizations
56
57	\| Rationalization \| Reality \|
58	\| --- \| --- \|
59	\| "Everything should become ELT." \| Some workloads still need pre-load shaping, masking, or protocol-specific extraction controls. \|
60	\| "The ETL tool is the problem." \| The real issue may be unclear ownership, poor contracts, or duplicated logic across layers. \|
61	\| "We can rewrite all transformations at once." \| Big-bang modernization usually breaks parity, runbooks, and downstream trust. \|
62
63	## Red Flags
64
65	- the same business logic exists in extraction jobs, Spark, and warehouse SQL
66	- ETL versus ELT is chosen by tool preference instead of workload needs
67	- modernization plans skip parity, cutover, or rollback
68	- sensitive fields are moved into ELT layers without revisiting controls
69
70	## Verification
71
72	- [ ] The transformation boundary matches the real workload constraints
73	- [ ] ETL, ELT, and hybrid choices are explicit rather than assumed
74	- [ ] Modernization sequencing, parity proof, and rollback are defined
75	- [ ] Cost, lineage, controls, and support ownership were considered together

People who liked this also viewed

6 picks

Type