apache-hudi-lakehouse

Name: apache-hudi-lakehouse
Author: vaquarkhan

19stars4forks

How to install apache-hudi-lakehouse?

vaquarkhan/data-engineering-agent-skills/apache-hudi-lakehouse

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill apache-hudi-lakehouse

Installs into the current project.

Summary

Guides agents through Apache Hudi lakehouse design. Use when managing incremental upserts, record-level mutations, timeline behavior, compaction, and Hudi-based lakehouse tables.

Files · 1

View on GitHub

SKILL.md

1	# Apache Hudi Lakehouse
2
3	## Overview
4
5	Use this skill when `Apache Hudi` is the primary table layer for incremental lakehouse workloads. It helps agents reason about mutation-heavy patterns, table type selection, compaction behavior, timeline safety, and consumer expectations across read-optimized and real-time query paths.
6
7	## When to Use
8
9	- choosing or operating `Apache Hudi` for lakehouse tables
10	- building record-level upsert or delete pipelines
11	- managing compaction, clustering, and incremental consumption
12	- supporting lakehouse tables with heavy mutations (CDC sinks, slowly changing dimensions)
13	- planning multi-engine access (Spark, Presto, Trino, Athena, Hive)
14
15	Do not use this when the workload is append-only with no mutation requirements and simpler formats like Parquet or Iceberg would suffice.
16
17	## Workflow
18
19	1. Define mutation patterns and read access expectations.
20	Include:
21	- primary record key and partition path
22	- expected operations: inserts, upserts, deletes, or bulk replaces
23	- read latency expectations: are readers okay with merge-on-read or do they need read-optimized snapshots?
24	- query engines that must access the table
25	- expected write throughput and record mutation rate
26
27	2. Choose the right table type and indexing strategy.
28	- `Copy-on-Write (COW)`: best for read-heavy workloads, produces columnar snapshots on write
29	- `Merge-on-Read (MOR)`: best for write-heavy workloads, defers merge to read time or compaction
30	- choose record index type: `BLOOM`, `GLOBAL_BLOOM`, `SIMPLE`, `BUCKET`, or `RECORD_INDEX`
31	- index choice affects upsert performance and scaling behavior
32	- document why the table type was chosen — revisiting later is expensive
33
34	3. Plan compaction and clustering explicitly.
35	- for MOR tables: compaction converts log files to columnar — it is not optional
36	- define compaction strategy: synchronous (inline) or asynchronous (scheduled)
37	- set compaction triggers: by number of commits, time, or log file size
38	- clustering reorganizes data layout for query performance — plan separately from compaction
39	- budget compute for compaction and clustering in cost planning
40
41	4. Design incremental consumption and downstream contracts.
42	- Hudi supports incremental queries by commit timeline
43	- define the consumer contract: which commit instant do consumers start from?
44	- plan for consumer resets and bootstrap reads
45	- document how schema changes affect incremental consumers
46	- test that consumers handle compaction and rollback instants correctly
47
48	5. Handle schema evolution and timeline safety.
49	- Hudi supports schema evolution but not all changes are safe across readers
50	- column adds are generally safe; renames and type changes require care
51	- define compatibility expectations per reader engine
52	- rollback instants can confuse consumers — document rollback behavior
53	- archive policy affects timeline visibility for late consumers
54
55	6. Plan operations, monitoring, and recovery.
56	- monitor timeline growth, pending compactions, and inflight commits
57	- alert on compaction backlog and write failures
58	- plan for rollback: Hudi supports instant-level rollback, but consumers must handle gaps
59	- define retention and archival for the Hudi timeline
60	- document backup and restore procedures for critical tables
61
62	## Common Rationalizations
63
64	\| Rationalization \| Reality \|
65	\| --- \| --- \|
66	\| "Hudi handles upserts so we don't need to think about keys." \| Record key and partition path design determines correctness, performance, and scaling. Wrong keys cause silent data loss or duplication. \|
67	\| "MOR is always better because writes are faster." \| MOR defers work to compaction and read time. Without compaction planning, read performance degrades unboundedly. \|
68	\| "Compaction will just happen in the background." \| Compaction requires explicit scheduling, compute budget, and monitoring. Unmanaged compaction leads to reader degradation and timeline bloat. \|
69	\| "All query engines see the same data." \| COW and MOR tables expose different snapshots to different query types. Read-optimized queries on MOR tables see only compacted data. \|
70
71	## Red Flags
72
73	- record key chosen without understanding uniqueness guarantees
74	- MOR table with no compaction schedule or monitoring
75	- incremental consumers have no documented starting instant or reset procedure
76	- schema changes deployed without testing across all reader engines
77	- no monitoring of timeline growth, pending compactions, or inflight commits
78	- clustering is never run despite increasing query scan ranges
79	- rollback behavior is undocumented and consumers assume a linear timeline
80	- index type is default without analysis of key cardinality and write patterns
81
82	## Verification
83
84	- [ ] Record key, parti

People who liked this also viewed

6 picks

Type

apache-hudi-lakehouse

19stars4forks

View on GitHub

How to install apache-hudi-lakehouse?

vaquarkhan/data-engineering-agent-skills/apache-hudi-lakehouse

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill apache-hudi-lakehouse

Installs into the current project.

Summary

Guides agents through Apache Hudi lakehouse design. Use when managing incremental upserts, record-level mutations, timeline behavior, compaction, and Hudi-based lakehouse tables.

Files · 1

View on GitHub

SKILL.md

1	# Apache Hudi Lakehouse
2
3	## Overview
4
5	Use this skill when `Apache Hudi` is the primary table layer for incremental lakehouse workloads. It helps agents reason about mutation-heavy patterns, table type selection, compaction behavior, timeline safety, and consumer expectations across read-optimized and real-time query paths.
6
7	## When to Use
8
9	- choosing or operating `Apache Hudi` for lakehouse tables
10	- building record-level upsert or delete pipelines
11	- managing compaction, clustering, and incremental consumption
12	- supporting lakehouse tables with heavy mutations (CDC sinks, slowly changing dimensions)
13	- planning multi-engine access (Spark, Presto, Trino, Athena, Hive)
14
15	Do not use this when the workload is append-only with no mutation requirements and simpler formats like Parquet or Iceberg would suffice.
16
17	## Workflow
18
19	1. Define mutation patterns and read access expectations.
20	Include:
21	- primary record key and partition path
22	- expected operations: inserts, upserts, deletes, or bulk replaces
23	- read latency expectations: are readers okay with merge-on-read or do they need read-optimized snapshots?
24	- query engines that must access the table
25	- expected write throughput and record mutation rate
26
27	2. Choose the right table type and indexing strategy.
28	- `Copy-on-Write (COW)`: best for read-heavy workloads, produces columnar snapshots on write
29	- `Merge-on-Read (MOR)`: best for write-heavy workloads, defers merge to read time or compaction
30	- choose record index type: `BLOOM`, `GLOBAL_BLOOM`, `SIMPLE`, `BUCKET`, or `RECORD_INDEX`
31	- index choice affects upsert performance and scaling behavior
32	- document why the table type was chosen — revisiting later is expensive
33
34	3. Plan compaction and clustering explicitly.
35	- for MOR tables: compaction converts log files to columnar — it is not optional
36	- define compaction strategy: synchronous (inline) or asynchronous (scheduled)
37	- set compaction triggers: by number of commits, time, or log file size
38	- clustering reorganizes data layout for query performance — plan separately from compaction
39	- budget compute for compaction and clustering in cost planning
40
41	4. Design incremental consumption and downstream contracts.
42	- Hudi supports incremental queries by commit timeline
43	- define the consumer contract: which commit instant do consumers start from?
44	- plan for consumer resets and bootstrap reads
45	- document how schema changes affect incremental consumers
46	- test that consumers handle compaction and rollback instants correctly
47
48	5. Handle schema evolution and timeline safety.
49	- Hudi supports schema evolution but not all changes are safe across readers
50	- column adds are generally safe; renames and type changes require care
51	- define compatibility expectations per reader engine
52	- rollback instants can confuse consumers — document rollback behavior
53	- archive policy affects timeline visibility for late consumers
54
55	6. Plan operations, monitoring, and recovery.
56	- monitor timeline growth, pending compactions, and inflight commits
57	- alert on compaction backlog and write failures
58	- plan for rollback: Hudi supports instant-level rollback, but consumers must handle gaps
59	- define retention and archival for the Hudi timeline
60	- document backup and restore procedures for critical tables
61
62	## Common Rationalizations
63
64	\| Rationalization \| Reality \|
65	\| --- \| --- \|
66	\| "Hudi handles upserts so we don't need to think about keys." \| Record key and partition path design determines correctness, performance, and scaling. Wrong keys cause silent data loss or duplication. \|
67	\| "MOR is always better because writes are faster." \| MOR defers work to compaction and read time. Without compaction planning, read performance degrades unboundedly. \|
68	\| "Compaction will just happen in the background." \| Compaction requires explicit scheduling, compute budget, and monitoring. Unmanaged compaction leads to reader degradation and timeline bloat. \|
69	\| "All query engines see the same data." \| COW and MOR tables expose different snapshots to different query types. Read-optimized queries on MOR tables see only compacted data. \|
70
71	## Red Flags
72
73	- record key chosen without understanding uniqueness guarantees
74	- MOR table with no compaction schedule or monitoring
75	- incremental consumers have no documented starting instant or reset procedure
76	- schema changes deployed without testing across all reader engines
77	- no monitoring of timeline growth, pending compactions, or inflight commits
78	- clustering is never run despite increasing query scan ranges
79	- rollback behavior is undocumented and consumers assume a linear timeline
80	- index type is default without analysis of key cardinality and write patterns
81
82	## Verification
83
84	- [ ] Record key, parti

People who liked this also viewed

6 picks

Type