lakehouse-table-format-engineering

Name: lakehouse-table-format-engineering
Author: vaquarkhan

19stars4forks

How to install lakehouse-table-format-engineering?

vaquarkhan/data-engineering-agent-skills/lakehouse-table-format-engineering

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill lakehouse-table-format-engineering

Installs into the current project.

Summary

Guides agents through lakehouse table design and open table format decisions. Use when designing or changing Iceberg, Delta, Hudi, partitioning, schema evolution, compaction, or batch and streaming interoperability.

Files · 1

View on GitHub

SKILL.md

1	# Lakehouse Table Format Engineering
2
3	## Overview
4
5	Use this skill when the storage layer itself is an architecture decision. It helps agents reason about `Iceberg`, `Delta`, `Hudi`, table evolution, partitioning, compaction, snapshot behavior, and how batch and streaming jobs interact with shared lakehouse tables.
6
7	## When to Use
8
9	- choosing between `Iceberg`, `Delta`, and `Hudi`
10	- designing lakehouse datasets with update or merge behavior
11	- changing partition strategy, snapshot retention, or compaction behavior
12	- integrating `Spark`, `Flink`, or warehouse readers with shared table formats
13	- implementing CDC-style or incremental lakehouse pipelines
14
15	Do not use this for unmanaged flat-file datasets that are not intended to behave like governed tables.
16
17	## Workflow
18
19	1. Define the table contract.
20	Include:
21	- grain and keys
22	- mutation model
23	- read patterns
24	- retention and snapshot policy
25	- compatibility requirements across engines
26
27	2. Pick the table format intentionally.
28	- `Iceberg`: strong open-table interoperability and metadata-driven planning
29	- `Delta`: strong lakehouse ergonomics and transactional patterns in Databricks-centered stacks
30	- `Hudi`: strong incremental and record-level update use cases
31
32	3. Design physical layout for lifecycle, not only day-one queries.
33	Consider:
34	- partition evolution
35	- small-file control
36	- compaction
37	- metadata growth
38	- merge and delete behavior
39
40	4. Align compute engines with the table behavior.
41	Batch and streaming writers must not conflict silently on checkpoints, commits, or schema changes.
42
43	5. Define maintenance and recovery operations.
44	Table formats require operational housekeeping, not just one-time creation.
45
46	## Common Rationalizations
47
48	\| Rationalization \| Reality \|
49	\| --- \| --- \|
50	\| "A table format is just a storage detail." \| Format choice affects interoperability, maintenance, mutation semantics, and long-term cost. \|
51	\| "We can decide compaction later." \| Small files and metadata bloat become operational pain quickly. \|
52	\| "Any engine can read and write the table the same way." \| Cross-engine support varies, especially for advanced write and mutation behavior. \|
53
54	## Red Flags
55
56	- format selection is based only on familiarity
57	- mutation and delete semantics are undefined
58	- snapshot retention and cleanup are ignored
59	- batch and stream writers target the same table without coordination
60	- interoperability requirements are discovered after adoption
61
62	## Verification
63
64	- [ ] Table format choice is tied to interoperability and workload needs
65	- [ ] Mutation, partitioning, retention, and compaction rules are explicit
66	- [ ] Engine compatibility assumptions are documented
67	- [ ] Maintenance and recovery procedures exist for the table lifecycle

People who liked this also viewed

6 picks

Type

lakehouse-table-format-engineering

19stars4forks

View on GitHub

How to install lakehouse-table-format-engineering?

vaquarkhan/data-engineering-agent-skills/lakehouse-table-format-engineering

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill lakehouse-table-format-engineering

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Lakehouse Table Format Engineering
2
3	## Overview
4
5	Use this skill when the storage layer itself is an architecture decision. It helps agents reason about `Iceberg`, `Delta`, `Hudi`, table evolution, partitioning, compaction, snapshot behavior, and how batch and streaming jobs interact with shared lakehouse tables.
6
7	## When to Use
8
9	- choosing between `Iceberg`, `Delta`, and `Hudi`
10	- designing lakehouse datasets with update or merge behavior
11	- changing partition strategy, snapshot retention, or compaction behavior
12	- integrating `Spark`, `Flink`, or warehouse readers with shared table formats
13	- implementing CDC-style or incremental lakehouse pipelines
14
15	Do not use this for unmanaged flat-file datasets that are not intended to behave like governed tables.
16
17	## Workflow
18
19	1. Define the table contract.
20	Include:
21	- grain and keys
22	- mutation model
23	- read patterns
24	- retention and snapshot policy
25	- compatibility requirements across engines
26
27	2. Pick the table format intentionally.
28	- `Iceberg`: strong open-table interoperability and metadata-driven planning
29	- `Delta`: strong lakehouse ergonomics and transactional patterns in Databricks-centered stacks
30	- `Hudi`: strong incremental and record-level update use cases
31
32	3. Design physical layout for lifecycle, not only day-one queries.
33	Consider:
34	- partition evolution
35	- small-file control
36	- compaction
37	- metadata growth
38	- merge and delete behavior
39
40	4. Align compute engines with the table behavior.
41	Batch and streaming writers must not conflict silently on checkpoints, commits, or schema changes.
42
43	5. Define maintenance and recovery operations.
44	Table formats require operational housekeeping, not just one-time creation.
45
46	## Common Rationalizations
47
48	\| Rationalization \| Reality \|
49	\| --- \| --- \|
50	\| "A table format is just a storage detail." \| Format choice affects interoperability, maintenance, mutation semantics, and long-term cost. \|
51	\| "We can decide compaction later." \| Small files and metadata bloat become operational pain quickly. \|
52	\| "Any engine can read and write the table the same way." \| Cross-engine support varies, especially for advanced write and mutation behavior. \|
53
54	## Red Flags
55
56	- format selection is based only on familiarity
57	- mutation and delete semantics are undefined
58	- snapshot retention and cleanup are ignored
59	- batch and stream writers target the same table without coordination
60	- interoperability requirements are discovered after adoption
61
62	## Verification
63
64	- [ ] Table format choice is tied to interoperability and workload needs
65	- [ ] Mutation, partitioning, retention, and compaction rules are explicit
66	- [ ] Engine compatibility assumptions are documented
67	- [ ] Maintenance and recovery procedures exist for the table lifecycle

People who liked this also viewed

6 picks

Type