data-lake-and-zone-architecture

Name: data-lake-and-zone-architecture
Author: vaquarkhan

19stars4forks

How to install data-lake-and-zone-architecture?

vaquarkhan/data-engineering-agent-skills/data-lake-and-zone-architecture

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill data-lake-and-zone-architecture

Installs into the current project.

Summary

Guides agents through data lake and zone architecture design. Use when defining raw, refined, curated, or publish layers; storage organization; retention; and operational boundaries for a data lake.

Files · 1

View on GitHub

SKILL.md

1	# Data Lake And Zone Architecture
2
3	## Overview
4
5	Use this skill when the storage platform needs structure before pipelines scale into chaos. It helps agents design clear lake zones, dataset boundaries, ownership, lifecycle rules, and publish-safe storage conventions.
6
7	## When to Use
8
9	- designing a new data lake
10	- reorganizing raw, staging, refined, or curated zones
11	- defining object storage layout and lifecycle rules
12	- separating landing, transformation, and publish responsibilities
13	- reducing data swamp behavior in shared lake storage
14
15	Do not use this to justify creating extra layers with no operational purpose.
16
17	## Workflow
18
19	1. Define the lake purpose and consumers.
20	Clarify:
21	- source landing needs
22	- internal producer teams
23	- publish consumers
24	- compliance and retention expectations
25
26	2. Define the zone model intentionally.
27	Typical zones include:
28	- raw or landing
29	- standardized or staging
30	- refined or modeled
31	- publish or serving
32
33	3. Assign responsibilities to each zone.
34	Decide:
35	- who writes to it
36	- who reads from it
37	- what quality guarantees exist
38	- whether mutation is allowed
39
40	4. Design storage conventions.
41	Include:
42	- path or catalog naming
43	- partition strategy
44	- retention lifecycle
45	- file-size expectations
46	- ownership tags and metadata
47
48	5. Keep publish rules separate from lake convenience.
49	Not every dataset in the lake is ready for shared consumption.
50
51	## Cross-Cloud Architecture
52
53	Use `references/cloud-data-engineering-architecture-patterns.md` when the task is not only zone design, but choosing the overall cloud architecture pattern across lake, warehouse, lakehouse, streaming, and hybrid shapes.
54
55	## Common Rationalizations
56
57	\| Rationalization \| Reality \|
58	\| --- \| --- \|
59	\| "We can dump everything into one bucket or container and organize later." \| That is how data lakes turn into data swamps. \|
60	\| "More zones always means better governance." \| Extra layers without distinct purpose add complexity and slow teams down. \|
61	\| "If the file exists in the lake, it is available for analytics." \| Raw landing data rarely has the quality or contract guarantees needed for shared use. \|
62
63	## Red Flags
64
65	- zone meanings overlap or are undocumented
66	- ownership is unclear at the dataset or zone level
67	- publish and landing data are mixed together
68	- retention, cleanup, or lifecycle policy is absent
69
70	## Verification
71
72	- [ ] The zone model has clear purposes and boundaries
73	- [ ] Ownership, read/write expectations, and quality guarantees are explicit
74	- [ ] Storage conventions and lifecycle rules are documented
75	- [ ] Shared publish datasets are separated from raw landing data

People who liked this also viewed

6 picks

Type

data-lake-and-zone-architecture

19stars4forks

View on GitHub

How to install data-lake-and-zone-architecture?

vaquarkhan/data-engineering-agent-skills/data-lake-and-zone-architecture

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill data-lake-and-zone-architecture

Installs into the current project.

Summary

Guides agents through data lake and zone architecture design. Use when defining raw, refined, curated, or publish layers; storage organization; retention; and operational boundaries for a data lake.

Files · 1

View on GitHub

SKILL.md

1	# Data Lake And Zone Architecture
2
3	## Overview
4
5	Use this skill when the storage platform needs structure before pipelines scale into chaos. It helps agents design clear lake zones, dataset boundaries, ownership, lifecycle rules, and publish-safe storage conventions.
6
7	## When to Use
8
9	- designing a new data lake
10	- reorganizing raw, staging, refined, or curated zones
11	- defining object storage layout and lifecycle rules
12	- separating landing, transformation, and publish responsibilities
13	- reducing data swamp behavior in shared lake storage
14
15	Do not use this to justify creating extra layers with no operational purpose.
16
17	## Workflow
18
19	1. Define the lake purpose and consumers.
20	Clarify:
21	- source landing needs
22	- internal producer teams
23	- publish consumers
24	- compliance and retention expectations
25
26	2. Define the zone model intentionally.
27	Typical zones include:
28	- raw or landing
29	- standardized or staging
30	- refined or modeled
31	- publish or serving
32
33	3. Assign responsibilities to each zone.
34	Decide:
35	- who writes to it
36	- who reads from it
37	- what quality guarantees exist
38	- whether mutation is allowed
39
40	4. Design storage conventions.
41	Include:
42	- path or catalog naming
43	- partition strategy
44	- retention lifecycle
45	- file-size expectations
46	- ownership tags and metadata
47
48	5. Keep publish rules separate from lake convenience.
49	Not every dataset in the lake is ready for shared consumption.
50
51	## Cross-Cloud Architecture
52
53	Use `references/cloud-data-engineering-architecture-patterns.md` when the task is not only zone design, but choosing the overall cloud architecture pattern across lake, warehouse, lakehouse, streaming, and hybrid shapes.
54
55	## Common Rationalizations
56
57	\| Rationalization \| Reality \|
58	\| --- \| --- \|
59	\| "We can dump everything into one bucket or container and organize later." \| That is how data lakes turn into data swamps. \|
60	\| "More zones always means better governance." \| Extra layers without distinct purpose add complexity and slow teams down. \|
61	\| "If the file exists in the lake, it is available for analytics." \| Raw landing data rarely has the quality or contract guarantees needed for shared use. \|
62
63	## Red Flags
64
65	- zone meanings overlap or are undocumented
66	- ownership is unclear at the dataset or zone level
67	- publish and landing data are mixed together
68	- retention, cleanup, or lifecycle policy is absent
69
70	## Verification
71
72	- [ ] The zone model has clear purposes and boundaries
73	- [ ] Ownership, read/write expectations, and quality guarantees are explicit
74	- [ ] Storage conventions and lifecycle rules are documented
75	- [ ] Shared publish datasets are separated from raw landing data

People who liked this also viewed

6 picks

Type