glue-data-catalog-and-lake-formation-governance

Name: glue-data-catalog-and-lake-formation-governance
Author: vaquarkhan

19stars4forks

How to install glue-data-catalog-and-lake-formation-governance?

vaquarkhan/data-engineering-agent-skills/glue-data-catalog-and-lake-formation-governance

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill glue-data-catalog-and-lake-formation-governance

Installs into the current project.

Summary

Guides agents through AWS-native data catalog and lake governance workflows. Use when designing or reviewing Glue Data Catalog, Lake Formation permissions, governed sharing, metadata quality, and access boundaries for S3, Athena, Redshift, EMR, or Glue pipelines.

Files · 1

View on GitHub

SKILL.md

1	# Glue Data Catalog And Lake Formation Governance
2
3	## Overview
4
5	Use this skill when `AWS` governance is anchored in `Glue Data Catalog` and `Lake Formation`, not only in generic metadata tools. It helps agents design catalog structure, access controls, governed sharing, and publish-safe dataset access across lake and warehouse workflows.
6
7	## When to Use
8
9	- designing `Glue Data Catalog` database or table organization
10	- defining `Lake Formation` permissions, tag-based access, or sharing boundaries
11	- reviewing governed access for `S3`, `Athena`, `Glue`, `EMR`, or `Redshift`
12	- improving metadata quality for AWS-native data discovery
13	- aligning platform-native governance with regulated-data and publish controls
14
15	Do not treat `Lake Formation` and the catalog as only platform-admin setup. They are part of delivery design.
16
17	## Workflow
18
19	1. Define the governed asset boundary.
20	Clarify which datasets, tables, zones, and consumers need AWS-native governance.
21
22	2. Design the catalog structure.
23	Decide:
24	- database boundaries
25	- table naming and ownership
26	- metadata quality expectations
27	- partition and location conventions
28
29	3. Define the access model.
30	Include:
31	- principals and roles
32	- tag-based access where appropriate
33	- row or column restrictions when required
34	- cross-account or consumer sharing behavior
35
36	4. Align publish behavior with governance.
37	Require:
38	- certified versus raw asset distinctions
39	- explicit publish approval or validation gates where needed
40	- lineage and ownership visibility for shared assets
41
42	5. Validate operational behavior.
43	Check how permissions, schema evolution, new partitions, and cross-service access behave under real delivery conditions.
44
45	## Common Rationalizations
46
47	\| Rationalization \| Reality \|
48	\| --- \| --- \|
49	\| "IAM alone is enough." \| Dataset governance often needs finer-grained sharing, tagging, and lake access patterns than broad service-level IAM. \|
50	\| "We can clean up catalog metadata later." \| Poor metadata and unclear ownership make governed data hard to discover and trust. \|
51	\| "Lake Formation is only for the platform team." \| Producers still need to design publish boundaries and access assumptions around it. \|
52
53	## Red Flags
54
55	- database and table ownership are unclear
56	- `Lake Formation` permissions are bolted on after publish design
57	- catalog metadata does not distinguish trusted and experimental assets
58	- cross-account sharing or consumer access paths are undocumented
59	- schema and partition changes are not validated against governance behavior
60
61	## Verification
62
63	- [ ] Catalog structure and ownership are intentionally designed
64	- [ ] `Lake Formation` permissions and sharing boundaries are explicit
65	- [ ] Metadata quality supports discovery and governance
66	- [ ] Publish and access behavior are aligned
67	- [ ] Operational behavior is validated for schema and access changes

People who liked this also viewed

6 picks

Type

glue-data-catalog-and-lake-formation-governance

19stars4forks

View on GitHub

How to install glue-data-catalog-and-lake-formation-governance?

vaquarkhan/data-engineering-agent-skills/glue-data-catalog-and-lake-formation-governance

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill glue-data-catalog-and-lake-formation-governance

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Glue Data Catalog And Lake Formation Governance
2
3	## Overview
4
5	Use this skill when `AWS` governance is anchored in `Glue Data Catalog` and `Lake Formation`, not only in generic metadata tools. It helps agents design catalog structure, access controls, governed sharing, and publish-safe dataset access across lake and warehouse workflows.
6
7	## When to Use
8
9	- designing `Glue Data Catalog` database or table organization
10	- defining `Lake Formation` permissions, tag-based access, or sharing boundaries
11	- reviewing governed access for `S3`, `Athena`, `Glue`, `EMR`, or `Redshift`
12	- improving metadata quality for AWS-native data discovery
13	- aligning platform-native governance with regulated-data and publish controls
14
15	Do not treat `Lake Formation` and the catalog as only platform-admin setup. They are part of delivery design.
16
17	## Workflow
18
19	1. Define the governed asset boundary.
20	Clarify which datasets, tables, zones, and consumers need AWS-native governance.
21
22	2. Design the catalog structure.
23	Decide:
24	- database boundaries
25	- table naming and ownership
26	- metadata quality expectations
27	- partition and location conventions
28
29	3. Define the access model.
30	Include:
31	- principals and roles
32	- tag-based access where appropriate
33	- row or column restrictions when required
34	- cross-account or consumer sharing behavior
35
36	4. Align publish behavior with governance.
37	Require:
38	- certified versus raw asset distinctions
39	- explicit publish approval or validation gates where needed
40	- lineage and ownership visibility for shared assets
41
42	5. Validate operational behavior.
43	Check how permissions, schema evolution, new partitions, and cross-service access behave under real delivery conditions.
44
45	## Common Rationalizations
46
47	\| Rationalization \| Reality \|
48	\| --- \| --- \|
49	\| "IAM alone is enough." \| Dataset governance often needs finer-grained sharing, tagging, and lake access patterns than broad service-level IAM. \|
50	\| "We can clean up catalog metadata later." \| Poor metadata and unclear ownership make governed data hard to discover and trust. \|
51	\| "Lake Formation is only for the platform team." \| Producers still need to design publish boundaries and access assumptions around it. \|
52
53	## Red Flags
54
55	- database and table ownership are unclear
56	- `Lake Formation` permissions are bolted on after publish design
57	- catalog metadata does not distinguish trusted and experimental assets
58	- cross-account sharing or consumer access paths are undocumented
59	- schema and partition changes are not validated against governance behavior
60
61	## Verification
62
63	- [ ] Catalog structure and ownership are intentionally designed
64	- [ ] `Lake Formation` permissions and sharing boundaries are explicit
65	- [ ] Metadata quality supports discovery and governance
66	- [ ] Publish and access behavior are aligned
67	- [ ] Operational behavior is validated for schema and access changes

People who liked this also viewed

6 picks

Type