openmetadata-datahub-and-openlineage

Name: openmetadata-datahub-and-openlineage
Author: vaquarkhan

19stars4forks

How to install openmetadata-datahub-and-openlineage?

vaquarkhan/data-engineering-agent-skills/openmetadata-datahub-and-openlineage

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill openmetadata-datahub-and-openlineage

Installs into the current project.

Summary

Guides agents through metadata platform and lineage workflows using OpenMetadata, DataHub, or OpenLineage-compatible systems. Use when improving discovery, lineage quality, metadata governance, or producer-to-catalog integration.

Files · 1

View on GitHub

SKILL.md

1	# OpenMetadata DataHub And OpenLineage
2
3	## Overview
4
5	Use this skill when metadata and lineage must be operationalized through open tooling such as `OpenMetadata`, `DataHub`, or `OpenLineage`. It helps agents align producers with catalog expectations, ensure lineage accuracy, and make discovery trustworthy rather than decorative.
6
7	## When to Use
8
9	- integrating metadata platforms into delivery workflows
10	- improving dataset discovery and trust signals
11	- capturing and validating lineage across pipelines
12	- publishing governed datasets into open catalog ecosystems
13	- connecting `OpenLineage` events from Spark, Airflow, or dbt to a catalog
14	- defining metadata governance policies for multi-team environments
15
16	Do not use this when the team has no shared catalog requirement or when lineage is handled entirely within a closed platform (e.g., Unity Catalog, Purview) with no open integration needs.
17
18	## Workflow
19
20	1. Define metadata ownership and minimum required fields.
21	Include:
22	- who owns each dataset's metadata (producing team, platform team, or steward)
23	- minimum required fields: owner, description, classification, freshness, grain
24	- which fields are auto-populated from lineage vs manually maintained
25	- how metadata quality is measured and enforced
26	- what "complete" means for a dataset to appear as discoverable
27
28	2. Connect lineage capture to actual execution surfaces.
29	- `OpenLineage` integration with orchestrators (Airflow, Dagster)
30	- `OpenLineage` integration with processing engines (Spark, dbt, Flink)
31	- validate that lineage events fire correctly on job start, complete, and failure
32	- test that column-level lineage propagates through transformation layers
33	- monitor for gaps: jobs that run but produce no lineage events
34
35	3. Make discovery trustworthy through quality signals.
36	- surface trust indicators: last validated, freshness, quality score, owner responsiveness
37	- distinguish production-governed datasets from experimental or deprecated ones
38	- tag datasets with classification levels (public, internal, restricted, sensitive)
39	- ensure search relevance: descriptions, tags, and usage signals improve discoverability
40	- retire stale assets — dead datasets in the catalog erode trust
41
42	4. Integrate metadata publishing into the pipeline lifecycle.
43	- metadata should update when pipelines deploy, not in separate manual workflows
44	- use CI/CD hooks to validate metadata completeness before release
45	- publish schema changes to the catalog automatically through lineage events
46	- trigger alerts when metadata freshness falls behind actual data freshness
47	- make metadata a release-gate requirement: no catalog entry, no production access
48
49	5. Plan for multi-team governance and federated ownership.
50	- define roles: data producer, data steward, platform admin, consumer
51	- establish policies for who can modify metadata for shared datasets
52	- create templates for common dataset types so teams start with good defaults
53	- define escalation paths when metadata quality issues are found
54	- audit metadata coverage regularly and report gaps per team
55
56	6. Handle lineage accuracy and drift.
57	- lineage can become stale when pipelines change without updating integration
58	- schedule lineage validation: compare catalog lineage with actual pipeline structure
59	- alert when expected lineage events stop firing
60	- plan for lineage in complex scenarios: dynamic tables, views, materialized CTEs
61	- document lineage limitations: what the system captures vs what it cannot
62
63	## Common Rationalizations
64
65	\| Rationalization \| Reality \|
66	\| --- \| --- \|
67	\| "We'll add metadata later after the pipeline is stable." \| Metadata added later is often wrong because context is lost. Capturing metadata during development is cheaper and more accurate. \|
68	\| "Lineage is automatic — we just need to install the integration." \| Integrations capture lineage from supported execution paths only. Custom code, dynamic SQL, and non-instrumented tools create gaps that require explicit attention. \|
69	\| "The catalog is just for discovery — it doesn't need to be accurate." \| An inaccurate catalog is worse than no catalog. Teams lose trust and stop using it, making the investment worthless. \|
70	\| "Metadata governance is overhead for small teams." \| Small teams benefit most from metadata discipline because there's less institutional knowledge to fall back on when someone leaves. \|
71
72	## Red Flags
73
74	- catalog has datasets with no owner, no description, or stale freshness indicators
75	- lineage graph has gaps for major pipelines or transformation layers
76	- metadata is manually maintained in a spreadsheet separate from the platform
77	- no process for retiri

People who liked this also viewed

6 picks

Type

openmetadata-datahub-and-openlineage

19stars4forks

View on GitHub

How to install openmetadata-datahub-and-openlineage?

vaquarkhan/data-engineering-agent-skills/openmetadata-datahub-and-openlineage

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill openmetadata-datahub-and-openlineage

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# OpenMetadata DataHub And OpenLineage
2
3	## Overview
4
5	Use this skill when metadata and lineage must be operationalized through open tooling such as `OpenMetadata`, `DataHub`, or `OpenLineage`. It helps agents align producers with catalog expectations, ensure lineage accuracy, and make discovery trustworthy rather than decorative.
6
7	## When to Use
8
9	- integrating metadata platforms into delivery workflows
10	- improving dataset discovery and trust signals
11	- capturing and validating lineage across pipelines
12	- publishing governed datasets into open catalog ecosystems
13	- connecting `OpenLineage` events from Spark, Airflow, or dbt to a catalog
14	- defining metadata governance policies for multi-team environments
15
16	Do not use this when the team has no shared catalog requirement or when lineage is handled entirely within a closed platform (e.g., Unity Catalog, Purview) with no open integration needs.
17
18	## Workflow
19
20	1. Define metadata ownership and minimum required fields.
21	Include:
22	- who owns each dataset's metadata (producing team, platform team, or steward)
23	- minimum required fields: owner, description, classification, freshness, grain
24	- which fields are auto-populated from lineage vs manually maintained
25	- how metadata quality is measured and enforced
26	- what "complete" means for a dataset to appear as discoverable
27
28	2. Connect lineage capture to actual execution surfaces.
29	- `OpenLineage` integration with orchestrators (Airflow, Dagster)
30	- `OpenLineage` integration with processing engines (Spark, dbt, Flink)
31	- validate that lineage events fire correctly on job start, complete, and failure
32	- test that column-level lineage propagates through transformation layers
33	- monitor for gaps: jobs that run but produce no lineage events
34
35	3. Make discovery trustworthy through quality signals.
36	- surface trust indicators: last validated, freshness, quality score, owner responsiveness
37	- distinguish production-governed datasets from experimental or deprecated ones
38	- tag datasets with classification levels (public, internal, restricted, sensitive)
39	- ensure search relevance: descriptions, tags, and usage signals improve discoverability
40	- retire stale assets — dead datasets in the catalog erode trust
41
42	4. Integrate metadata publishing into the pipeline lifecycle.
43	- metadata should update when pipelines deploy, not in separate manual workflows
44	- use CI/CD hooks to validate metadata completeness before release
45	- publish schema changes to the catalog automatically through lineage events
46	- trigger alerts when metadata freshness falls behind actual data freshness
47	- make metadata a release-gate requirement: no catalog entry, no production access
48
49	5. Plan for multi-team governance and federated ownership.
50	- define roles: data producer, data steward, platform admin, consumer
51	- establish policies for who can modify metadata for shared datasets
52	- create templates for common dataset types so teams start with good defaults
53	- define escalation paths when metadata quality issues are found
54	- audit metadata coverage regularly and report gaps per team
55
56	6. Handle lineage accuracy and drift.
57	- lineage can become stale when pipelines change without updating integration
58	- schedule lineage validation: compare catalog lineage with actual pipeline structure
59	- alert when expected lineage events stop firing
60	- plan for lineage in complex scenarios: dynamic tables, views, materialized CTEs
61	- document lineage limitations: what the system captures vs what it cannot
62
63	## Common Rationalizations
64
65	\| Rationalization \| Reality \|
66	\| --- \| --- \|
67	\| "We'll add metadata later after the pipeline is stable." \| Metadata added later is often wrong because context is lost. Capturing metadata during development is cheaper and more accurate. \|
68	\| "Lineage is automatic — we just need to install the integration." \| Integrations capture lineage from supported execution paths only. Custom code, dynamic SQL, and non-instrumented tools create gaps that require explicit attention. \|
69	\| "The catalog is just for discovery — it doesn't need to be accurate." \| An inaccurate catalog is worse than no catalog. Teams lose trust and stop using it, making the investment worthless. \|
70	\| "Metadata governance is overhead for small teams." \| Small teams benefit most from metadata discipline because there's less institutional knowledge to fall back on when someone leaves. \|
71
72	## Red Flags
73
74	- catalog has datasets with no owner, no description, or stale freshness indicators
75	- lineage graph has gaps for major pipelines or transformation layers
76	- metadata is manually maintained in a spreadsheet separate from the platform
77	- no process for retiri

People who liked this also viewed

6 picks

Type