tracing-upstream-lineage

Name: tracing-upstream-lineage
Author: astronomer

885installs55forks

How to install tracing-upstream-lineage?

astronomer/agents/tracing-upstream-lineage

$npx -y skills add astronomer/agents --skill tracing-upstream-lineage

Installs into the current project.

Summary

Trace upstream data lineage. Use when the user asks where data comes from, what feeds a table, upstream dependencies, data sources, or needs to understand data origins.

Files · 1

View on GitHub

SKILL.md

1	# Upstream Lineage: Sources
2
3	Trace the origins of data - answer "Where does this data come from?"
4
5	## Lineage Investigation
6
7	### Step 1: Identify the Target Type
8
9	Determine what we're tracing:
10	- Table: Trace what populates this table
11	- Column: Trace where this specific column comes from
12	- DAG: Trace what data sources this DAG reads from
13
14	### Step 2: Find the Producing DAG
15
16	Tables are typically populated by Airflow DAGs. Find the connection:
17
18	1. Search DAGs by name: Use `af dags list` and look for DAG names matching the table name
19	- `load_customers` -> `customers` table
20	- `etl_daily_orders` -> `orders` table
21
22	2. Explore DAG source code: Use `af dags source <dag_id>` to read the DAG definition
23	- Look for INSERT, MERGE, CREATE TABLE statements
24	- Find the target table in the code
25
26	3. Check DAG tasks: Use `af tasks list <dag_id>` to see what operations the DAG performs
27
28	### On Astro
29
30	If you're running on Astro, the Lineage tab in the Astro UI provides visual lineage exploration across DAGs and datasets. Use it to quickly trace upstream dependencies without manually searching DAG source code.
31
32	### On OSS Airflow
33
34	Use DAG source code and task logs to trace lineage (no built-in cross-DAG UI).
35
36	### Step 3: Trace Data Sources
37
38	From the DAG code, identify source tables and systems:
39
40	SQL Sources (look for FROM clauses):
41	```python
42	# In DAG code:
43	SELECT * FROM source_schema.source_table # <- This is an upstream source
44	```
45
46	External Sources (look for connection references):
47	- `S3Operator` -> S3 bucket source
48	- `PostgresOperator` -> Postgres database source
49	- `SalesforceOperator` -> Salesforce API source
50	- `HttpOperator` -> REST API source
51
52	File Sources:
53	- CSV/Parquet files in object storage
54	- SFTP drops
55	- Local file paths
56
57	### Step 4: Build the Lineage Chain
58
59	Recursively trace each source:
60
61	```
62	TARGET: analytics.orders_daily
63	^
64	+-- DAG: etl_daily_orders
65	^
66	+-- SOURCE: raw.orders (table)
67	\| ^
68	\| +-- DAG: ingest_orders
69	\| ^
70	\| +-- SOURCE: Salesforce API (external)
71	\|
72	+-- SOURCE: dim.customers (table)
73	^
74	+-- DAG: load_customers
75	^
76	+-- SOURCE: PostgreSQL (external DB)
77	```
78
79	### Step 5: Check Source Health
80
81	For each upstream source:
82	- Tables: Check freshness with the checking-freshness skill
83	- DAGs: Check recent run status with `af dags stats`
84	- External systems: Note connection info from DAG code
85
86	## Lineage for Columns
87
88	When tracing a specific column:
89
90	1. Find the column in the target table schema
91	2. Search DAG source code for references to that column name
92	3. Trace through transformations:
93	- Direct mappings: `source.col AS target_col`
94	- Transformations: `COALESCE(a.col, b.col) AS target_col`
95	- Aggregations: `SUM(detail.amount) AS total_amount`
96
97	## Output: Lineage Report
98
99	### Summary
100	One-line answer: "This table is populated by DAG X from sources Y and Z"
101
102	### Lineage Diagram
103	```
104	[Salesforce] --> [raw.opportunities] --> [stg.opportunities] --> [fct.sales]
105	\| \|
106	DAG: ingest_sfdc DAG: transform_sales
107	```
108
109	### Source Details
110
111	\| Source \| Type \| Connection \| Freshness \| Owner \|
112	\|--------\|------\|------------\|-----------\|-------\|
113	\| raw.orders \| Table \| Internal \| 2h ago \| data-team \|
114	\| Salesforce \| API \| salesforce_conn \| Real-time \| sales-ops \|
115
116	### Transformation Chain
117	Describe how data flows and transforms:
118	1. Raw data lands in `raw.orders` via Salesforce API sync
119	2. DAG `transform_orders` cleans and dedupes into `stg.orders`
120	3. DAG `build_order_facts` joins with dimensions into `fct.orders`
121
122	### Data Quality Implications
123	- Single points of failure?
124	- Stale upstream sources?
125	- Complex transformation chains that could break?
126
127	### Related Skills
128	- Check source freshness: checking-freshness skill
129	- Debug source DAG: debugging-dags skill
130	- Trace downstream impacts: tracing-downstream-lineage skill
131	- Add manual lineage annotations: annotating-task-lineage skill
132	- Build custom lineage extractors: creating-openlineage-extractors skill

People who liked this also viewed

6 picks

Type

tracing-upstream-lineage

885installs55forks

View on GitHub

How to install tracing-upstream-lineage?

astronomer/agents/tracing-upstream-lineage

$npx -y skills add astronomer/agents --skill tracing-upstream-lineage

Installs into the current project.

Summary

Trace upstream data lineage. Use when the user asks where data comes from, what feeds a table, upstream dependencies, data sources, or needs to understand data origins.

Files · 1

View on GitHub

SKILL.md

1	# Upstream Lineage: Sources
2
3	Trace the origins of data - answer "Where does this data come from?"
4
5	## Lineage Investigation
6
7	### Step 1: Identify the Target Type
8
9	Determine what we're tracing:
10	- Table: Trace what populates this table
11	- Column: Trace where this specific column comes from
12	- DAG: Trace what data sources this DAG reads from
13
14	### Step 2: Find the Producing DAG
15
16	Tables are typically populated by Airflow DAGs. Find the connection:
17
18	1. Search DAGs by name: Use `af dags list` and look for DAG names matching the table name
19	- `load_customers` -> `customers` table
20	- `etl_daily_orders` -> `orders` table
21
22	2. Explore DAG source code: Use `af dags source <dag_id>` to read the DAG definition
23	- Look for INSERT, MERGE, CREATE TABLE statements
24	- Find the target table in the code
25
26	3. Check DAG tasks: Use `af tasks list <dag_id>` to see what operations the DAG performs
27
28	### On Astro
29
30	If you're running on Astro, the Lineage tab in the Astro UI provides visual lineage exploration across DAGs and datasets. Use it to quickly trace upstream dependencies without manually searching DAG source code.
31
32	### On OSS Airflow
33
34	Use DAG source code and task logs to trace lineage (no built-in cross-DAG UI).
35
36	### Step 3: Trace Data Sources
37
38	From the DAG code, identify source tables and systems:
39
40	SQL Sources (look for FROM clauses):
41	```python
42	# In DAG code:
43	SELECT * FROM source_schema.source_table # <- This is an upstream source
44	```
45
46	External Sources (look for connection references):
47	- `S3Operator` -> S3 bucket source
48	- `PostgresOperator` -> Postgres database source
49	- `SalesforceOperator` -> Salesforce API source
50	- `HttpOperator` -> REST API source
51
52	File Sources:
53	- CSV/Parquet files in object storage
54	- SFTP drops
55	- Local file paths
56
57	### Step 4: Build the Lineage Chain
58
59	Recursively trace each source:
60
61	```
62	TARGET: analytics.orders_daily
63	^
64	+-- DAG: etl_daily_orders
65	^
66	+-- SOURCE: raw.orders (table)
67	\| ^
68	\| +-- DAG: ingest_orders
69	\| ^
70	\| +-- SOURCE: Salesforce API (external)
71	\|
72	+-- SOURCE: dim.customers (table)
73	^
74	+-- DAG: load_customers
75	^
76	+-- SOURCE: PostgreSQL (external DB)
77	```
78
79	### Step 5: Check Source Health
80
81	For each upstream source:
82	- Tables: Check freshness with the checking-freshness skill
83	- DAGs: Check recent run status with `af dags stats`
84	- External systems: Note connection info from DAG code
85
86	## Lineage for Columns
87
88	When tracing a specific column:
89
90	1. Find the column in the target table schema
91	2. Search DAG source code for references to that column name
92	3. Trace through transformations:
93	- Direct mappings: `source.col AS target_col`
94	- Transformations: `COALESCE(a.col, b.col) AS target_col`
95	- Aggregations: `SUM(detail.amount) AS total_amount`
96
97	## Output: Lineage Report
98
99	### Summary
100	One-line answer: "This table is populated by DAG X from sources Y and Z"
101
102	### Lineage Diagram
103	```
104	[Salesforce] --> [raw.opportunities] --> [stg.opportunities] --> [fct.sales]
105	\| \|
106	DAG: ingest_sfdc DAG: transform_sales
107	```
108
109	### Source Details
110
111	\| Source \| Type \| Connection \| Freshness \| Owner \|
112	\|--------\|------\|------------\|-----------\|-------\|
113	\| raw.orders \| Table \| Internal \| 2h ago \| data-team \|
114	\| Salesforce \| API \| salesforce_conn \| Real-time \| sales-ops \|
115
116	### Transformation Chain
117	Describe how data flows and transforms:
118	1. Raw data lands in `raw.orders` via Salesforce API sync
119	2. DAG `transform_orders` cleans and dedupes into `stg.orders`
120	3. DAG `build_order_facts` joins with dimensions into `fct.orders`
121
122	### Data Quality Implications
123	- Single points of failure?
124	- Stale upstream sources?
125	- Complex transformation chains that could break?
126
127	### Related Skills
128	- Check source freshness: checking-freshness skill
129	- Debug source DAG: debugging-dags skill
130	- Trace downstream impacts: tracing-downstream-lineage skill
131	- Add manual lineage annotations: annotating-task-lineage skill
132	- Build custom lineage extractors: creating-openlineage-extractors skill

People who liked this also viewed

6 picks

Type