python-data-engineering-and-pipeline-packaging

Name: python-data-engineering-and-pipeline-packaging
Author: vaquarkhan

19stars4forks

How to install python-data-engineering-and-pipeline-packaging?

vaquarkhan/data-engineering-agent-skills/python-data-engineering-and-pipeline-packaging

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill python-data-engineering-and-pipeline-packaging

Installs into the current project.

Summary

Guides agents through Python-based data engineering implementation. Use when building or modifying Python ingestion jobs, orchestration helpers, PySpark entry points, validation code, packaging, dependency management, or operational CLI workflows.

Files · 1

View on GitHub

SKILL.md

1	# Python Data Engineering And Pipeline Packaging
2
3	## Overview
4
5	Use this skill when `Python` is the main implementation language for data pipelines or operational data tooling. It helps agents structure jobs as maintainable packages instead of loose scripts, choose the right execution boundary, manage dependencies explicitly, and keep runtime behavior testable and production-safe.
6
7	## When to Use
8
9	- building or modifying `Python` data pipelines
10	- packaging `PySpark`, ingestion, validation, or orchestration helper code
11	- moving from notebooks or scripts into production-ready modules
12	- managing dependency, environment, and runtime issues in `Python`
13	- adding CLI entry points, test harnesses, or local development workflows
14
15	Do not treat a working script as a production design just because it runs once.
16
17	## Workflow
18
19	1. Define the role of the Python code.
20	Clarify whether it is:
21	- a single-node transform
22	- a `PySpark` job entry point
23	- an orchestration helper
24	- a validation or reconciliation tool
25	- an integration or extraction service
26
27	2. Package logic into explicit modules.
28	Prefer:
29	- versioned packages
30	- reusable modules
31	- clear CLI or job entry points
32	- isolated configuration
33	- minimal hidden global state
34
35	3. Make dependency management real.
36	Define:
37	- environment model
38	- pinned dependency strategy
39	- native or system dependency assumptions
40	- compatibility with runtime platforms such as `Airflow`, `Spark`, or container images
41
42	4. Keep runtime boundaries explicit.
43	Decide:
44	- what runs locally versus distributed
45	- what belongs in orchestration versus the job package
46	- how configuration, secrets, and environment values are supplied
47	- how logs, retries, and exits behave operationally
48
49	5. Prove the package is maintainable.
50	Require:
51	- targeted tests
52	- representative input cases
53	- type or interface clarity where useful
54	- reproducible local execution
55
56	## Common Rationalizations
57
58	\| Rationalization \| Reality \|
59	\| --- \| --- \|
60	\| "It is only a small Python script." \| Small scripts often become critical pipeline entry points with no packaging or test discipline. \|
61	\| "We can keep the business logic in the DAG or notebook." \| Hidden logic in orchestration or notebook state becomes hard to test, reuse, and debug. \|
62	\| "Requirements are enough documentation." \| Dependency files do not explain runtime assumptions, entry points, or platform compatibility. \|
63
64	## Red Flags
65
66	- pipeline logic lives mostly in one script with no reusable module structure
67	- notebooks, DAGs, and job code duplicate the same transformation logic
68	- dependency versions are implicit or environment-specific
69	- local runs and deployed runs behave differently with no explanation
70	- secrets or environment assumptions are embedded in code
71
72	## Verification
73
74	- [ ] The Python code has a clear package and entry-point shape
75	- [ ] Runtime, dependency, and environment assumptions are explicit
76	- [ ] Orchestration code and business logic are separated
77	- [ ] Tests or reproducible execution paths exist for the important logic

People who liked this also viewed

6 picks

Type

python-data-engineering-and-pipeline-packaging

19stars4forks

View on GitHub

How to install python-data-engineering-and-pipeline-packaging?

vaquarkhan/data-engineering-agent-skills/python-data-engineering-and-pipeline-packaging

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill python-data-engineering-and-pipeline-packaging

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Python Data Engineering And Pipeline Packaging
2
3	## Overview
4
5	Use this skill when `Python` is the main implementation language for data pipelines or operational data tooling. It helps agents structure jobs as maintainable packages instead of loose scripts, choose the right execution boundary, manage dependencies explicitly, and keep runtime behavior testable and production-safe.
6
7	## When to Use
8
9	- building or modifying `Python` data pipelines
10	- packaging `PySpark`, ingestion, validation, or orchestration helper code
11	- moving from notebooks or scripts into production-ready modules
12	- managing dependency, environment, and runtime issues in `Python`
13	- adding CLI entry points, test harnesses, or local development workflows
14
15	Do not treat a working script as a production design just because it runs once.
16
17	## Workflow
18
19	1. Define the role of the Python code.
20	Clarify whether it is:
21	- a single-node transform
22	- a `PySpark` job entry point
23	- an orchestration helper
24	- a validation or reconciliation tool
25	- an integration or extraction service
26
27	2. Package logic into explicit modules.
28	Prefer:
29	- versioned packages
30	- reusable modules
31	- clear CLI or job entry points
32	- isolated configuration
33	- minimal hidden global state
34
35	3. Make dependency management real.
36	Define:
37	- environment model
38	- pinned dependency strategy
39	- native or system dependency assumptions
40	- compatibility with runtime platforms such as `Airflow`, `Spark`, or container images
41
42	4. Keep runtime boundaries explicit.
43	Decide:
44	- what runs locally versus distributed
45	- what belongs in orchestration versus the job package
46	- how configuration, secrets, and environment values are supplied
47	- how logs, retries, and exits behave operationally
48
49	5. Prove the package is maintainable.
50	Require:
51	- targeted tests
52	- representative input cases
53	- type or interface clarity where useful
54	- reproducible local execution
55
56	## Common Rationalizations
57
58	\| Rationalization \| Reality \|
59	\| --- \| --- \|
60	\| "It is only a small Python script." \| Small scripts often become critical pipeline entry points with no packaging or test discipline. \|
61	\| "We can keep the business logic in the DAG or notebook." \| Hidden logic in orchestration or notebook state becomes hard to test, reuse, and debug. \|
62	\| "Requirements are enough documentation." \| Dependency files do not explain runtime assumptions, entry points, or platform compatibility. \|
63
64	## Red Flags
65
66	- pipeline logic lives mostly in one script with no reusable module structure
67	- notebooks, DAGs, and job code duplicate the same transformation logic
68	- dependency versions are implicit or environment-specific
69	- local runs and deployed runs behave differently with no explanation
70	- secrets or environment assumptions are embedded in code
71
72	## Verification
73
74	- [ ] The Python code has a clear package and entry-point shape
75	- [ ] Runtime, dependency, and environment assumptions are explicit
76	- [ ] Orchestration code and business logic are separated
77	- [ ] Tests or reproducible execution paths exist for the important logic

People who liked this also viewed

6 picks

Type