ml-pipeline

Name: ml-pipeline
Author: Jeffallan

2.8kinstalls971forks

How to install ml-pipeline?

jeffallan/claude-skills/ml-pipeline

$npx -y skills add Jeffallan/claude-skills --skill ml-pipeline

Installs into the current project.

Summary

Designs and implements production-grade ML pipeline infrastructure: configures experiment tracking with MLflow or Weights & Biases, creates Kubeflow or Airflow DAGs for training orchestration, builds feature store schemas with Feast, deploys model registries, and automates retrai

Files · 1

View on GitHub

SKILL.md

1	# ML Pipeline Expert
2
3	Senior ML pipeline engineer specializing in production-grade machine learning infrastructure, orchestration systems, and automated training workflows.
4
5	## Core Workflow
6
7	1. Design pipeline architecture — Map data flow, identify stages, define interfaces between components
8	2. Validate data schema — Run schema checks and distribution validation before any training begins; halt and report on failures
9	3. Implement feature engineering — Build transformation pipelines, feature stores, and validation checks
10	4. Orchestrate training — Configure distributed training, hyperparameter tuning, and resource allocation
11	5. Track experiments — Log metrics, parameters, and artifacts; enable comparison and reproducibility
12	6. Validate and deploy — Run model evaluation gates; implement A/B testing or shadow deployment before promotion
13
14	## Reference Guide
15
16	Load detailed guidance based on context:
17
18	\| Topic \| Reference \| Load When \|
19	\|-------\|-----------\|-----------\|
20	\| Feature Engineering \| `references/feature-engineering.md` \| Feature pipelines, transformations, feature stores, Feast, data validation \|
21	\| Training Pipelines \| `references/training-pipelines.md` \| Training orchestration, distributed training, hyperparameter tuning, resource management \|
22	\| Experiment Tracking \| `references/experiment-tracking.md` \| MLflow, Weights & Biases, experiment logging, model registry \|
23	\| Pipeline Orchestration \| `references/pipeline-orchestration.md` \| Kubeflow Pipelines, Airflow, Prefect, DAG design, workflow automation \|
24	\| Model Validation \| `references/model-validation.md` \| Evaluation strategies, validation workflows, A/B testing, shadow deployment \|
25
26	## Code Templates
27
28	### MLflow Experiment Logging (minimal reproducible example)
29
30	```python
31	import mlflow
32	import mlflow.sklearn
33	from sklearn.ensemble import RandomForestClassifier
34	from sklearn.model_selection import train_test_split
35	from sklearn.metrics import accuracy_score, f1_score
36	import numpy as np
37
38	# Pin random state for reproducibility
39	SEED = 42
40	np.random.seed(SEED)
41
42	mlflow.set_experiment("my-classifier-experiment")
43
44	with mlflow.start_run():
45	# Log all hyperparameters — never hardcode silently
46	params = {"n_estimators": 100, "max_depth": 5, "random_state": SEED}
47	mlflow.log_params(params)
48
49	model = RandomForestClassifier(**params)
50	model.fit(X_train, y_train)
51	preds = model.predict(X_test)
52
53	# Log metrics
54	mlflow.log_metric("accuracy", accuracy_score(y_test, preds))
55	mlflow.log_metric("f1", f1_score(y_test, preds, average="weighted"))
56
57	# Log and register the model artifact
58	mlflow.sklearn.log_model(model, artifact_path="model",
59	registered_model_name="my-classifier")
60	```
61
62	### Kubeflow Pipeline Component (single-step template)
63
64	```python
65	from kfp.v2 import dsl
66	from kfp.v2.dsl import component, Input, Output, Dataset, Model, Metrics
67
68	@component(base_image="python:3.10", packages_to_install=["scikit-learn", "mlflow"])
69	def train_model(
70	train_data: Input[Dataset],
71	model_output: Output[Model],
72	metrics_output: Output[Metrics],
73	n_estimators: int = 100,
74	max_depth: int = 5,
75	):
76	import pandas as pd
77	from sklearn.ensemble import RandomForestClassifier
78	import pickle, json
79
80	df = pd.read_csv(train_data.path)
81	X, y = df.drop("label", axis=1), df["label"]
82
83	model = RandomForestClassifier(n_estimators=n_estimators,
84	max_depth=max_depth, random_state=42)
85	model.fit(X, y)
86
87	with open(model_output.path, "wb") as f:
88	pickle.dump(model, f)
89
90	metrics_output.log_metric("train_samples", len(df))
91
92	@dsl.pipeline(name="training-pipeline")
93	def training_pipeline(data_path: str, n_estimators: int = 100):
94	train_step = train_model(n_estimators=n_estimators)
95	# Chain additional steps (validate,

People who liked this also viewed

6 picks

Type

ml-pipeline

2.8kinstalls971forks

View on GitHub

How to install ml-pipeline?

jeffallan/claude-skills/ml-pipeline

$npx -y skills add Jeffallan/claude-skills --skill ml-pipeline

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# ML Pipeline Expert
2
3	Senior ML pipeline engineer specializing in production-grade machine learning infrastructure, orchestration systems, and automated training workflows.
4
5	## Core Workflow
6
7	1. Design pipeline architecture — Map data flow, identify stages, define interfaces between components
8	2. Validate data schema — Run schema checks and distribution validation before any training begins; halt and report on failures
9	3. Implement feature engineering — Build transformation pipelines, feature stores, and validation checks
10	4. Orchestrate training — Configure distributed training, hyperparameter tuning, and resource allocation
11	5. Track experiments — Log metrics, parameters, and artifacts; enable comparison and reproducibility
12	6. Validate and deploy — Run model evaluation gates; implement A/B testing or shadow deployment before promotion
13
14	## Reference Guide
15
16	Load detailed guidance based on context:
17
18	\| Topic \| Reference \| Load When \|
19	\|-------\|-----------\|-----------\|
20	\| Feature Engineering \| `references/feature-engineering.md` \| Feature pipelines, transformations, feature stores, Feast, data validation \|
21	\| Training Pipelines \| `references/training-pipelines.md` \| Training orchestration, distributed training, hyperparameter tuning, resource management \|
22	\| Experiment Tracking \| `references/experiment-tracking.md` \| MLflow, Weights & Biases, experiment logging, model registry \|
23	\| Pipeline Orchestration \| `references/pipeline-orchestration.md` \| Kubeflow Pipelines, Airflow, Prefect, DAG design, workflow automation \|
24	\| Model Validation \| `references/model-validation.md` \| Evaluation strategies, validation workflows, A/B testing, shadow deployment \|
25
26	## Code Templates
27
28	### MLflow Experiment Logging (minimal reproducible example)
29
30	```python
31	import mlflow
32	import mlflow.sklearn
33	from sklearn.ensemble import RandomForestClassifier
34	from sklearn.model_selection import train_test_split
35	from sklearn.metrics import accuracy_score, f1_score
36	import numpy as np
37
38	# Pin random state for reproducibility
39	SEED = 42
40	np.random.seed(SEED)
41
42	mlflow.set_experiment("my-classifier-experiment")
43
44	with mlflow.start_run():
45	# Log all hyperparameters — never hardcode silently
46	params = {"n_estimators": 100, "max_depth": 5, "random_state": SEED}
47	mlflow.log_params(params)
48
49	model = RandomForestClassifier(**params)
50	model.fit(X_train, y_train)
51	preds = model.predict(X_test)
52
53	# Log metrics
54	mlflow.log_metric("accuracy", accuracy_score(y_test, preds))
55	mlflow.log_metric("f1", f1_score(y_test, preds, average="weighted"))
56
57	# Log and register the model artifact
58	mlflow.sklearn.log_model(model, artifact_path="model",
59	registered_model_name="my-classifier")
60	```
61
62	### Kubeflow Pipeline Component (single-step template)
63
64	```python
65	from kfp.v2 import dsl
66	from kfp.v2.dsl import component, Input, Output, Dataset, Model, Metrics
67
68	@component(base_image="python:3.10", packages_to_install=["scikit-learn", "mlflow"])
69	def train_model(
70	train_data: Input[Dataset],
71	model_output: Output[Model],
72	metrics_output: Output[Metrics],
73	n_estimators: int = 100,
74	max_depth: int = 5,
75	):
76	import pandas as pd
77	from sklearn.ensemble import RandomForestClassifier
78	import pickle, json
79
80	df = pd.read_csv(train_data.path)
81	X, y = df.drop("label", axis=1), df["label"]
82
83	model = RandomForestClassifier(n_estimators=n_estimators,
84	max_depth=max_depth, random_state=42)
85	model.fit(X, y)
86
87	with open(model_output.path, "wb") as f:
88	pickle.dump(model, f)
89
90	metrics_output.log_metric("train_samples", len(df))
91
92	@dsl.pipeline(name="training-pipeline")
93	def training_pipeline(data_path: str, n_estimators: int = 100):
94	train_step = train_model(n_estimators=n_estimators)
95	# Chain additional steps (validate,

People who liked this also viewed

6 picks

Type