experiment-log

Name: experiment-log
Author: zakelfassi

17stars0forks

How to install experiment-log?

zakelfassi/skills-driven-development/experiment-log

$npx -y skills add zakelfassi/skills-driven-development --skill experiment-log

Installs into the current project.

Summary

Record a model training run — log parameters, metrics, and artifact paths, then append a structured row to the experiments ledger. Use when a training run completes, when asked to "log this experiment", or when comparing runs to decide which model to promote.

Files · 1

View on GitHub

SKILL.md

1	# Experiment Log
2
3	Record a training run's parameters, metrics, and artifact paths in the experiments ledger.
4
5	## Inputs
6	- Experiment name (e.g., `churn-v3-lgbm`)
7	- Run ID (auto-generated if omitted: `{name}-{YYYYMMDD-HHmmss}`)
8	- Parameters (key-value pairs, e.g., `learning_rate=0.05 n_estimators=500`)
9	- Metrics (key-value pairs, e.g., `auc=0.83 f1=0.71 precision=0.79 recall=0.64`)
10	- Artifact path (model checkpoint, serialized pipeline, etc.)
11	- Notes (free-form, optional)
12
13	## Steps
14
15	1. Collect run metadata
16	```python
17	import datetime, hashlib, json, os
18
19	run_id = "{name}-" + datetime.datetime.utcnow().strftime("%Y%m%d-%H%M%S")
20	git_sha = os.popen("git rev-parse --short HEAD").read().strip()
21	dataset_version = open("data/.version").read().strip() # semver or hash
22	```
23
24	2. Capture parameters and metrics
25	If using a training script that outputs JSON:
26	```bash
27	python train.py --config config/{name}.yaml --output-metrics /tmp/metrics.json
28	```
29	Otherwise, capture values directly from the trainer object:
30	```python
31	params = model.get_params()
32	metrics = {"auc": roc_auc_score(y_test, y_pred), "f1": f1_score(y_test, y_pred)}
33	```
34
35	3. Save artifacts
36	```python
37	import joblib
38	artifact_path = f"artifacts/{run_id}/model.pkl"
39	os.makedirs(os.path.dirname(artifact_path), exist_ok=True)
40	joblib.dump(model, artifact_path)
41	```
42
43	4. Append to the experiments ledger
44	```bash
45	scripts/log-experiment.sh \
46	--name "{name}" \
47	--run-id "{run_id}" \
48	--params '{"learning_rate": 0.05}' \
49	--metrics '{"auc": 0.83, "f1": 0.71}' \
50	--artifact "{artifact_path}" \
51	--notes "{notes}"
52	```
53	The ledger is `experiments/log.csv` — a flat CSV with one row per run.
54
55	5. Verify the entry
56	```bash
57	tail -n 5 experiments/log.csv
58	```
59	Confirm: run_id is unique, metrics are numeric, artifact path exists.
60
61	6. Compare with previous runs (optional)
62	```bash
63	python scripts/compare-runs.py --metric auc --top 5
64	```
65
66	7. Promote if best (invoke `pipeline-stage` skill to wire the new model into serving)
67	Only promote after peer review of the metrics and a sanity-check on the holdout set.
68
69	## Conventions
70	- Ledger location: `experiments/log.csv` — never delete rows; mark superseded runs with `status=superseded`
71	- Artifact naming: `artifacts/{run_id}/` — one directory per run
72	- Required fields: `run_id`, `experiment_name`, `timestamp`, `git_sha`, `auc` (or primary metric), `artifact_path`
73	- All metrics are floats; parameters are JSON strings
74	- Every run references the dataset version it trained on (`dataset_version` column)
75
76	## Edge Cases
77	- Duplicate run_id: Append a counter suffix (`-2`, `-3`) rather than overwriting; log a warning.
78	- Training failed mid-run: Log the entry with `status=failed` and the error message in `notes`; don't leave the ledger entry absent.
79	- Very large artifact: Store only the path; add a `artifact_size_mb` column for reference.
80	- Remote artifact store (S3/GCS): Use the full URI as `artifact_path`; the log script accepts both local paths and URIs.

People who liked this also viewed

6 picks

Type

experiment-log

17stars0forks

View on GitHub

How to install experiment-log?

zakelfassi/skills-driven-development/experiment-log

$npx -y skills add zakelfassi/skills-driven-development --skill experiment-log

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Experiment Log
2
3	Record a training run's parameters, metrics, and artifact paths in the experiments ledger.
4
5	## Inputs
6	- Experiment name (e.g., `churn-v3-lgbm`)
7	- Run ID (auto-generated if omitted: `{name}-{YYYYMMDD-HHmmss}`)
8	- Parameters (key-value pairs, e.g., `learning_rate=0.05 n_estimators=500`)
9	- Metrics (key-value pairs, e.g., `auc=0.83 f1=0.71 precision=0.79 recall=0.64`)
10	- Artifact path (model checkpoint, serialized pipeline, etc.)
11	- Notes (free-form, optional)
12
13	## Steps
14
15	1. Collect run metadata
16	```python
17	import datetime, hashlib, json, os
18
19	run_id = "{name}-" + datetime.datetime.utcnow().strftime("%Y%m%d-%H%M%S")
20	git_sha = os.popen("git rev-parse --short HEAD").read().strip()
21	dataset_version = open("data/.version").read().strip() # semver or hash
22	```
23
24	2. Capture parameters and metrics
25	If using a training script that outputs JSON:
26	```bash
27	python train.py --config config/{name}.yaml --output-metrics /tmp/metrics.json
28	```
29	Otherwise, capture values directly from the trainer object:
30	```python
31	params = model.get_params()
32	metrics = {"auc": roc_auc_score(y_test, y_pred), "f1": f1_score(y_test, y_pred)}
33	```
34
35	3. Save artifacts
36	```python
37	import joblib
38	artifact_path = f"artifacts/{run_id}/model.pkl"
39	os.makedirs(os.path.dirname(artifact_path), exist_ok=True)
40	joblib.dump(model, artifact_path)
41	```
42
43	4. Append to the experiments ledger
44	```bash
45	scripts/log-experiment.sh \
46	--name "{name}" \
47	--run-id "{run_id}" \
48	--params '{"learning_rate": 0.05}' \
49	--metrics '{"auc": 0.83, "f1": 0.71}' \
50	--artifact "{artifact_path}" \
51	--notes "{notes}"
52	```
53	The ledger is `experiments/log.csv` — a flat CSV with one row per run.
54
55	5. Verify the entry
56	```bash
57	tail -n 5 experiments/log.csv
58	```
59	Confirm: run_id is unique, metrics are numeric, artifact path exists.
60
61	6. Compare with previous runs (optional)
62	```bash
63	python scripts/compare-runs.py --metric auc --top 5
64	```
65
66	7. Promote if best (invoke `pipeline-stage` skill to wire the new model into serving)
67	Only promote after peer review of the metrics and a sanity-check on the holdout set.
68
69	## Conventions
70	- Ledger location: `experiments/log.csv` — never delete rows; mark superseded runs with `status=superseded`
71	- Artifact naming: `artifacts/{run_id}/` — one directory per run
72	- Required fields: `run_id`, `experiment_name`, `timestamp`, `git_sha`, `auc` (or primary metric), `artifact_path`
73	- All metrics are floats; parameters are JSON strings
74	- Every run references the dataset version it trained on (`dataset_version` column)
75
76	## Edge Cases
77	- Duplicate run_id: Append a counter suffix (`-2`, `-3`) rather than overwriting; log a warning.
78	- Training failed mid-run: Log the entry with `status=failed` and the error message in `notes`; don't leave the ledger entry absent.
79	- Very large artifact: Store only the path; add a `artifact_size_mb` column for reference.
80	- Remote artifact store (S3/GCS): Use the full URI as `artifact_path`; the log script accepts both local paths and URIs.

People who liked this also viewed

6 picks

Type