version-dataset

Name: version-dataset
Author: Aperivue

33installs53forks

How to install version-dataset?

aperivue/medsci-skills/version-dataset

$npx -y skills add Aperivue/medsci-skills --skill version-dataset

Installs into the current project.

Summary

Dataset version control for research reproducibility. Builds a deterministic content-hash manifest of a dataset (file SHA-256 + tabular schema + per-column value hashes), verifies a later copy against it to detect drift (schema change, row-count change, value changes), and diffs

Files · 1

View on GitHub

SKILL.md

1	# Version Dataset Skill
2
3	You help a medical researcher put a dataset under version control: fingerprint it,
4	detect when it changes, and lock a reproducible version. This guards the
5	data-integrity rule — an analysis must run on the data it claims to, with a fixed
6	seed — by making any drift between runs loud instead of silent.
7
8	## Communication Rules
9
10	- Communicate with the user in their preferred language.
11	- Manifest fields, drift reports, and provenance notes are in English.
12
13	## Philosophy
14
15	A dataset is an input to a result; if it changes silently, every downstream
16	number is suspect. This skill records a deterministic fingerprint (file SHA-256 +,
17	for tabular files, schema and per-column value hashes) so a later run can prove
18	the inputs are unchanged. It does not alter data, and it records nothing
19	non-deterministic (no timestamps unless explicitly passed), so the same data
20	always yields the same manifest.
21
22	## Reference Files
23
24	- Manifest schema + workflow: `${CLAUDE_SKILL_DIR}/references/manifest_schema.md` —
25	the manifest.json structure, what each drift category means, and the non-
26	deterministic-artifact policy (PPTX/DOCX timestamps). Read before interpreting drift.
27
28	## Deterministic Script
29
30	```bash
31	# Build a manifest (record the analysis seed + provenance)
32	python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" manifest data.csv \
33	--out manifest.json --seed 42 --provenance "KNHANES 2018 extract v1"
34
35	# Verify a later copy against it (CI / pre-analysis gate)
36	python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" verify --manifest manifest.json --strict
37
38	# Compare two manifests (what changed between versions)
39	python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" diff --old v1.json --new v2.json
40	```
41
42	File hashing is stdlib-only; tabular schema/column hashing uses pandas when present.
43	`--ignore-cols` excludes volatile columns; `--base` makes manifest keys relative.
44
45	## Workflow
46
47	### Step 1: Lock the version (gate)
48
49	Build the manifest at the moment the dataset is frozen for analysis. Gate:
50	confirm with the user the seed and provenance note are correct before locking —
51	the manifest is the record they will cite as "this is the data the results came from."
52
53	### Step 2: Verify before each run (gate)
54
55	Before re-running an analysis (or in CI), `verify --strict`. Gate: if drift is
56	reported, stop and show the user the drift report; do not proceed on changed data
57	without their explicit acknowledgement and a re-lock. Silent re-run on drifted data
58	is the failure this skill exists to prevent.
59
60	### Step 3: Diff across versions
61
62	When a dataset is intentionally updated, `diff` the old and new manifests and
63	present the change set (added/removed/changed columns, row-count delta) so the
64	user can record what changed and re-lock. Gate: the user approves the new
65	version before it replaces the locked one.
66
67	## Non-Deterministic Artifacts
68
69	Some outputs (PPTX/DOCX with embedded timestamps, figures with render metadata)
70	change byte-for-byte on every build even when the analysis is identical. Do not
71	put these under strict byte verification — manifest only the deterministic inputs
72	and tabular outputs (data files, result CSVs), or use `--ignore-cols` for volatile
73	columns. See references for the policy.
74
75	## Scope Limitations
76
77	### Supported
78	- Content-hash manifest of any file; schema + per-column hashes for tabular files
79	(CSV/TSV/Parquet/Stata/SAS/Excel).
80	- Drift verification and manifest-to-manifest diff.
81
82	### NOT Supported
83	- Storing or transmitting the data itself (manifests hold hashes, not contents).
84	- Cleaning, profiling, or de-identifying — use `/clean-data`, `/generate-codebook`, `/deidentify`.
85	- Full pipeline-output reproducibility for non-deterministic binaries (see above).
86
87	## Cross-Skill Integration
88
89	- /generate-codebook documents what is in the data; version-dataset locks which version.
90	- /deidentify should run before a manifest is shared (example values are not stored, but provenance notes may carry context).
91	- Demo reproducibility: each bundled `demo/*/` carries a `manifest.lock.json` (input data + deterministic result tables) that `verify --strict` checks.
92
93	## Worked Example
94
95	Lock a freshly-frozen extract:
96
97	```bash
98	python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" manifest cohort.csv \
99	--out manifest.json --se

People who liked this also viewed

6 picks

Type

version-dataset

33installs53forks

View on GitHub

How to install version-dataset?

aperivue/medsci-skills/version-dataset

$npx -y skills add Aperivue/medsci-skills --skill version-dataset

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Version Dataset Skill
2
3	You help a medical researcher put a dataset under version control: fingerprint it,
4	detect when it changes, and lock a reproducible version. This guards the
5	data-integrity rule — an analysis must run on the data it claims to, with a fixed
6	seed — by making any drift between runs loud instead of silent.
7
8	## Communication Rules
9
10	- Communicate with the user in their preferred language.
11	- Manifest fields, drift reports, and provenance notes are in English.
12
13	## Philosophy
14
15	A dataset is an input to a result; if it changes silently, every downstream
16	number is suspect. This skill records a deterministic fingerprint (file SHA-256 +,
17	for tabular files, schema and per-column value hashes) so a later run can prove
18	the inputs are unchanged. It does not alter data, and it records nothing
19	non-deterministic (no timestamps unless explicitly passed), so the same data
20	always yields the same manifest.
21
22	## Reference Files
23
24	- Manifest schema + workflow: `${CLAUDE_SKILL_DIR}/references/manifest_schema.md` —
25	the manifest.json structure, what each drift category means, and the non-
26	deterministic-artifact policy (PPTX/DOCX timestamps). Read before interpreting drift.
27
28	## Deterministic Script
29
30	```bash
31	# Build a manifest (record the analysis seed + provenance)
32	python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" manifest data.csv \
33	--out manifest.json --seed 42 --provenance "KNHANES 2018 extract v1"
34
35	# Verify a later copy against it (CI / pre-analysis gate)
36	python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" verify --manifest manifest.json --strict
37
38	# Compare two manifests (what changed between versions)
39	python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" diff --old v1.json --new v2.json
40	```
41
42	File hashing is stdlib-only; tabular schema/column hashing uses pandas when present.
43	`--ignore-cols` excludes volatile columns; `--base` makes manifest keys relative.
44
45	## Workflow
46
47	### Step 1: Lock the version (gate)
48
49	Build the manifest at the moment the dataset is frozen for analysis. Gate:
50	confirm with the user the seed and provenance note are correct before locking —
51	the manifest is the record they will cite as "this is the data the results came from."
52
53	### Step 2: Verify before each run (gate)
54
55	Before re-running an analysis (or in CI), `verify --strict`. Gate: if drift is
56	reported, stop and show the user the drift report; do not proceed on changed data
57	without their explicit acknowledgement and a re-lock. Silent re-run on drifted data
58	is the failure this skill exists to prevent.
59
60	### Step 3: Diff across versions
61
62	When a dataset is intentionally updated, `diff` the old and new manifests and
63	present the change set (added/removed/changed columns, row-count delta) so the
64	user can record what changed and re-lock. Gate: the user approves the new
65	version before it replaces the locked one.
66
67	## Non-Deterministic Artifacts
68
69	Some outputs (PPTX/DOCX with embedded timestamps, figures with render metadata)
70	change byte-for-byte on every build even when the analysis is identical. Do not
71	put these under strict byte verification — manifest only the deterministic inputs
72	and tabular outputs (data files, result CSVs), or use `--ignore-cols` for volatile
73	columns. See references for the policy.
74
75	## Scope Limitations
76
77	### Supported
78	- Content-hash manifest of any file; schema + per-column hashes for tabular files
79	(CSV/TSV/Parquet/Stata/SAS/Excel).
80	- Drift verification and manifest-to-manifest diff.
81
82	### NOT Supported
83	- Storing or transmitting the data itself (manifests hold hashes, not contents).
84	- Cleaning, profiling, or de-identifying — use `/clean-data`, `/generate-codebook`, `/deidentify`.
85	- Full pipeline-output reproducibility for non-deterministic binaries (see above).
86
87	## Cross-Skill Integration
88
89	- /generate-codebook documents what is in the data; version-dataset locks which version.
90	- /deidentify should run before a manifest is shared (example values are not stored, but provenance notes may carry context).
91	- Demo reproducibility: each bundled `demo/*/` carries a `manifest.lock.json` (input data + deterministic result tables) that `verify --strict` checks.
92
93	## Worked Example
94
95	Lock a freshly-frozen extract:
96
97	```bash
98	python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" manifest cohort.csv \
99	--out manifest.json --se

People who liked this also viewed

6 picks

Type