find-data

Name: find-data
Author: QinghongLin

146stars19forks

How to install find-data?

qinghonglin/data2story-skill/find-data

$npx -y skills add QinghongLin/data2story-skill --skill find-data

Installs into the current project.

Summary

Find a dataset for a Data2Story blog. Accepts a topic, a URL, or a DIP-style category. Downloads + validates against 4 completeness gates before handing off to /data2story-pro. Local-first: searches Economist/Pudding/TidyTuesday clones before going online. Supports --validate-onl

Files · 1

View on GitHub

SKILL.md

1	# find-data
2
3	Turn an idea, URL, or category into a `phase2/datasets/<name>/` folder
4	that the `/data2story-pro` pipeline can run on without crashing.
5
6	You are the gatekeeper before the 7-agent newsroom. Detective, Analyst,
7	Editor, Designer, Programmer, Auditor, Inspector all assume the data is
8	already there, parseable, and provenanced. Your job is to make sure that's
9	true before they start.
10
11	Refuse to mark a folder ready until it passes 4 gates. See
12	`references/completeness_gates.md` for the criteria. The gates are codified
13	in `tools/audit.py`, not in this prose.
14
15	## Prerequisites
16
17	Python deps for the `tools/`:
18
19	- `pandas` — required (`audit.py` reads/inspects CSV/JSON).
20	- `openpyxl` — required only when a source is `.xlsx` (`audit.py`'s
21	`pd.ExcelFile`); a mid-run missing-`openpyxl` is the usual cause of an XLSX
22	audit error.
23	- `urllib` — stdlib, no install (`fetch.py` downloads, `audit.py` HEAD-checks).
24
25	One install line:
26
27	```bash
28	pip install pandas openpyxl
29	```
30
31	After editing anything under `tools/`, run the no-network regression suite:
32	`py tools/selftest.py` (exit 0 = pass).
33
34	## Resolve paths first
35
36	- `SKILL_DIR` = directory containing this `SKILL.md`
37	- `WORKSPACE` = ancestor that contains `phase2/datasets/` (the parent repo root), if one exists
38	- `DATASETS_ROOT` = `WORKSPACE/phase2/datasets` when `WORKSPACE` exists; otherwise `./datasets`
39	(clone-relative, under the current working dir) — the open-source clone has no `phase2/datasets`
40	- `BLOGS_ROOT` = `WORKSPACE/phase2/blogs` when `WORKSPACE` exists
41	- `OUT_DIR` default = `DATASETS_ROOT/<name>` (so `./datasets/<name>` on an OSS clone). Callers may
42	override with an explicit `--out`, which always wins.
43	- `INPUT` = first positional argument from `$ARGUMENTS`
44	- Parse flags from `$ARGUMENTS`: `--mode`, `--source`, `--out`, `--validate-only`
45
46	Never hard-code machine paths. Resolve at runtime by walking up from `SKILL_DIR`.
47
48	## Step 0 — Classify the input
49
50	Decision tree (the FIRST condition that matches wins):
51
52	1. `--validate-only` flag present → MODE = `validate-only`. `INPUT` must be an existing folder path.
53	2. `INPUT` is an existing folder path (and `--validate-only` absent) → still ask the user before re-fetching; default to `validate-only` behaviour with a confirmation.
54	3. `INPUT` starts with `http://` or `https://` → MODE = `url`.
55	4. `INPUT` (case-insensitive) exactly matches a DIP category from `phase2/datasets/data_is_plural/dip_category_summary.md` → MODE = `category`.
56	5. Otherwise → MODE = `topic`.
57
58	State your classification out loud in one line before doing anything else.
59
60	No local corpora (open-source clone): on a machine without the `Economist/`, `Pudding/`,
61	`tidytuesday/` clones and without the Data-is-Plural CSV under `phase2/datasets/`, `browse_local.py`
62	and `dip_query.py` both return empty (no crash). In that case there is nothing to match locally, so
63	`topic`/`category` discovery degrades straight to WebSearch (Step 1). No flag is needed — detect it
64	from the empty `browse_local.py` + `dip_query.py` results. Keep the local-first path intact for
65	machines that DO have the corpora.
66
67	## Step 1 — Branch on mode
68
69	### validate-only mode
70
71	Just audit. Skip discovery, skip fetch.
72
73	```bash
74	python "SKILL_DIR/tools/audit.py" "<folder>"
75	```
76
77	This writes `<folder>/validate.json` and prints the gate summary. Read it back
78	with `Read`, surface the verdict, and stop.
79
80	If `overall.ready_for_data2story` is true → print the `/data2story-pro <folder>`
81	command for the user to copy.
82
83	If it's false → list the specific gate failures and recommend remediations.
84
85	### url mode
86
87	Determine OUT_DIR (use `--out` if given, else derive a slug from the URL path's
88	last meaningful segment, store under `DATASETS_ROOT/<slug>/` — i.e. `./datasets/<slug>/`
89	on an open-source clone with no `phase2/datasets`).
90
91	Branch on URL shape:
92
93	\| URL pattern \| Tool \|
94	\|---\|---\|
95	\| `github.com/{owner}/{repo}/tree/{branch}/{path}` \| `python fetch.py github-folder <url> <out_dir>` \|
96	\| `github.com/{owner}/{repo}/blob/{branch}/{path}` \| Same (fetch.py auto-detects blob) \|
97	\| `github.com/{owner}/{repo}` with no path \| Refuse — tell user to narrow to a folder \|
98	\| Anything else \| `python fetch.py url <url> <out_dir>` \|
99
100	`fetch.py` is source-aware: if the URL points to `TheEconomist/graphic-detail-data`,
101	`the-pudding/data`, or `rfordatascience/tidytuesday` AND

People who liked this also viewed

6 picks

Type

find-data

146stars19forks

View on GitHub

How to install find-data?

qinghonglin/data2story-skill/find-data

$npx -y skills add QinghongLin/data2story-skill --skill find-data

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# find-data
2
3	Turn an idea, URL, or category into a `phase2/datasets/<name>/` folder
4	that the `/data2story-pro` pipeline can run on without crashing.
5
6	You are the gatekeeper before the 7-agent newsroom. Detective, Analyst,
7	Editor, Designer, Programmer, Auditor, Inspector all assume the data is
8	already there, parseable, and provenanced. Your job is to make sure that's
9	true before they start.
10
11	Refuse to mark a folder ready until it passes 4 gates. See
12	`references/completeness_gates.md` for the criteria. The gates are codified
13	in `tools/audit.py`, not in this prose.
14
15	## Prerequisites
16
17	Python deps for the `tools/`:
18
19	- `pandas` — required (`audit.py` reads/inspects CSV/JSON).
20	- `openpyxl` — required only when a source is `.xlsx` (`audit.py`'s
21	`pd.ExcelFile`); a mid-run missing-`openpyxl` is the usual cause of an XLSX
22	audit error.
23	- `urllib` — stdlib, no install (`fetch.py` downloads, `audit.py` HEAD-checks).
24
25	One install line:
26
27	```bash
28	pip install pandas openpyxl
29	```
30
31	After editing anything under `tools/`, run the no-network regression suite:
32	`py tools/selftest.py` (exit 0 = pass).
33
34	## Resolve paths first
35
36	- `SKILL_DIR` = directory containing this `SKILL.md`
37	- `WORKSPACE` = ancestor that contains `phase2/datasets/` (the parent repo root), if one exists
38	- `DATASETS_ROOT` = `WORKSPACE/phase2/datasets` when `WORKSPACE` exists; otherwise `./datasets`
39	(clone-relative, under the current working dir) — the open-source clone has no `phase2/datasets`
40	- `BLOGS_ROOT` = `WORKSPACE/phase2/blogs` when `WORKSPACE` exists
41	- `OUT_DIR` default = `DATASETS_ROOT/<name>` (so `./datasets/<name>` on an OSS clone). Callers may
42	override with an explicit `--out`, which always wins.
43	- `INPUT` = first positional argument from `$ARGUMENTS`
44	- Parse flags from `$ARGUMENTS`: `--mode`, `--source`, `--out`, `--validate-only`
45
46	Never hard-code machine paths. Resolve at runtime by walking up from `SKILL_DIR`.
47
48	## Step 0 — Classify the input
49
50	Decision tree (the FIRST condition that matches wins):
51
52	1. `--validate-only` flag present → MODE = `validate-only`. `INPUT` must be an existing folder path.
53	2. `INPUT` is an existing folder path (and `--validate-only` absent) → still ask the user before re-fetching; default to `validate-only` behaviour with a confirmation.
54	3. `INPUT` starts with `http://` or `https://` → MODE = `url`.
55	4. `INPUT` (case-insensitive) exactly matches a DIP category from `phase2/datasets/data_is_plural/dip_category_summary.md` → MODE = `category`.
56	5. Otherwise → MODE = `topic`.
57
58	State your classification out loud in one line before doing anything else.
59
60	No local corpora (open-source clone): on a machine without the `Economist/`, `Pudding/`,
61	`tidytuesday/` clones and without the Data-is-Plural CSV under `phase2/datasets/`, `browse_local.py`
62	and `dip_query.py` both return empty (no crash). In that case there is nothing to match locally, so
63	`topic`/`category` discovery degrades straight to WebSearch (Step 1). No flag is needed — detect it
64	from the empty `browse_local.py` + `dip_query.py` results. Keep the local-first path intact for
65	machines that DO have the corpora.
66
67	## Step 1 — Branch on mode
68
69	### validate-only mode
70
71	Just audit. Skip discovery, skip fetch.
72
73	```bash
74	python "SKILL_DIR/tools/audit.py" "<folder>"
75	```
76
77	This writes `<folder>/validate.json` and prints the gate summary. Read it back
78	with `Read`, surface the verdict, and stop.
79
80	If `overall.ready_for_data2story` is true → print the `/data2story-pro <folder>`
81	command for the user to copy.
82
83	If it's false → list the specific gate failures and recommend remediations.
84
85	### url mode
86
87	Determine OUT_DIR (use `--out` if given, else derive a slug from the URL path's
88	last meaningful segment, store under `DATASETS_ROOT/<slug>/` — i.e. `./datasets/<slug>/`
89	on an open-source clone with no `phase2/datasets`).
90
91	Branch on URL shape:
92
93	\| URL pattern \| Tool \|
94	\|---\|---\|
95	\| `github.com/{owner}/{repo}/tree/{branch}/{path}` \| `python fetch.py github-folder <url> <out_dir>` \|
96	\| `github.com/{owner}/{repo}/blob/{branch}/{path}` \| Same (fetch.py auto-detects blob) \|
97	\| `github.com/{owner}/{repo}` with no path \| Refuse — tell user to narrow to a folder \|
98	\| Anything else \| `python fetch.py url <url> <out_dir>` \|
99
100	`fetch.py` is source-aware: if the URL points to `TheEconomist/graphic-detail-data`,
101	`the-pudding/data`, or `rfordatascience/tidytuesday` AND

People who liked this also viewed

6 picks

Type