literature-engineer

Name: literature-engineer
Author: WILLOSCAR

489stars39forks

How to install literature-engineer?

willoscar/research-units-pipeline-skills/literature-engineer

$npx -y skills add WILLOSCAR/research-units-pipeline-skills --skill literature-engineer

Installs into the current project.

Summary

Multi-route literature expansion + metadata normalization for evidence-first surveys. Trigger: evidence collector, literature engineer, 文献扩充, 多路召回, snowballing, cited by, references, 元信息增强, provenance. Use when: Workflow 需要按锁定的 retrieval policy 扩充候选文献并补齐可追溯 metadata。

Files · 1

View on GitHub

SKILL.md

1	# Literature Engineer (evidence collector)
2
3	Goal: build a large, verifiable candidate pool for downstream dedupe/rank, mapping, notes, citations, and drafting.
4
5	This skill is intentionally evidence-first: if you can't reach the target size with verifiable IDs/provenance, the correct behavior is to block and ask for more exports / enable network, not to fabricate.
6
7	## Load Order
8
9	Always read:
10	- `references/domain_pack_overview.md` — how domain packs drive topic-specific behavior
11
12	Domain packs (loaded by topic match):
13	- `assets/domain_packs/llm_agents.json` — pinned classic/survey arXiv IDs for LLM agent topics
14
15	## Script Boundary
16
17	Use `scripts/run.py` only for:
18	- multi-route offline import, normalization, and provenance tagging
19	- online arXiv/Semantic Scholar API retrieval
20	- snowball expansion and deduplication
21	- retrieval report generation
22
23	Do not treat `run.py` as the place for:
24	- hardcoded pinned arXiv ID lists (use domain packs)
25	- hardcoded topic detection logic (use domain packs)
26
27	## Inputs
28
29	- `queries.md`
30	- `keywords`, `exclude`, `max_results`, `time window`
31	- Optional offline sources (any combination; all are merged):
32	- `papers/import.(csv\|json\|jsonl\|bib)`
33	- `papers/arxiv_export.(csv\|json\|jsonl\|bib)`
34	- `papers/imports/*.(csv\|json\|jsonl\|bib)`
35	- Optional snowball exports (offline):
36	- `papers/snowball/*.(csv\|json\|jsonl\|bib)`
37
38	## Outputs
39
40	- `papers/papers_raw.jsonl`
41	- 1 record per line; minimum fields:
42	- `title` (str), `authors` (list[str]), `year` (int\|""), `url` (str)
43	- stable identifier(s): `arxiv_id` and/or `doi`
44	- `abstract` (str; may be empty in offline mode)
45	- `source` (str) + `provenance` (list[dict])
46	- `papers/papers_raw.csv` (human scan)
47	- `papers/retrieval_report.md` (route counts, missing-meta stats, next actions)
48
49	## Workflow (multi-route)
50
51	1. Offline-first merge: ingest all available offline exports (and label provenance per file).
52	2. Online retrieval (optional): if enabled, run arXiv API retrieval for each keyword query.
53	3. Snowballing (optional): expand from seed papers via references/cited-by (online), or merge offline snowball exports.
54	4. Normalize + dedupe: canonicalize IDs/URLs, merge duplicates while unioning `provenance`.
55	5. Report: write a concise retrieval report with coverage buckets and missing-meta counts.
56
57	## Quality checklist
58
59	- [ ] Candidate pool meets the active Workflow's declared retrieval floor without fabrication. For Workflows with `retrieval_policy.minimum_records`, use that value; survey profiles may instead derive a stricter pool target from `core_size`.
60	- [ ] Each record has a stable identifier (`arxiv_id` or `doi`, plus `url`).
61	- [ ] Each record has provenance: which route/file/API produced it.
62
63	## Script
64
65	### Quick Start
66
67	- `uv run python .codex/skills/literature-engineer/scripts/run.py --help`
68
69
70	### All Options
71
72	- See `uv run python .codex/skills/literature-engineer/scripts/run.py --help`.
73	- Reads retrieval config from `queries.md`.
74	- Offline inputs (merged if present): `papers/import.(csv\|json\|jsonl\|bib)`, `papers/arxiv_export.(csv\|json\|jsonl\|bib)`, `papers/imports/*.(csv\|json\|jsonl\|bib)`.
75	- Optional offline snowball inputs: `papers/snowball/*.(csv\|json\|jsonl\|bib)`.
76	- Online expansion requires network: use `--online` and/or `--snowball`.
77	- Online retrieval is best-effort: arXiv API can be flaky in some environments; the script will also attempt a Semantic Scholar route when needed.
78	- For LLM-agent topics, the script also performs a best-effort pinned arXiv id_list fetch (canonical classics like ReAct/Toolformer/Reflexion/Voyager/Tree-of-Thoughts + a small prior-survey seed set) so `ref.bib` can include must-cite anchors even when keyword search misses them.
79	- If HTTPS/TLS to external domains is unstable, the Semantic Scholar route is fetched via the `r.jina.ai` proxy so the pipeline can still self-boot without manual exports.
80	- When an online run returns `0` records due to transient network errors, a simple rerun is often sufficient (the pipeline should not fabricate).
81
82
83	### Examples
84
85	- Offline imports only:
86	- Put exports under `papers/imports/` then run:
87	- `uv run python .codex/skills/literature-engineer/scripts/run.py --workspace <workspace>`
88
89	- Explicit offline inputs (multi-route):
90	- `uv run python .codex/skills/literature-engineer/scripts/run.py --workspace <workspace> --input path/to/a.bib --input path/to/b.jsonl`
91
92	- Online arXiv retrieval (needs network):
93	- `uv run python .codex/skills/literature-engineer/scripts/run.py --workspace <workspace> --online`
94
95	- Snowballing (needs network unless you provide offline snowball exports):
96	- `uv run python .codex

People who liked this also viewed

6 picks

Type

literature-engineer

489stars39forks

View on GitHub

How to install literature-engineer?

willoscar/research-units-pipeline-skills/literature-engineer

$npx -y skills add WILLOSCAR/research-units-pipeline-skills --skill literature-engineer

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Literature Engineer (evidence collector)
2
3	Goal: build a large, verifiable candidate pool for downstream dedupe/rank, mapping, notes, citations, and drafting.
4
5	This skill is intentionally evidence-first: if you can't reach the target size with verifiable IDs/provenance, the correct behavior is to block and ask for more exports / enable network, not to fabricate.
6
7	## Load Order
8
9	Always read:
10	- `references/domain_pack_overview.md` — how domain packs drive topic-specific behavior
11
12	Domain packs (loaded by topic match):
13	- `assets/domain_packs/llm_agents.json` — pinned classic/survey arXiv IDs for LLM agent topics
14
15	## Script Boundary
16
17	Use `scripts/run.py` only for:
18	- multi-route offline import, normalization, and provenance tagging
19	- online arXiv/Semantic Scholar API retrieval
20	- snowball expansion and deduplication
21	- retrieval report generation
22
23	Do not treat `run.py` as the place for:
24	- hardcoded pinned arXiv ID lists (use domain packs)
25	- hardcoded topic detection logic (use domain packs)
26
27	## Inputs
28
29	- `queries.md`
30	- `keywords`, `exclude`, `max_results`, `time window`
31	- Optional offline sources (any combination; all are merged):
32	- `papers/import.(csv\|json\|jsonl\|bib)`
33	- `papers/arxiv_export.(csv\|json\|jsonl\|bib)`
34	- `papers/imports/*.(csv\|json\|jsonl\|bib)`
35	- Optional snowball exports (offline):
36	- `papers/snowball/*.(csv\|json\|jsonl\|bib)`
37
38	## Outputs
39
40	- `papers/papers_raw.jsonl`
41	- 1 record per line; minimum fields:
42	- `title` (str), `authors` (list[str]), `year` (int\|""), `url` (str)
43	- stable identifier(s): `arxiv_id` and/or `doi`
44	- `abstract` (str; may be empty in offline mode)
45	- `source` (str) + `provenance` (list[dict])
46	- `papers/papers_raw.csv` (human scan)
47	- `papers/retrieval_report.md` (route counts, missing-meta stats, next actions)
48
49	## Workflow (multi-route)
50
51	1. Offline-first merge: ingest all available offline exports (and label provenance per file).
52	2. Online retrieval (optional): if enabled, run arXiv API retrieval for each keyword query.
53	3. Snowballing (optional): expand from seed papers via references/cited-by (online), or merge offline snowball exports.
54	4. Normalize + dedupe: canonicalize IDs/URLs, merge duplicates while unioning `provenance`.
55	5. Report: write a concise retrieval report with coverage buckets and missing-meta counts.
56
57	## Quality checklist
58
59	- [ ] Candidate pool meets the active Workflow's declared retrieval floor without fabrication. For Workflows with `retrieval_policy.minimum_records`, use that value; survey profiles may instead derive a stricter pool target from `core_size`.
60	- [ ] Each record has a stable identifier (`arxiv_id` or `doi`, plus `url`).
61	- [ ] Each record has provenance: which route/file/API produced it.
62
63	## Script
64
65	### Quick Start
66
67	- `uv run python .codex/skills/literature-engineer/scripts/run.py --help`
68
69
70	### All Options
71
72	- See `uv run python .codex/skills/literature-engineer/scripts/run.py --help`.
73	- Reads retrieval config from `queries.md`.
74	- Offline inputs (merged if present): `papers/import.(csv\|json\|jsonl\|bib)`, `papers/arxiv_export.(csv\|json\|jsonl\|bib)`, `papers/imports/*.(csv\|json\|jsonl\|bib)`.
75	- Optional offline snowball inputs: `papers/snowball/*.(csv\|json\|jsonl\|bib)`.
76	- Online expansion requires network: use `--online` and/or `--snowball`.
77	- Online retrieval is best-effort: arXiv API can be flaky in some environments; the script will also attempt a Semantic Scholar route when needed.
78	- For LLM-agent topics, the script also performs a best-effort pinned arXiv id_list fetch (canonical classics like ReAct/Toolformer/Reflexion/Voyager/Tree-of-Thoughts + a small prior-survey seed set) so `ref.bib` can include must-cite anchors even when keyword search misses them.
79	- If HTTPS/TLS to external domains is unstable, the Semantic Scholar route is fetched via the `r.jina.ai` proxy so the pipeline can still self-boot without manual exports.
80	- When an online run returns `0` records due to transient network errors, a simple rerun is often sufficient (the pipeline should not fabricate).
81
82
83	### Examples
84
85	- Offline imports only:
86	- Put exports under `papers/imports/` then run:
87	- `uv run python .codex/skills/literature-engineer/scripts/run.py --workspace <workspace>`
88
89	- Explicit offline inputs (multi-route):
90	- `uv run python .codex/skills/literature-engineer/scripts/run.py --workspace <workspace> --input path/to/a.bib --input path/to/b.jsonl`
91
92	- Online arXiv retrieval (needs network):
93	- `uv run python .codex/skills/literature-engineer/scripts/run.py --workspace <workspace> --online`
94
95	- Snowballing (needs network unless you provide offline snowball exports):
96	- `uv run python .codex

People who liked this also viewed

6 picks

Type