pdf-text-extractor

Name: pdf-text-extractor
Author: WILLOSCAR

489stars39forks

How to install pdf-text-extractor?

willoscar/research-units-pipeline-skills/pdf-text-extractor

$npx -y skills add WILLOSCAR/research-units-pipeline-skills --skill pdf-text-extractor

Installs into the current project.

Summary

Download PDFs (when available) and extract plain text to support full-text evidence, writing papers/fulltext_index.jsonl and papers/fulltext/*.txt. Trigger: PDF download, fulltext, extract text, papers/pdfs, 全文抽取, 下载PDF. Use when: queries.md 设置 `evidence_mode: fullt

Files · 1

View on GitHub

SKILL.md

1	# PDF Text Extractor
2
3	Optionally collect full-text snippets to deepen evidence beyond abstracts.
4
5	This skill is intentionally conservative: in many survey runs, abstract/snippet mode is enough and avoids heavy downloads.
6
7	## Inputs
8
9	- `papers/core_set.csv` (expects `paper_id`, `title`, and ideally `pdf_url`/`arxiv_id`/`url`)
10	- Optional: `outline/mapping.tsv` (to prioritize mapped papers)
11
12	## Outputs
13
14	- `papers/fulltext_index.jsonl` (one record per attempted paper)
15	- Side artifacts:
16	- `papers/pdfs/<paper_id>.pdf` (cached downloads)
17	- `papers/fulltext/<paper_id>.txt` (extracted text)
18
19	## Decision: evidence mode
20
21	- `queries.md` can set `evidence_mode: "abstract" \| "fulltext"`.
22	- `abstract` (default template): do not download; write an index that clearly records skipping.
23	- `fulltext`: download PDFs (when possible) and extract text to `papers/fulltext/`.
24
25	## Local PDFs Mode
26
27	When you cannot/should not download PDFs (restricted network, rate limits, no permission), provide PDFs manually and run in “local PDFs only” mode.
28
29	- PDF naming convention: `papers/pdfs/<paper_id>.pdf` where `<paper_id>` matches `papers/core_set.csv`.
30	- Set `- evidence_mode: "fulltext"` in `queries.md`.
31	- Run: `uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace> --local-pdfs-only`
32
33	If PDFs are missing, the script writes a to-do list:
34
35	- `output/MISSING_PDFS.md` (human-readable summary)
36	- `papers/missing_pdfs.csv` (machine-readable list)
37
38	## Workflow (heuristic)
39
40	1. Read `papers/core_set.csv`.
41	2. If `outline/mapping.tsv` exists, prioritize mapped papers first.
42	3. For each selected paper (fulltext mode):
43	- resolve `pdf_url` (use `pdf_url`, else derive from `arxiv_id`/`url` when possible)
44	- download to `papers/pdfs/<paper_id>.pdf` if missing
45	- extract a reasonable prefix of text to `papers/fulltext/<paper_id>.txt`
46	- append/update a JSONL record in `papers/fulltext_index.jsonl` with status + stats
47	4. Never overwrite existing extracted text unless explicitly requested (delete the `.txt` to re-extract).
48
49	## Quality checklist
50
51	- [ ] `papers/fulltext_index.jsonl` exists and is non-empty.
52	- [ ] If `evidence_mode: "fulltext"`: at least a small but non-trivial subset has extracted text (strict mode blocks if extraction coverage is near-zero).
53	- [ ] If `evidence_mode: "abstract"`: the index covers every `papers/core_set.csv` paper and every record clearly reflects `skip_mode_abstract` (no downloads attempted). `fulltext_max_papers` does not truncate this zero-download index.
54
55	## Script
56
57	### Quick Start
58
59	- `uv run python .codex/skills/pdf-text-extractor/scripts/run.py --help`
60	- `uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace>`
61
62	### All Options
63
64	- `--max-papers <n>`: cap number of papers processed (can be overridden by `queries.md`)
65	- `--max-pages <n>`: extract at most N pages per PDF
66	- `--min-chars <n>`: minimum extracted chars to count as OK
67	- `--sleep <sec>`: delay between downloads
68	- `--local-pdfs-only`: do not download; only use `papers/pdfs/<paper_id>.pdf` if present
69	- `queries.md` supports: `evidence_mode`, `fulltext_max_papers`, `fulltext_max_pages`, `fulltext_min_chars`
70
71	### Examples
72
73	- Abstract mode (no downloads):
74	- Set `- evidence_mode: "abstract"` in `queries.md`, then run the script (it will emit `papers/fulltext_index.jsonl` with skip statuses)
75	- Fulltext mode with local PDFs only:
76	- Set `- evidence_mode: "fulltext"` in `queries.md`, put PDFs under `papers/pdfs/`, then run: `uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace> --local-pdfs-only`
77	- Fulltext mode with smaller budget:
78	- `uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace> --max-papers 20 --max-pages 4 --min-chars 1200`
79
80	### Notes
81
82	- Downloads are cached under `papers/pdfs/`; extracted text is cached under `papers/fulltext/`.
83	- The script does not overwrite existing extracted text unless you delete the `.txt` file.
84
85	## Troubleshooting
86
87	### Issue: no PDFs are available to download
88
89	Fix:
90	- Use `evidence_mode: abstract` (default) or provide local PDFs under `papers/pdfs/` and rerun with `--local-pdfs-only`.
91
92	### Issue: extracted text is empty/garbled
93
94	Fix:
95	- Try a different extraction backend if supported; otherwise mark the paper as `abstract` evidence level and avoid strong fulltext claims.

People who liked this also viewed

6 picks

Type

pdf-text-extractor

489stars39forks

View on GitHub

How to install pdf-text-extractor?

willoscar/research-units-pipeline-skills/pdf-text-extractor

$npx -y skills add WILLOSCAR/research-units-pipeline-skills --skill pdf-text-extractor

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# PDF Text Extractor
2
3	Optionally collect full-text snippets to deepen evidence beyond abstracts.
4
5	This skill is intentionally conservative: in many survey runs, abstract/snippet mode is enough and avoids heavy downloads.
6
7	## Inputs
8
9	- `papers/core_set.csv` (expects `paper_id`, `title`, and ideally `pdf_url`/`arxiv_id`/`url`)
10	- Optional: `outline/mapping.tsv` (to prioritize mapped papers)
11
12	## Outputs
13
14	- `papers/fulltext_index.jsonl` (one record per attempted paper)
15	- Side artifacts:
16	- `papers/pdfs/<paper_id>.pdf` (cached downloads)
17	- `papers/fulltext/<paper_id>.txt` (extracted text)
18
19	## Decision: evidence mode
20
21	- `queries.md` can set `evidence_mode: "abstract" \| "fulltext"`.
22	- `abstract` (default template): do not download; write an index that clearly records skipping.
23	- `fulltext`: download PDFs (when possible) and extract text to `papers/fulltext/`.
24
25	## Local PDFs Mode
26
27	When you cannot/should not download PDFs (restricted network, rate limits, no permission), provide PDFs manually and run in “local PDFs only” mode.
28
29	- PDF naming convention: `papers/pdfs/<paper_id>.pdf` where `<paper_id>` matches `papers/core_set.csv`.
30	- Set `- evidence_mode: "fulltext"` in `queries.md`.
31	- Run: `uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace> --local-pdfs-only`
32
33	If PDFs are missing, the script writes a to-do list:
34
35	- `output/MISSING_PDFS.md` (human-readable summary)
36	- `papers/missing_pdfs.csv` (machine-readable list)
37
38	## Workflow (heuristic)
39
40	1. Read `papers/core_set.csv`.
41	2. If `outline/mapping.tsv` exists, prioritize mapped papers first.
42	3. For each selected paper (fulltext mode):
43	- resolve `pdf_url` (use `pdf_url`, else derive from `arxiv_id`/`url` when possible)
44	- download to `papers/pdfs/<paper_id>.pdf` if missing
45	- extract a reasonable prefix of text to `papers/fulltext/<paper_id>.txt`
46	- append/update a JSONL record in `papers/fulltext_index.jsonl` with status + stats
47	4. Never overwrite existing extracted text unless explicitly requested (delete the `.txt` to re-extract).
48
49	## Quality checklist
50
51	- [ ] `papers/fulltext_index.jsonl` exists and is non-empty.
52	- [ ] If `evidence_mode: "fulltext"`: at least a small but non-trivial subset has extracted text (strict mode blocks if extraction coverage is near-zero).
53	- [ ] If `evidence_mode: "abstract"`: the index covers every `papers/core_set.csv` paper and every record clearly reflects `skip_mode_abstract` (no downloads attempted). `fulltext_max_papers` does not truncate this zero-download index.
54
55	## Script
56
57	### Quick Start
58
59	- `uv run python .codex/skills/pdf-text-extractor/scripts/run.py --help`
60	- `uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace>`
61
62	### All Options
63
64	- `--max-papers <n>`: cap number of papers processed (can be overridden by `queries.md`)
65	- `--max-pages <n>`: extract at most N pages per PDF
66	- `--min-chars <n>`: minimum extracted chars to count as OK
67	- `--sleep <sec>`: delay between downloads
68	- `--local-pdfs-only`: do not download; only use `papers/pdfs/<paper_id>.pdf` if present
69	- `queries.md` supports: `evidence_mode`, `fulltext_max_papers`, `fulltext_max_pages`, `fulltext_min_chars`
70
71	### Examples
72
73	- Abstract mode (no downloads):
74	- Set `- evidence_mode: "abstract"` in `queries.md`, then run the script (it will emit `papers/fulltext_index.jsonl` with skip statuses)
75	- Fulltext mode with local PDFs only:
76	- Set `- evidence_mode: "fulltext"` in `queries.md`, put PDFs under `papers/pdfs/`, then run: `uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace> --local-pdfs-only`
77	- Fulltext mode with smaller budget:
78	- `uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace> --max-papers 20 --max-pages 4 --min-chars 1200`
79
80	### Notes
81
82	- Downloads are cached under `papers/pdfs/`; extracted text is cached under `papers/fulltext/`.
83	- The script does not overwrite existing extracted text unless you delete the `.txt` file.
84
85	## Troubleshooting
86
87	### Issue: no PDFs are available to download
88
89	Fix:
90	- Use `evidence_mode: abstract` (default) or provide local PDFs under `papers/pdfs/` and rerun with `--local-pdfs-only`.
91
92	### Issue: extracted text is empty/garbled
93
94	Fix:
95	- Try a different extraction backend if supported; otherwise mark the paper as `abstract` evidence level and avoid strong fulltext claims.

People who liked this also viewed

6 picks

Type