arxiv-search

Name: arxiv-search
Author: WILLOSCAR

51installs39forks

How to install arxiv-search?

willoscar/research-units-pipeline-skills/arxiv-search

$npx -y skills add WILLOSCAR/research-units-pipeline-skills --skill arxiv-search

Installs into the current project.

Summary

Retrieve paper metadata from arXiv using keyword queries and save results as JSONL (papers/papers_raw.jsonl). Trigger: arXiv, arxiv, paper search, metadata retrieval, 文献检索, 论文检索, 拉取元数据, 离线导入. Use when: 需要一个初始论文集合（survey/snapshot 的 Stage C1），来源为 arXiv（在线检索或离线导入 export）。

Files · 1

View on GitHub

SKILL.md

1	# arXiv Search (metadata-first)
2
3	Collect an initial paper set with enough metadata to support downstream ranking, taxonomy building, and citation generation.
4
5	When online, prefer rich arXiv metadata (categories, arxiv_id, pdf_url, published/updated, etc.). When offline, accept an export and convert it cleanly.
6
7	## Load Order
8
9	Always read:
10	- `references/domain_pack_overview.md` — how domain packs drive topic-specific behavior
11
12	Domain packs (loaded by topic match):
13	- `assets/domain_packs/llm_agents.json` — pinned IDs, query rewrite rules for LLM agent topics
14
15	## Script Boundary
16
17	Use `scripts/run.py` only for:
18	- arXiv API retrieval and XML parsing
19	- offline export conversion (CSV/JSON/JSONL normalization)
20	- metadata enrichment via `id_list` backfill
21
22	Do not treat `run.py` as the place for:
23	- hardcoded topic detection or query rewriting (use domain packs)
24	- domain-specific pinned paper lists (externalize to `assets/domain_packs/`)
25
26	## Contract-driven behavior
27
28	- Domain-pack query rewriting is the default for broad discovery Workflows.
29	- A focused Workflow may set
30	`quality_contract.retrieval_policy.domain_pack_query_mode: explicit`; in that
31	mode, the query list in `queries.md` remains authoritative and the domain
32	pack must not replace its topic focus.
33	- `quality_contract.retrieval_policy.minimum_records` turns a Workflow's raw
34	candidate-pool floor into a strict quality-gate check.
35
36	## Input
37
38	- `queries.md` (keywords, excludes, time window)
39
40	## Outputs
41
42	- `papers/papers_raw.jsonl` (JSONL; 1 paper per line)
43	- Each record includes at least: `title`, `authors`, `year`, `url`, `abstract`
44	- When using the arXiv API online mode, records also include helpful metadata: `arxiv_id`, `pdf_url`, `categories`, `primary_category`, `published`, `updated`, `doi`, `journal_ref`, `comment`
45	- Convenience index (optional but generated by the script):
46	- `papers/papers_raw.csv`
47
48	## Decision: online vs offline
49
50	- If you have network access: run arXiv API retrieval.
51	- If not: import an export the user provides (CSV/JSON/JSONL) and normalize fields.
52	- Hybrid: if you import offline but still have network later, you can enrich missing fields (abstract/authors/categories) via arXiv `id_list` using `--enrich-metadata` or `queries.md` `enrich_metadata: true`.
53
54	## Workflow (heuristic)
55
56	1. Read `queries.md` and expand into concrete query strings.
57	2. Retrieve results (online) or import an export (offline).
58	3. Normalize every record to include at least:
59	- `title`, `authors` (array), `year`, `url`, `abstract`
60	4. Keep the set broad at this stage; dedupe/ranking comes next.
61	5. Apply time window and `max_results` if specified.
62
63	## Quality checklist
64
65	- [ ] `papers/papers_raw.jsonl` exists.
66	- [ ] Each line is valid JSON and contains `title`, `authors`, `year`, `url`.
67
68	## Side effects
69
70	- Allowed: create/overwrite `papers/papers_raw.jsonl`; append notes to `STATUS.md`.
71	- Not allowed: write prose sections in `output/` before writing is approved.
72
73	## Script
74
75	### Quick Start
76
77	- `uv run python .codex/skills/arxiv-search/scripts/run.py --help`
78	- Online: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --query "<query>" --max-results 200`
79	- Offline import: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --input <export.csv\|json\|jsonl>`
80
81	### All Options
82
83	- `--query <q>`: repeatable; multiple queries are unioned
84	- `--exclude <term>`: repeatable; excludes applied after retrieval
85	- `--max-results <n>`: cap total retrieved
86	- `--input <export.*>`: offline mode (CSV/JSON/JSONL)
87	- `--enrich-metadata`: best-effort enrich via arXiv `id_list` (needs network)
88	- `queries.md` also supports: `keywords`, `exclude`, `time window`, `max_results`, `enrich_metadata`
89
90	### Examples
91
92	- Online (multi-query + excludes):
93	- `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --query "LLM agent" --query "tool use" --exclude "survey" --max-results 300`
94	- Fetch a single paper by arXiv ID (direct `id_list` fetch):
95	- `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --query 2509.02547 --max-results 1`
96	- Offline auto-detect (no flags):
97	- Place `papers/import.csv` (or `.json/.jsonl`) under the workspace, then run: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace>`
98	- Offline import + time window (via `queries.md`):
99	- Set `- time window: { from: 2022, to: 2025 }` then run offline import normally
100
101	## Troubleshooting
102
103	### Common Issues
104
105	#### Issue: `papers/papers_raw.jsonl` is empty
106
107	Symptom:
108	- Script exits with “No results returned …” or output file is empty.
109
110	Causes:
111	- Network is blo

People who liked this also viewed

6 picks

Type

arxiv-search

51installs39forks

View on GitHub

How to install arxiv-search?

willoscar/research-units-pipeline-skills/arxiv-search

$npx -y skills add WILLOSCAR/research-units-pipeline-skills --skill arxiv-search

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# arXiv Search (metadata-first)
2
3	Collect an initial paper set with enough metadata to support downstream ranking, taxonomy building, and citation generation.
4
5	When online, prefer rich arXiv metadata (categories, arxiv_id, pdf_url, published/updated, etc.). When offline, accept an export and convert it cleanly.
6
7	## Load Order
8
9	Always read:
10	- `references/domain_pack_overview.md` — how domain packs drive topic-specific behavior
11
12	Domain packs (loaded by topic match):
13	- `assets/domain_packs/llm_agents.json` — pinned IDs, query rewrite rules for LLM agent topics
14
15	## Script Boundary
16
17	Use `scripts/run.py` only for:
18	- arXiv API retrieval and XML parsing
19	- offline export conversion (CSV/JSON/JSONL normalization)
20	- metadata enrichment via `id_list` backfill
21
22	Do not treat `run.py` as the place for:
23	- hardcoded topic detection or query rewriting (use domain packs)
24	- domain-specific pinned paper lists (externalize to `assets/domain_packs/`)
25
26	## Contract-driven behavior
27
28	- Domain-pack query rewriting is the default for broad discovery Workflows.
29	- A focused Workflow may set
30	`quality_contract.retrieval_policy.domain_pack_query_mode: explicit`; in that
31	mode, the query list in `queries.md` remains authoritative and the domain
32	pack must not replace its topic focus.
33	- `quality_contract.retrieval_policy.minimum_records` turns a Workflow's raw
34	candidate-pool floor into a strict quality-gate check.
35
36	## Input
37
38	- `queries.md` (keywords, excludes, time window)
39
40	## Outputs
41
42	- `papers/papers_raw.jsonl` (JSONL; 1 paper per line)
43	- Each record includes at least: `title`, `authors`, `year`, `url`, `abstract`
44	- When using the arXiv API online mode, records also include helpful metadata: `arxiv_id`, `pdf_url`, `categories`, `primary_category`, `published`, `updated`, `doi`, `journal_ref`, `comment`
45	- Convenience index (optional but generated by the script):
46	- `papers/papers_raw.csv`
47
48	## Decision: online vs offline
49
50	- If you have network access: run arXiv API retrieval.
51	- If not: import an export the user provides (CSV/JSON/JSONL) and normalize fields.
52	- Hybrid: if you import offline but still have network later, you can enrich missing fields (abstract/authors/categories) via arXiv `id_list` using `--enrich-metadata` or `queries.md` `enrich_metadata: true`.
53
54	## Workflow (heuristic)
55
56	1. Read `queries.md` and expand into concrete query strings.
57	2. Retrieve results (online) or import an export (offline).
58	3. Normalize every record to include at least:
59	- `title`, `authors` (array), `year`, `url`, `abstract`
60	4. Keep the set broad at this stage; dedupe/ranking comes next.
61	5. Apply time window and `max_results` if specified.
62
63	## Quality checklist
64
65	- [ ] `papers/papers_raw.jsonl` exists.
66	- [ ] Each line is valid JSON and contains `title`, `authors`, `year`, `url`.
67
68	## Side effects
69
70	- Allowed: create/overwrite `papers/papers_raw.jsonl`; append notes to `STATUS.md`.
71	- Not allowed: write prose sections in `output/` before writing is approved.
72
73	## Script
74
75	### Quick Start
76
77	- `uv run python .codex/skills/arxiv-search/scripts/run.py --help`
78	- Online: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --query "<query>" --max-results 200`
79	- Offline import: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --input <export.csv\|json\|jsonl>`
80
81	### All Options
82
83	- `--query <q>`: repeatable; multiple queries are unioned
84	- `--exclude <term>`: repeatable; excludes applied after retrieval
85	- `--max-results <n>`: cap total retrieved
86	- `--input <export.*>`: offline mode (CSV/JSON/JSONL)
87	- `--enrich-metadata`: best-effort enrich via arXiv `id_list` (needs network)
88	- `queries.md` also supports: `keywords`, `exclude`, `time window`, `max_results`, `enrich_metadata`
89
90	### Examples
91
92	- Online (multi-query + excludes):
93	- `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --query "LLM agent" --query "tool use" --exclude "survey" --max-results 300`
94	- Fetch a single paper by arXiv ID (direct `id_list` fetch):
95	- `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --query 2509.02547 --max-results 1`
96	- Offline auto-detect (no flags):
97	- Place `papers/import.csv` (or `.json/.jsonl`) under the workspace, then run: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace>`
98	- Offline import + time window (via `queries.md`):
99	- Set `- time window: { from: 2022, to: 2025 }` then run offline import normally
100
101	## Troubleshooting
102
103	### Common Issues
104
105	#### Issue: `papers/papers_raw.jsonl` is empty
106
107	Symptom:
108	- Script exits with “No results returned …” or output file is empty.
109
110	Causes:
111	- Network is blo

People who liked this also viewed

6 picks

Type