fulltext-retrieval
installsforks
How to install fulltext-retrieval?
aperivue/medsci-skills/fulltext-retrieval
$npx -y skills add Aperivue/medsci-skills --skill fulltext-retrievalBatch download open-access PDFs by DOI using legitimate OA APIs (Unpaywall, PMC, OpenAlex, Crossref). Optional PDF→Markdown conversion for token-efficient LLM analysis.
SKILL.md
| 1 | # Fulltext Retrieval Skill |
| 2 | |
| 3 | Batch download open-access full-text PDFs from a DOI list using legitimate OA APIs only. |
| 4 | |
| 5 | ## Pipeline |
| 6 | |
| 7 | ``` |
| 8 | DOI → arXiv (10.48550/arXiv.* DOIs) → Unpaywall → PMC (Europe PMC / OA FTP / web) → OpenAlex → Crossref → landing page |
| 9 | ``` |
| 10 | |
| 11 | Each DOI goes through these sources in order until a valid PDF (≥10 KB, `%PDF-` header) is found. arXiv DOIs (`10.48550/arXiv.2401.01234`, version suffixes, old-style `hep-th/9901001`, or a bare `arXiv:` id) resolve directly to the arXiv PDF first. |
| 12 | |
| 13 | ## Quick Start |
| 14 | |
| 15 | ```bash |
| 16 | # Prepare a DOI list (one per line) |
| 17 | cat > dois.txt << 'EOF' |
| 18 | 10.1007/s00330-010-1783-x |
| 19 | 10.1002/mp.12524 |
| 20 | 10.1148/radiol.13131265 |
| 21 | EOF |
| 22 | |
| 23 | # Run |
| 24 | python fetch_oa.py dois.txt --output pdfs/ --email your@email.com |
| 25 | |
| 26 | # Verbose mode for debugging |
| 27 | python fetch_oa.py dois.txt -o pdfs/ -e your@email.com --verbose |
| 28 | ``` |
| 29 | |
| 30 | ## Input Formats |
| 31 | |
| 32 | **Plain text** — one DOI per line: |
| 33 | ``` |
| 34 | 10.1007/s00330-010-1783-x |
| 35 | 10.1002/mp.12524 |
| 36 | ``` |
| 37 | |
| 38 | **TSV / CSV with header** — must contain a `DOI` column; optional `PMID` and `Title` columns: |
| 39 | ```tsv |
| 40 | ID Title DOI PMID Year |
| 41 | 1 Some paper 10.1007/s00330-010-1783-x 20628747 2010 |
| 42 | ``` |
| 43 | |
| 44 | **Markdown table** — a pipe table with a `DOI` column also works: |
| 45 | ```markdown |
| 46 | | DOI | PMID | Title | |
| 47 | |-----|------|-------| |
| 48 | | 10.1007/s00330-010-1783-x | 20628747 | Some paper | |
| 49 | ``` |
| 50 | |
| 51 | When a PMID is available, the PMC lookup is more reliable (PMID → PMCID conversion). When a `Title` column is present, downloaded PDFs get a best-effort title cross-check (see *Retrieval report* below). |
| 52 | |
| 53 | ## PMC Download (JS-Challenge Resistant) |
| 54 | |
| 55 | PMC web pages may block automated downloads with JavaScript proof-of-work challenges. This tool uses three fallback methods: |
| 56 | |
| 57 | ### Method A: Europe PMC REST API (most reliable) |
| 58 | |
| 59 | ```bash |
| 60 | PMCID="PMC9733600" |
| 61 | curl -sLo output.pdf \ |
| 62 | "https://europepmc.org/backend/ptpmcrender.fcgi?accid=${PMCID}&blobtype=pdf" |
| 63 | ``` |
| 64 | |
| 65 | ### Method B: PMC OA FTP Service |
| 66 | |
| 67 | ```bash |
| 68 | curl -s "https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=${PMCID}" | \ |
| 69 | grep -oE 'href="[^"]*\.pdf"' | head -1 | \ |
| 70 | sed 's/href="//;s/"//' | xargs curl -sLo output.pdf |
| 71 | ``` |
| 72 | |
| 73 | ### DOI/PMID → PMCID Conversion |
| 74 | |
| 75 | ```bash |
| 76 | # Works with both DOI and PMID |
| 77 | curl -s "https://www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/?ids=${DOI}&format=json" | \ |
| 78 | python3 -c "import sys,json; print(json.load(sys.stdin)['records'][0].get('pmcid',''))" |
| 79 | ``` |
| 80 | |
| 81 | ## Output |
| 82 | |
| 83 | - PDFs saved as `{DOI_safe}.pdf` (slashes replaced with underscores) |
| 84 | - `pdfs/retrieval_report.json` — structured per-DOI report (see below) |
| 85 | - `manual_needed.txt` — DOIs that could not be retrieved via OA |
| 86 | - Summary with arXiv/OA/PMC/fail/skip counts |
| 87 | |
| 88 | ## Retrieval report (`--report`) |
| 89 | |
| 90 | Every run writes a structured report (default `<output>/retrieval_report.json`, |
| 91 | override with `--report PATH`): |
| 92 | |
| 93 | ```json |
| 94 | { |
| 95 | "schema_version": 1, |
| 96 | "generated_by": "fetch_oa.py", |
| 97 | "counts": {"total": 10, "retrieved": 6, "not_retrieved": 4, "title_mismatch": 1}, |
| 98 | "items": [ |
| 99 | {"doi": "10.1007/...", "pmid": "20628747", "title": "...", |
| 100 | "status": "oa", "source": "unpaywall", "file": "10.1007_....pdf", |
| 101 | "size_bytes": 482113, "title_match": "match"} |
| 102 | ] |
| 103 | } |
| 104 | ``` |
| 105 | |
| 106 | - `status` ∈ `arxiv | oa | pmc | skip | fail`; `source` names the resolver that succeeded. |
| 107 | - `title_match` ∈ `match | mismatch | unavailable` (tri-state). It is **best-effort**: |
| 108 | it needs a `Title` column **and** `pdftotext` (poppler). When either is missing it is |
| 109 | `unavailable`; a `mismatch` is **flagged** for review and **never** auto-rejects a PDF |
| 110 | (guards against a publisher serving a wrong/redirect PDF that still passes the `%PDF-` check). |
| 111 | |
| 112 | ## Attach PDFs into Zotero ("Find Available PDF") |
| 113 | |
| 114 | OA-only resolvers miss paywalled-but-licensed papers. To attach full text **inside |
| 115 | Zotero** at a much higher yield, use `references/find_available_pdf.js` — a user-run |
| 116 | snippet for Zotero's *Tools → Developer → Run JavaScript*. It triggers Zotero's own |
| 117 | `addAvailablePDF` / `addAvailablePDFs` and therefore reuses **your** OpenURL resolver / |
| 118 | institutional proxy config; **no credentials, proxy hosts, or institutional identifiers |
| 119 | are hard-coded or leave your Zotero client**. The no-code equivalent is right-click → |
| 120 | "Find Available PDF". |
| 121 | |
| 122 | This path is **user-initiated** and depends on your live Zotero session, so its results |
| 123 | are recorded manually (not reproducible CI evidence). `/lit-sync` Phase 2.7 orchestrates |
| 124 | both routes (disk OA via this script + in-library via the snippet) and reconciles them in |
| 125 | a report. |
| 126 | |
| 127 | ## Requirements |
| 128 | |
| 129 | - Python 3.10+ (stdlib only, no pip dependencies) |
| 130 | - Contact email (required by Unpaywall Terms of Service) |
| 131 | |
| 132 | ## API Policies |
| 133 | |
| 134 | | Source | Rate Li |
People who liked this also viewed
6 picksType
- azure-aiUse for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector searSkillsJul 2026
- just-scrapeSearch, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON fromSkillsJul 2026
- firecrawlCLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agentsSkillsJul 2026
- firecrawl-searchWeb search with full page content extraction. Use this skill whenever the user asks to search the web, find articles, research a topic, look something up, find recent news, discover sources, or says "SkillsJul 2026
- firecrawl-agentAI-powered autonomous data extraction that navigates complex sites and returns structured JSON. Use this skill when the user wants structured data from websites, needs to extract pricing tiers, producSkillsJul 2026
- firecrawl-mapDiscover and list all URLs on a website, with optional search filtering. Use this skill when the user wants to find a specific page on a large site, list all URLs, see the site structure, find where sSkillsJul 2026
