vlm-ocr-pipeline

Name: vlm-ocr-pipeline
Author: scdenney

34stars0forks

How to install vlm-ocr-pipeline?

scdenney/open-science-skills/vlm-ocr-pipeline

$npx -y skills add scdenney/open-science-skills --skill vlm-ocr-pipeline

Installs into the current project.

Summary

Design VLM-based OCR pipelines. Use for model selection, prompting, architecture, multilingual documents, structured output, and evaluation planning.

Files · 1

View on GitHub

SKILL.md

1	# VLM-Based OCR Pipeline for Scanned Document Corpora
2
3	## Instructions
4
5	For a worked language-specific transcription prompt (pre-reform Cyrillic) and a per-page JSON output schema with `uncertain_spans`, `layout_markers`, and `flags`, see `reference/prompt-and-schema.md`.
6
7	### 1. Model Selection
8
9	- Start from OCR benchmarks, not general VLM leaderboards. OCRBench (Liu et al. 2024) tests across 29 document OCR dimensions; OCRBench v2 (Fu et al. 2025) extends to multilingual scripts and multi-page documents. General vision-language benchmarks (MMMU, VQAv2) do not predict OCR accuracy.
10	- Verify language support explicitly. Confirm the target language appears in the model's training set with per-language accuracy data. Qwen3-VL (Bai et al. 2025) enumerates 39 languages with OCR benchmarks; models that claim "multilingual support" without language-specific evidence may fail on non-Latin scripts.
11	- Compare across models for your script family. E-ARMOR (2025) compares five VLMs and two traditional OCR systems across 54 languages on a hand-annotated dataset. Use this or similar comparative studies rather than relying on a single model's self-reported scores.
12	- Assess quantization tradeoffs before committing. Low-bit quantization (e.g., W3A16, W4A8) enables single-GPU deployment but can degrade VLM accuracy non-uniformly across modalities. Li et al. (2025) show that language tokens are an order of magnitude more sensitive to quantization than vision tokens; treating them equally during calibration over-weights the insensitive modality and hurts performance. Always compare quantized output against a full-precision baseline on a pilot sample before committing to a bulk run.
13	- Test the instruct variant against reasoning variants. For OCR tasks, instruction-following models typically outperform chain-of-thought variants, which may add latency and fabricate content rather than improving transcription fidelity.
14
15	### 2. Image Handling and Preprocessing
16
17	- Separate OCR input resolution from archival preservation. For archival capture, FADGI (2016) prescribes 300-400 ppi across books, journals, and manuscripts (with 400+ ppi for 4-star compliance), and Metamorfoze (van Dormolen 2012) applies similar European standards. For VLM OCR input, resolution requirements are lower but not well-pinned by the KB: calibrate on a pilot, do not drop below ~150-200 ppi, and raise DPI for small type, faded ink, or pre-industrial typography.
18	- Prefer native image extraction over rasterization. Scanned PDFs store each page as an embedded image at the original scan resolution. Extract these byte-for-byte using PDF library methods rather than re-rasterizing, which downsamples and re-encodes (Pitt OCR Best Practices). Reserve rasterization for VLM input when a specific DPI is needed.
19	- Test preprocessing on a sample before applying corpus-wide. Learned image restoration combined with neural post-correction can yield large CER reductions on degraded historical documents — Guan et al. (2025) report 63.9-70.3% CER reduction for the full PreP-OCR pipeline (ResShift image restoration + ByT5 post-correction), not for classical deskew/binarization/contrast alone. Classical preprocessing alone may not help and can actively hurt: Machidon & Machidon (2025) find that grayscale conversion, binarization, and dilation did not improve OCR on degraded folkloristic scans and in some cases introduced artifacts that worsened recognition. Always run with and without preprocessing on a diverse sample and compare.
20	- Store archival images separately from OCR derivatives. Save native-resolution images for preservation and generate OCR-resolution images in memory for the VLM. Do not save OCR-resolution rasterizations as the archival copy.
21
22	### 3. Prompt Engineering
23
24	- Build language-specific prompts that enumerate expected characters. For diacritics-heavy languages (Polish, Czech, Vietnamese, Turkish), list every expected diacritical character explicitly in the prompt. For CJK scripts, instruct the model to handle mixed-script content (e.g., Korean hangul with classical Chinese hanja).
25	- Specify structured output format. Instruct the model to output markdown preserving headings, paragraphs, footnotes, and tables. Structured prompts with explicit output format significantly outperform generic "extract text" instructions.
26	- Include negative instructions. "Do not translate," "do not interpret," "do not add content not present in the image." VLMs will summarize, translate, or describe images unless explicitly constrained.
27	- Handle page-type edge cases in the prompt. Instruct the model on what to return for blank pages, illustration-only pages, and pages with only page numbers. Without this guidance, models may hallucinate con

People who liked this also viewed

6 picks

Type

paperclip-create-agentCreate new agents in Paperclip with governance-aware hiring. Use when you need to inspect adapter configuration options, compare existing agent configs, draft a new agent prompt/config, and submit a hSkillsJul 2026157k5
subagent-driven-developmentUse when executing implementation plans with independent tasks in the current sessionSkillsJul 2026154k259k
dispatching-parallel-agentsUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesSkillsJul 2026143k259k
browser-act-skill-forgeForges reusable Skill packages (SKILL.md + scripts) from website exploration via browser-act — no re-exploration later. Use when: user wants a reusable Skill for any website, needs to understand a sitSkillsJul 202685k4.6k
wizardGenerate an interactive bash wizard that walks a human through a manual procedure — third-party setup, a one-off migration, an A→B state transition — opening URLs, capturing values, confirming each stSkillsJul 202675k182k
wonda-cliUsing the Wonda CLI to generate images, videos, music, and audio from the terminal — plus LinkedIn, Reddit, and X/Twitter research and automationSkillsJul 202666k137

vlm-ocr-pipeline

34stars0forks

View on GitHub

How to install vlm-ocr-pipeline?

scdenney/open-science-skills/vlm-ocr-pipeline

$npx -y skills add scdenney/open-science-skills --skill vlm-ocr-pipeline

Installs into the current project.

Summary

Design VLM-based OCR pipelines. Use for model selection, prompting, architecture, multilingual documents, structured output, and evaluation planning.

Files · 1

View on GitHub

SKILL.md

1	# VLM-Based OCR Pipeline for Scanned Document Corpora
2
3	## Instructions
4
5	For a worked language-specific transcription prompt (pre-reform Cyrillic) and a per-page JSON output schema with `uncertain_spans`, `layout_markers`, and `flags`, see `reference/prompt-and-schema.md`.
6
7	### 1. Model Selection
8
9	- Start from OCR benchmarks, not general VLM leaderboards. OCRBench (Liu et al. 2024) tests across 29 document OCR dimensions; OCRBench v2 (Fu et al. 2025) extends to multilingual scripts and multi-page documents. General vision-language benchmarks (MMMU, VQAv2) do not predict OCR accuracy.
10	- Verify language support explicitly. Confirm the target language appears in the model's training set with per-language accuracy data. Qwen3-VL (Bai et al. 2025) enumerates 39 languages with OCR benchmarks; models that claim "multilingual support" without language-specific evidence may fail on non-Latin scripts.
11	- Compare across models for your script family. E-ARMOR (2025) compares five VLMs and two traditional OCR systems across 54 languages on a hand-annotated dataset. Use this or similar comparative studies rather than relying on a single model's self-reported scores.
12	- Assess quantization tradeoffs before committing. Low-bit quantization (e.g., W3A16, W4A8) enables single-GPU deployment but can degrade VLM accuracy non-uniformly across modalities. Li et al. (2025) show that language tokens are an order of magnitude more sensitive to quantization than vision tokens; treating them equally during calibration over-weights the insensitive modality and hurts performance. Always compare quantized output against a full-precision baseline on a pilot sample before committing to a bulk run.
13	- Test the instruct variant against reasoning variants. For OCR tasks, instruction-following models typically outperform chain-of-thought variants, which may add latency and fabricate content rather than improving transcription fidelity.
14
15	### 2. Image Handling and Preprocessing
16
17	- Separate OCR input resolution from archival preservation. For archival capture, FADGI (2016) prescribes 300-400 ppi across books, journals, and manuscripts (with 400+ ppi for 4-star compliance), and Metamorfoze (van Dormolen 2012) applies similar European standards. For VLM OCR input, resolution requirements are lower but not well-pinned by the KB: calibrate on a pilot, do not drop below ~150-200 ppi, and raise DPI for small type, faded ink, or pre-industrial typography.
18	- Prefer native image extraction over rasterization. Scanned PDFs store each page as an embedded image at the original scan resolution. Extract these byte-for-byte using PDF library methods rather than re-rasterizing, which downsamples and re-encodes (Pitt OCR Best Practices). Reserve rasterization for VLM input when a specific DPI is needed.
19	- Test preprocessing on a sample before applying corpus-wide. Learned image restoration combined with neural post-correction can yield large CER reductions on degraded historical documents — Guan et al. (2025) report 63.9-70.3% CER reduction for the full PreP-OCR pipeline (ResShift image restoration + ByT5 post-correction), not for classical deskew/binarization/contrast alone. Classical preprocessing alone may not help and can actively hurt: Machidon & Machidon (2025) find that grayscale conversion, binarization, and dilation did not improve OCR on degraded folkloristic scans and in some cases introduced artifacts that worsened recognition. Always run with and without preprocessing on a diverse sample and compare.
20	- Store archival images separately from OCR derivatives. Save native-resolution images for preservation and generate OCR-resolution images in memory for the VLM. Do not save OCR-resolution rasterizations as the archival copy.
21
22	### 3. Prompt Engineering
23
24	- Build language-specific prompts that enumerate expected characters. For diacritics-heavy languages (Polish, Czech, Vietnamese, Turkish), list every expected diacritical character explicitly in the prompt. For CJK scripts, instruct the model to handle mixed-script content (e.g., Korean hangul with classical Chinese hanja).
25	- Specify structured output format. Instruct the model to output markdown preserving headings, paragraphs, footnotes, and tables. Structured prompts with explicit output format significantly outperform generic "extract text" instructions.
26	- Include negative instructions. "Do not translate," "do not interpret," "do not add content not present in the image." VLMs will summarize, translate, or describe images unless explicitly constrained.
27	- Handle page-type edge cases in the prompt. Instruct the model on what to return for blank pages, illustration-only pages, and pages with only page numbers. Without this guidance, models may hallucinate con

People who liked this also viewed

6 picks

Type