vlm-ocr-evaluation

Name: vlm-ocr-evaluation
Author: scdenney

34stars0forks

How to install vlm-ocr-evaluation?

scdenney/open-science-skills/vlm-ocr-evaluation

$npx -y skills add scdenney/open-science-skills --skill vlm-ocr-evaluation

Installs into the current project.

Summary

Compare OCR systems before a bulk run: candidate set, stratified ground truth, CER/WER, normalization, per-language and per-stratum accuracy.

Files · 1

View on GitHub

SKILL.md

1	# VLM-OCR Evaluation: Comparing OCR Systems Before You Commit
2
3	## Instructions
4
5	Before running any OCR model across a whole corpus, run a controlled comparison on a small, human-transcribed sample and let the measured error rates pick the model. This skill is the selection gate that precedes the `vlm-ocr-pipeline` skill: use this to choose a model and document why, then use `vlm-ocr-pipeline` to run the chosen model at scale and `post-ocr-cleanup` to clean its output. For the hardest pages, where no single model is reliable, the multi-model voting logic in `model-council-voting` can be applied to OCR transcriptions as well.
6
7	### 1. Run a Comparison Before You Commit
8
9	- Treat model choice as an empirical question, not a default — test, do not guess. A model that tops a general vision-language leaderboard, or that read one language well, can still fail on your script, era, or page layout. The only trustworthy signal is its error rate on pages like yours.
10	- The comparison is cheap insurance. Evaluating a 50–100 page sample once prevents discovering, after a 50,000-page bulk run, that the chosen model silently dropped every table or collapsed on degraded scans.
11	- Scope the comparison to the decision. The user's `ocr_pipeline/comparison/` setup runs nine systems on 64 pages spanning two languages and seven decades before committing to a bulk pipeline — enough to rank the candidates on the strata that matter, not a full corpus.
12	- The output is a go/no-go gate: keep the model whose accuracy clears your threshold on the strata you care about, or conclude that no single model does and split the corpus by script or era (§6).
13
14	### 2. Assemble the Candidate Set
15
16	- Include three kinds of system: several open-weight VLMs, one or two proprietary APIs, and a traditional OCR baseline (Tesseract). The gei comparison's `MODELS` registry in `run_comparison.py` holds six open-weight VLMs (Qwen3.5-35B, Qwen3-VL-32B, Qwen3.5-9B, Gemma, MiniCPM-V, DeepSeek-OCR) plus Tesseract, with two proprietary APIs (GPT-4.1, Claude) run by separate scripts — nine systems total.
17	- Pick the VLM candidates from OCR benchmarks (e.g., OCRBench), not general multimodal leaderboards. General vision-language ability does not predict transcription fidelity on dense historical print; the `vlm-ocr-pipeline` skill covers benchmark-grounded model selection in detail.
18	- Always keep a traditional baseline. Even when it loses, Tesseract anchors what "hard" means for your corpus and shows where a VLM actually earns its extra cost and latency.
19	- Record an exact registry entry per system — name, HuggingFace id or dated API identifier, quantization, and loader — as the gei registry does (`name`/`hf_id`/`type`/`loader` per model). Family-name-only reporting ("we used Qwen and Gemma") is not reproducible.
20
21	### 3. Build a Stratified Ground-Truth Set
22
23	- Human-transcribe a stratified sample, not a convenience sample. Stratify on the dimensions that actually drive OCR difficulty: language/script, era (a decade bracket), and content type — running body text, multi-column tables, illustrated or captioned pages, degraded or water-damaged print, and front matter / title pages. The gei manifest tags each page with `language`, `year`, `decade`, `subject`, and `content_type` and samples across all of them.
24	- Size: roughly 50–100 pages is a workable default for a handful of candidate systems (a house default that balances transcription effort against per-stratum cell sizes, not a cited figure). More strata require more pages so that each cell holds enough pages to mean something.
25	- Store ground truth as one UTF-8 `.txt` per page, keyed by a stable page id (`ground_truth/<page_id>.txt` in `compute_accuracy.py`). Transcribe faithfully — preserve the characters actually on the page (hanja alongside hangul, diacritics) — and decide up front how to render non-text regions (tables, figures) so the reference and the OCR output are scored on the same basis.
26	- Transcribe before looking at any model output, so the reference is not anchored to a model's guesses. Two independent transcribers on a subset, with disagreements reconciled, guard against a single transcriber's systematic errors (the inter-coder logic in the `text-classification` skill applies here too).
27
28	### 4. Character and Word Error Rate
29
30	- CER and WER are edit-distance metrics. CER = Levenshtein(reference, hypothesis) ÷ length(reference); WER is the same at the word level (a word-level dynamic-programming edit distance ÷ reference word count). `compute_accuracy.py` implements both — a character Levenshtein distance for CER and a word-level DP for WER (Levenshtein 1966).
31	- Declare normalization before scoring; it changes the numbers. The gei `normalize_text` applies Unicode NFC, strips

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

vlm-ocr-evaluation

34stars0forks

View on GitHub

How to install vlm-ocr-evaluation?

scdenney/open-science-skills/vlm-ocr-evaluation

$npx -y skills add scdenney/open-science-skills --skill vlm-ocr-evaluation

Installs into the current project.

Summary

Compare OCR systems before a bulk run: candidate set, stratified ground truth, CER/WER, normalization, per-language and per-stratum accuracy.

Files · 1

View on GitHub

SKILL.md

1	# VLM-OCR Evaluation: Comparing OCR Systems Before You Commit
2
3	## Instructions
4
5	Before running any OCR model across a whole corpus, run a controlled comparison on a small, human-transcribed sample and let the measured error rates pick the model. This skill is the selection gate that precedes the `vlm-ocr-pipeline` skill: use this to choose a model and document why, then use `vlm-ocr-pipeline` to run the chosen model at scale and `post-ocr-cleanup` to clean its output. For the hardest pages, where no single model is reliable, the multi-model voting logic in `model-council-voting` can be applied to OCR transcriptions as well.
6
7	### 1. Run a Comparison Before You Commit
8
9	- Treat model choice as an empirical question, not a default — test, do not guess. A model that tops a general vision-language leaderboard, or that read one language well, can still fail on your script, era, or page layout. The only trustworthy signal is its error rate on pages like yours.
10	- The comparison is cheap insurance. Evaluating a 50–100 page sample once prevents discovering, after a 50,000-page bulk run, that the chosen model silently dropped every table or collapsed on degraded scans.
11	- Scope the comparison to the decision. The user's `ocr_pipeline/comparison/` setup runs nine systems on 64 pages spanning two languages and seven decades before committing to a bulk pipeline — enough to rank the candidates on the strata that matter, not a full corpus.
12	- The output is a go/no-go gate: keep the model whose accuracy clears your threshold on the strata you care about, or conclude that no single model does and split the corpus by script or era (§6).
13
14	### 2. Assemble the Candidate Set
15
16	- Include three kinds of system: several open-weight VLMs, one or two proprietary APIs, and a traditional OCR baseline (Tesseract). The gei comparison's `MODELS` registry in `run_comparison.py` holds six open-weight VLMs (Qwen3.5-35B, Qwen3-VL-32B, Qwen3.5-9B, Gemma, MiniCPM-V, DeepSeek-OCR) plus Tesseract, with two proprietary APIs (GPT-4.1, Claude) run by separate scripts — nine systems total.
17	- Pick the VLM candidates from OCR benchmarks (e.g., OCRBench), not general multimodal leaderboards. General vision-language ability does not predict transcription fidelity on dense historical print; the `vlm-ocr-pipeline` skill covers benchmark-grounded model selection in detail.
18	- Always keep a traditional baseline. Even when it loses, Tesseract anchors what "hard" means for your corpus and shows where a VLM actually earns its extra cost and latency.
19	- Record an exact registry entry per system — name, HuggingFace id or dated API identifier, quantization, and loader — as the gei registry does (`name`/`hf_id`/`type`/`loader` per model). Family-name-only reporting ("we used Qwen and Gemma") is not reproducible.
20
21	### 3. Build a Stratified Ground-Truth Set
22
23	- Human-transcribe a stratified sample, not a convenience sample. Stratify on the dimensions that actually drive OCR difficulty: language/script, era (a decade bracket), and content type — running body text, multi-column tables, illustrated or captioned pages, degraded or water-damaged print, and front matter / title pages. The gei manifest tags each page with `language`, `year`, `decade`, `subject`, and `content_type` and samples across all of them.
24	- Size: roughly 50–100 pages is a workable default for a handful of candidate systems (a house default that balances transcription effort against per-stratum cell sizes, not a cited figure). More strata require more pages so that each cell holds enough pages to mean something.
25	- Store ground truth as one UTF-8 `.txt` per page, keyed by a stable page id (`ground_truth/<page_id>.txt` in `compute_accuracy.py`). Transcribe faithfully — preserve the characters actually on the page (hanja alongside hangul, diacritics) — and decide up front how to render non-text regions (tables, figures) so the reference and the OCR output are scored on the same basis.
26	- Transcribe before looking at any model output, so the reference is not anchored to a model's guesses. Two independent transcribers on a subset, with disagreements reconciled, guard against a single transcriber's systematic errors (the inter-coder logic in the `text-classification` skill applies here too).
27
28	### 4. Character and Word Error Rate
29
30	- CER and WER are edit-distance metrics. CER = Levenshtein(reference, hypothesis) ÷ length(reference); WER is the same at the word level (a word-level dynamic-programming edit distance ÷ reference word count). `compute_accuracy.py` implements both — a character Levenshtein distance for CER and a word-level DP for WER (Levenshtein 1966).
31	- Declare normalization before scoring; it changes the numbers. The gei `normalize_text` applies Unicode NFC, strips

People who liked this also viewed

6 picks

Type