document-processing

Name: document-processing
Author: neuromechanist

37stars7forks

How to install document-processing?

neuromechanist/research-skills/document-processing

$npx -y skills add neuromechanist/research-skills --skill document-processing

Installs into the current project.

Summary

This skill should be used when the user says \"process documents\", \"extract text from PDF\", \"OCR this document\", \"convert PDF to markdown\", \"extract emails from documents\", \"parse document\", \"document conversion\", \"batch OCR\", \"extract structured data from PDF\",

Files · 1

View on GitHub

SKILL.md

1	# Document Processing
2
3	Extract, convert, and structure content from PDFs, images, and other document formats. Handles OCR, text extraction, markdown conversion, email extraction, and structured data output.
4
5	## When to Use
6
7	- Converting scanned documents to searchable text
8	- Extracting text from PDFs (native or scanned)
9	- Converting documents to markdown for further processing
10	- Extracting emails, addresses, or other structured data from documents
11	- Batch processing document collections
12
13	## Processing Pipeline
14
15	### Step 1: Identify Document Type
16
17	Determine the processing approach:
18
19	\| Input \| Method \| Tool \|
20	\|---\|---\|---\|
21	\| Native PDF (has text layer) \| Direct extraction \| `pdftotext`, `pymupdf` \|
22	\| Scanned PDF (images only) \| OCR \| Mistral OCR API, `tesseract` \|
23	\| Image files (PNG, JPG, TIFF) \| OCR \| Mistral OCR API, `tesseract` \|
24	\| Word documents (.docx) \| Conversion \| `python-docx`, `pandoc` \|
25	\| HTML \| Conversion \| `pandoc`, `beautifulsoup4` \|
26
27	Detection:
28	```bash
29	# Check if PDF has text content
30	pdftotext input.pdf - \| head -20
31	# If output is empty or garbled, it's a scanned PDF -> use OCR
32	```
33
34	### Step 2: Extract Content
35
36	#### Native PDF Extraction
37
38	```python
39	import pymupdf
40
41	doc = pymupdf.open("input.pdf")
42	for page in doc:
43	text = page.get_text("markdown") # or "text", "html"
44	print(text)
45	```
46
47	#### OCR with Mistral (for scanned documents)
48
49	Requires `MISTRAL_API_KEY` environment variable. Falls back to tesseract for offline processing if unavailable.
50
51	```python
52	import base64
53	import httpx
54
55	def ocr_page(image_path: str, api_key: str) -> str:
56	with open(image_path, "rb") as f:
57	image_data = base64.b64encode(f.read()).decode()
58
59	response = httpx.post(
60	"https://api.mistral.ai/v1/chat/completions",
61	headers={"Authorization": f"Bearer {api_key}"},
62	json={
63	"model": "mistral-ocr-latest",
64	"messages": [{
65	"role": "user",
66	"content": [
67	{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_data}"}},
68	{"type": "text", "text": "Extract all text from this image. Preserve formatting, tables, and structure. Output as markdown."}
69	]
70	}]
71	}
72	)
73	return response.json()["choices"][0]["message"]["content"]
74	```
75
76	#### OCR with Tesseract (offline fallback)
77
78	```bash
79	# Single page
80	tesseract input.png output -l eng --oem 3 --psm 6
81
82	# PDF to text via tesseract
83	pdftoppm input.pdf page -png
84	for f in page-*.png; do tesseract "$f" "${f%.png}" -l eng; done
85	cat page-*.txt > output.txt
86	```
87
88	### Step 3: Structure Output
89
90	Convert extracted text to structured formats:
91
92	#### Markdown cleanup
93	- Fix OCR artifacts (broken words, spurious line breaks)
94	- Reconstruct tables from aligned text
95	- Identify headers from font size/weight changes
96	- Preserve list formatting
97
98	#### Structured data extraction
99
100	```python
101	# Extract emails
102	import re
103	emails = re.findall(r'[\w.+-]+@[\w-]+\.[\w.]+', text)
104
105	# Extract dates
106	dates = re.findall(r'\b\d{1,2}[/-]\d{1,2}[/-]\d{2,4}\b', text)
107
108	# Extract phone numbers
109	phones = re.findall(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b', text)
110	```
111
112	### Step 4: Output
113
114	Save results in the requested format:
115	- Markdown (`.md`) - default for text content
116	- JSON - for structured data extraction
117	- Plain text (`.txt`) - for simple text extraction
118
119	## Batch Processing
120
121	For document collections:
122
123	```bash
124	# Process all PDFs in a directory
125	for pdf in /path/to/docs/*.pdf; do
126	name=$(basename "$pdf" .pdf)
127	pdftotext "$pdf" "/path/to/output/${name}.txt"
128	done
129	```
130
131	For large collections, track progress:
132	1. Create a manifest of input files
133	2. Process each file, recording success/failure
134	3. Report summary (processed, failed, skipped)
135
136	## Quality Checks
137
138	After extraction, verify:
139	- [ ] Text is readable (not garbled encoding)
140	- [ ] Tables preserved their structure
141	- [ ] No pages were skipped
142	- [ ] Special characters rendered correctly
143	- [ ] Headers and sections identified
144
145	## Additional Resources
146
147	- Reference: [references/ocr-configuration.md](references/ocr-configuration.md) - Tesseract languages, page segmentation modes, preprocessing
148	- Reference: [references/pdf-tools.md](references/pdf-tools.md) - Comparison of PDF extraction libraries and their tradeoffs

People who liked this also viewed

6 picks

Type

document-processing

37stars7forks

View on GitHub

How to install document-processing?

neuromechanist/research-skills/document-processing

$npx -y skills add neuromechanist/research-skills --skill document-processing

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Document Processing
2
3	Extract, convert, and structure content from PDFs, images, and other document formats. Handles OCR, text extraction, markdown conversion, email extraction, and structured data output.
4
5	## When to Use
6
7	- Converting scanned documents to searchable text
8	- Extracting text from PDFs (native or scanned)
9	- Converting documents to markdown for further processing
10	- Extracting emails, addresses, or other structured data from documents
11	- Batch processing document collections
12
13	## Processing Pipeline
14
15	### Step 1: Identify Document Type
16
17	Determine the processing approach:
18
19	\| Input \| Method \| Tool \|
20	\|---\|---\|---\|
21	\| Native PDF (has text layer) \| Direct extraction \| `pdftotext`, `pymupdf` \|
22	\| Scanned PDF (images only) \| OCR \| Mistral OCR API, `tesseract` \|
23	\| Image files (PNG, JPG, TIFF) \| OCR \| Mistral OCR API, `tesseract` \|
24	\| Word documents (.docx) \| Conversion \| `python-docx`, `pandoc` \|
25	\| HTML \| Conversion \| `pandoc`, `beautifulsoup4` \|
26
27	Detection:
28	```bash
29	# Check if PDF has text content
30	pdftotext input.pdf - \| head -20
31	# If output is empty or garbled, it's a scanned PDF -> use OCR
32	```
33
34	### Step 2: Extract Content
35
36	#### Native PDF Extraction
37
38	```python
39	import pymupdf
40
41	doc = pymupdf.open("input.pdf")
42	for page in doc:
43	text = page.get_text("markdown") # or "text", "html"
44	print(text)
45	```
46
47	#### OCR with Mistral (for scanned documents)
48
49	Requires `MISTRAL_API_KEY` environment variable. Falls back to tesseract for offline processing if unavailable.
50
51	```python
52	import base64
53	import httpx
54
55	def ocr_page(image_path: str, api_key: str) -> str:
56	with open(image_path, "rb") as f:
57	image_data = base64.b64encode(f.read()).decode()
58
59	response = httpx.post(
60	"https://api.mistral.ai/v1/chat/completions",
61	headers={"Authorization": f"Bearer {api_key}"},
62	json={
63	"model": "mistral-ocr-latest",
64	"messages": [{
65	"role": "user",
66	"content": [
67	{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_data}"}},
68	{"type": "text", "text": "Extract all text from this image. Preserve formatting, tables, and structure. Output as markdown."}
69	]
70	}]
71	}
72	)
73	return response.json()["choices"][0]["message"]["content"]
74	```
75
76	#### OCR with Tesseract (offline fallback)
77
78	```bash
79	# Single page
80	tesseract input.png output -l eng --oem 3 --psm 6
81
82	# PDF to text via tesseract
83	pdftoppm input.pdf page -png
84	for f in page-*.png; do tesseract "$f" "${f%.png}" -l eng; done
85	cat page-*.txt > output.txt
86	```
87
88	### Step 3: Structure Output
89
90	Convert extracted text to structured formats:
91
92	#### Markdown cleanup
93	- Fix OCR artifacts (broken words, spurious line breaks)
94	- Reconstruct tables from aligned text
95	- Identify headers from font size/weight changes
96	- Preserve list formatting
97
98	#### Structured data extraction
99
100	```python
101	# Extract emails
102	import re
103	emails = re.findall(r'[\w.+-]+@[\w-]+\.[\w.]+', text)
104
105	# Extract dates
106	dates = re.findall(r'\b\d{1,2}[/-]\d{1,2}[/-]\d{2,4}\b', text)
107
108	# Extract phone numbers
109	phones = re.findall(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b', text)
110	```
111
112	### Step 4: Output
113
114	Save results in the requested format:
115	- Markdown (`.md`) - default for text content
116	- JSON - for structured data extraction
117	- Plain text (`.txt`) - for simple text extraction
118
119	## Batch Processing
120
121	For document collections:
122
123	```bash
124	# Process all PDFs in a directory
125	for pdf in /path/to/docs/*.pdf; do
126	name=$(basename "$pdf" .pdf)
127	pdftotext "$pdf" "/path/to/output/${name}.txt"
128	done
129	```
130
131	For large collections, track progress:
132	1. Create a manifest of input files
133	2. Process each file, recording success/failure
134	3. Report summary (processed, failed, skipped)
135
136	## Quality Checks
137
138	After extraction, verify:
139	- [ ] Text is readable (not garbled encoding)
140	- [ ] Tables preserved their structure
141	- [ ] No pages were skipped
142	- [ ] Special characters rendered correctly
143	- [ ] Headers and sections identified
144
145	## Additional Resources
146
147	- Reference: [references/ocr-configuration.md](references/ocr-configuration.md) - Tesseract languages, page segmentation modes, preprocessing
148	- Reference: [references/pdf-tools.md](references/pdf-tools.md) - Comparison of PDF extraction libraries and their tradeoffs

People who liked this also viewed

6 picks

Type