article-extractor

Name: article-extractor
Author: michalparkola

498stars65forks

How to install article-extractor?

michalparkola/tapestry-skills/article-extractor

$npx -y skills add michalparkola/tapestry-skills --skill article-extractor

Installs into the current project.

Summary

Extract clean article content from URLs (blog posts, articles, tutorials) and save as readable text. Use when user wants to download, extract, or save an article/blog post from a URL without ads, navigation, or clutter.

Files · 1

View on GitHub

SKILL.md

1	# Article Extractor
2
3	This skill extracts the main content from web articles and blog posts, removing navigation, ads, newsletter signups, and other clutter. Saves clean, readable text.
4
5	## When to Use This Skill
6
7	Activate when the user:
8	- Provides an article/blog URL and wants the text content
9	- Asks to "download this article"
10	- Wants to "extract the content from [URL]"
11	- Asks to "save this blog post as text"
12	- Needs clean article text without distractions
13
14	## How It Works
15
16	### Priority Order:
17	1. Check if tools are installed (reader or trafilatura)
18	2. Download and extract article using best available tool
19	3. Clean up the content (remove extra whitespace, format properly)
20	4. Save to file with article title as filename
21	5. Confirm location and show preview
22
23	## Installation Check
24
25	Check for article extraction tools in this order:
26
27	### Option 1: reader (Recommended - Mozilla's Readability)
28
29	```bash
30	command -v reader
31	```
32
33	If not installed:
34	```bash
35	npm install -g @mozilla/readability-cli
36	# or
37	npm install -g reader-cli
38	```
39
40	### Option 2: trafilatura (Python-based, very good)
41
42	```bash
43	command -v trafilatura
44	```
45
46	If not installed:
47	```bash
48	pip3 install trafilatura
49	```
50
51	### Option 3: Fallback (curl + simple parsing)
52
53	If no tools available, use basic curl + text extraction (less reliable but works)
54
55	## Extraction Methods
56
57	### Method 1: Using reader (Best for most articles)
58
59	```bash
60	# Extract article
61	reader "URL" > article.txt
62	```
63
64	Pros:
65	- Based on Mozilla's Readability algorithm
66	- Excellent at removing clutter
67	- Preserves article structure
68
69	### Method 2: Using trafilatura (Best for blogs/news)
70
71	```bash
72	# Extract article
73	trafilatura --URL "URL" --output-format txt > article.txt
74
75	# Or with more options
76	trafilatura --URL "URL" --output-format txt --no-comments --no-tables > article.txt
77	```
78
79	Pros:
80	- Very accurate extraction
81	- Good with various site structures
82	- Handles multiple languages
83
84	Options:
85	- `--no-comments`: Skip comment sections
86	- `--no-tables`: Skip data tables
87	- `--precision`: Favor precision over recall
88	- `--recall`: Extract more content (may include some noise)
89
90	### Method 3: Fallback (curl + basic parsing)
91
92	```bash
93	# Download and extract basic content
94	curl -s "URL" \| python3 -c "
95	from html.parser import HTMLParser
96	import sys
97
98	class ArticleExtractor(HTMLParser):
99	def __init__(self):
100	super().__init__()
101	self.in_content = False
102	self.content = []
103	self.skip_tags = {'script', 'style', 'nav', 'header', 'footer', 'aside'}
104	self.current_tag = None
105
106	def handle_starttag(self, tag, attrs):
107	if tag not in self.skip_tags:
108	if tag in {'p', 'article', 'main', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6'}:
109	self.in_content = True
110	self.current_tag = tag
111
112	def handle_data(self, data):
113	if self.in_content and data.strip():
114	self.content.append(data.strip())
115
116	def get_content(self):
117	return '\n\n'.join(self.content)
118
119	parser = ArticleExtractor()
120	parser.feed(sys.stdin.read())
121	print(parser.get_content())
122	" > article.txt
123	```
124
125	Note: This is less reliable but works without dependencies.
126
127	## Getting Article Title
128
129	Extract title for filename:
130
131	### Using reader:
132	```bash
133	# reader outputs markdown with title at top
134	TITLE=$(reader "URL" \| head -n 1 \| sed 's/^# //')
135	```
136
137	### Using trafilatura:
138	```bash
139	# Get metadata including title
140	TITLE=$(trafilatura --URL "URL" --json \| python3 -c "import json, sys; print(json.load(sys.stdin)['title'])")
141	```
142
143	### Using curl (fallback):
144	```bash
145	TITLE=$(curl -s "URL" \| grep -oP '<title>\K[^<]+' \| sed 's/ - .//' \| sed 's/ \| .//')
146	```
147
148	## Filename Creation
149
150	Clean title for filesystem:
151
152	```bash
153	# Get title
154	TITLE="Article Title from Website"
155
156	# Clean for filesystem (remove special chars, limit length)
157	FILENAME=$(echo "$TITLE" \| tr '/' '-' \| tr ':' '-' \| tr '?' '' \| tr '"' '' \| tr '<' '' \| tr '>' '' \| tr '\|' '-' \| cut -c 1-100 \| sed 's/ *$//')
158
159	# Add extension
160	FILENAME="${FILENAME}.txt"
161	```
162
163	## Complete Workflow
164
165	```bash
166	ARTICLE_URL="https://example.com/article"
167
168	# Check for tools
169	if command -v reader &> /dev/null; then
170	TOOL="reader"
171	echo "Using reader (Mozilla Readability)"
172	elif command -v trafilatura &> /dev/null; then
173	TOOL="trafilatura"
174	echo "Using trafilatura"
175	else
176	TOOL="fallback"
177	echo "Using fallback method (may be less accurate)"
178	fi
179
180	# Extract article
181	case $TOOL in
182	reader)
183	# Get content
184	reader "$ARTICLE_URL" > temp_article.txt
185
186	# Get title (first line after # in markdown)
187	TITLE=$(head -n 1 temp_article.txt \| sed 's/^# //')
188	;;
189
190	trafilatura)
191	# Get title from metadata
192	METADATA=$(trafilatura --URL

People who liked this also viewed

6 picks

Type

article-extractor

498stars65forks

View on GitHub

How to install article-extractor?

michalparkola/tapestry-skills/article-extractor

$npx -y skills add michalparkola/tapestry-skills --skill article-extractor

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Article Extractor
2
3	This skill extracts the main content from web articles and blog posts, removing navigation, ads, newsletter signups, and other clutter. Saves clean, readable text.
4
5	## When to Use This Skill
6
7	Activate when the user:
8	- Provides an article/blog URL and wants the text content
9	- Asks to "download this article"
10	- Wants to "extract the content from [URL]"
11	- Asks to "save this blog post as text"
12	- Needs clean article text without distractions
13
14	## How It Works
15
16	### Priority Order:
17	1. Check if tools are installed (reader or trafilatura)
18	2. Download and extract article using best available tool
19	3. Clean up the content (remove extra whitespace, format properly)
20	4. Save to file with article title as filename
21	5. Confirm location and show preview
22
23	## Installation Check
24
25	Check for article extraction tools in this order:
26
27	### Option 1: reader (Recommended - Mozilla's Readability)
28
29	```bash
30	command -v reader
31	```
32
33	If not installed:
34	```bash
35	npm install -g @mozilla/readability-cli
36	# or
37	npm install -g reader-cli
38	```
39
40	### Option 2: trafilatura (Python-based, very good)
41
42	```bash
43	command -v trafilatura
44	```
45
46	If not installed:
47	```bash
48	pip3 install trafilatura
49	```
50
51	### Option 3: Fallback (curl + simple parsing)
52
53	If no tools available, use basic curl + text extraction (less reliable but works)
54
55	## Extraction Methods
56
57	### Method 1: Using reader (Best for most articles)
58
59	```bash
60	# Extract article
61	reader "URL" > article.txt
62	```
63
64	Pros:
65	- Based on Mozilla's Readability algorithm
66	- Excellent at removing clutter
67	- Preserves article structure
68
69	### Method 2: Using trafilatura (Best for blogs/news)
70
71	```bash
72	# Extract article
73	trafilatura --URL "URL" --output-format txt > article.txt
74
75	# Or with more options
76	trafilatura --URL "URL" --output-format txt --no-comments --no-tables > article.txt
77	```
78
79	Pros:
80	- Very accurate extraction
81	- Good with various site structures
82	- Handles multiple languages
83
84	Options:
85	- `--no-comments`: Skip comment sections
86	- `--no-tables`: Skip data tables
87	- `--precision`: Favor precision over recall
88	- `--recall`: Extract more content (may include some noise)
89
90	### Method 3: Fallback (curl + basic parsing)
91
92	```bash
93	# Download and extract basic content
94	curl -s "URL" \| python3 -c "
95	from html.parser import HTMLParser
96	import sys
97
98	class ArticleExtractor(HTMLParser):
99	def __init__(self):
100	super().__init__()
101	self.in_content = False
102	self.content = []
103	self.skip_tags = {'script', 'style', 'nav', 'header', 'footer', 'aside'}
104	self.current_tag = None
105
106	def handle_starttag(self, tag, attrs):
107	if tag not in self.skip_tags:
108	if tag in {'p', 'article', 'main', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6'}:
109	self.in_content = True
110	self.current_tag = tag
111
112	def handle_data(self, data):
113	if self.in_content and data.strip():
114	self.content.append(data.strip())
115
116	def get_content(self):
117	return '\n\n'.join(self.content)
118
119	parser = ArticleExtractor()
120	parser.feed(sys.stdin.read())
121	print(parser.get_content())
122	" > article.txt
123	```
124
125	Note: This is less reliable but works without dependencies.
126
127	## Getting Article Title
128
129	Extract title for filename:
130
131	### Using reader:
132	```bash
133	# reader outputs markdown with title at top
134	TITLE=$(reader "URL" \| head -n 1 \| sed 's/^# //')
135	```
136
137	### Using trafilatura:
138	```bash
139	# Get metadata including title
140	TITLE=$(trafilatura --URL "URL" --json \| python3 -c "import json, sys; print(json.load(sys.stdin)['title'])")
141	```
142
143	### Using curl (fallback):
144	```bash
145	TITLE=$(curl -s "URL" \| grep -oP '<title>\K[^<]+' \| sed 's/ - .//' \| sed 's/ \| .//')
146	```
147
148	## Filename Creation
149
150	Clean title for filesystem:
151
152	```bash
153	# Get title
154	TITLE="Article Title from Website"
155
156	# Clean for filesystem (remove special chars, limit length)
157	FILENAME=$(echo "$TITLE" \| tr '/' '-' \| tr ':' '-' \| tr '?' '' \| tr '"' '' \| tr '<' '' \| tr '>' '' \| tr '\|' '-' \| cut -c 1-100 \| sed 's/ *$//')
158
159	# Add extension
160	FILENAME="${FILENAME}.txt"
161	```
162
163	## Complete Workflow
164
165	```bash
166	ARTICLE_URL="https://example.com/article"
167
168	# Check for tools
169	if command -v reader &> /dev/null; then
170	TOOL="reader"
171	echo "Using reader (Mozilla Readability)"
172	elif command -v trafilatura &> /dev/null; then
173	TOOL="trafilatura"
174	echo "Using trafilatura"
175	else
176	TOOL="fallback"
177	echo "Using fallback method (may be less accurate)"
178	fi
179
180	# Extract article
181	case $TOOL in
182	reader)
183	# Get content
184	reader "$ARTICLE_URL" > temp_article.txt
185
186	# Get title (first line after # in markdown)
187	TITLE=$(head -n 1 temp_article.txt \| sed 's/^# //')
188	;;
189
190	trafilatura)
191	# Get title from metadata
192	METADATA=$(trafilatura --URL

People who liked this also viewed

6 picks

Type