read-web-scraper

Name: read-web-scraper
Author: SpaceZephyr

56stars18forks

How to install read-web-scraper?

spacezephyr/read-buddy/read-web-scraper

$npx -y skills add SpaceZephyr/read-buddy --skill read-web-scraper

Installs into the current project.

Summary

Fetch and extract content from web pages, converting HTML to clean markdown. Use when users want to read web articles, extract information from URLs, scrape web content, or when the built-in WebFetch tool fails due to network restrictions. Trigger when user provides URLs to read,

Files · 1

View on GitHub

SKILL.md

1	# Web Scraper
2
3	Fetch web page content and convert to clean markdown format.
4
5	## Usage
6
7	Run the fetch script to get web content:
8
9	```bash
10	python3 scripts/fetch_url.py <url> [options]
11	```
12
13	### Options
14
15	- `--timeout <seconds>`: Request timeout (default: 30)
16	- `--max-length <chars>`: Maximum output length (default: 100000)
17	- `--raw`: Output raw HTML instead of markdown
18
19	### Examples
20
21	Fetch single URL:
22	```bash
23	python3 scripts/fetch_url.py "https://example.com/article"
24	```
25
26	Fetch with custom timeout:
27	```bash
28	python3 scripts/fetch_url.py "https://example.com/article" --timeout 60
29	```
30
31	Fetch multiple URLs in parallel:
32	```bash
33	for url in "https://url1.com" "https://url2.com"; do
34	python3 scripts/fetch_url.py "$url" &
35	done
36	wait
37	```
38
39	## Workflow
40
41	1. Single URL: Run `fetch_url.py` with the URL
42	2. Multiple URLs: Run multiple fetch commands in parallel using background processes
43	3. Handle errors: If a URL fails, check:
44	- Network connectivity
45	- URL validity
46	- Website may block automated requests (try different User-Agent or use browser automation)
47
48	## Output Format
49
50	The script converts HTML to clean markdown:
51	- Headings → `#`, `##`, `###`, etc.
52	- Lists → `-` for unordered, `1.` for ordered
53	- Bold/Italic → `bold`, `italic`
54	- Code blocks preserved
55	- Navigation, footer, and ads removed
56
57	## Troubleshooting
58
59	403 Forbidden: Website blocks automated requests. Consider:
60	- Some sites require JavaScript rendering (not supported by this script)
61	- Try accessing from a different network
62
63	Timeout errors: Increase timeout with `--timeout 60`
64
65	Empty content: Website may require JavaScript to render content

People who liked this also viewed

6 picks

Type

read-web-scraper

56stars18forks

View on GitHub

How to install read-web-scraper?

spacezephyr/read-buddy/read-web-scraper

$npx -y skills add SpaceZephyr/read-buddy --skill read-web-scraper

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Web Scraper
2
3	Fetch web page content and convert to clean markdown format.
4
5	## Usage
6
7	Run the fetch script to get web content:
8
9	```bash
10	python3 scripts/fetch_url.py <url> [options]
11	```
12
13	### Options
14
15	- `--timeout <seconds>`: Request timeout (default: 30)
16	- `--max-length <chars>`: Maximum output length (default: 100000)
17	- `--raw`: Output raw HTML instead of markdown
18
19	### Examples
20
21	Fetch single URL:
22	```bash
23	python3 scripts/fetch_url.py "https://example.com/article"
24	```
25
26	Fetch with custom timeout:
27	```bash
28	python3 scripts/fetch_url.py "https://example.com/article" --timeout 60
29	```
30
31	Fetch multiple URLs in parallel:
32	```bash
33	for url in "https://url1.com" "https://url2.com"; do
34	python3 scripts/fetch_url.py "$url" &
35	done
36	wait
37	```
38
39	## Workflow
40
41	1. Single URL: Run `fetch_url.py` with the URL
42	2. Multiple URLs: Run multiple fetch commands in parallel using background processes
43	3. Handle errors: If a URL fails, check:
44	- Network connectivity
45	- URL validity
46	- Website may block automated requests (try different User-Agent or use browser automation)
47
48	## Output Format
49
50	The script converts HTML to clean markdown:
51	- Headings → `#`, `##`, `###`, etc.
52	- Lists → `-` for unordered, `1.` for ordered
53	- Bold/Italic → `bold`, `italic`
54	- Code blocks preserved
55	- Navigation, footer, and ads removed
56
57	## Troubleshooting
58
59	403 Forbidden: Website blocks automated requests. Consider:
60	- Some sites require JavaScript rendering (not supported by this script)
61	- Try accessing from a different network
62
63	Timeout errors: Increase timeout with `--timeout 60`
64
65	Empty content: Website may require JavaScript to render content

People who liked this also viewed

6 picks

Type