docs-crawler

Name: docs-crawler
Author: CaesiumY

23stars2forks

How to install docs-crawler?

caesiumy/ko-design-md/docs-crawler

$npx -y skills add CaesiumY/ko-design-md --skill docs-crawler

Installs into the current project.

Summary

Crawl an entire documentation or design-system website into one LLM-ready Markdown corpus. Discovers every page from the site's sitemap.xml (with a same-origin link-following fallback when there is no sitemap), extracts each page's main content as clean Markdown, downloads each p

Files · 1

View on GitHub

SKILL.md

1	# docs-crawler
2
3	Crawl a documentation website into a single Markdown corpus that an LLM can
4	consume as context.
5
6	## When to use
7
8	The user wants to capture a multi-page documentation, design-system, or
9	reference site as Markdown — to feed an LLM, archive it, or research a design
10	system. This skill is also invoked by the `design-md` skill's research phase.
11
12	## Inputs
13
14	- Site URL (required) — the documentation site, e.g.
15	`https://socarframe.socar.kr/`. Any page on the site works; page discovery
16	starts from the site's `sitemap.xml`.
17	- Output directory (optional) — where to write the corpus. Defaults to
18	`./{host}-docs/` in the current working directory.
19
20	If the user has not provided a URL, ask for one — do not guess.
21
22	## Running the crawl
23
24	From the repository root:
25
26	```
27	pnpm crawl:docs <site-url> [--out <dir>] [--external-images]
28	```
29
30	Equivalent direct form:
31
32	```
33	pnpm exec tsx .claude/skills/docs-crawler/scripts/crawl.ts <site-url> [--out <dir>]
34	```
35
36	The engine discovers pages via `sitemap.xml` (falling back to same-origin
37	link-following), fetches each page, extracts the main content, and converts it
38	to Markdown. It prints per-page progress and a final summary line.
39
40	By default it also localizes images: every external image and inline base64
41	`data:` image is downloaded into `crawl/images/` and the Markdown is rewritten
42	to relative paths, so the corpus is self-contained for renderers that can't
43	fetch external URLs (Claude Design, offline previews). Pass `--external-images`
44	to skip downloading and keep the original external URLs instead.
45
46	## Output
47
48	These artifacts are written under the output directory:
49
50	- `crawl-corpus.md` — every successful page merged into one document, with
51	a table of contents and a `Source:` URL per page. This is the primary
52	deliverable.
53	- `crawl/pages/{NNN}-{slug}.md` — one file per page, each with
54	`source_url` / `title` / `method` frontmatter.
55	- `crawl/images/` — downloaded image files (skipped with
56	`--external-images`), referenced from the corpus and page files by relative
57	paths.
58	- `crawl/manifest.json` — a per-URL audit log (status, fetch method,
59	extracted character count, error reason).
60
61	## After crawling
62
63	Report back to the user:
64
65	- How many pages succeeded versus failed (from the summary line or
66	`crawl/manifest.json`).
67	- The path to `crawl-corpus.md`.
68	- If any pages failed, list them from the manifest with the reason (e.g.
69	`disallowed by robots.txt`, `no extractable content`).
70
71	Do not paste the whole corpus into the conversation — point the user at the
72	file. The corpus can be large.
73
74	## Output location and the design-md skill
75
76	The `--out` argument controls where everything is written. When the `design-md`
77	skill calls this crawler during its research phase, it passes `--out` pointing
78	at its cache directory (`.claude/cache/design-md/{slug}/`), so the resulting
79	`crawl-corpus.md` (and the localized `crawl/images/`) land where
80	`research-collector` can read and cite them. The whole cache directory is
81	gitignored, so downloaded images stay out of version control. Nothing special
82	is needed for that case — just run the crawl with the given `--out`.
83
84	## Notes and edge cases
85
86	- No sitemap — discovery automatically falls back to following same-origin
87	links from the start URL.
88	- JavaScript-rendered sites — if a page returns an empty shell, the engine
89	re-fetches it through a headless browser. Chromium is installed automatically
90	on first need (~150 MB, one-time). Static sites never launch a browser.
91	- Images are downloaded into `crawl/images/` by default and referenced by
92	relative paths, so the corpus is

People who liked this also viewed

6 picks

Type

docs-crawler

23stars2forks

View on GitHub

How to install docs-crawler?

caesiumy/ko-design-md/docs-crawler

$npx -y skills add CaesiumY/ko-design-md --skill docs-crawler

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# docs-crawler
2
3	Crawl a documentation website into a single Markdown corpus that an LLM can
4	consume as context.
5
6	## When to use
7
8	The user wants to capture a multi-page documentation, design-system, or
9	reference site as Markdown — to feed an LLM, archive it, or research a design
10	system. This skill is also invoked by the `design-md` skill's research phase.
11
12	## Inputs
13
14	- Site URL (required) — the documentation site, e.g.
15	`https://socarframe.socar.kr/`. Any page on the site works; page discovery
16	starts from the site's `sitemap.xml`.
17	- Output directory (optional) — where to write the corpus. Defaults to
18	`./{host}-docs/` in the current working directory.
19
20	If the user has not provided a URL, ask for one — do not guess.
21
22	## Running the crawl
23
24	From the repository root:
25
26	```
27	pnpm crawl:docs <site-url> [--out <dir>] [--external-images]
28	```
29
30	Equivalent direct form:
31
32	```
33	pnpm exec tsx .claude/skills/docs-crawler/scripts/crawl.ts <site-url> [--out <dir>]
34	```
35
36	The engine discovers pages via `sitemap.xml` (falling back to same-origin
37	link-following), fetches each page, extracts the main content, and converts it
38	to Markdown. It prints per-page progress and a final summary line.
39
40	By default it also localizes images: every external image and inline base64
41	`data:` image is downloaded into `crawl/images/` and the Markdown is rewritten
42	to relative paths, so the corpus is self-contained for renderers that can't
43	fetch external URLs (Claude Design, offline previews). Pass `--external-images`
44	to skip downloading and keep the original external URLs instead.
45
46	## Output
47
48	These artifacts are written under the output directory:
49
50	- `crawl-corpus.md` — every successful page merged into one document, with
51	a table of contents and a `Source:` URL per page. This is the primary
52	deliverable.
53	- `crawl/pages/{NNN}-{slug}.md` — one file per page, each with
54	`source_url` / `title` / `method` frontmatter.
55	- `crawl/images/` — downloaded image files (skipped with
56	`--external-images`), referenced from the corpus and page files by relative
57	paths.
58	- `crawl/manifest.json` — a per-URL audit log (status, fetch method,
59	extracted character count, error reason).
60
61	## After crawling
62
63	Report back to the user:
64
65	- How many pages succeeded versus failed (from the summary line or
66	`crawl/manifest.json`).
67	- The path to `crawl-corpus.md`.
68	- If any pages failed, list them from the manifest with the reason (e.g.
69	`disallowed by robots.txt`, `no extractable content`).
70
71	Do not paste the whole corpus into the conversation — point the user at the
72	file. The corpus can be large.
73
74	## Output location and the design-md skill
75
76	The `--out` argument controls where everything is written. When the `design-md`
77	skill calls this crawler during its research phase, it passes `--out` pointing
78	at its cache directory (`.claude/cache/design-md/{slug}/`), so the resulting
79	`crawl-corpus.md` (and the localized `crawl/images/`) land where
80	`research-collector` can read and cite them. The whole cache directory is
81	gitignored, so downloaded images stay out of version control. Nothing special
82	is needed for that case — just run the crawl with the given `--out`.
83
84	## Notes and edge cases
85
86	- No sitemap — discovery automatically falls back to following same-origin
87	links from the start URL.
88	- JavaScript-rendered sites — if a page returns an empty shell, the engine
89	re-fetches it through a headless browser. Chromium is installed automatically
90	on first need (~150 MB, one-time). Static sites never launch a browser.
91	- Images are downloaded into `crawl/images/` by default and referenced by
92	relative paths, so the corpus is

People who liked this also viewed

6 picks

Type