seo-crawlability

Name: seo-crawlability
Author: Hainrixz

29stars2forks

How to install seo-crawlability?

hainrixz/claude-seo-ai/seo-crawlability

$npx -y skills add Hainrixz/claude-seo-ai --skill seo-crawlability

Installs into the current project.

Summary

Audit and generate robots.txt and general crawl access for a page — verify robots.txt reachability and syntax, detect Disallow rules that block CSS/JS or important content, sanity-check crawl-delay, confirm a Sitemap directive, and assert overall crawl access for Googlebot/Bingbo

Files · 1

View on GitHub

SKILL.md

1	# seo-crawlability (M1)
2
3	Crawl access is the precondition for every other search signal: if Googlebot/Bingbot can't fetch the page and its assets, nothing else ranks. This module covers general-purpose crawl access only. AI-specific bot directives (GPTBot, Claude-SearchBot, etc.) and `llms.txt` live in `seo-ai-crawlers` (M14/M21); see `references/ai-crawlers.md` for that boundary.
4
5	## Audits
6	Working from the PageSnapshot (`rendered_dom` if present, else `raw_html`) plus a fetch of `/robots.txt`:
7	1. Reachability: `/robots.txt` returns 200 (a 404 means "allow all" but is worth flagging; a 5xx can suspend crawling).
8	2. Syntax: each line is a valid directive (`User-agent`, `Disallow`, `Allow`, `Sitemap`, `Crawl-delay`); flag unknown tokens, missing `User-agent` group headers, and BOM/encoding issues.
9	3. Asset blocking: any `Disallow` that blocks CSS/JS, fonts, or `/wp-includes/`-style paths — this breaks rendering and is a leading cause of "page looks broken to Google" (cross-check with M-render).
10	4. Content blocking: `Disallow` rules that hide important indexable paths from `Googlebot`/`Bingbot`.
11	5. Crawl-delay sanity: a large `Crawl-delay` (or one applied to the global group) can starve crawl budget; note that Googlebot ignores `Crawl-delay` but Bingbot honors it.
12	6. Sitemap directive: presence of at least one absolute `Sitemap:` URL.
13	7. Overall access: resolve the effective ruleset for `Googlebot` and `Bingbot` against the audited URL — does it end up allowed?
14
15	## Fixes
16	Generated edits are a diff for `fix`, mapped to the schema `fixable` field:
17	- AUTO: remove an accidental `Disallow` of CSS/JS or a key content path (additive un-block, verifiable); add a missing absolute `Sitemap:` line; repair malformed syntax (e.g. `Dissallow` typo, missing colon, group with no `User-agent` header).
18	- PROPOSED: tightening `Crawl-delay` or restructuring `User-agent` groups — drafted, requires per-item accept because intent may be deliberate.
19	- ADVISORY: changing what is intentionally disallowed (private/staging paths) — never written by the tool.
20	Never fabricate a sitemap URL or path: if the canonical sitemap location is unknown, emit a clearly-marked `TODO` placeholder for the user to fill, or ask.
21
22	## Verification
23	- `node ${CLAUDE_SKILL_DIR}/../../scripts/parse-robots-sitemap.mjs --url <u>` — fetches and parses `/robots.txt`, resolves the effective allow/deny for the named agent + URL, and checks for the `Sitemap:` directive (method `robots_parse`).
24	- When `/robots.txt` cannot be fetched (network/auth/edge block) the status is `needs_api`, never a false `pass`.
25
26	## Findings
27	Emit findings per `schema/finding.schema.json`. Examples:
28	- `M1.robots.blocks_css_js` — a `Disallow` matches CSS/JS the rendered page loads (severity 5, fail, `fixable: auto`, axis `search`, confidence `established`).
29	- `M1.robots.blocks_googlebot` — effective ruleset disallows the audited URL for `Googlebot` (severity 5, fail, `fixable: proposed`, axis `search`, confidence `established`).
30	- `M1.sitemap.missing_directive` — no `Sitemap:` line in robots.txt (severity 5, warn, `fixable: auto`, axis `search`, confidence `directional`).
31	Each finding: `evidence.observed` quotes the offending robots.txt line (or the resolved verdict) verbatim; `verification.reproduce` is the runnable command above; `expected_impact` is banded + confidence-tagged (no naked %).
32
33	## Honesty
34	- `robots.txt` controls crawling, not indexing: a `Disallow`-ed page can still be indexed (URL-only) from external links. To keep a page out of the index use a `noindex` meta tag and do not also `Disallow` it, or the crawler can't see the `noindex`.
35	- A missing or 404 `robots.txt` is not a defect — it means "crawl everything." Don't report it as a fail; flag only as informational.
36	- `Crawl-delay` is ignored by Googlebot; recommending it as a Google ranking/crawl lever is a myth — scope advice to Bingbot and other honoring agents.

People who liked this also viewed

6 picks

Type

seo-crawlability

29stars2forks

View on GitHub

How to install seo-crawlability?

hainrixz/claude-seo-ai/seo-crawlability

$npx -y skills add Hainrixz/claude-seo-ai --skill seo-crawlability

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# seo-crawlability (M1)
2
3	Crawl access is the precondition for every other search signal: if Googlebot/Bingbot can't fetch the page and its assets, nothing else ranks. This module covers general-purpose crawl access only. AI-specific bot directives (GPTBot, Claude-SearchBot, etc.) and `llms.txt` live in `seo-ai-crawlers` (M14/M21); see `references/ai-crawlers.md` for that boundary.
4
5	## Audits
6	Working from the PageSnapshot (`rendered_dom` if present, else `raw_html`) plus a fetch of `/robots.txt`:
7	1. Reachability: `/robots.txt` returns 200 (a 404 means "allow all" but is worth flagging; a 5xx can suspend crawling).
8	2. Syntax: each line is a valid directive (`User-agent`, `Disallow`, `Allow`, `Sitemap`, `Crawl-delay`); flag unknown tokens, missing `User-agent` group headers, and BOM/encoding issues.
9	3. Asset blocking: any `Disallow` that blocks CSS/JS, fonts, or `/wp-includes/`-style paths — this breaks rendering and is a leading cause of "page looks broken to Google" (cross-check with M-render).
10	4. Content blocking: `Disallow` rules that hide important indexable paths from `Googlebot`/`Bingbot`.
11	5. Crawl-delay sanity: a large `Crawl-delay` (or one applied to the global group) can starve crawl budget; note that Googlebot ignores `Crawl-delay` but Bingbot honors it.
12	6. Sitemap directive: presence of at least one absolute `Sitemap:` URL.
13	7. Overall access: resolve the effective ruleset for `Googlebot` and `Bingbot` against the audited URL — does it end up allowed?
14
15	## Fixes
16	Generated edits are a diff for `fix`, mapped to the schema `fixable` field:
17	- AUTO: remove an accidental `Disallow` of CSS/JS or a key content path (additive un-block, verifiable); add a missing absolute `Sitemap:` line; repair malformed syntax (e.g. `Dissallow` typo, missing colon, group with no `User-agent` header).
18	- PROPOSED: tightening `Crawl-delay` or restructuring `User-agent` groups — drafted, requires per-item accept because intent may be deliberate.
19	- ADVISORY: changing what is intentionally disallowed (private/staging paths) — never written by the tool.
20	Never fabricate a sitemap URL or path: if the canonical sitemap location is unknown, emit a clearly-marked `TODO` placeholder for the user to fill, or ask.
21
22	## Verification
23	- `node ${CLAUDE_SKILL_DIR}/../../scripts/parse-robots-sitemap.mjs --url <u>` — fetches and parses `/robots.txt`, resolves the effective allow/deny for the named agent + URL, and checks for the `Sitemap:` directive (method `robots_parse`).
24	- When `/robots.txt` cannot be fetched (network/auth/edge block) the status is `needs_api`, never a false `pass`.
25
26	## Findings
27	Emit findings per `schema/finding.schema.json`. Examples:
28	- `M1.robots.blocks_css_js` — a `Disallow` matches CSS/JS the rendered page loads (severity 5, fail, `fixable: auto`, axis `search`, confidence `established`).
29	- `M1.robots.blocks_googlebot` — effective ruleset disallows the audited URL for `Googlebot` (severity 5, fail, `fixable: proposed`, axis `search`, confidence `established`).
30	- `M1.sitemap.missing_directive` — no `Sitemap:` line in robots.txt (severity 5, warn, `fixable: auto`, axis `search`, confidence `directional`).
31	Each finding: `evidence.observed` quotes the offending robots.txt line (or the resolved verdict) verbatim; `verification.reproduce` is the runnable command above; `expected_impact` is banded + confidence-tagged (no naked %).
32
33	## Honesty
34	- `robots.txt` controls crawling, not indexing: a `Disallow`-ed page can still be indexed (URL-only) from external links. To keep a page out of the index use a `noindex` meta tag and do not also `Disallow` it, or the crawler can't see the `noindex`.
35	- A missing or 404 `robots.txt` is not a defect — it means "crawl everything." Don't report it as a fail; flag only as informational.
36	- `Crawl-delay` is ignored by Googlebot; recommending it as a Google ranking/crawl lever is a myth — scope advice to Bingbot and other honoring agents.

People who liked this also viewed

6 picks

Type