seo-technical

Name: seo-technical
Author: AgriciDaniel

3.9kinstalls1.8kforks

How to install seo-technical?

agricidaniel/claude-seo/seo-technical

$npx -y skills add AgriciDaniel/claude-seo --skill seo-technical

Installs into the current project.

Summary

Technical SEO audit across 9 categories: crawlability, indexability, security, URL structure, mobile, Core Web Vitals, structured data, JavaScript rendering, and IndexNow protocol. Use when user says "technical SEO", "crawl issues", "robots.txt", "Core Web Vitals", "site speed",

Files · 1

View on GitHub

SKILL.md

1	# Technical SEO Audit
2
3	## Categories
4
5	### 1. Crawlability
6	- robots.txt: exists, valid, not blocking important resources
7	- XML sitemap: run `claude-seo run sitemap_discovery.py <url> --json`; require a
8	valid entry in `found`, and report stale or unsafe robots.txt declarations
9	separately from working fallback locations
10	- Noindex tags: intentional vs accidental
11	- Crawl depth: important pages within 3 clicks of homepage
12	- JavaScript rendering: check if critical content requires JS execution
13	- Crawl budget: for large sites (>10k pages), efficiency matters
14	- Googlebot fetch limits: Googlebot fetches the first 2MB of HTML and first 64MB of a PDF (uncompressed; 15MB is the broader crawler-infra default). Long-standing, not a 2026 change, but inline base64 images, oversized inline CSS/JS, or bloated nav can push critical content/JSON-LD past the cap and out of the index. Keep key content + structured data within the first 2MB.
15	- Crawl rate auto-adjusts (backs off on 5xx/slow responses); there is no manual crawl-rate control (the legacy Search Console setting was removed Jan 2024). Influence crawling via sitemaps, server responsiveness, and robots controls.
16	- Google's canonical crawling/robots reference moved to developers.google.com/crawling (migrated 2025-11-20); IP-range files relocated to `/crawling/ipranges/` and `googlebot.json` was renamed `common-crawlers.json`.
17
18	#### AI Crawler Management
19
20	As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.
21
22	Known AI crawlers:
23
24	\| Crawler \| Company \| robots.txt token \| Purpose \|
25	\|---------\|---------\|-----------------\|---------\|
26	\| GPTBot \| OpenAI \| `GPTBot` \| Model training \|
27	\| ChatGPT-User \| OpenAI \| `ChatGPT-User` \| Real-time browsing \|
28	\| ClaudeBot \| Anthropic \| `ClaudeBot` \| Model training \|
29	\| PerplexityBot \| Perplexity \| `PerplexityBot` \| Search index + training \|
30	\| Bytespider \| ByteDance \| `Bytespider` \| Model training \|
31	\| Google-Extended \| Google \| `Google-Extended` \| Gemini training (NOT search) \|
32	\| CCBot \| Common Crawl \| `CCBot` \| Open dataset \|
33
34	Key distinctions:
35	- Blocking `Google-Extended` prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`)
36	- Blocking `GPTBot` prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (`ChatGPT-User`)
37	- ~3-5% of websites now use AI-specific robots.txt rules
38
39	Example, selective AI crawler blocking:
40	```
41	# Allow search indexing, block AI training crawlers
42	User-agent: GPTBot
43	Disallow: /
44
45	User-agent: Google-Extended
46	Disallow: /
47
48	User-agent: Bytespider
49	Disallow: /
50
51	# Allow all other crawlers (including Googlebot for search)
52	User-agent: *
53	Allow: /
54	```
55
56	Recommendation: Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the `seo-geo` skill for the full AI crawler/fetcher taxonomy.
57
58	> User-triggered fetchers ignore robots.txt by design. Google now documents Google-Agent (Project Mariner, agentic browsing) plus Google-NotebookLM and Google Messages as user-triggered fetchers that cannot be blocked via robots.txt. Use server-side access controls instead. By contrast, `Google-Extended` and `Google-CloudVertexBot` obey robots.txt. Emerging: Web Bot Auth (RFC 9421) lets bots authenticate cryptographically via a `Signature-Agent` header + key directory at `agent.bot.goog` (used by Google-Agent); reverse-DNS verification remains the fallback.
59
60	### 2. Indexability
61	- Canonical tags: self-referencing, no conflicts with noindex
62	- Duplicate content: near-duplicates, parameter URLs, www vs non-www
63	- Thin content: pages below minimum word counts per type
64	- Pagination: rel=next/prev or load-more pattern
65	- Hreflang: correct for multi-language/multi-region sites
66	- Index bloat: unnecessary pages consuming crawl budget
67
68	### 3. Security
69	- HTTPS: enforced, valid SSL certificate, no mixed content
70	- Security headers:
71	- Content-Security-Policy (CSP)
72	- Strict-Transport-Security (HSTS)
73	- X-Frame-Options
74	- X-Content-Type-Options
75	- Referrer-Policy
76	- HSTS preload: check preload list inclusion for high-security sites
77	- Back-button hijacking (spam-policy violation, malicious practices): flag pages that defeat the Back button via `history.pushState`/`replaceState` (including scripts injected by third-party ad/library platfo

People who liked this also viewed

6 picks

Type

seo-technical

3.9kinstalls1.8kforks

View on GitHub

How to install seo-technical?

agricidaniel/claude-seo/seo-technical

$npx -y skills add AgriciDaniel/claude-seo --skill seo-technical

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Technical SEO Audit
2
3	## Categories
4
5	### 1. Crawlability
6	- robots.txt: exists, valid, not blocking important resources
7	- XML sitemap: run `claude-seo run sitemap_discovery.py <url> --json`; require a
8	valid entry in `found`, and report stale or unsafe robots.txt declarations
9	separately from working fallback locations
10	- Noindex tags: intentional vs accidental
11	- Crawl depth: important pages within 3 clicks of homepage
12	- JavaScript rendering: check if critical content requires JS execution
13	- Crawl budget: for large sites (>10k pages), efficiency matters
14	- Googlebot fetch limits: Googlebot fetches the first 2MB of HTML and first 64MB of a PDF (uncompressed; 15MB is the broader crawler-infra default). Long-standing, not a 2026 change, but inline base64 images, oversized inline CSS/JS, or bloated nav can push critical content/JSON-LD past the cap and out of the index. Keep key content + structured data within the first 2MB.
15	- Crawl rate auto-adjusts (backs off on 5xx/slow responses); there is no manual crawl-rate control (the legacy Search Console setting was removed Jan 2024). Influence crawling via sitemaps, server responsiveness, and robots controls.
16	- Google's canonical crawling/robots reference moved to developers.google.com/crawling (migrated 2025-11-20); IP-range files relocated to `/crawling/ipranges/` and `googlebot.json` was renamed `common-crawlers.json`.
17
18	#### AI Crawler Management
19
20	As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.
21
22	Known AI crawlers:
23
24	\| Crawler \| Company \| robots.txt token \| Purpose \|
25	\|---------\|---------\|-----------------\|---------\|
26	\| GPTBot \| OpenAI \| `GPTBot` \| Model training \|
27	\| ChatGPT-User \| OpenAI \| `ChatGPT-User` \| Real-time browsing \|
28	\| ClaudeBot \| Anthropic \| `ClaudeBot` \| Model training \|
29	\| PerplexityBot \| Perplexity \| `PerplexityBot` \| Search index + training \|
30	\| Bytespider \| ByteDance \| `Bytespider` \| Model training \|
31	\| Google-Extended \| Google \| `Google-Extended` \| Gemini training (NOT search) \|
32	\| CCBot \| Common Crawl \| `CCBot` \| Open dataset \|
33
34	Key distinctions:
35	- Blocking `Google-Extended` prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`)
36	- Blocking `GPTBot` prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (`ChatGPT-User`)
37	- ~3-5% of websites now use AI-specific robots.txt rules
38
39	Example, selective AI crawler blocking:
40	```
41	# Allow search indexing, block AI training crawlers
42	User-agent: GPTBot
43	Disallow: /
44
45	User-agent: Google-Extended
46	Disallow: /
47
48	User-agent: Bytespider
49	Disallow: /
50
51	# Allow all other crawlers (including Googlebot for search)
52	User-agent: *
53	Allow: /
54	```
55
56	Recommendation: Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the `seo-geo` skill for the full AI crawler/fetcher taxonomy.
57
58	> User-triggered fetchers ignore robots.txt by design. Google now documents Google-Agent (Project Mariner, agentic browsing) plus Google-NotebookLM and Google Messages as user-triggered fetchers that cannot be blocked via robots.txt. Use server-side access controls instead. By contrast, `Google-Extended` and `Google-CloudVertexBot` obey robots.txt. Emerging: Web Bot Auth (RFC 9421) lets bots authenticate cryptographically via a `Signature-Agent` header + key directory at `agent.bot.goog` (used by Google-Agent); reverse-DNS verification remains the fallback.
59
60	### 2. Indexability
61	- Canonical tags: self-referencing, no conflicts with noindex
62	- Duplicate content: near-duplicates, parameter URLs, www vs non-www
63	- Thin content: pages below minimum word counts per type
64	- Pagination: rel=next/prev or load-more pattern
65	- Hreflang: correct for multi-language/multi-region sites
66	- Index bloat: unnecessary pages consuming crawl budget
67
68	### 3. Security
69	- HTTPS: enforced, valid SSL certificate, no mixed content
70	- Security headers:
71	- Content-Security-Policy (CSP)
72	- Strict-Transport-Security (HSTS)
73	- X-Frame-Options
74	- X-Content-Type-Options
75	- Referrer-Policy
76	- HSTS preload: check preload list inclusion for high-security sites
77	- Back-button hijacking (spam-policy violation, malicious practices): flag pages that defeat the Back button via `history.pushState`/`replaceState` (including scripts injected by third-party ad/library platfo

People who liked this also viewed

6 picks

Type