geo-crawlers
starsforks
How to install geo-crawlers?
zubair-trabzada/geo-seo-claude/geo-crawlers
$npx -y skills add zubair-trabzada/geo-seo-claude --skill geo-crawlersAI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which AI crawlers can access the site. Provides a complete access map and recommendations for maximizing AI visibility while maintaining appropriate control.
SKILL.md
| 1 | # AI Crawler Access Analysis Skill |
| 2 | |
| 3 | ## Purpose |
| 4 | |
| 5 | This skill analyzes a website's accessibility to AI crawlers -- the bots that AI companies use to discover, index, and train on web content. If AI crawlers are blocked, the site's content cannot appear in AI-generated responses regardless of its quality. Crawler access is the foundational technical requirement for GEO. |
| 6 | |
| 7 | ## Key Insight |
| 8 | |
| 9 | As of early 2026, many websites inadvertently block AI crawlers through overly aggressive robots.txt rules, inherited from legacy SEO configurations. An Originality.ai 2025 study found that over 35% of the top 1,000 websites block at least one major AI crawler, and 5-10% block all AI crawlers. Blocking AI crawlers is the single fastest way to become invisible in AI-generated search results. |
| 10 | |
| 11 | --- |
| 12 | |
| 13 | ## Complete AI Crawler Reference |
| 14 | |
| 15 | ### Tier 1: Critical for AI Search Visibility (RECOMMEND: ALLOW) |
| 16 | |
| 17 | These crawlers power the AI search products where users actively look for answers. Blocking them directly reduces your visibility in AI-generated responses. |
| 18 | |
| 19 | #### GPTBot |
| 20 | - **Operator:** OpenAI |
| 21 | - **User-Agent:** `GPTBot` |
| 22 | - **Full User-Agent String:** `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)` |
| 23 | - **Purpose:** Fetches content for ChatGPT's web browsing, plugins, and search features. Content accessed by GPTBot may be used to improve OpenAI models. |
| 24 | - **Impact of Blocking:** Content will NOT appear in ChatGPT Search results or be accessible when users ask ChatGPT to browse the web. This is the highest-impact AI crawler to allow. |
| 25 | - **Recommendation:** **ALLOW** -- ChatGPT has 300M+ weekly active users as of 2025. Blocking GPTBot removes your content from one of the largest AI search surfaces. |
| 26 | |
| 27 | #### OAI-SearchBot |
| 28 | - **Operator:** OpenAI |
| 29 | - **User-Agent:** `OAI-SearchBot` |
| 30 | - **Full User-Agent String:** `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://docs.openai.com/bots/overview)` |
| 31 | - **Purpose:** Specifically powers ChatGPT's search feature. Unlike GPTBot, content accessed by OAI-SearchBot is NOT used for model training -- only for live search results. |
| 32 | - **Impact of Blocking:** Content will not appear in ChatGPT's search results even if GPTBot is allowed. |
| 33 | - **Recommendation:** **ALLOW** -- This is a search-only crawler with no training implications. There is no strategic reason to block it. |
| 34 | |
| 35 | #### ChatGPT-User |
| 36 | - **Operator:** OpenAI |
| 37 | - **User-Agent:** `ChatGPT-User` |
| 38 | - **Full User-Agent String:** `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)` |
| 39 | - **Purpose:** Used when a ChatGPT user explicitly asks the model to visit a specific URL. Acts like a browser agent on behalf of the user. |
| 40 | - **Impact of Blocking:** ChatGPT cannot visit your pages when users ask it to read or summarize them. This prevents direct user-initiated traffic. |
| 41 | - **Recommendation:** **ALLOW** -- Blocking this bot prevents users who are actively trying to engage with your content from accessing it through ChatGPT. |
| 42 | |
| 43 | #### ClaudeBot |
| 44 | - **Operator:** Anthropic |
| 45 | - **User-Agent:** `ClaudeBot` |
| 46 | - **Full User-Agent String:** `ClaudeBot/1.0; +https://www.anthropic.com/claude-bot` |
| 47 | - **Purpose:** Fetches web content for Claude's features including web search, citations, and analysis tools. |
| 48 | - **Impact of Blocking:** Content will not be accessible to Claude for web search or when users ask Claude to analyze specific URLs. |
| 49 | - **Recommendation:** **ALLOW** -- Claude is a major AI assistant with growing market share. Blocking ClaudeBot reduces your AI search footprint. |
| 50 | |
| 51 | #### PerplexityBot |
| 52 | - **Operator:** Perplexity AI |
| 53 | - **User-Agent:** `PerplexityBot` |
| 54 | - **Full User-Agent String:** `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)` |
| 55 | - **Purpose:** Powers Perplexity's AI search engine, which provides sourced answers with direct citations and links back to source pages. |
| 56 | - **Impact of Blocking:** Content will not appear in Perplexity search results. Perplexity is one of the best referral traffic sources among AI search products because it always displays source links. |
| 57 | - **Recommendation:** **ALLOW** -- Perplexity drives actual referral traffic and always attributes sources. High-value AI crawler for publishers and businesses. |
| 58 | |
| 59 | --- |
| 60 | |
| 61 | ### Tier 2: Important for Broader AI Ecosystem (RECOMMEND: ALLOW) |
| 62 | |
| 63 | These crawlers serve large AI platforms or search ecosystems. Allowing them increases your content's reach. |
| 64 | |
| 65 | #### Google-Extended |
| 66 | - **Operator:** Google |
| 67 | - **User-Agent:** `Google-Extended` |
| 68 | - **Purpose:** Controls whether Google uses you |
People who liked this also viewed
6 picksType
- seo-auditWhen the user wants to audit, review, or diagnose SEO issues on their site. Also use when the user mentions "SEO audit," "technical SEO," "why am I not ranking," "SEO issues," "on-page SEO," "meta tagSkillsJul 2026
- copywritingWhen the user wants to write, rewrite, or improve marketing copy for any page — including homepage, landing pages, pricing pages, feature pages, about pages, or product pages. Also use when the user sSkillsJul 2026
- marketing-psychologyWhen the user wants to apply psychological principles, mental models, or behavioral science to marketing. Also use when the user mentions 'psychology,' 'mental models,' 'cognitive bias,' 'persuasion,'SkillsJul 2026
- content-strategyWhen the user wants to plan a content strategy, decide what content to create, or figure out what topics to cover. Also use when the user mentions "content strategy," "what should I write about," "conSkillsJul 2026
- programmatic-seoWhen the user wants to create SEO-driven pages at scale using templates and data. Also use when the user mentions "programmatic SEO," "template pages," "pages at scale," "directory pages," "location pSkillsJul 2026
- marketing-ideasWhen the user needs marketing ideas, inspiration, or strategies for their SaaS or software product. Also use when the user asks for 'marketing ideas,' 'growth ideas,' 'how to market,' 'marketing stratSkillsJul 2026
