wigolo-crawl
byknockoutez· 12 skills
Local-first multi-page crawl with sitemap, BFS, DFS, and URL-map strategies, anchor-fragment dedup, rate limiting, robots.txt respect, and automatic local cache population. Use when the user wants to index documentation, crawl a docs site, extract all pages under a path, or says "crawl", "index this site", "get all the docs", "bulk extract". Prefer when crawled pages should land in a reusable local cache for later cache / find_similar queries.
How to install wigolo-crawl?
knockoutez/wigolo/wigolo-crawl
$npx -y skills add knockoutez/wigolo --skill wigolo-crawlInstalls into the current project.
›Prefer a prompt? Paste this to your agent
Run `npx skills use "https://github.com/knockoutez/wigolo" --skill "knockoutez/wigolo/wigolo-crawl"` and follow the generated skill instructions now. Read its complete output, redirecting it to a temporary file first if necessary. Resolve relative paths from the supporting-files directory it provides.
Use the skills in "https://github.com/knockoutez/wigolo" that are relevant to the current task. Run `npx skills add "https://github.com/knockoutez/wigolo"` and select the relevant skills, then follow their instructions.
Related
6 picksType
- agent-browserBrowser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking…Jul 2026
- just-scrapeSearch, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI.Jul 2026
- browser-actBrowser automation CLI for AI agents. NEVER run browser-act commands directly via Bash — always invoke this skill first.Jul 2026
- playwright-cliAutomate browser interactions, test web pages and work with Playwright tests.Jul 2026
- browser-useDirect browser control via CDP for web interaction: automation, scraping, testing, screenshots, and site/app work.Jul 2026
- browser-act-skill-forgeForges reusable Skill packages (SKILL.md + scripts) from website exploration via browser-act — no re-exploration later.Jul 2026
