crawling

Name: crawling
Author: omnigentx

31stars13forks

How to install crawling?

omnigentx/jarvis/crawling

$npx -y skills add omnigentx/jarvis --skill crawling

Installs into the current project.

Summary

Crawl and download stories from ANY website into the local library — site-agnostic, language-agnostic. Use when the user wants to add a new story, download from the web, or check crawl progress. Flow: detect structure → build recipe → verify 3 chapters → full crawl (worker enumer

Files · 1

View on GitHub

SKILL.md

1	# STORY CRAWLING WORKFLOW (recipe-based, AI-driven)
2
3	You analyse page STRUCTURE and emit a small RECIPE. The backend worker enumerates
4	every chapter from that recipe and downloads them — the chapter list never passes
5	through you, so this scales to 1000+ chapter stories without burning tokens.
6
7	## Decision Tree
8	```
9	Add new story → has URL? use it : search_stories(query)
10	Check progress → get_crawl_status(job_id)
11	View library → local_list_stories()
12	```
13
14	## Step 0: GET A URL
15	- User gave a URL → use it directly (skip search).
16	- No URL → serpapi (Google search, fast) to find the story page. scrapling
17	is for fetching a known URL, not searching. (Legacy `web_search_stories` is a
18	slow per-site HTTP fallback — avoid unless serpapi is down.)
19
20	## Step 1: DETECT STRUCTURE
21	`detect_story_structure(url)` renders the page (JS-aware) and returns, all
22	LANGUAGE-AGNOSTIC:
23	- `link_patterns`: same-domain links grouped by URL shape (digits→#) with counts +
24	`within_story_root` flag + `selector_hint`. A high-count within-story pattern =
25	the chapter links.
26	- `content_candidates`: scored by paragraph density (pick narrative, not nav).
27	- `pagination_hints`, `story_root_guess`.
28
29	## Step 2: BUILD RECIPE (JSON)
30	- LIST mode (page lists many chapters):
31	`{mode:"list", story_root:"/slug/", overview_url, chapter_url_pattern:"<sample chapter URL>", content_selector, title_selector, next_page_selector?}`
32	- CHAIN mode (single chapter page + a "next" link):
33	`{mode:"chain", story_root:"/slug/", first_chapter_url, next_chapter_selector, content_selector, title_selector}`
34	- `story_root` is REQUIRED in both — the same-story guard that blocks drift into
35	recommended/other stories.
36	- Find `content_selector` on a CHAPTER page (overview pages have no chapter body).
37	Re-run detect_story_structure on a sample chapter URL if needed.
38
39	## Step 3: VERIFY — MANDATORY (3 chapters)
40	`verify_chapters(recipe_json)` fetches the first 3 chapters and checks length,
41	that chapters differ, and URL continuity.
42	- ok=false → read `issues`, fix the recipe (usually content_selector), re-verify.
43	- NEVER crawl an unverified recipe.
44
45	## Step 4: FULL CRAWL
46	`crawl_story(recipe_json, max_chapters=..., speed=1.0)` → returns job_id.
47	Report to the user ending with `[[[CRAWL_STARTED: job_id]]]`.
48
49	## Step 5: TRACKING / SELF-HEAL
50	- Progress on request → `get_crawl_status(job_id)`.
51	- If a crawl gets stuck, the worker pauses and AUTO-WAKES you with a
52	`[CRAWL SELF-HEAL]` message (stuck URL + current recipe). Then:
53	detect_story_structure(stuck_url) → diagnose → verify_chapters(fixed) →
54	`resume_crawl(job_id, recipe_patch_json=<fix>)`. If it's the real end of the
55	story → `resume_crawl(job_id, mark_done=true)`. If the site is blocked / changed
56	structurally → tell the user.
57
58	<violation>
59	- Crawling WITHOUT verify_chapters first → VIOLATION.
60	- A recipe without `story_root` (no same-story guard) → VIOLATION.
61	- Relying on language-specific words (e.g. "Chương") for detection → use URL/DOM
62	structure instead.
63	</violation>

People who liked this also viewed

6 picks

Type

crawling

31stars13forks

View on GitHub

How to install crawling?

omnigentx/jarvis/crawling

$npx -y skills add omnigentx/jarvis --skill crawling

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# STORY CRAWLING WORKFLOW (recipe-based, AI-driven)
2
3	You analyse page STRUCTURE and emit a small RECIPE. The backend worker enumerates
4	every chapter from that recipe and downloads them — the chapter list never passes
5	through you, so this scales to 1000+ chapter stories without burning tokens.
6
7	## Decision Tree
8	```
9	Add new story → has URL? use it : search_stories(query)
10	Check progress → get_crawl_status(job_id)
11	View library → local_list_stories()
12	```
13
14	## Step 0: GET A URL
15	- User gave a URL → use it directly (skip search).
16	- No URL → serpapi (Google search, fast) to find the story page. scrapling
17	is for fetching a known URL, not searching. (Legacy `web_search_stories` is a
18	slow per-site HTTP fallback — avoid unless serpapi is down.)
19
20	## Step 1: DETECT STRUCTURE
21	`detect_story_structure(url)` renders the page (JS-aware) and returns, all
22	LANGUAGE-AGNOSTIC:
23	- `link_patterns`: same-domain links grouped by URL shape (digits→#) with counts +
24	`within_story_root` flag + `selector_hint`. A high-count within-story pattern =
25	the chapter links.
26	- `content_candidates`: scored by paragraph density (pick narrative, not nav).
27	- `pagination_hints`, `story_root_guess`.
28
29	## Step 2: BUILD RECIPE (JSON)
30	- LIST mode (page lists many chapters):
31	`{mode:"list", story_root:"/slug/", overview_url, chapter_url_pattern:"<sample chapter URL>", content_selector, title_selector, next_page_selector?}`
32	- CHAIN mode (single chapter page + a "next" link):
33	`{mode:"chain", story_root:"/slug/", first_chapter_url, next_chapter_selector, content_selector, title_selector}`
34	- `story_root` is REQUIRED in both — the same-story guard that blocks drift into
35	recommended/other stories.
36	- Find `content_selector` on a CHAPTER page (overview pages have no chapter body).
37	Re-run detect_story_structure on a sample chapter URL if needed.
38
39	## Step 3: VERIFY — MANDATORY (3 chapters)
40	`verify_chapters(recipe_json)` fetches the first 3 chapters and checks length,
41	that chapters differ, and URL continuity.
42	- ok=false → read `issues`, fix the recipe (usually content_selector), re-verify.
43	- NEVER crawl an unverified recipe.
44
45	## Step 4: FULL CRAWL
46	`crawl_story(recipe_json, max_chapters=..., speed=1.0)` → returns job_id.
47	Report to the user ending with `[[[CRAWL_STARTED: job_id]]]`.
48
49	## Step 5: TRACKING / SELF-HEAL
50	- Progress on request → `get_crawl_status(job_id)`.
51	- If a crawl gets stuck, the worker pauses and AUTO-WAKES you with a
52	`[CRAWL SELF-HEAL]` message (stuck URL + current recipe). Then:
53	detect_story_structure(stuck_url) → diagnose → verify_chapters(fixed) →
54	`resume_crawl(job_id, recipe_patch_json=<fix>)`. If it's the real end of the
55	story → `resume_crawl(job_id, mark_done=true)`. If the site is blocked / changed
56	structurally → tell the user.
57
58	<violation>
59	- Crawling WITHOUT verify_chapters first → VIOLATION.
60	- A recipe without `story_root` (no same-story guard) → VIOLATION.
61	- Relying on language-specific words (e.g. "Chương") for detection → use URL/DOM
62	structure instead.
63	</violation>

People who liked this also viewed

6 picks

Type