just-scrape

Name: just-scrape
Author: ScrapeGraphAI

245kinstalls7forks

How to install just-scrape?

scrapegraphai/just-scrape/just-scrape

$npx -y skills add scrapegraphai/just-scrape --skill just-scrape

Installs into the current project.

Summary

Search, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON from a site, crawl documentation or site sections, monitor a page for changes, inspe

Files · 1

View on GitHub

SKILL.md

1	# just-scrape CLI
2
3	Search, scrape, crawl, extract structured JSON, and monitor page changes using the just-scrape CLI.
4
5	Run `just-scrape --help` or `just-scrape <command> --help` for full option details.
6
7	If the task is to integrate ScrapeGraph AI into application code, add `SGAI_API_KEY` to a project, or choose endpoint usage in product code, inspect the project first and use the ScrapeGraph AI SDK/API docs directly instead of this CLI skill.
8
9	## Prerequisites
10
11	Must be installed and authenticated. Check with `just-scrape validate` and `just-scrape credits`.
12
13	```bash
14	command -v just-scrape >/dev/null 2>&1 \|\| npm install -g just-scrape@latest
15	just-scrape validate
16	just-scrape credits
17	```
18
19	- API key: Set `SGAI_API_KEY`, use a `.env` file, use `~/.scrapegraphai/config.json`, or complete the interactive prompt.
20	- Credits: Remaining ScrapeGraph AI credits. Each operation consumes credits.
21
22	Before doing real work, verify the setup with one small request:
23
24	```bash
25	mkdir -p .just-scrape
26	just-scrape scrape "https://example.com" --json > .just-scrape/install-check.json
27	```
28
29	```bash
30	just-scrape search "query" --num-results 3 --json > .just-scrape/search-check.json
31	```
32
33	## Workflow
34
35	Follow this escalation pattern:
36
37	1. Search - No specific URL yet. Find pages, answer questions, discover sources.
38	2. Scrape - Have a URL. Extract markdown, html, screenshots, links, images, summaries, or branding.
39	3. Extract - Need structured JSON from a known URL with an AI prompt and optional schema.
40	4. Crawl - Need bulk content from an entire site section.
41	5. Monitor - Need scheduled page-change tracking with optional webhook notifications.
42
43	\| Need \| Command \| When \|
44	\| --------------------------- \| ---------- \| ------------------------------------------ \|
45	\| Find pages on a topic \| `search` \| No specific URL yet \|
46	\| Get a page's content \| `scrape` \| Have a URL, need one or more page formats \|
47	\| AI-powered data extraction \| `extract` \| Need structured data from a known URL \|
48	\| Bulk extract a site section \| `crawl` \| Need many pages or docs sections \|
49	\| Track changes over time \| `monitor` \| Need recurring scraping and webhooks \|
50	\| Inspect prior requests \| `history` \| Need past request IDs, status, or payloads \|
51	\| Check credit balance \| `credits` \| Need remaining API credits \|
52	\| Validate API setup \| `validate` \| Need health check and API key validation \|
53
54	For detailed command reference, run `just-scrape <command> --help`.
55
56	Scrape vs extract:
57
58	- Use `scrape` for raw page formats: `markdown`, `html`, `screenshot`, `branding`, `links`, `images`, `summary`.
59	- Use `scrape -f json -p "<prompt>"` or `extract -p "<prompt>"` for AI-structured output.
60	- Use `extract` when the task is only structured data. Use `scrape` when mixed formats are needed in one call.
61
62	Avoid redundant fetches:
63
64	- `search -p` can extract structured data from search results. Do not re-scrape those URLs unless results are incomplete.
65	- `crawl` already fetches per-page formats. Do not re-scrape every crawled URL unless a second pass is required.
66	- Check `.just-scrape/` for existing data before fetching again.
67
68	## Commands
69
70	### Search
71
72	```bash
73	just-scrape search "query"
74	just-scrape search "query" --num-results 10
75	just-scrape search "query" -p "Extract provider names and prices"
76	just-scrape search "query" -p "Extract provider names and prices" --schema '<json-schema>'
77	just-scrape search "query" --format html
78	just-scrape search "query" --country us
79	just-scrape search "query" --time-range past_week
80	```
81
82	Time ranges: `past_hour`, `past_24_hours`, `past_week`, `past_month`, `past_year`.
83
84	### Scrape
85
86	```bash
87	just-scrape scrape "<url>"
88	just-scrape scrape "<url>" -f markdown
89	just-scrape scrape "<url>" -f html
90	just-scrape scrape "<url>" -f markdown,html,links --json
91	just-scrape scrape "<url>" -f screenshot
92	just-scrape scrape "<url>" -f branding
93	just-scrape scrape "<url>" -f summary
94	just-scrape scrape "<url>" -f json -p "Extract all products"
95	just-scrape scrape "<url>" -f json -p "Extract all products" --schema '<json-schema>

People who liked this also viewed

6 picks

Type

just-scrape

245kinstalls7forks

View on GitHub

How to install just-scrape?

scrapegraphai/just-scrape/just-scrape

$npx -y skills add scrapegraphai/just-scrape --skill just-scrape

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# just-scrape CLI
2
3	Search, scrape, crawl, extract structured JSON, and monitor page changes using the just-scrape CLI.
4
5	Run `just-scrape --help` or `just-scrape <command> --help` for full option details.
6
7	If the task is to integrate ScrapeGraph AI into application code, add `SGAI_API_KEY` to a project, or choose endpoint usage in product code, inspect the project first and use the ScrapeGraph AI SDK/API docs directly instead of this CLI skill.
8
9	## Prerequisites
10
11	Must be installed and authenticated. Check with `just-scrape validate` and `just-scrape credits`.
12
13	```bash
14	command -v just-scrape >/dev/null 2>&1 \|\| npm install -g just-scrape@latest
15	just-scrape validate
16	just-scrape credits
17	```
18
19	- API key: Set `SGAI_API_KEY`, use a `.env` file, use `~/.scrapegraphai/config.json`, or complete the interactive prompt.
20	- Credits: Remaining ScrapeGraph AI credits. Each operation consumes credits.
21
22	Before doing real work, verify the setup with one small request:
23
24	```bash
25	mkdir -p .just-scrape
26	just-scrape scrape "https://example.com" --json > .just-scrape/install-check.json
27	```
28
29	```bash
30	just-scrape search "query" --num-results 3 --json > .just-scrape/search-check.json
31	```
32
33	## Workflow
34
35	Follow this escalation pattern:
36
37	1. Search - No specific URL yet. Find pages, answer questions, discover sources.
38	2. Scrape - Have a URL. Extract markdown, html, screenshots, links, images, summaries, or branding.
39	3. Extract - Need structured JSON from a known URL with an AI prompt and optional schema.
40	4. Crawl - Need bulk content from an entire site section.
41	5. Monitor - Need scheduled page-change tracking with optional webhook notifications.
42
43	\| Need \| Command \| When \|
44	\| --------------------------- \| ---------- \| ------------------------------------------ \|
45	\| Find pages on a topic \| `search` \| No specific URL yet \|
46	\| Get a page's content \| `scrape` \| Have a URL, need one or more page formats \|
47	\| AI-powered data extraction \| `extract` \| Need structured data from a known URL \|
48	\| Bulk extract a site section \| `crawl` \| Need many pages or docs sections \|
49	\| Track changes over time \| `monitor` \| Need recurring scraping and webhooks \|
50	\| Inspect prior requests \| `history` \| Need past request IDs, status, or payloads \|
51	\| Check credit balance \| `credits` \| Need remaining API credits \|
52	\| Validate API setup \| `validate` \| Need health check and API key validation \|
53
54	For detailed command reference, run `just-scrape <command> --help`.
55
56	Scrape vs extract:
57
58	- Use `scrape` for raw page formats: `markdown`, `html`, `screenshot`, `branding`, `links`, `images`, `summary`.
59	- Use `scrape -f json -p "<prompt>"` or `extract -p "<prompt>"` for AI-structured output.
60	- Use `extract` when the task is only structured data. Use `scrape` when mixed formats are needed in one call.
61
62	Avoid redundant fetches:
63
64	- `search -p` can extract structured data from search results. Do not re-scrape those URLs unless results are incomplete.
65	- `crawl` already fetches per-page formats. Do not re-scrape every crawled URL unless a second pass is required.
66	- Check `.just-scrape/` for existing data before fetching again.
67
68	## Commands
69
70	### Search
71
72	```bash
73	just-scrape search "query"
74	just-scrape search "query" --num-results 10
75	just-scrape search "query" -p "Extract provider names and prices"
76	just-scrape search "query" -p "Extract provider names and prices" --schema '<json-schema>'
77	just-scrape search "query" --format html
78	just-scrape search "query" --country us
79	just-scrape search "query" --time-range past_week
80	```
81
82	Time ranges: `past_hour`, `past_24_hours`, `past_week`, `past_month`, `past_year`.
83
84	### Scrape
85
86	```bash
87	just-scrape scrape "<url>"
88	just-scrape scrape "<url>" -f markdown
89	just-scrape scrape "<url>" -f html
90	just-scrape scrape "<url>" -f markdown,html,links --json
91	just-scrape scrape "<url>" -f screenshot
92	just-scrape scrape "<url>" -f branding
93	just-scrape scrape "<url>" -f summary
94	just-scrape scrape "<url>" -f json -p "Extract all products"
95	just-scrape scrape "<url>" -f json -p "Extract all products" --schema '<json-schema>

People who liked this also viewed

6 picks

Type