web-scraper-api

Name: web-scraper-api
Author: oxylabs

317stars1forks

How to install web-scraper-api?

oxylabs/agent-skills/web-scraper-api

$npx -y skills add oxylabs/agent-skills --skill web-scraper-api

Installs into the current project.

Summary

Production-grade web scraping with automatic anti-bot bypass, structured JSON parsing for 40+ targets, and geo-targeting. Use when the user needs to scrape web pages, extract product data, get search results, or collect structured data from supported e-commerce and search platfor

Files · 1

View on GitHub

SKILL.md

1	# Oxylabs Web Scraper API
2
3	## Authentication
4
5	Requires HTTP Basic Auth with credentials from environment variables:
6
7	```bash
8	curl -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" ...
9	```
10
11	## Endpoint
12
13	```
14	POST https://realtime.oxylabs.io/v1/queries # immediate response
15	POST https://data.oxylabs.io/v1/queries # Push-Pull jobs, callbacks, storage
16	Content-Type: application/json
17	```
18
19	## Core Parameters
20
21	\| Parameter \| Required \| Description \|
22	\|-----------\|----------\|-------------\|
23	\| `source` \| Yes \| Target scraper (e.g., `universal`, `amazon_product`, `google_search`) \|
24	\| `url` \| Conditional \| URL to scrape (for `universal` and `*_url` sources) \|
25	\| `query` \| Conditional \| Search query or product ID (for `_search` and `_product` sources) \|
26	\| `parse` \| No \| Enable structured data parsing (recommended for supported sources) \|
27	\| `render` \| No \| JavaScript rendering: `html` or `png` \|
28	\| `geo_location` \| No \| Geographic targeting: country/state/city, ZIP/postcode, coordinates, or Criteria ID where supported \|
29	\| `session_id` \| No \| Reuse the same proxy IP across multiple jobs \|
30	\| `content_encoding` \| No \| Set to `base64` when downloading image files via Realtime or Push-Pull \|
31	\| `user_agent_type` \| No \| Device/browser preset, e.g., `desktop_chrome`, `mobile_ios`, `tablet_android` \|
32	\| `locale` \| No \| Interface language / `Accept-Language`, e.g., `de-DE` \|
33	\| `callback_url` \| No \| Push-Pull callback endpoint \|
34	\| `storage_type`, `storage_url` \| No \| Push-Pull cloud upload target (`gcs`, `s3`, `tos`, `s3_compatible`) \|
35	\| `markdown`, `xhr` \| No \| Enable markdown or captured XHR result types \|
36	\| `browser_instructions` \| No \| Rendered browser actions; requires `render: "html"` \|
37	\| `parsing_instructions`, `parser_preset` \| No \| Custom parser rules or saved preset; pair with `parse: true` \|
38	\| `client_notes` \| No \| Client-side job tag saved with the job metadata \|
39	\| `domain`, `subdomain`, `start_page`, `pages`, `limit`, `store_id`, `delivery_zip`, `fulfillment_type` \| Source-specific \| Marketplace/search/store localization and pagination fields \|
40
41	`user_agent_type` values: `desktop`, `desktop_chrome`, `desktop_edge`, `desktop_firefox`, `desktop_opera`, `desktop_safari`, `mobile`, `mobile_android`, `mobile_ios`, `tablet`, `tablet_android`, `tablet_ios`.
42
43	## Context Parameters
44
45	Add these as `{ "key": "...", "value": ... }` objects in `context`:
46
47	\| Key \| Use \|
48	\|-----\|-----\|
49	\| `force_headers`, `headers` \| Merge custom headers with managed headers \|
50	\| `force_cookies`, `cookies` \| Merge custom cookies with managed cookies \|
51	\| `http_method`, `content` \| Use `post` with Base64-encoded body content \|
52	\| `follow_redirects` \| Follow 3xx redirect chains \|
53	\| `successful_status_codes` \| Treat specific non-standard HTTP codes as successful \|
54
55	For multi-format output, enable types in the payload (`parse`, `markdown`, `xhr`, `render: "png"`) and request them with `?type=raw,parsed,png,markdown,xhr`.
56
57	For batch Push-Pull jobs, use `POST /v1/queries/batch` with arrays only for `query` or `url`; keep all other parameters singular. Maximum batch size is 5,000 values.
58
59	## Quick Start
60
61	Scrape any URL:
62	```bash
63	curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
64	-u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
65	-H 'Content-Type: application/json' \
66	-d '{"source": "universal", "url": "https://example.com"}'
67	```
68
69	Google search with parsing:
70	```bash
71	curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
72	-u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
73	-H 'Content-Type: application/json' \
74	-d '{"source": "google_search", "query": "best laptops", "parse": true}'
75	```
76
77	Amazon product by ASIN:
78	```bash
79	curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
80	-u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
81	-H 'Content-Type: application/json' \
82	-d '{"source": "amazon_product", "query": "B07FZ8S74R", "parse": true}'
83	```
84
85	## Choosing the Right Source
86
87	1. Use specific sources when available (`amazon_product`, `google_search`) - better parsing and reliability
88	2. Use `universal` for unsupported sites - works with any URL
89	3. Enable `parse: true` for structured JSON output on supported sources
90
91	## Response Structure
92
93	```json
94	{
95	"results": [{
96	"content": "...",
97	"status_code": 200,
98	"url": "https://..."
99	}]
100	}
101	```
102
103	With `parse: true`, `content` contains structured data (title, price, reviews, etc.) instead of raw HTML.
104
105	## Available Sources
106
107	For the complete list of 40+ supported sources organized by category, see [sources.md](sources.md).
108
109	## More Examples
110
111	For detailed request/response examples including geo-location, JavaScript rende

People who liked this also viewed

6 picks

Type

web-scraper-api

317stars1forks

View on GitHub

How to install web-scraper-api?

oxylabs/agent-skills/web-scraper-api

$npx -y skills add oxylabs/agent-skills --skill web-scraper-api

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Oxylabs Web Scraper API
2
3	## Authentication
4
5	Requires HTTP Basic Auth with credentials from environment variables:
6
7	```bash
8	curl -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" ...
9	```
10
11	## Endpoint
12
13	```
14	POST https://realtime.oxylabs.io/v1/queries # immediate response
15	POST https://data.oxylabs.io/v1/queries # Push-Pull jobs, callbacks, storage
16	Content-Type: application/json
17	```
18
19	## Core Parameters
20
21	\| Parameter \| Required \| Description \|
22	\|-----------\|----------\|-------------\|
23	\| `source` \| Yes \| Target scraper (e.g., `universal`, `amazon_product`, `google_search`) \|
24	\| `url` \| Conditional \| URL to scrape (for `universal` and `*_url` sources) \|
25	\| `query` \| Conditional \| Search query or product ID (for `_search` and `_product` sources) \|
26	\| `parse` \| No \| Enable structured data parsing (recommended for supported sources) \|
27	\| `render` \| No \| JavaScript rendering: `html` or `png` \|
28	\| `geo_location` \| No \| Geographic targeting: country/state/city, ZIP/postcode, coordinates, or Criteria ID where supported \|
29	\| `session_id` \| No \| Reuse the same proxy IP across multiple jobs \|
30	\| `content_encoding` \| No \| Set to `base64` when downloading image files via Realtime or Push-Pull \|
31	\| `user_agent_type` \| No \| Device/browser preset, e.g., `desktop_chrome`, `mobile_ios`, `tablet_android` \|
32	\| `locale` \| No \| Interface language / `Accept-Language`, e.g., `de-DE` \|
33	\| `callback_url` \| No \| Push-Pull callback endpoint \|
34	\| `storage_type`, `storage_url` \| No \| Push-Pull cloud upload target (`gcs`, `s3`, `tos`, `s3_compatible`) \|
35	\| `markdown`, `xhr` \| No \| Enable markdown or captured XHR result types \|
36	\| `browser_instructions` \| No \| Rendered browser actions; requires `render: "html"` \|
37	\| `parsing_instructions`, `parser_preset` \| No \| Custom parser rules or saved preset; pair with `parse: true` \|
38	\| `client_notes` \| No \| Client-side job tag saved with the job metadata \|
39	\| `domain`, `subdomain`, `start_page`, `pages`, `limit`, `store_id`, `delivery_zip`, `fulfillment_type` \| Source-specific \| Marketplace/search/store localization and pagination fields \|
40
41	`user_agent_type` values: `desktop`, `desktop_chrome`, `desktop_edge`, `desktop_firefox`, `desktop_opera`, `desktop_safari`, `mobile`, `mobile_android`, `mobile_ios`, `tablet`, `tablet_android`, `tablet_ios`.
42
43	## Context Parameters
44
45	Add these as `{ "key": "...", "value": ... }` objects in `context`:
46
47	\| Key \| Use \|
48	\|-----\|-----\|
49	\| `force_headers`, `headers` \| Merge custom headers with managed headers \|
50	\| `force_cookies`, `cookies` \| Merge custom cookies with managed cookies \|
51	\| `http_method`, `content` \| Use `post` with Base64-encoded body content \|
52	\| `follow_redirects` \| Follow 3xx redirect chains \|
53	\| `successful_status_codes` \| Treat specific non-standard HTTP codes as successful \|
54
55	For multi-format output, enable types in the payload (`parse`, `markdown`, `xhr`, `render: "png"`) and request them with `?type=raw,parsed,png,markdown,xhr`.
56
57	For batch Push-Pull jobs, use `POST /v1/queries/batch` with arrays only for `query` or `url`; keep all other parameters singular. Maximum batch size is 5,000 values.
58
59	## Quick Start
60
61	Scrape any URL:
62	```bash
63	curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
64	-u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
65	-H 'Content-Type: application/json' \
66	-d '{"source": "universal", "url": "https://example.com"}'
67	```
68
69	Google search with parsing:
70	```bash
71	curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
72	-u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
73	-H 'Content-Type: application/json' \
74	-d '{"source": "google_search", "query": "best laptops", "parse": true}'
75	```
76
77	Amazon product by ASIN:
78	```bash
79	curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
80	-u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
81	-H 'Content-Type: application/json' \
82	-d '{"source": "amazon_product", "query": "B07FZ8S74R", "parse": true}'
83	```
84
85	## Choosing the Right Source
86
87	1. Use specific sources when available (`amazon_product`, `google_search`) - better parsing and reliability
88	2. Use `universal` for unsupported sites - works with any URL
89	3. Enable `parse: true` for structured JSON output on supported sources
90
91	## Response Structure
92
93	```json
94	{
95	"results": [{
96	"content": "...",
97	"status_code": 200,
98	"url": "https://..."
99	}]
100	}
101	```
102
103	With `parse: true`, `content` contains structured data (title, price, reviews, etc.) instead of raw HTML.
104
105	## Available Sources
106
107	For the complete list of 40+ supported sources organized by category, see [sources.md](sources.md).
108
109	## More Examples
110
111	For detailed request/response examples including geo-location, JavaScript rende

People who liked this also viewed

6 picks

Type