apify-ultimate-scraper

Name: apify-ultimate-scraper
Author: apify

14kinstalls247forks

How to install apify-ultimate-scraper?

apify/agent-skills/apify-ultimate-scraper

$npx -y skills add apify/agent-skills --skill apify-ultimate-scraper

Installs into the current project.

Summary

Universal AI-powered web scraper for any platform. Scrape data from Instagram, Facebook, TikTok, YouTube, LinkedIn, X/Twitter, Google Maps, Google Search, Google Trends, Reddit, Airbnb, Yelp, and 15+ more platforms. Use for lead generation, brand monitoring, competitor analysis,

Files · 1

View on GitHub

SKILL.md

1	# Universal web scraper
2
3	AI-driven data extraction from ~100 Actors across 15+ platforms via the Apify CLI.
4
5	Rules for every `apify` command:
6	1. Pass `--json` for machine-readable output (stable across CLI versions).
7	2. Pass `--user-agent apify-agent-skills/apify-ultimate-scraper` for telemetry attribution.
8	3. Redirect stderr with `2>/dev/null` (stderr contains progress messages that break JSON parsers).
9
10	## Prerequisites
11
12	- Apify CLI v1.5.0+ (`npm install -g apify-cli`)
13	- Authenticated session (see below)
14
15	## Authentication
16
17	If a CLI command fails with an auth error, authenticate using one of these methods:
18
19	1. OAuth (interactive): `apify login` (opens browser)
20	2. Environment variable: `export APIFY_TOKEN=your_token_here`
21	3. From .env file: `source .env` (if the file contains `APIFY_TOKEN=...`)
22
23	Generate token: https://console.apify.com/settings/integrations
24
25	## Workflow
26
27	### Step 1: Understand goal and select Actor
28
29	Identify the target platform and use case. Read `references/actor-index.md` to find the right Actor.
30
31	If the task involves a multi-step pipeline, also read the matching workflow guide:
32
33	\| Task involves... \| Read \|
34	\|-----------------\|------\|
35	\| leads, contacts, emails, B2B \| `references/workflows/lead-generation.md` \|
36	\| competitor, ads, pricing \| `references/workflows/competitive-intel.md` \|
37	\| influencer, creator \| `references/workflows/influencer-vetting.md` \|
38	\| brand, mentions, sentiment \| `references/workflows/brand-monitoring.md` \|
39	\| reviews, ratings, reputation \| `references/workflows/review-analysis.md` \|
40	\| SEO, SERP, crawl, content, RAG \| `references/workflows/content-and-seo.md` \|
41	\| analytics, engagement, performance \| `references/workflows/social-media-analytics.md` \|
42	\| trends, keywords, hashtags \| `references/workflows/trend-research.md` \|
43	\| jobs, recruiting, candidates \| `references/workflows/job-market-and-recruitment.md` \|
44	\| real estate, listings, hotels \| `references/workflows/real-estate-and-hospitality.md` \|
45	\| price monitoring, e-commerce, products \| `references/workflows/ecommerce-price-monitoring.md` \|
46	\| contact enrichment, email extraction \| `references/workflows/contact-enrichment.md` \|
47	\| knowledge base, RAG, LLM data feed \| `references/workflows/knowledge-base-and-rag.md` \|
48	\| company research, due diligence \| `references/workflows/company-research.md` \|
49
50	If no Actor matches in the index, search dynamically:
51
52	apify actors search "KEYWORDS" --user-agent apify-agent-skills/apify-ultimate-scraper --json --limit 10 2>/dev/null
53
54	From results: `items[].username`/`items[].name` (Actor ID), `items[].title`, `items[].stats.totalUsers30Days`, `items[].currentPricingInfo.pricingModel`.
55
56	### Step 2: Fetch Actor schema and check gotchas
57
58	Fetch the input schema dynamically:
59
60	apify actors info "ACTOR_ID" --user-agent apify-agent-skills/apify-ultimate-scraper --input --json 2>/dev/null
61
62	Also read `references/gotchas.md` to check for common pitfalls for the selected Actor.
63
64	For Actor documentation: `apify actors info "ACTOR_ID" --user-agent apify-agent-skills/apify-ultimate-scraper --readme`
65
66	### Step 3: Configure and run
67
68	Skip user preferences for simple lookups (e.g., "Nike's follower count"). Go straight to running with quick answer mode.
69
70	For larger tasks, confirm output format (quick answer / CSV / JSON) and result count.
71
72	Standard run (blocking):
73
74	apify actors call "ACTOR_ID" --input-file input.json --user-agent apify-agent-skills/apify-ultimate-scraper --json 2>/dev/null
75
76	Prefer `--input-file input.json` for large or complex inputs. For tiny inputs, inline JSON is acceptable with shell quoting: `--input '{"maxItems":10}'`.
77
78	From output: `.id` (run ID), `.status`, `.defaultDatasetId`, `.stats.durationMillis`
79
80	Fetch results:
81
82	apify datasets get-items DATASET_ID --user-agent apify-agent-skills/apify-ultimate-scraper --format json
83
84	For CSV: `apify datasets get-items DATASET_ID --user-agent apify-agent-skills/apify-ultimate-scraper --format csv`
85
86	Quick answer mode: Fetch results as JSON, pick top 5, present formatted in chat.
87
88	Save to file: Fetch results, use Write tool to save as `YYYY-MM-DD_descriptive-name.csv` or `.json`.
89
90	Large/long-running scrapes:
91
92	apify actors start "ACTOR_ID" --input-file input.json --user-agent apify-agent-skills/apify-ultimate-scraper --json 2>/dev/null
93
94	Poll: `apify runs info RUN_ID --user-agent apify-agent-skills/apify-ultimate-scraper --json 2>/dev/null` (check `.status` for `SUCCEEDED`).
95
96	### Step 4: Deliver results
97
98	Report:

People who liked this also viewed

6 picks

Type

apify-ultimate-scraper

14kinstalls247forks

View on GitHub

How to install apify-ultimate-scraper?

apify/agent-skills/apify-ultimate-scraper

$npx -y skills add apify/agent-skills --skill apify-ultimate-scraper

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Universal web scraper
2
3	AI-driven data extraction from ~100 Actors across 15+ platforms via the Apify CLI.
4
5	Rules for every `apify` command:
6	1. Pass `--json` for machine-readable output (stable across CLI versions).
7	2. Pass `--user-agent apify-agent-skills/apify-ultimate-scraper` for telemetry attribution.
8	3. Redirect stderr with `2>/dev/null` (stderr contains progress messages that break JSON parsers).
9
10	## Prerequisites
11
12	- Apify CLI v1.5.0+ (`npm install -g apify-cli`)
13	- Authenticated session (see below)
14
15	## Authentication
16
17	If a CLI command fails with an auth error, authenticate using one of these methods:
18
19	1. OAuth (interactive): `apify login` (opens browser)
20	2. Environment variable: `export APIFY_TOKEN=your_token_here`
21	3. From .env file: `source .env` (if the file contains `APIFY_TOKEN=...`)
22
23	Generate token: https://console.apify.com/settings/integrations
24
25	## Workflow
26
27	### Step 1: Understand goal and select Actor
28
29	Identify the target platform and use case. Read `references/actor-index.md` to find the right Actor.
30
31	If the task involves a multi-step pipeline, also read the matching workflow guide:
32
33	\| Task involves... \| Read \|
34	\|-----------------\|------\|
35	\| leads, contacts, emails, B2B \| `references/workflows/lead-generation.md` \|
36	\| competitor, ads, pricing \| `references/workflows/competitive-intel.md` \|
37	\| influencer, creator \| `references/workflows/influencer-vetting.md` \|
38	\| brand, mentions, sentiment \| `references/workflows/brand-monitoring.md` \|
39	\| reviews, ratings, reputation \| `references/workflows/review-analysis.md` \|
40	\| SEO, SERP, crawl, content, RAG \| `references/workflows/content-and-seo.md` \|
41	\| analytics, engagement, performance \| `references/workflows/social-media-analytics.md` \|
42	\| trends, keywords, hashtags \| `references/workflows/trend-research.md` \|
43	\| jobs, recruiting, candidates \| `references/workflows/job-market-and-recruitment.md` \|
44	\| real estate, listings, hotels \| `references/workflows/real-estate-and-hospitality.md` \|
45	\| price monitoring, e-commerce, products \| `references/workflows/ecommerce-price-monitoring.md` \|
46	\| contact enrichment, email extraction \| `references/workflows/contact-enrichment.md` \|
47	\| knowledge base, RAG, LLM data feed \| `references/workflows/knowledge-base-and-rag.md` \|
48	\| company research, due diligence \| `references/workflows/company-research.md` \|
49
50	If no Actor matches in the index, search dynamically:
51
52	apify actors search "KEYWORDS" --user-agent apify-agent-skills/apify-ultimate-scraper --json --limit 10 2>/dev/null
53
54	From results: `items[].username`/`items[].name` (Actor ID), `items[].title`, `items[].stats.totalUsers30Days`, `items[].currentPricingInfo.pricingModel`.
55
56	### Step 2: Fetch Actor schema and check gotchas
57
58	Fetch the input schema dynamically:
59
60	apify actors info "ACTOR_ID" --user-agent apify-agent-skills/apify-ultimate-scraper --input --json 2>/dev/null
61
62	Also read `references/gotchas.md` to check for common pitfalls for the selected Actor.
63
64	For Actor documentation: `apify actors info "ACTOR_ID" --user-agent apify-agent-skills/apify-ultimate-scraper --readme`
65
66	### Step 3: Configure and run
67
68	Skip user preferences for simple lookups (e.g., "Nike's follower count"). Go straight to running with quick answer mode.
69
70	For larger tasks, confirm output format (quick answer / CSV / JSON) and result count.
71
72	Standard run (blocking):
73
74	apify actors call "ACTOR_ID" --input-file input.json --user-agent apify-agent-skills/apify-ultimate-scraper --json 2>/dev/null
75
76	Prefer `--input-file input.json` for large or complex inputs. For tiny inputs, inline JSON is acceptable with shell quoting: `--input '{"maxItems":10}'`.
77
78	From output: `.id` (run ID), `.status`, `.defaultDatasetId`, `.stats.durationMillis`
79
80	Fetch results:
81
82	apify datasets get-items DATASET_ID --user-agent apify-agent-skills/apify-ultimate-scraper --format json
83
84	For CSV: `apify datasets get-items DATASET_ID --user-agent apify-agent-skills/apify-ultimate-scraper --format csv`
85
86	Quick answer mode: Fetch results as JSON, pick top 5, present formatted in chat.
87
88	Save to file: Fetch results, use Write tool to save as `YYYY-MM-DD_descriptive-name.csv` or `.json`.
89
90	Large/long-running scrapes:
91
92	apify actors start "ACTOR_ID" --input-file input.json --user-agent apify-agent-skills/apify-ultimate-scraper --json 2>/dev/null
93
94	Poll: `apify runs info RUN_ID --user-agent apify-agent-skills/apify-ultimate-scraper --json 2>/dev/null` (check `.status` for `SUCCEEDED`).
95
96	### Step 4: Deliver results
97
98	Report:

People who liked this also viewed

6 picks

Type