data-scraper-agent

Name: data-scraper-agent
Author: affaan-m

4.8kinstalls35kforks

How to install data-scraper-agent?

affaan-m/everything-claude-code/data-scraper-agent

$npx -y skills add affaan-m/everything-claude-code --skill data-scraper-agent

Installs into the current project.

Summary

Build a fully automated AI-powered data collection agent for any public source — job boards, prices, news, GitHub, sports, anything. Runs on a schedule, enriches data with a free LLM (Gemini Flash), stores results in Notion/Sheets/Supabase, and learns from user feedback. Runs 100

Files · 1

View on GitHub

SKILL.md

1	# Data Scraper Agent
2
3	Build a production-ready, AI-powered data collection agent for any public data source.
4	Runs on a schedule, enriches results with a free LLM, stores to a database, and improves over time.
5
6	Stack: Python · Gemini Flash (free) · GitHub Actions (free) · Notion / Sheets / Supabase
7
8	## When to Activate
9
10	- User wants to gather or monitor any public website or API
11	- User says "build a bot that checks...", "monitor X for me", "collect data from..."
12	- User wants to track jobs, prices, news, repos, sports scores, events, listings
13	- User asks how to automate data collection without paying for hosting
14	- User wants an agent that gets smarter over time based on their decisions
15
16	## Core Concepts
17
18	### The Three Layers
19
20	Every data collection agent has three layers:
21
22	```
23	COLLECT → ENRICH → STORE
24	│ │ │
25	Scraper AI (LLM) Database
26	runs on scores/ Notion /
27	schedule summarises Sheets /
28	& classifies Supabase
29	```
30
31	### Free Stack
32
33	\| Layer \| Tool \| Why \|
34	\|---\|---\|---\|
35	\| Scraping \| `requests` + `BeautifulSoup` \| No cost, covers 80% of public sites \|
36	\| JS-rendered sites \| `playwright` (free) \| When HTML fetching fails \|
37	\| AI enrichment \| Gemini Flash via REST API \| 500 req/day, 1M tokens/day — free \|
38	\| Storage \| Notion API \| Free tier, great UI for review \|
39	\| Schedule \| GitHub Actions cron \| Free for public repos \|
40	\| Learning \| JSON feedback file in repo \| Zero infra, persists in git \|
41
42	### AI Model Fallback Chain
43
44	Build agents to auto-fallback across Gemini models on quota exhaustion:
45
46	```
47	gemini-2.0-flash-lite (30 RPM) →
48	gemini-2.0-flash (15 RPM) →
49	gemini-2.5-flash (10 RPM) →
50	gemini-flash-lite-latest (fallback)
51	```
52
53	### Batch API Calls for Efficiency
54
55	Never call the LLM once per item. Always batch:
56
57	```python
58	# BAD: 33 API calls for 33 items
59	for item in items:
60	result = call_ai(item) # 33 calls → hits rate limit
61
62	# GOOD: 7 API calls for 33 items (batch size 5)
63	for batch in chunks(items, size=5):
64	results = call_ai(batch) # 7 calls → stays within free tier
65	```
66
67	---
68
69	## Workflow
70
71	### Step 1: Understand the Goal
72
73	Ask the user:
74
75	1. What to collect: "What data source? URL / API / RSS / public endpoint?"
76	2. What to extract: "What fields matter? Title, price, URL, date, score?"
77	3. How to store: "Where should results go? Notion, Google Sheets, Supabase, or local file?"
78	4. How to enrich: "Do you want AI to score, summarise, classify, or match each item?"
79	5. Frequency: "How often should it run? Every hour, daily, weekly?"
80
81	Common examples to prompt:
82	- Job boards → score relevance to resume
83	- Product prices → alert on drops
84	- GitHub repos → summarise new releases
85	- News feeds → classify by topic + sentiment
86	- Sports results → extract stats to tracker
87	- Events calendar → filter by interest
88
89	---
90
91	### Step 2: Design the Collection Architecture
92
93	Generate this directory structure for the user:
94
95	```
96	my-agent/
97	├── config.yaml # User customises this (keywords, filters, preferences)
98	├── profile/
99	│ └── context.md # User context the AI uses (resume, interests, criteria)
100	├── scraper/
101	│ ├── __init__.py
102	│ ├── main.py # Orchestrator: scrape → enrich → store
103	│ ├── filters.py # Rule-based pre-filter (fast, before AI)
104	│ └── sources/
105	│ ├── __init__.py
106	│ └── source_name.py # One file per data source
107	├── ai/
108	│ ├── __init__.py
109	│ ├── client.py # Gemini REST client with model fallback
110	│ ├── pipeline.py # Batch AI analysis
111	│ ├── jd_fetcher.py # Fetch full content from URLs (optional)
112	│ └── memory.py # Learn from user feedback
113	├── storage/
114	│ ├── __init__.py
115	│ └── notion_sync.py # Or sheets_sync.py / supabase_sync.py
116	├── data/
117	│ └── feedback.json # User decision history (auto-updated)
118	├── .env.example
119	├── setup.py # One-time DB/schema creation
120	├── enrich_existing.py # Backfill AI scores on old rows
121	├── requirements.txt
122	└── .github/
123	└── workflows/
124	└── scraper.yml # GitHub Actions schedule
125	```
126
127	---
128
129	### Step 3: Build the Source Connector
130
131	Template for any data source:
132
133	```python
134	# scraper/sources/my_source.py
135	"""
136	[Source Name] — gathers [what] from [where].
137	Method: [REST API / HTML scraping / RSS feed]
138	"""
139	import requests
140	from bs4 import BeautifulSoup
141	from datetime import datetime, timezone
142	from scraper.filters import is_relevant
143
144	HEADERS = {
145	"User-Agent": "Mozilla/5.0 (compatible; research-bot/1.0)",
146	}
147
148
149	def fetch() -> li

People who liked this also viewed

6 picks

Type

data-scraper-agent

4.8kinstalls35kforks

View on GitHub

How to install data-scraper-agent?

affaan-m/everything-claude-code/data-scraper-agent

$npx -y skills add affaan-m/everything-claude-code --skill data-scraper-agent

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Data Scraper Agent
2
3	Build a production-ready, AI-powered data collection agent for any public data source.
4	Runs on a schedule, enriches results with a free LLM, stores to a database, and improves over time.
5
6	Stack: Python · Gemini Flash (free) · GitHub Actions (free) · Notion / Sheets / Supabase
7
8	## When to Activate
9
10	- User wants to gather or monitor any public website or API
11	- User says "build a bot that checks...", "monitor X for me", "collect data from..."
12	- User wants to track jobs, prices, news, repos, sports scores, events, listings
13	- User asks how to automate data collection without paying for hosting
14	- User wants an agent that gets smarter over time based on their decisions
15
16	## Core Concepts
17
18	### The Three Layers
19
20	Every data collection agent has three layers:
21
22	```
23	COLLECT → ENRICH → STORE
24	│ │ │
25	Scraper AI (LLM) Database
26	runs on scores/ Notion /
27	schedule summarises Sheets /
28	& classifies Supabase
29	```
30
31	### Free Stack
32
33	\| Layer \| Tool \| Why \|
34	\|---\|---\|---\|
35	\| Scraping \| `requests` + `BeautifulSoup` \| No cost, covers 80% of public sites \|
36	\| JS-rendered sites \| `playwright` (free) \| When HTML fetching fails \|
37	\| AI enrichment \| Gemini Flash via REST API \| 500 req/day, 1M tokens/day — free \|
38	\| Storage \| Notion API \| Free tier, great UI for review \|
39	\| Schedule \| GitHub Actions cron \| Free for public repos \|
40	\| Learning \| JSON feedback file in repo \| Zero infra, persists in git \|
41
42	### AI Model Fallback Chain
43
44	Build agents to auto-fallback across Gemini models on quota exhaustion:
45
46	```
47	gemini-2.0-flash-lite (30 RPM) →
48	gemini-2.0-flash (15 RPM) →
49	gemini-2.5-flash (10 RPM) →
50	gemini-flash-lite-latest (fallback)
51	```
52
53	### Batch API Calls for Efficiency
54
55	Never call the LLM once per item. Always batch:
56
57	```python
58	# BAD: 33 API calls for 33 items
59	for item in items:
60	result = call_ai(item) # 33 calls → hits rate limit
61
62	# GOOD: 7 API calls for 33 items (batch size 5)
63	for batch in chunks(items, size=5):
64	results = call_ai(batch) # 7 calls → stays within free tier
65	```
66
67	---
68
69	## Workflow
70
71	### Step 1: Understand the Goal
72
73	Ask the user:
74
75	1. What to collect: "What data source? URL / API / RSS / public endpoint?"
76	2. What to extract: "What fields matter? Title, price, URL, date, score?"
77	3. How to store: "Where should results go? Notion, Google Sheets, Supabase, or local file?"
78	4. How to enrich: "Do you want AI to score, summarise, classify, or match each item?"
79	5. Frequency: "How often should it run? Every hour, daily, weekly?"
80
81	Common examples to prompt:
82	- Job boards → score relevance to resume
83	- Product prices → alert on drops
84	- GitHub repos → summarise new releases
85	- News feeds → classify by topic + sentiment
86	- Sports results → extract stats to tracker
87	- Events calendar → filter by interest
88
89	---
90
91	### Step 2: Design the Collection Architecture
92
93	Generate this directory structure for the user:
94
95	```
96	my-agent/
97	├── config.yaml # User customises this (keywords, filters, preferences)
98	├── profile/
99	│ └── context.md # User context the AI uses (resume, interests, criteria)
100	├── scraper/
101	│ ├── __init__.py
102	│ ├── main.py # Orchestrator: scrape → enrich → store
103	│ ├── filters.py # Rule-based pre-filter (fast, before AI)
104	│ └── sources/
105	│ ├── __init__.py
106	│ └── source_name.py # One file per data source
107	├── ai/
108	│ ├── __init__.py
109	│ ├── client.py # Gemini REST client with model fallback
110	│ ├── pipeline.py # Batch AI analysis
111	│ ├── jd_fetcher.py # Fetch full content from URLs (optional)
112	│ └── memory.py # Learn from user feedback
113	├── storage/
114	│ ├── __init__.py
115	│ └── notion_sync.py # Or sheets_sync.py / supabase_sync.py
116	├── data/
117	│ └── feedback.json # User decision history (auto-updated)
118	├── .env.example
119	├── setup.py # One-time DB/schema creation
120	├── enrich_existing.py # Backfill AI scores on old rows
121	├── requirements.txt
122	└── .github/
123	└── workflows/
124	└── scraper.yml # GitHub Actions schedule
125	```
126
127	---
128
129	### Step 3: Build the Source Connector
130
131	Template for any data source:
132
133	```python
134	# scraper/sources/my_source.py
135	"""
136	[Source Name] — gathers [what] from [where].
137	Method: [REST API / HTML scraping / RSS feed]
138	"""
139	import requests
140	from bs4 import BeautifulSoup
141	from datetime import datetime, timezone
142	from scraper.filters import is_relevant
143
144	HEADERS = {
145	"User-Agent": "Mozilla/5.0 (compatible; research-bot/1.0)",
146	}
147
148
149	def fetch() -> li

People who liked this also viewed

6 picks

Type