job-scraper-pipeline

Name: job-scraper-pipeline
Author: arimanyus

33stars2forks

How to install job-scraper-pipeline?

arimanyus/hermes-merchant/job-scraper-pipeline

$npx -y skills add arimanyus/hermes-merchant --skill job-scraper-pipeline

Installs into the current project.

Summary

Automated ML/AI job scraping, deduplication, and gatekeeper scoring pipeline. Scrapes AI company career pages on Greenhouse, AshbyHQ, and Lever, avoids duplicates, and scores jobs 0-100 against profile keywords, classifying tier-1/2/3.

Files · 1

View on GitHub

SKILL.md

1	# Job Scraper + Gatekeeper Pipeline
2
3	Automated job scraping, deduplication, and gatekeeper scoring for ML/AI job hunting.
4
5	## When to Use
6	- Cron job that scrapes AI company career pages on Greenhouse, AshbyHQ, and Lever
7	- Runs on a schedule (e.g. every 15m, 6h, or daily)
8	- Appends new jobs to the queue file
9	- Scores pending jobs against profile keywords
10
11	## Configuration
12
13	All PII, paths, and scoring keywords live in `profile.yaml` at the repo root. Copy `profile.yaml.example` to `profile.yaml` and edit. Keys this skill reads:
14
15	- `paths.queue` — queue JSON location (default `~/hermes-merchant/state/queue/jobs.json`)
16	- `paths.scraper_log` — log file
17	- `scoring.required_keywords`, `scoring.adjacent_roles`, `scoring.reject_keywords`, `scoring.ai_companies`, `scoring.tiers`
18
19	Minimal loader (Python):
20
21	```python
22	import yaml, os
23	from pathlib import Path
24
25	cfg = yaml.safe_load(Path("profile.yaml").read_text())
26	queue_path = Path(os.path.expanduser(cfg["paths"]["queue"]))
27	queue_path.parent.mkdir(parents=True, exist_ok=True)
28	```
29
30	## Scraping Strategy
31
32	### Sources — What Works / What Doesn't
33
34	\| Source \| Method \| Status \| Notes \|
35	\|--------\|--------\|--------\|-------\|
36	\| Naukri.com \| browser_navigate \| Blocked \| Search results require JS rendering; homepage loads but listings aren't accessible. \|
37	\| Indeed.com \| browser_navigate \| Blocked by Cloudflare \| "Additional Verification Required"; cannot bypass without residential proxies. \|
38	\| Together AI \| Browser → `https://job-boards.greenhouse.io/togetherai` \| Works \| Greenhouse format — use the JS snippet below to extract all jobs. \|
39	\| Cohere \| Browser → `https://jobs.ashbyhq.com/cohere` \| Works (~100+ roles) \| AshbyHQ — largest pipeline source. Departments include Agentic Platform, Modeling, Applied-ML, Inference, Model Serving. \|
40	\| Baseten \| Browser → `https://jobs.ashbyhq.com/baseten` \| Works \| AshbyHQ. Many roles previously bounced on email — prefer web form. \|
41	\| Modal Labs \| Browser → `https://www.modal.com/careers` \| Works (~24 roles) \| AshbyHQ embedded on page — scroll to see listings. \|
42	\| Cerebras \| Browser → `https://www.cerebras.ai/join-us` → click ML dept \| Works \| Greenhouse-based, limited ML roles. \|
43	\| Mistral AI \| Browser → `https://jobs.lever.co/mistral` \| Works \| Lever ATS — do NOT add query params; go direct and filter in-page. \|
44	\| LangChain \| Browser → `https://jobs.ashbyhq.com/langchain` \| Works \| AshbyHQ; iframe on their own careers page — go direct. \|
45	\| Pinecone \| Browser → `https://www.pinecone.io/careers` \| Works \| AshbyHQ-hosted, JS console extraction works. \|
46	\| Groq \| Browser → `https://www.groq.com/careers/` \| Moved \| All engineering roles now on an external ATS (Gem.com). \|
47	\| Hugging Face \| Browser → `https://huggingface.co/jobs` \| Login-gated \| Requires auth to view. \|
48	\| Anthropic \| Browser → `https://job-boards.greenhouse.io/anthropic` \| Works partially \| Greenhouse — some listings require a second fetch. \|
49	\| Qdrant \| Browser \| Blocked \| `careers.qdrant.tech` resolves to a private network. \|
50
51	### ATS Platform Taxonomy
52
53	Three ATS platforms dominate AI company career pages. Each wants a different scraping approach:
54
55	\| ATS \| URL Pattern \| Best Method \|
56	\|-----\|-------------\|-------------\|
57	\| Greenhouse \| `job-boards.greenhouse.io/{company}` \| `browser_navigate` → `browser_console` JS snippet \|
58	\| AshbyHQ \| `jobs.ashbyhq.com/{company}` \| `browser_navigate` → `browser_snapshot` \|
59	\| Lever \| `jobs.lever.co/{company}` \| `browser_navigate` → `browser_snapshot` (no query params) \|
60
61	Known Greenhouse: Anthropic, Together AI, Cerebras.
62	Known AshbyHQ: Cohere, Baseten, LangChain, Modal, Pinecone.
63	Known Lever: Mistral AI.
64
65	### Greenhouse Job Extraction
66
67	Many AI companies host on Greenhouse at `https://job-boards.greenhouse.io/{company}`.
68
69	JavaScript snippet to extract ALL job links from a Greenhouse page (run via `browser_console`):
70
71	```javascript
72	"use strict"; (() => {
73	const links = document.querySelectorAll('a[href*="/jobs/"]');
74	const seen = new Set();
75	return Array.from(links)
76	.filter(l => l.href && !seen.has(l.href) && seen.add(l.href))
77	.map(l => l.href + '\t' + l.textContent.trim())
78	.join('\n');
79	})()
80	```
81
82	Returns a newline-separated list of `URL\tJob Title` pairs. Greenhouse pages load all jobs on one page — no pagination.
83
84	### web_extract is unreliable for job scraping
85
86	- `web_extract` frequently fails with `409 BILLING_ERROR` on job-heavy targets (Naukri, Indeed, Anthropic, Cohere, Mistral, Together AI, Groq, LangChain, Baseten).
87	- `web_search` is the working fallback — it returns titles, descriptions, and listin

People who liked this also viewed

6 picks

Type

job-scraper-pipeline

33stars2forks

View on GitHub

How to install job-scraper-pipeline?

arimanyus/hermes-merchant/job-scraper-pipeline

$npx -y skills add arimanyus/hermes-merchant --skill job-scraper-pipeline

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Job Scraper + Gatekeeper Pipeline
2
3	Automated job scraping, deduplication, and gatekeeper scoring for ML/AI job hunting.
4
5	## When to Use
6	- Cron job that scrapes AI company career pages on Greenhouse, AshbyHQ, and Lever
7	- Runs on a schedule (e.g. every 15m, 6h, or daily)
8	- Appends new jobs to the queue file
9	- Scores pending jobs against profile keywords
10
11	## Configuration
12
13	All PII, paths, and scoring keywords live in `profile.yaml` at the repo root. Copy `profile.yaml.example` to `profile.yaml` and edit. Keys this skill reads:
14
15	- `paths.queue` — queue JSON location (default `~/hermes-merchant/state/queue/jobs.json`)
16	- `paths.scraper_log` — log file
17	- `scoring.required_keywords`, `scoring.adjacent_roles`, `scoring.reject_keywords`, `scoring.ai_companies`, `scoring.tiers`
18
19	Minimal loader (Python):
20
21	```python
22	import yaml, os
23	from pathlib import Path
24
25	cfg = yaml.safe_load(Path("profile.yaml").read_text())
26	queue_path = Path(os.path.expanduser(cfg["paths"]["queue"]))
27	queue_path.parent.mkdir(parents=True, exist_ok=True)
28	```
29
30	## Scraping Strategy
31
32	### Sources — What Works / What Doesn't
33
34	\| Source \| Method \| Status \| Notes \|
35	\|--------\|--------\|--------\|-------\|
36	\| Naukri.com \| browser_navigate \| Blocked \| Search results require JS rendering; homepage loads but listings aren't accessible. \|
37	\| Indeed.com \| browser_navigate \| Blocked by Cloudflare \| "Additional Verification Required"; cannot bypass without residential proxies. \|
38	\| Together AI \| Browser → `https://job-boards.greenhouse.io/togetherai` \| Works \| Greenhouse format — use the JS snippet below to extract all jobs. \|
39	\| Cohere \| Browser → `https://jobs.ashbyhq.com/cohere` \| Works (~100+ roles) \| AshbyHQ — largest pipeline source. Departments include Agentic Platform, Modeling, Applied-ML, Inference, Model Serving. \|
40	\| Baseten \| Browser → `https://jobs.ashbyhq.com/baseten` \| Works \| AshbyHQ. Many roles previously bounced on email — prefer web form. \|
41	\| Modal Labs \| Browser → `https://www.modal.com/careers` \| Works (~24 roles) \| AshbyHQ embedded on page — scroll to see listings. \|
42	\| Cerebras \| Browser → `https://www.cerebras.ai/join-us` → click ML dept \| Works \| Greenhouse-based, limited ML roles. \|
43	\| Mistral AI \| Browser → `https://jobs.lever.co/mistral` \| Works \| Lever ATS — do NOT add query params; go direct and filter in-page. \|
44	\| LangChain \| Browser → `https://jobs.ashbyhq.com/langchain` \| Works \| AshbyHQ; iframe on their own careers page — go direct. \|
45	\| Pinecone \| Browser → `https://www.pinecone.io/careers` \| Works \| AshbyHQ-hosted, JS console extraction works. \|
46	\| Groq \| Browser → `https://www.groq.com/careers/` \| Moved \| All engineering roles now on an external ATS (Gem.com). \|
47	\| Hugging Face \| Browser → `https://huggingface.co/jobs` \| Login-gated \| Requires auth to view. \|
48	\| Anthropic \| Browser → `https://job-boards.greenhouse.io/anthropic` \| Works partially \| Greenhouse — some listings require a second fetch. \|
49	\| Qdrant \| Browser \| Blocked \| `careers.qdrant.tech` resolves to a private network. \|
50
51	### ATS Platform Taxonomy
52
53	Three ATS platforms dominate AI company career pages. Each wants a different scraping approach:
54
55	\| ATS \| URL Pattern \| Best Method \|
56	\|-----\|-------------\|-------------\|
57	\| Greenhouse \| `job-boards.greenhouse.io/{company}` \| `browser_navigate` → `browser_console` JS snippet \|
58	\| AshbyHQ \| `jobs.ashbyhq.com/{company}` \| `browser_navigate` → `browser_snapshot` \|
59	\| Lever \| `jobs.lever.co/{company}` \| `browser_navigate` → `browser_snapshot` (no query params) \|
60
61	Known Greenhouse: Anthropic, Together AI, Cerebras.
62	Known AshbyHQ: Cohere, Baseten, LangChain, Modal, Pinecone.
63	Known Lever: Mistral AI.
64
65	### Greenhouse Job Extraction
66
67	Many AI companies host on Greenhouse at `https://job-boards.greenhouse.io/{company}`.
68
69	JavaScript snippet to extract ALL job links from a Greenhouse page (run via `browser_console`):
70
71	```javascript
72	"use strict"; (() => {
73	const links = document.querySelectorAll('a[href*="/jobs/"]');
74	const seen = new Set();
75	return Array.from(links)
76	.filter(l => l.href && !seen.has(l.href) && seen.add(l.href))
77	.map(l => l.href + '\t' + l.textContent.trim())
78	.join('\n');
79	})()
80	```
81
82	Returns a newline-separated list of `URL\tJob Title` pairs. Greenhouse pages load all jobs on one page — no pagination.
83
84	### web_extract is unreliable for job scraping
85
86	- `web_extract` frequently fails with `409 BILLING_ERROR` on job-heavy targets (Naukri, Indeed, Anthropic, Cohere, Mistral, Together AI, Groq, LangChain, Baseten).
87	- `web_search` is the working fallback — it returns titles, descriptions, and listin

People who liked this also viewed

6 picks

Type