read-personal-data-harvester

Name: read-personal-data-harvester
Author: SpaceZephyr

56stars18forks

How to install read-personal-data-harvester?

spacezephyr/read-buddy/read-personal-data-harvester

$npx -y skills add SpaceZephyr/read-buddy --skill read-personal-data-harvester

Installs into the current project.

Summary

Autonomously collect and structure a user's personal content history from Chinese and international platforms (豆瓣, 小红书, B站, 微信读书, 抖音, Kindle, etc.) using browser automation, local file parsing, and system share-sheet intake. Use this skill whenever the user wants to: build a pers

Files · 1

View on GitHub

SKILL.md

1	# Personal Data Harvester
2
3	Helps Claude autonomously build and maintain a local pipeline that collects a user's personal content
4	history across platforms, stores it in a structured SQLite database, and keeps it fresh over time.
5
6	## Core philosophy
7
8	- User's own data, user's own device. All collection happens under the user's authenticated session
9	or from locally cached files. Never store credentials; reuse existing browser sessions.
10	- Graceful degradation. Each platform has a primary and fallback strategy. If automation breaks
11	(platform redesign, rate-limit), fall back to file import without losing prior data.
12	- Privacy-first. Data stays local by default. No uploads unless the user explicitly requests it.
13
14	---
15
16	## Step 0 — Assess the environment
17
18	Before writing any code, run this checklist:
19
20	```bash
21	python3 --version # need >= 3.10
22	pip show playwright # if missing: pip install playwright && playwright install chromium
23	pip show beautifulsoup4 # if missing: pip install beautifulsoup4 requests pydantic
24	ls ~/Library/Application\ Support/微信读书/ 2>/dev/null \|\| echo "no local wechat-read cache"
25	```
26
27	Ask the user:
28	1. Which platforms to harvest first (prioritise by what they use most)?
29	2. Desktop or mobile primary device?
30	3. Acceptable harvest frequency (daily cron, manual trigger, or always-on daemon)?
31
32	Read `references/platforms.md` for per-platform technical details before writing any scraper.
33
34	---
35
36	## Step 1 — Initialise the local database
37
38	Always create this schema first. All scrapers write to the same DB.
39
40	```python
41	# scripts/init_db.py
42	import sqlite3, pathlib
43
44	DB = pathlib.Path.home() / ".personal-harvest" / "data.db"
45	DB.parent.mkdir(exist_ok=True)
46
47	with sqlite3.connect(DB) as con:
48	con.executescript("""
49	CREATE TABLE IF NOT EXISTS items (
50	id TEXT PRIMARY KEY, -- platform:platform_id
51	platform TEXT NOT NULL, -- douban \| bilibili \| xiaohongshu \| wechatread \| kindle
52	type TEXT NOT NULL, -- book \| video \| note \| article \| post
53	title TEXT,
54	url TEXT,
55	creator TEXT,
56	tags TEXT, -- JSON array
57	user_rating INTEGER, -- 1-5 or null
58	user_status TEXT, -- want \| doing \| done \| liked \| saved
59	user_note TEXT, -- user's own annotation
60	summary TEXT, -- AI-generated or platform description
61	collected_at TEXT, -- ISO8601
62	harvested_at TEXT DEFAULT (datetime('now'))
63	);
64	CREATE INDEX IF NOT EXISTS idx_platform ON items(platform);
65	CREATE INDEX IF NOT EXISTS idx_type ON items(type);
66	CREATE INDEX IF NOT EXISTS idx_status ON items(user_status);
67	""")
68	print(f"DB ready at {DB}")
69	```
70
71	---
72
73	## Step 2 — Choose collection strategy per platform
74
75	\| Platform \| Primary strategy \| Fallback \|
76	\|---\|---\|---\|
77	\| 豆瓣 \| Playwright browser automation (logged-in session) \| HTML export + parse \|
78	\| 小红书 \| Browser plugin DOM capture / Playwright \| Manual share-sheet link intake \|
79	\| B 站 \| Playwright (favorites API endpoint visible in DevTools) \| Official data export \|
80	\| 微信读书 \| Local SQLite cache file \| Playwright web version \|
81	\| 抖音 \| iOS/Android share-sheet → link resolver \| Playwright (harder, rate-limited) \|
82	\| Kindle \| `My Clippings.txt` file parse \| Goodreads export \|
83	\| 豆瓣读书 App \| Playwright web douban.com \| Official "我的豆瓣"数据导出 \|
84
85	See `references/platforms.md` for exact file paths, API endpoints, and selector patterns.
86
87	---
88
89	## Step 3 — Implement scrapers
90
91	### Pattern A — Playwright browser automation (豆瓣, B站, 小红书)
92
93	```python
94	# scripts/scrape_douban.py
95	"""
96	Collect 豆瓣读过/在读/想读 books and watched films.
97	Requires: user already logged in to douban.com in the system Chromium profile.
98	"""
99	import asyncio, json, sqlite3, pathlib
100	from datetime import datetime
101	from playwright.async_api import async_playwrig

People who liked this also viewed

6 picks

Type

read-personal-data-harvester

56stars18forks

View on GitHub

How to install read-personal-data-harvester?

spacezephyr/read-buddy/read-personal-data-harvester

$npx -y skills add SpaceZephyr/read-buddy --skill read-personal-data-harvester

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Personal Data Harvester
2
3	Helps Claude autonomously build and maintain a local pipeline that collects a user's personal content
4	history across platforms, stores it in a structured SQLite database, and keeps it fresh over time.
5
6	## Core philosophy
7
8	- User's own data, user's own device. All collection happens under the user's authenticated session
9	or from locally cached files. Never store credentials; reuse existing browser sessions.
10	- Graceful degradation. Each platform has a primary and fallback strategy. If automation breaks
11	(platform redesign, rate-limit), fall back to file import without losing prior data.
12	- Privacy-first. Data stays local by default. No uploads unless the user explicitly requests it.
13
14	---
15
16	## Step 0 — Assess the environment
17
18	Before writing any code, run this checklist:
19
20	```bash
21	python3 --version # need >= 3.10
22	pip show playwright # if missing: pip install playwright && playwright install chromium
23	pip show beautifulsoup4 # if missing: pip install beautifulsoup4 requests pydantic
24	ls ~/Library/Application\ Support/微信读书/ 2>/dev/null \|\| echo "no local wechat-read cache"
25	```
26
27	Ask the user:
28	1. Which platforms to harvest first (prioritise by what they use most)?
29	2. Desktop or mobile primary device?
30	3. Acceptable harvest frequency (daily cron, manual trigger, or always-on daemon)?
31
32	Read `references/platforms.md` for per-platform technical details before writing any scraper.
33
34	---
35
36	## Step 1 — Initialise the local database
37
38	Always create this schema first. All scrapers write to the same DB.
39
40	```python
41	# scripts/init_db.py
42	import sqlite3, pathlib
43
44	DB = pathlib.Path.home() / ".personal-harvest" / "data.db"
45	DB.parent.mkdir(exist_ok=True)
46
47	with sqlite3.connect(DB) as con:
48	con.executescript("""
49	CREATE TABLE IF NOT EXISTS items (
50	id TEXT PRIMARY KEY, -- platform:platform_id
51	platform TEXT NOT NULL, -- douban \| bilibili \| xiaohongshu \| wechatread \| kindle
52	type TEXT NOT NULL, -- book \| video \| note \| article \| post
53	title TEXT,
54	url TEXT,
55	creator TEXT,
56	tags TEXT, -- JSON array
57	user_rating INTEGER, -- 1-5 or null
58	user_status TEXT, -- want \| doing \| done \| liked \| saved
59	user_note TEXT, -- user's own annotation
60	summary TEXT, -- AI-generated or platform description
61	collected_at TEXT, -- ISO8601
62	harvested_at TEXT DEFAULT (datetime('now'))
63	);
64	CREATE INDEX IF NOT EXISTS idx_platform ON items(platform);
65	CREATE INDEX IF NOT EXISTS idx_type ON items(type);
66	CREATE INDEX IF NOT EXISTS idx_status ON items(user_status);
67	""")
68	print(f"DB ready at {DB}")
69	```
70
71	---
72
73	## Step 2 — Choose collection strategy per platform
74
75	\| Platform \| Primary strategy \| Fallback \|
76	\|---\|---\|---\|
77	\| 豆瓣 \| Playwright browser automation (logged-in session) \| HTML export + parse \|
78	\| 小红书 \| Browser plugin DOM capture / Playwright \| Manual share-sheet link intake \|
79	\| B 站 \| Playwright (favorites API endpoint visible in DevTools) \| Official data export \|
80	\| 微信读书 \| Local SQLite cache file \| Playwright web version \|
81	\| 抖音 \| iOS/Android share-sheet → link resolver \| Playwright (harder, rate-limited) \|
82	\| Kindle \| `My Clippings.txt` file parse \| Goodreads export \|
83	\| 豆瓣读书 App \| Playwright web douban.com \| Official "我的豆瓣"数据导出 \|
84
85	See `references/platforms.md` for exact file paths, API endpoints, and selector patterns.
86
87	---
88
89	## Step 3 — Implement scrapers
90
91	### Pattern A — Playwright browser automation (豆瓣, B站, 小红书)
92
93	```python
94	# scripts/scrape_douban.py
95	"""
96	Collect 豆瓣读过/在读/想读 books and watched films.
97	Requires: user already logged in to douban.com in the system Chromium profile.
98	"""
99	import asyncio, json, sqlite3, pathlib
100	from datetime import datetime
101	from playwright.async_api import async_playwrig

People who liked this also viewed

6 picks

Type