sync-finance-data

Name: sync-finance-data
Author: ya5huk

20stars2forks

How to install sync-finance-data?

ya5huk/findash/sync-finance-data

$npx -y skills add ya5huk/findash --skill sync-finance-data

Installs into the current project.

Summary

Use when the user says "sync finance", "sync my finance", "ingest new docs", or any morning-update equivalent. Gathers new Drive drops into local staging (best-effort), ingests everything staged — bank fetch pairs + manual drops — into the SQLite database, sorts dump/ drops int

Files · 1

View on GitHub

SKILL.md

1	# sync-finance-data
2
3	You ingest new financial data into the local SQLite database: files staged locally by `fetch-bank-data` plus documents the user dropped in the Google Drive vault. You use judgment, not pattern-matching. Categorization, deduplication, and reconciliation across docs are your job — there are no rules in this file beyond the flow.
4
5	Drive is best-effort throughout; local ingest is the reliable core. Any Drive failure (listing, download, move, backup) becomes a `⚠️` bullet in the summary — never an abort. The run only fails if local ingestion itself crashes.
6
7	## Where things live
8
9	- Drive folder ID: the `[drive]` section of `.secrets/findash` (key `root_folder_id=…`, chmod 600). Folder structure: [`docs/drive-layout.md`](../../docs/drive-layout.md).
10	- SQLite schema + conventions: [`docs/sqlite-schema.md`](../../docs/sqlite-schema.md)
11	- How each archetype maps to tables: [`docs/doc-types/`](../../docs/doc-types/README.md)
12	- Password for protected payslip PDFs: the `[pdf-passwords]` section of `.secrets/findash` (one `pattern=password` line per file pattern)
13	- Drive access: `scripts/drive.sh` (`ls [-R]` / `pull` / `push` / `move` / `delete`) — it owns `./rclone.conf` and the root-ID from `.secrets/findash`, fast-fails on network trouble, and any non-zero exit means "Drive degraded" (warn + continue)
14	- Local DB: `data/finance.db`
15	- Staging — the ingest queue: `inbox/staging/fetched/` (pairs staged by `fetch-bank-data`) and `inbox/staging/drive/` (Drive pulls from step 1, filenames prefixed `<driveId>__` so identity survives a crashed session)
16
17	## Flow
18
19	### 1. Gather new Drive drops into staging (best-effort)
20
21	Pull anything new from Drive into local staging. Every Drive call here is best-effort: on failure, add a `⚠️` bullet (step 8) and continue — never abort.
22
23	- List the inbox: `scripts/drive.sh ls dump/`
24	- List the vault: for each top folder in `<DRIVE_ROOT>` (the five archetype folders — `payslips`, `investments`, `long-term-savings`, `full-statements`, `fx-conversions` — plus any new-domain folder), `scripts/drive.sh ls -R <folder>/`. Collect `(ID, Path, ModTime, Size)`.
25	- If listing fails (auth lapse, network down): one warning bullet — `⚠️ Drive unreachable — manual drops skipped this run` — then go straight to step 2 and ingest whatever is already staged.
26	- Dedup before download: query `documents.drive_id`; skip anything already present. Exception: a `dump/`-listed file whose `drive_id` is already in `documents` was ingested on a previous run but its Drive-side move failed — put it on the deferred-move list for step 5 (its destination is stored in `documents.drive_path`); don't re-download.
27	- Download each genuinely new file: `scripts/drive.sh pull <path> inbox/staging/drive/<ID>__<filename>` — the `<ID>__` prefix keeps the drive_id attached to the file even if this session dies before ingest. Sanity-check the downloaded size against the listing. A per-file failure → warning bullet, skip that file, keep going.
28
29	### 2. Ingest everything in staging (local, reliable)
30
31	The core of the skill — no Drive dependency. Enumerate both staging dirs, `inbox/staging/fetched/` and `inbox/staging/drive/`, including files left over from a previous crashed run.
32
33	Per file, first the already-ingested check: derive its document id — for Drive files, the `<ID>__` filename prefix; for fetched pairs, `local:fetch:<json-filename>:<sha256-first-12>` recomputed from the `.json` bytes. If that id is already in `documents`, delete the staged file (a previous run committed it but died before cleanup) and move on.
34
35	Otherwise process it:
36
37	- Recognize the archetype from its origin folder (Drive files) or its `-api-fetch` naming (fetched pairs), then confirm by reading the content. See [`docs/doc-types/`](../../docs/doc-types/README.md) for the shapes you'll see.
38	- Dump-sourced files: judge which archetype folder the file belongs to and give it a descriptive destination name (no required format). Record the decision in the document row: `drive_path = <dest-folder>/<new-name>` — step 5 executes the actual Drive-side move from that value. Set `documents.doc_type` to a short free-text label (e.g. `harel-pension-statement`) — a human-readable note, not a value from a closed set.
39
40	Unfamiliar files: map to the nearest archetype by reading the content — most things fit one of the five folders. If a file genuinely belongs to a new financial domain (e.g. insurance), use judgment: give it a folder, add a short prose note to the matching catalogue page (what it holds, how to read it, which tables

People who liked this also viewed

6 picks

Type

sync-finance-data

20stars2forks

View on GitHub

How to install sync-finance-data?

ya5huk/findash/sync-finance-data

$npx -y skills add ya5huk/findash --skill sync-finance-data

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# sync-finance-data
2
3	You ingest new financial data into the local SQLite database: files staged locally by `fetch-bank-data` plus documents the user dropped in the Google Drive vault. You use judgment, not pattern-matching. Categorization, deduplication, and reconciliation across docs are your job — there are no rules in this file beyond the flow.
4
5	Drive is best-effort throughout; local ingest is the reliable core. Any Drive failure (listing, download, move, backup) becomes a `⚠️` bullet in the summary — never an abort. The run only fails if local ingestion itself crashes.
6
7	## Where things live
8
9	- Drive folder ID: the `[drive]` section of `.secrets/findash` (key `root_folder_id=…`, chmod 600). Folder structure: [`docs/drive-layout.md`](../../docs/drive-layout.md).
10	- SQLite schema + conventions: [`docs/sqlite-schema.md`](../../docs/sqlite-schema.md)
11	- How each archetype maps to tables: [`docs/doc-types/`](../../docs/doc-types/README.md)
12	- Password for protected payslip PDFs: the `[pdf-passwords]` section of `.secrets/findash` (one `pattern=password` line per file pattern)
13	- Drive access: `scripts/drive.sh` (`ls [-R]` / `pull` / `push` / `move` / `delete`) — it owns `./rclone.conf` and the root-ID from `.secrets/findash`, fast-fails on network trouble, and any non-zero exit means "Drive degraded" (warn + continue)
14	- Local DB: `data/finance.db`
15	- Staging — the ingest queue: `inbox/staging/fetched/` (pairs staged by `fetch-bank-data`) and `inbox/staging/drive/` (Drive pulls from step 1, filenames prefixed `<driveId>__` so identity survives a crashed session)
16
17	## Flow
18
19	### 1. Gather new Drive drops into staging (best-effort)
20
21	Pull anything new from Drive into local staging. Every Drive call here is best-effort: on failure, add a `⚠️` bullet (step 8) and continue — never abort.
22
23	- List the inbox: `scripts/drive.sh ls dump/`
24	- List the vault: for each top folder in `<DRIVE_ROOT>` (the five archetype folders — `payslips`, `investments`, `long-term-savings`, `full-statements`, `fx-conversions` — plus any new-domain folder), `scripts/drive.sh ls -R <folder>/`. Collect `(ID, Path, ModTime, Size)`.
25	- If listing fails (auth lapse, network down): one warning bullet — `⚠️ Drive unreachable — manual drops skipped this run` — then go straight to step 2 and ingest whatever is already staged.
26	- Dedup before download: query `documents.drive_id`; skip anything already present. Exception: a `dump/`-listed file whose `drive_id` is already in `documents` was ingested on a previous run but its Drive-side move failed — put it on the deferred-move list for step 5 (its destination is stored in `documents.drive_path`); don't re-download.
27	- Download each genuinely new file: `scripts/drive.sh pull <path> inbox/staging/drive/<ID>__<filename>` — the `<ID>__` prefix keeps the drive_id attached to the file even if this session dies before ingest. Sanity-check the downloaded size against the listing. A per-file failure → warning bullet, skip that file, keep going.
28
29	### 2. Ingest everything in staging (local, reliable)
30
31	The core of the skill — no Drive dependency. Enumerate both staging dirs, `inbox/staging/fetched/` and `inbox/staging/drive/`, including files left over from a previous crashed run.
32
33	Per file, first the already-ingested check: derive its document id — for Drive files, the `<ID>__` filename prefix; for fetched pairs, `local:fetch:<json-filename>:<sha256-first-12>` recomputed from the `.json` bytes. If that id is already in `documents`, delete the staged file (a previous run committed it but died before cleanup) and move on.
34
35	Otherwise process it:
36
37	- Recognize the archetype from its origin folder (Drive files) or its `-api-fetch` naming (fetched pairs), then confirm by reading the content. See [`docs/doc-types/`](../../docs/doc-types/README.md) for the shapes you'll see.
38	- Dump-sourced files: judge which archetype folder the file belongs to and give it a descriptive destination name (no required format). Record the decision in the document row: `drive_path = <dest-folder>/<new-name>` — step 5 executes the actual Drive-side move from that value. Set `documents.doc_type` to a short free-text label (e.g. `harel-pension-statement`) — a human-readable note, not a value from a closed set.
39
40	Unfamiliar files: map to the nearest archetype by reading the content — most things fit one of the five folders. If a file genuinely belongs to a new financial domain (e.g. insurance), use judgment: give it a folder, add a short prose note to the matching catalogue page (what it holds, how to read it, which tables

People who liked this also viewed

6 picks

Type