wiki-retrieve

Name: wiki-retrieve
Author: AgriciDaniel

164installs1.1kforks

How to install wiki-retrieve?

agricidaniel/claude-obsidian/wiki-retrieve

$npx -y skills add AgriciDaniel/claude-obsidian --skill wiki-retrieve

Installs into the current project.

Summary

Hybrid retrieval primitive for the Compound Vault. Replaces the v1.6 static hot→index→drill read order with contextual-prefix + BM25 + cosine-rerank, modeled on Anthropic's Sept 2024 Contextual Retrieval research (35-49-67% retrieval-failure reduction). Opt-in via `bash bin/setup

Files · 1

View on GitHub

SKILL.md

1	# wiki-retrieve: Hybrid Retrieval over the Vault
2
3	The v1.6 query path was `Read(hot.md) → Read(index.md) → Read(3-5 pages) → synthesize`. It worked, but page-level granularity loses to chunk-level granularity any time the answer lives in a specific passage rather than a whole page. The v1.7 `wiki-retrieve` skill is the chunk-level upgrade — opt-in, feature-gated, and replaces nothing if you don't run the setup.
4
5	Origin: This skill is original to claude-obsidian. There is no upstream kepano equivalent. The technique is from [Anthropic's Sept 2024 Contextual Retrieval research](https://www.anthropic.com/news/contextual-retrieval) — we implement it as agent-skill plumbing.
6
7	---
8
9	## Data privacy (v1.7.1+)
10
11	Tier 1 (Anthropic API) and tier 2 (claude CLI subprocess) of the contextual-prefix generator send wiki page bodies off-machine. As of v1.7.1, both tiers are GATED behind explicit user consent at two layers:
12
13	- `scripts/contextual-prefix.py --allow-egress` (default off). Without the flag, `pick_prefix_tier()` returns `"synthetic"` regardless of `ANTHROPIC_API_KEY` or `claude` binary presence.
14	- `bin/setup-retrieve.sh` prompts before any non-synthetic Stage 1 run; default is abort.
15
16	To run fully on-machine (tier 3 synthetic prefix + local ollama rerank), use `bash bin/setup-retrieve.sh --no-llm`. This is also the effective behavior if you decline the consent prompt or omit `--allow-egress`.
17
18	The guard mirrors `scripts/tiling-check.py:351` `--allow-remote-ollama`. v1.6 vaults that never provisioned this skill see zero behavior change.
19
20	---
21
22	## Architecture
23
24	```
25	INGEST (one-time, then incremental):
26
27	wiki/<page>.md
28	│
29	▼
30	scripts/contextual-prefix.py
31	│ ├─ chunk on paragraph boundaries (~500 token target, 200 char overlap)
32	│ ├─ generate 1-2 sentence prefix per chunk
33	│ │ tier 1: ANTHROPIC_API_KEY → Anthropic API (Haiku, prompt-cached
34	│ │ when body ≥ ~16 KB / Haiku 4.5 floor)
35	│ │ tier 2: `claude` on PATH → claude -p subprocess
36	│ │ tier 3: synthetic → frontmatter title + first paragraph
37	│ └─ write .vault-meta/chunks/<address>/chunk-NNN.json
38	│
39	▼
40	scripts/bm25-index.py build
41	└─ inverted index over chunks' contextualized_text → .vault-meta/bm25/index.json
42
43	QUERY:
44
45	query string
46	│
47	▼
48	scripts/retrieve.py "<query>" --top 5
49	├─ bm25-index.py query "<query>" --top 20 (sparse candidate set)
50	├─ rerank.py "<query>" --candidates - (dense rerank via ollama cosine)
51	│ cosine(query_embedding, chunk_embedding)
52	│ embeddings cached in .vault-meta/embed-cache.json keyed by body_hash
53	└─ dedupe by page-address, return top-N candidates with absolute_path
54	│
55	▼
56	caller (wiki-query / autoresearch) reads the cited pages and synthesizes
57	```
58
59	---
60
61	## Feature gating
62
63	Other skills must detect this skill before using it. The canonical detection:
64
65	```bash
66	[ -x scripts/retrieve.py ] && [ -d .vault-meta/chunks ] && \
67	[ -f .vault-meta/bm25/index.json ] && \
68	echo "wiki-retrieve installed" \|\| echo "fallback: legacy hot→index→drill"
69	```
70
71	If detection fails, callers MUST fall back to the v1.6 read order. This skill never breaks the base plugin.
72
73	---
74
75	## Setup
76
77	```bash
78	bash bin/setup-retrieve.sh
79	```
80
81	What it does, in order:
82	1. Sanity-checks the 4 scripts are present and executable.
83	2. Creates `.vault-meta/chunks/` and `.vault-meta/bm25/`.
84	3. Probes ollama at `http://127.0.0.1:11434` for `nomic-embed-text` (rerank prerequisite). Reports status; does not install.
85	4. Reports which contextual-prefix tier will be used (Anthropic API / claude CLI / synthetic).
86	5. Runs `contextual-prefix.py --all` to chunk + contextualize every wiki page.
87	6. Runs `bm25-index.py build`.
88	7. Smoke-tests `retrieve.py` against the query "wiki".
89
90	Flags:
91	- `--check` — diagnostics only, no provisioning.
92	- `--no-llm` — force tier-3 synthetic prefix (cheapest, zero LLM dependency).
93	- `--rebuild` — re-chunk every page even if body_hash matches.
94
95	---
96
97	## Cost ceiling
98
99	Per Anthropic's published research, contextual-prefix generation costs approximately $12 per 1,000 documents with Haiku + prompt caching. For a 100-page vault with ~3 chunks per page, that's ~$3.60 one-time, with incremental updates much cheaper (only changed pages re-process).
100
101	If you want to vali

People who liked this also viewed

6 picks

Type

wiki-retrieve

164installs1.1kforks

View on GitHub

How to install wiki-retrieve?

agricidaniel/claude-obsidian/wiki-retrieve

$npx -y skills add AgriciDaniel/claude-obsidian --skill wiki-retrieve

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# wiki-retrieve: Hybrid Retrieval over the Vault
2
3	The v1.6 query path was `Read(hot.md) → Read(index.md) → Read(3-5 pages) → synthesize`. It worked, but page-level granularity loses to chunk-level granularity any time the answer lives in a specific passage rather than a whole page. The v1.7 `wiki-retrieve` skill is the chunk-level upgrade — opt-in, feature-gated, and replaces nothing if you don't run the setup.
4
5	Origin: This skill is original to claude-obsidian. There is no upstream kepano equivalent. The technique is from [Anthropic's Sept 2024 Contextual Retrieval research](https://www.anthropic.com/news/contextual-retrieval) — we implement it as agent-skill plumbing.
6
7	---
8
9	## Data privacy (v1.7.1+)
10
11	Tier 1 (Anthropic API) and tier 2 (claude CLI subprocess) of the contextual-prefix generator send wiki page bodies off-machine. As of v1.7.1, both tiers are GATED behind explicit user consent at two layers:
12
13	- `scripts/contextual-prefix.py --allow-egress` (default off). Without the flag, `pick_prefix_tier()` returns `"synthetic"` regardless of `ANTHROPIC_API_KEY` or `claude` binary presence.
14	- `bin/setup-retrieve.sh` prompts before any non-synthetic Stage 1 run; default is abort.
15
16	To run fully on-machine (tier 3 synthetic prefix + local ollama rerank), use `bash bin/setup-retrieve.sh --no-llm`. This is also the effective behavior if you decline the consent prompt or omit `--allow-egress`.
17
18	The guard mirrors `scripts/tiling-check.py:351` `--allow-remote-ollama`. v1.6 vaults that never provisioned this skill see zero behavior change.
19
20	---
21
22	## Architecture
23
24	```
25	INGEST (one-time, then incremental):
26
27	wiki/<page>.md
28	│
29	▼
30	scripts/contextual-prefix.py
31	│ ├─ chunk on paragraph boundaries (~500 token target, 200 char overlap)
32	│ ├─ generate 1-2 sentence prefix per chunk
33	│ │ tier 1: ANTHROPIC_API_KEY → Anthropic API (Haiku, prompt-cached
34	│ │ when body ≥ ~16 KB / Haiku 4.5 floor)
35	│ │ tier 2: `claude` on PATH → claude -p subprocess
36	│ │ tier 3: synthetic → frontmatter title + first paragraph
37	│ └─ write .vault-meta/chunks/<address>/chunk-NNN.json
38	│
39	▼
40	scripts/bm25-index.py build
41	└─ inverted index over chunks' contextualized_text → .vault-meta/bm25/index.json
42
43	QUERY:
44
45	query string
46	│
47	▼
48	scripts/retrieve.py "<query>" --top 5
49	├─ bm25-index.py query "<query>" --top 20 (sparse candidate set)
50	├─ rerank.py "<query>" --candidates - (dense rerank via ollama cosine)
51	│ cosine(query_embedding, chunk_embedding)
52	│ embeddings cached in .vault-meta/embed-cache.json keyed by body_hash
53	└─ dedupe by page-address, return top-N candidates with absolute_path
54	│
55	▼
56	caller (wiki-query / autoresearch) reads the cited pages and synthesizes
57	```
58
59	---
60
61	## Feature gating
62
63	Other skills must detect this skill before using it. The canonical detection:
64
65	```bash
66	[ -x scripts/retrieve.py ] && [ -d .vault-meta/chunks ] && \
67	[ -f .vault-meta/bm25/index.json ] && \
68	echo "wiki-retrieve installed" \|\| echo "fallback: legacy hot→index→drill"
69	```
70
71	If detection fails, callers MUST fall back to the v1.6 read order. This skill never breaks the base plugin.
72
73	---
74
75	## Setup
76
77	```bash
78	bash bin/setup-retrieve.sh
79	```
80
81	What it does, in order:
82	1. Sanity-checks the 4 scripts are present and executable.
83	2. Creates `.vault-meta/chunks/` and `.vault-meta/bm25/`.
84	3. Probes ollama at `http://127.0.0.1:11434` for `nomic-embed-text` (rerank prerequisite). Reports status; does not install.
85	4. Reports which contextual-prefix tier will be used (Anthropic API / claude CLI / synthetic).
86	5. Runs `contextual-prefix.py --all` to chunk + contextualize every wiki page.
87	6. Runs `bm25-index.py build`.
88	7. Smoke-tests `retrieve.py` against the query "wiki".
89
90	Flags:
91	- `--check` — diagnostics only, no provisioning.
92	- `--no-llm` — force tier-3 synthetic prefix (cheapest, zero LLM dependency).
93	- `--rebuild` — re-chunk every page even if body_hash matches.
94
95	---
96
97	## Cost ceiling
98
99	Per Anthropic's published research, contextual-prefix generation costs approximately $12 per 1,000 documents with Haiku + prompt caching. For a 100-page vault with ~3 chunks per page, that's ~$3.60 one-time, with incremental updates much cheaper (only changed pages re-process).
100
101	If you want to vali

People who liked this also viewed

6 picks

Type