distributed-search

Name: distributed-search
Author: proyecto26

44installs1forks

How to install distributed-search?

proyecto26/system-design-skills/distributed-search

$npx -y skills add proyecto26/system-design-skills --skill distributed-search

Installs into the current project.

Summary

This skill should be used when the user designs a "search system", needs "full-text search", asks about an "inverted index", "Elasticsearch / OpenSearch", "relevance ranking" (TF-IDF/BM25), "search autocomplete / typeahead", an "indexing pipeline", or "faceted search". It gives t

Files · 1

View on GitHub

SKILL.md

1	# Distributed search
2
3	Find the documents that best match a free-text query, ranked by relevance, fast,
4	across more data than one machine holds. Getting it wrong means either slow
5	`LIKE '%term%'` scans that melt the primary database, or a search box that
6	returns the wrong results and erodes user trust — both are silent until traffic
7	or corpus size exposes them.
8
9	## When to reach for this
10	Users type words and expect ranked, relevant matches — not exact-key lookups.
11	The corpus is text-heavy (documents, products, logs, messages), queries are
12	ad-hoc (any term, any combination), and results need ranking, highlighting,
13	facets, or typeahead. Reach for it when a `WHERE col LIKE` or full-table scan is
14	already the read bottleneck, or when you need fuzzy/partial matching a B-tree
15	index cannot serve.
16
17	## When NOT to
18	The access pattern is fetch-by-known-key or a fixed filter — a primary database
19	index serves that far more cheaply and consistently; keep it in `data-storage`.
20	The corpus is tiny (thousands of rows): an in-process filter or the database's
21	built-in full-text index is enough — a separate search cluster is pure
22	operational overhead (YAGNI). Search is a derived, eventually-consistent copy
23	of your data; never make it the system of record.
24
25	## Clarify first
26	- Corpus size and growth — document count, average doc size, total index
27	bytes? (→ `back-of-the-envelope`) This decides shard count.
28	- Query QPS and shape — read-heavy? term queries, phrase, fuzzy, facets,
29	autocomplete? Latency target (p99)?
30	- Indexing freshness — must a new/edited doc be searchable in seconds
31	(near-real-time) or is minutes/hours of lag fine?
32	- Relevance bar — is exact term-match enough, or do users expect "best"
33	results (ranking, synonyms, typo tolerance)?
34	- Write rate — how many docs/sec change? This sizes the indexing pipeline.
35
36	## The options
37
38	The pipeline (almost always present): a source emits document changes →
39	an indexing pipeline transforms/analyzes them → the inverted index stores
40	term→document postings → the query path matches and ranks. For a crawl-based
41	system (web search), prepend crawl → parse → dedupe; that crawler is its own
42	subsystem feeding the same pipeline.
43
44	Index build mode
45	- Batch / bulk reindex: rebuild the whole index periodically. Use when the
46	corpus changes slowly or freshness in hours is acceptable.
47	- Near-real-time (incremental): apply changes continuously so docs are
48	searchable in seconds. Use when users expect to find what they just wrote.
49
50	Ranking model
51	- Boolean / filter only: match, no scoring. Use for exact filtering (tags,
52	facets) where order doesn't matter.
53	- TF-IDF / BM25 (lexical): score by term frequency and rarity. The default
54	full-text relevance model; cheap and explainable.
55	- Hybrid (lexical + signals): blend BM25 with popularity, recency, or
56	business boosts. Use when "best" means more than word overlap.
57
58	Autocomplete
59	- Prefix trie / FST in memory: sub-millisecond typeahead from a prefix. Use
60	for suggestion-as-you-type.
61	- Edge-n-gram index: prefix matching inside the main index. Use when
62	suggestions must also respect filters/relevance, at higher cost.
63
64	Distribution: split the index into shards (each a self-contained
65	inverted index over a doc subset) for capacity, and replicas per shard for
66	read throughput and fault tolerance. Sharding theory lives in `data-storage`.
67
68	## Trade-offs
69
70	\| Option \| What it solves \| What it worsens \| Change it when \|
71	\|---\|---\|---\|---\|
72	\| Batch reindex \| Simple, atomic swap, no live-write complexity \| Stale until next build; full rebuild is costly \| Users need fresh results → near-real-time \|
73	\| Near-real-time \| Seconds-fresh; no full rebuild \| Segment churn, merge load, refresh cost on writes \| Write rate or merge cost overwhelms nodes → batch/larger refresh interval \|
74	\| Boolean/filter \| Cheapest; deterministic \| No notion of "best" result \| Users judge result quality → add BM25 \|
75	\| BM25 \| Good relevance, explainable, cheap \| Ignores popularity/recency/intent \| Word-overlap isn't enough → hybrid signals \|
76	\| Hybrid signals \| Matches business/user intent \| Complex, harder to debug, needs tuning data \| Tuning cost exceeds value → fall back to BM25 \|
77	\| Prefix trie/FST \| Fastest typeahead \| Separate structure to build/refresh; ignores filters \| Suggestions need filters/relevance → edge-n-gram \|
78	\| More shards \| Parallelism, fits

People who liked this also viewed

6 picks

Type

distributed-search

44installs1forks

View on GitHub

How to install distributed-search?

proyecto26/system-design-skills/distributed-search

$npx -y skills add proyecto26/system-design-skills --skill distributed-search

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Distributed search
2
3	Find the documents that best match a free-text query, ranked by relevance, fast,
4	across more data than one machine holds. Getting it wrong means either slow
5	`LIKE '%term%'` scans that melt the primary database, or a search box that
6	returns the wrong results and erodes user trust — both are silent until traffic
7	or corpus size exposes them.
8
9	## When to reach for this
10	Users type words and expect ranked, relevant matches — not exact-key lookups.
11	The corpus is text-heavy (documents, products, logs, messages), queries are
12	ad-hoc (any term, any combination), and results need ranking, highlighting,
13	facets, or typeahead. Reach for it when a `WHERE col LIKE` or full-table scan is
14	already the read bottleneck, or when you need fuzzy/partial matching a B-tree
15	index cannot serve.
16
17	## When NOT to
18	The access pattern is fetch-by-known-key or a fixed filter — a primary database
19	index serves that far more cheaply and consistently; keep it in `data-storage`.
20	The corpus is tiny (thousands of rows): an in-process filter or the database's
21	built-in full-text index is enough — a separate search cluster is pure
22	operational overhead (YAGNI). Search is a derived, eventually-consistent copy
23	of your data; never make it the system of record.
24
25	## Clarify first
26	- Corpus size and growth — document count, average doc size, total index
27	bytes? (→ `back-of-the-envelope`) This decides shard count.
28	- Query QPS and shape — read-heavy? term queries, phrase, fuzzy, facets,
29	autocomplete? Latency target (p99)?
30	- Indexing freshness — must a new/edited doc be searchable in seconds
31	(near-real-time) or is minutes/hours of lag fine?
32	- Relevance bar — is exact term-match enough, or do users expect "best"
33	results (ranking, synonyms, typo tolerance)?
34	- Write rate — how many docs/sec change? This sizes the indexing pipeline.
35
36	## The options
37
38	The pipeline (almost always present): a source emits document changes →
39	an indexing pipeline transforms/analyzes them → the inverted index stores
40	term→document postings → the query path matches and ranks. For a crawl-based
41	system (web search), prepend crawl → parse → dedupe; that crawler is its own
42	subsystem feeding the same pipeline.
43
44	Index build mode
45	- Batch / bulk reindex: rebuild the whole index periodically. Use when the
46	corpus changes slowly or freshness in hours is acceptable.
47	- Near-real-time (incremental): apply changes continuously so docs are
48	searchable in seconds. Use when users expect to find what they just wrote.
49
50	Ranking model
51	- Boolean / filter only: match, no scoring. Use for exact filtering (tags,
52	facets) where order doesn't matter.
53	- TF-IDF / BM25 (lexical): score by term frequency and rarity. The default
54	full-text relevance model; cheap and explainable.
55	- Hybrid (lexical + signals): blend BM25 with popularity, recency, or
56	business boosts. Use when "best" means more than word overlap.
57
58	Autocomplete
59	- Prefix trie / FST in memory: sub-millisecond typeahead from a prefix. Use
60	for suggestion-as-you-type.
61	- Edge-n-gram index: prefix matching inside the main index. Use when
62	suggestions must also respect filters/relevance, at higher cost.
63
64	Distribution: split the index into shards (each a self-contained
65	inverted index over a doc subset) for capacity, and replicas per shard for
66	read throughput and fault tolerance. Sharding theory lives in `data-storage`.
67
68	## Trade-offs
69
70	\| Option \| What it solves \| What it worsens \| Change it when \|
71	\|---\|---\|---\|---\|
72	\| Batch reindex \| Simple, atomic swap, no live-write complexity \| Stale until next build; full rebuild is costly \| Users need fresh results → near-real-time \|
73	\| Near-real-time \| Seconds-fresh; no full rebuild \| Segment churn, merge load, refresh cost on writes \| Write rate or merge cost overwhelms nodes → batch/larger refresh interval \|
74	\| Boolean/filter \| Cheapest; deterministic \| No notion of "best" result \| Users judge result quality → add BM25 \|
75	\| BM25 \| Good relevance, explainable, cheap \| Ignores popularity/recency/intent \| Word-overlap isn't enough → hybrid signals \|
76	\| Hybrid signals \| Matches business/user intent \| Complex, harder to debug, needs tuning data \| Tuning cost exceeds value → fall back to BM25 \|
77	\| Prefix trie/FST \| Fastest typeahead \| Separate structure to build/refresh; ignores filters \| Suggestions need filters/relevance → edge-n-gram \|
78	\| More shards \| Parallelism, fits

People who liked this also viewed

6 picks

Type