data-storage

Name: data-storage
Author: proyecto26

51installs1forks

How to install data-storage?

proyecto26/system-design-skills/data-storage

$npx -y skills add proyecto26/system-design-skills --skill data-storage

Installs into the current project.

Summary

This skill should be used when the user asks "SQL or NoSQL", "which database", how to design a "data model" or "schema design", picks an "indexing" strategy, needs "sharding" or "partitioning", sets up "replication" (leader-follower / multi-leader), defines a "primary key"/"sort

Files · 1

View on GitHub

SKILL.md

1	# Data Storage
2
3	Choose where records live, how they are keyed and queried, and how the store
4	grows past a single machine. Storage is the hardest layer to change later: a
5	wrong data model or shard key calcifies into a scaling ceiling, and getting
6	replication wrong silently serves stale or lost data.
7
8	## When to reach for this
9	Any system that persists state: picking SQL vs NoSQL, designing a schema and its
10	access paths, adding indexes, splitting a hot table, distributing data across
11	nodes (sharding/partitioning), adding read replicas, or deciding what to
12	denormalize. Reach here the moment "store the data" needs a concrete key and
13	query shape.
14
15	## When NOT to
16	Don't shard, add replicas, or reach for NoSQL before a number forces it (YAGNI).
17	A single well-indexed relational node handles ~1k QPS and tens of GB to low TB
18	comfortably — most systems never outgrow it. Sharding multiplies operational
19	cost and breaks joins/transactions; add it only when one node's write throughput
20	or dataset size is genuinely exceeded (→ `back-of-the-envelope`). Caching reads
21	(→ `caching`) and adding read replicas are cheaper first moves than sharding.
22
23	## Clarify first
24	Answer these before choosing a store or topology — they decide the design:
25
26	- Data shape & relationships — flat key-value? rich relations needing joins?
27	document blobs? a graph of connections? Drives SQL vs NoSQL.
28	- Access patterns — how is data read and written, not just where it lives.
29	Point lookups by key, range scans, ad-hoc queries, aggregations? Model the
30	store around the queries it must serve.
31	- Read:write ratio & scale — QPS each way, total size now and at retention.
32	(→ `back-of-the-envelope` for QPS, storage, and shard counts.)
33	- Consistency need — must reads see the latest write, or is eventual OK? Are
34	multi-record transactions required? (CAP/consistency theory → `consistency-coordination`.)
35	- Latency & durability targets — p99 read/write budget, and how much recent
36	data the system can afford to lose on a node failure.
37
38	## The options
39
40	Relational (SQL — Postgres, MySQL): strict schema, joins, ACID transactions.
41	Use when data is relational, integrity matters, and queries are varied/ad-hoc —
42	the safe default until a number rules it out.
43
44	Document (MongoDB, etc.): flexible schema, self-contained JSON-ish documents
45	queried by structure. Use when records are read/written as a whole and the
46	schema evolves; relationships are few.
47
48	Key-value (Redis, DynamoDB, Riak): O(1) get/put by key, no rich queries.
49	Use when access is purely by a known key and massive throughput is needed.
50
51	Wide-column (Cassandra, Bigtable, HBase): rows keyed by partition, columns
52	sparse, keys kept sorted for range scans. Use when writes are huge and the
53	table can be designed around a few known query patterns.
54
55	Graph (Neo4j): nodes and edges. Use when the core queries traverse
56	many-to-many relationships (social graph, recommendations).
57
58	Scaling moves (apply on top of any store):
59	- Indexing — add a secondary structure so a query stops scanning. First lever.
60	- Read replicas (leader-follower) — copy writes to followers that serve reads.
61	Use when reads dominate and slight staleness is OK.
62	- Federation — split DBs by function (users / products / forums). Use when
63	functional domains scale independently and rarely join.
64	- Sharding/partitioning — split one logical table across nodes by a shard key.
65	Use when a single node's writes or dataset are exceeded. (This skill owns it.)
66	- Denormalization — store redundant copies to skip joins. Use when reads
67	vastly outnumber writes and joins are the bottleneck.
68
69	Polyglot persistence: use more than one of the above, each for what it's best
70	at (e.g. Postgres for orders, Redis for sessions, a search index for full-text).
71	The cost is operating and reconciling several stores.
72
73	## Trade-offs
74
75	\| Option \| What it solves \| What it worsens \| Change it when \|
76	\|---\|---\|---\|---\|
77	\| Relational/SQL \| Joins, ACID, ad-hoc queries, integrity \| Single-node write ceiling; schema migrations; harder horizontal scale \| Writes/size exceed one node, or schema is truly fluid → NoSQL/shard \|
78	\| Document \| Schema flexibility; whole-object reads \| No joins; cross-document consistency is manual; query engine weaker \| Data turns relational or needs multi-doc transactions → SQL \|
79	\| Key-value \| Extreme throughput, simple ops \| Only key ac

People who liked this also viewed

6 picks

Type

data-storage

51installs1forks

View on GitHub

How to install data-storage?

proyecto26/system-design-skills/data-storage

$npx -y skills add proyecto26/system-design-skills --skill data-storage

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Data Storage
2
3	Choose where records live, how they are keyed and queried, and how the store
4	grows past a single machine. Storage is the hardest layer to change later: a
5	wrong data model or shard key calcifies into a scaling ceiling, and getting
6	replication wrong silently serves stale or lost data.
7
8	## When to reach for this
9	Any system that persists state: picking SQL vs NoSQL, designing a schema and its
10	access paths, adding indexes, splitting a hot table, distributing data across
11	nodes (sharding/partitioning), adding read replicas, or deciding what to
12	denormalize. Reach here the moment "store the data" needs a concrete key and
13	query shape.
14
15	## When NOT to
16	Don't shard, add replicas, or reach for NoSQL before a number forces it (YAGNI).
17	A single well-indexed relational node handles ~1k QPS and tens of GB to low TB
18	comfortably — most systems never outgrow it. Sharding multiplies operational
19	cost and breaks joins/transactions; add it only when one node's write throughput
20	or dataset size is genuinely exceeded (→ `back-of-the-envelope`). Caching reads
21	(→ `caching`) and adding read replicas are cheaper first moves than sharding.
22
23	## Clarify first
24	Answer these before choosing a store or topology — they decide the design:
25
26	- Data shape & relationships — flat key-value? rich relations needing joins?
27	document blobs? a graph of connections? Drives SQL vs NoSQL.
28	- Access patterns — how is data read and written, not just where it lives.
29	Point lookups by key, range scans, ad-hoc queries, aggregations? Model the
30	store around the queries it must serve.
31	- Read:write ratio & scale — QPS each way, total size now and at retention.
32	(→ `back-of-the-envelope` for QPS, storage, and shard counts.)
33	- Consistency need — must reads see the latest write, or is eventual OK? Are
34	multi-record transactions required? (CAP/consistency theory → `consistency-coordination`.)
35	- Latency & durability targets — p99 read/write budget, and how much recent
36	data the system can afford to lose on a node failure.
37
38	## The options
39
40	Relational (SQL — Postgres, MySQL): strict schema, joins, ACID transactions.
41	Use when data is relational, integrity matters, and queries are varied/ad-hoc —
42	the safe default until a number rules it out.
43
44	Document (MongoDB, etc.): flexible schema, self-contained JSON-ish documents
45	queried by structure. Use when records are read/written as a whole and the
46	schema evolves; relationships are few.
47
48	Key-value (Redis, DynamoDB, Riak): O(1) get/put by key, no rich queries.
49	Use when access is purely by a known key and massive throughput is needed.
50
51	Wide-column (Cassandra, Bigtable, HBase): rows keyed by partition, columns
52	sparse, keys kept sorted for range scans. Use when writes are huge and the
53	table can be designed around a few known query patterns.
54
55	Graph (Neo4j): nodes and edges. Use when the core queries traverse
56	many-to-many relationships (social graph, recommendations).
57
58	Scaling moves (apply on top of any store):
59	- Indexing — add a secondary structure so a query stops scanning. First lever.
60	- Read replicas (leader-follower) — copy writes to followers that serve reads.
61	Use when reads dominate and slight staleness is OK.
62	- Federation — split DBs by function (users / products / forums). Use when
63	functional domains scale independently and rarely join.
64	- Sharding/partitioning — split one logical table across nodes by a shard key.
65	Use when a single node's writes or dataset are exceeded. (This skill owns it.)
66	- Denormalization — store redundant copies to skip joins. Use when reads
67	vastly outnumber writes and joins are the bottleneck.
68
69	Polyglot persistence: use more than one of the above, each for what it's best
70	at (e.g. Postgres for orders, Redis for sessions, a search index for full-text).
71	The cost is operating and reconciling several stores.
72
73	## Trade-offs
74
75	\| Option \| What it solves \| What it worsens \| Change it when \|
76	\|---\|---\|---\|---\|
77	\| Relational/SQL \| Joins, ACID, ad-hoc queries, integrity \| Single-node write ceiling; schema migrations; harder horizontal scale \| Writes/size exceed one node, or schema is truly fluid → NoSQL/shard \|
78	\| Document \| Schema flexibility; whole-object reads \| No joins; cross-document consistency is manual; query engine weaker \| Data turns relational or needs multi-doc transactions → SQL \|
79	\| Key-value \| Extreme throughput, simple ops \| Only key ac

People who liked this also viewed

6 picks

Type