blob-store

Name: blob-store
Author: proyecto26

45installs1forks

How to install blob-store?

proyecto26/system-design-skills/blob-store

$npx -y skills add proyecto26/system-design-skills --skill blob-store

Installs into the current project.

Summary

This skill should be used when the user wants a "blob store" or "object storage", names "S3" or an S3-compatible store, needs to "store images / video / files", asks about "multipart upload" or "resumable upload", "signed / presigned URLs", "media storage", "unstructured data at

Files · 1

View on GitHub

SKILL.md

1	# Blob store
2
3	Store large, immutable, unstructured objects — images, video, backups, model
4	weights, document blobs — in a flat namespace keyed by a string, replicated for
5	durability and served by direct download. Getting it wrong means stuffing
6	multi-megabyte blobs into a row-oriented database (where they bloat the working
7	set, wreck cache locality, and cap throughput) or hand-rolling a file server that
8	loses data on the first disk failure.
9
10	## When to reach for this
11	Objects are large (KB to GB), written once and read many times, and you only ever
12	fetch them whole by key — never query inside them. Photo/video stores, user
13	uploads, backups, data-lake/ML datasets, static-site assets, log archives. The
14	access pattern is `PUT key → GET key`, durability matters, and the total volume is
15	too large or too cold to sit in a primary database.
16
17	## When NOT to
18	Small structured records you query, filter, sort, or join — that is `data-storage`.
19	Data that needs transactions, secondary indexes, or partial updates (blobs are
20	replace-whole, not edit-in-place). Low-latency reads of tiny values (a KV cache or
21	`caching` wins). A few files on one box that never grow — the local filesystem is
22	fine; a blob store is operational overhead you do not need yet (YAGNI). Naming
23	"object storage" for a workload that is really a database is failure mode #2.
24
25	## Clarify first
26	- Object size distribution — average and p99 size? (Decides chunking, multipart
27	thresholds, and whether reads stream or buffer.) → `back-of-the-envelope`.
28	- Read:write ratio and access recency — write-once/read-many? How fast does data
29	go cold? (Drives tiering and CDN fronting.)
30	- Durability and availability target — how many nines of durability? Can a read
31	briefly fail or must it always succeed? (Replication vs erasure coding, multi-region.)
32	- Access control — public, private, or time-limited per-object grants? (Signed URLs.)
33	- Mutability and history — do objects change? Must old versions be retained
34	(compliance, undo)? (Versioning + lifecycle.)
35	- Egress profile — who reads, from where, how often? (CDN offload, egress cost.)
36
37	## The options
38	Durability scheme (how many copies, what shape)
39	- N-way replication — store N full copies on different nodes/racks/AZs. Use when
40	objects are small, hot, and latency matters; simplest to reason about.
41	- Erasure coding (EC) — split an object into k data + m parity shards; any k
42	reconstruct it. Use for large/cold data at scale — same durability as replication
43	at ~1.4x overhead instead of 3x. (Mechanics in `references/deep-dive.md`.)
44
45	Storage tier (price/latency/retrieval trade)
46	- Hot/standard — millisecond reads, highest $/GB. Use for actively served objects.
47	- Cool/infrequent — cheaper storage, retrieval fee/slightly higher latency. Use
48	for backups and data read a few times a month.
49	- Archive/cold — cheapest storage, minutes-to-hours retrieval. Use for compliance
50	retention and rarely-touched data; never for anything on a request path.
51
52	Upload path
53	- Single PUT — one request. Use for small objects (under the multipart threshold).
54	- Multipart / resumable — split into parts, upload in parallel, retry per-part,
55	commit on completion. Use for large objects and flaky networks; the default above
56	the threshold.
57
58	Mutation model
59	- Immutable + versioning — each write is a new version; deletes are tombstones.
60	Use when history, undo, or accidental-overwrite protection matters.
61	- Overwrite-in-place (last-writer-wins) — simplest; no history. Use when only the
62	latest object matters and storage of old copies is waste.
63
64	## Trade-offs
65
66	\| Option \| What it solves \| What it worsens \| Change it when \|
67	\|---\|---\|---\|---\|
68	\| N-way replication \| Simple, fast reads, fast rebuild \| 3x+ storage cost \| Data is large/cold and cost dominates → erasure coding \|
69	\| Erasure coding \| Same durability at ~1.4x storage \| CPU + multi-node read on every fetch; slow small-object reads; costly rebuild \| Objects are small/hot and latency matters → replication \|
70	\| Hot tier \| Low-latency serving \| Highest $/GB \| Data goes cold and is rarely read → cool/archive \|
71	\| Archive tier \| Cheapest at-rest storage \| Minutes–hours to first byte; retrieval fees \| Anything ends up on a latency-sensitive path → hot/cool \|
72	\| Multipart/resumable upload \| Large files surv

People who liked this also viewed

6 picks

Type

blob-store

45installs1forks

View on GitHub

How to install blob-store?

proyecto26/system-design-skills/blob-store

$npx -y skills add proyecto26/system-design-skills --skill blob-store

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Blob store
2
3	Store large, immutable, unstructured objects — images, video, backups, model
4	weights, document blobs — in a flat namespace keyed by a string, replicated for
5	durability and served by direct download. Getting it wrong means stuffing
6	multi-megabyte blobs into a row-oriented database (where they bloat the working
7	set, wreck cache locality, and cap throughput) or hand-rolling a file server that
8	loses data on the first disk failure.
9
10	## When to reach for this
11	Objects are large (KB to GB), written once and read many times, and you only ever
12	fetch them whole by key — never query inside them. Photo/video stores, user
13	uploads, backups, data-lake/ML datasets, static-site assets, log archives. The
14	access pattern is `PUT key → GET key`, durability matters, and the total volume is
15	too large or too cold to sit in a primary database.
16
17	## When NOT to
18	Small structured records you query, filter, sort, or join — that is `data-storage`.
19	Data that needs transactions, secondary indexes, or partial updates (blobs are
20	replace-whole, not edit-in-place). Low-latency reads of tiny values (a KV cache or
21	`caching` wins). A few files on one box that never grow — the local filesystem is
22	fine; a blob store is operational overhead you do not need yet (YAGNI). Naming
23	"object storage" for a workload that is really a database is failure mode #2.
24
25	## Clarify first
26	- Object size distribution — average and p99 size? (Decides chunking, multipart
27	thresholds, and whether reads stream or buffer.) → `back-of-the-envelope`.
28	- Read:write ratio and access recency — write-once/read-many? How fast does data
29	go cold? (Drives tiering and CDN fronting.)
30	- Durability and availability target — how many nines of durability? Can a read
31	briefly fail or must it always succeed? (Replication vs erasure coding, multi-region.)
32	- Access control — public, private, or time-limited per-object grants? (Signed URLs.)
33	- Mutability and history — do objects change? Must old versions be retained
34	(compliance, undo)? (Versioning + lifecycle.)
35	- Egress profile — who reads, from where, how often? (CDN offload, egress cost.)
36
37	## The options
38	Durability scheme (how many copies, what shape)
39	- N-way replication — store N full copies on different nodes/racks/AZs. Use when
40	objects are small, hot, and latency matters; simplest to reason about.
41	- Erasure coding (EC) — split an object into k data + m parity shards; any k
42	reconstruct it. Use for large/cold data at scale — same durability as replication
43	at ~1.4x overhead instead of 3x. (Mechanics in `references/deep-dive.md`.)
44
45	Storage tier (price/latency/retrieval trade)
46	- Hot/standard — millisecond reads, highest $/GB. Use for actively served objects.
47	- Cool/infrequent — cheaper storage, retrieval fee/slightly higher latency. Use
48	for backups and data read a few times a month.
49	- Archive/cold — cheapest storage, minutes-to-hours retrieval. Use for compliance
50	retention and rarely-touched data; never for anything on a request path.
51
52	Upload path
53	- Single PUT — one request. Use for small objects (under the multipart threshold).
54	- Multipart / resumable — split into parts, upload in parallel, retry per-part,
55	commit on completion. Use for large objects and flaky networks; the default above
56	the threshold.
57
58	Mutation model
59	- Immutable + versioning — each write is a new version; deletes are tombstones.
60	Use when history, undo, or accidental-overwrite protection matters.
61	- Overwrite-in-place (last-writer-wins) — simplest; no history. Use when only the
62	latest object matters and storage of old copies is waste.
63
64	## Trade-offs
65
66	\| Option \| What it solves \| What it worsens \| Change it when \|
67	\|---\|---\|---\|---\|
68	\| N-way replication \| Simple, fast reads, fast rebuild \| 3x+ storage cost \| Data is large/cold and cost dominates → erasure coding \|
69	\| Erasure coding \| Same durability at ~1.4x storage \| CPU + multi-node read on every fetch; slow small-object reads; costly rebuild \| Objects are small/hot and latency matters → replication \|
70	\| Hot tier \| Low-latency serving \| Highest $/GB \| Data goes cold and is rarely read → cool/archive \|
71	\| Archive tier \| Cheapest at-rest storage \| Minutes–hours to first byte; retrieval fees \| Anything ends up on a latency-sensitive path → hot/cool \|
72	\| Multipart/resumable upload \| Large files surv

People who liked this also viewed

6 picks

Type