sharded-counters

Name: sharded-counters
Author: proyecto26

44installs1forks

How to install sharded-counters?

proyecto26/system-design-skills/sharded-counters

$npx -y skills add proyecto26/system-design-skills --skill sharded-counters

Installs into the current project.

Summary

This skill should be used when the user needs a "sharded counter", "distributed counter", to "count likes / views at scale", handles a "high-write counter" or "hot counter contention", asks about "approximate counting", "real-time counts", or "HyperLogLog". It gives the recipe fo

Files · 1

View on GitHub

SKILL.md

1	# Sharded counters
2
3	Count a thing that is incremented far faster than a single row, key, or
4	partition can serialize writes — likes, views, votes, rate tallies, inventory
5	decrements. The trap is the hot counter: every writer contends on one
6	record, so latency climbs and throughput plateaus no matter how big the box is.
7	Getting it wrong turns a trivial `+1` into the bottleneck of the whole feature.
8
9	## When to reach for this
10	Concurrent increments to a single logical count exceed what one row/key can
11	absorb — a viral post's like count, a live-event view counter, a global
12	rate tally. The symptom is write contention (lock waits, CAS retries, partition
13	hot-spotting) on one record while the rest of the store is idle. Reaching for
14	this means the write side is the problem, and an exact-to-the-millisecond
15	total is not required.
16
17	## When NOT to
18	Low write rate (a single atomic `INCR` handles thousands/sec — don't shard a
19	counter nobody is hammering; YAGNI). Counts that must be transactionally exact
20	and read-after-write consistent at every instant (bank balances, seat
21	inventory at sell-out) — that's a transactional decrement, see
22	`consistency-coordination`, not a fan-out tally. Counting distinct items
23	exactly (unique visitors) where you also need the member list — that's a set in
24	the store, not a counter. If reads dominate and writes are cheap, you need a
25	cached aggregate, not sharding.
26
27	## Clarify first
28	- Write rate to the hottest single count — peak increments/sec on one
29	logical counter, not the aggregate (→ `back-of-the-envelope`).
30	- Exact or approximate — is an off-by-a-few total acceptable, and for how
31	long may shards disagree (eventual)? Drives shard count and read path.
32	- Counting occurrences or distinct items — a running total vs. unique-count
33	(likes vs. unique viewers) decides plain shards vs. HyperLogLog.
34	- Read rate and freshness — how often is the total read, and how stale may
35	the served number be (sub-second? minutes?).
36	- Time-windowed or lifetime — "views in the last hour" needs bucketed keys
37	and expiry; a lifetime total does not.
38
39	## The options
40	- Single atomic counter — one row/key with atomic `INCR`/`UPDATE +1`. Use
41	when peak write rate on the hottest count is well within one node's serialized
42	write throughput. The default; don't outgrow it prematurely.
43	- Write-sharded (striped) counter — split one logical count into N physical
44	shards (`counter:{id}:shard:{0..N-1}`); each write increments a random/hashed
45	shard, reads sum all N. Use when single-key contention is the bottleneck
46	and the total may be eventually consistent.
47	- Approximate distinct count (HyperLogLog) — a fixed-size probabilistic
48	sketch (~12 KB) that counts unique items with ~2% error. Use for uniques at
49	scale where exact membership isn't needed (unique visitors, distinct search
50	terms).
51	- Time-windowed (bucketed) counters — key the counter by time bucket
52	(`views:{id}:2026-06-01T14`), increment the current bucket, sum recent buckets
53	on read, expire old ones. Use for "last N minutes/hours" rate-style counts.
54	- Aggregate-on-read + cached total — sum shards (or roll up) periodically and
55	serve the cached number. Use when reads vastly outnumber writes and a slightly
56	stale total is fine (pairs with `caching`).
57
58	## Trade-offs
59
60	\| Option \| What it solves \| What it worsens \| Change it when \|
61	\|---\|---\|---\|---\|
62	\| Single atomic counter \| Simplest; exact; read-after-write trivial \| One hot record caps write throughput; contention under spikes \| Increments on one count exceed one node → shard the writes \|
63	\| Write-sharded counter \| Spreads write load N-way; removes the hot spot \| Reads cost N lookups + sum; total is eventually consistent; pick N up front \| Read cost of summing N grows painful → cache the aggregate / roll up \|
64	\| HyperLogLog \| Counts uniques in fixed tiny memory at huge scale \| ~2% error; can't list members or do exact counts \| Exact uniques or the member set is required → use a stored set \|
65	\| Time-windowed buckets \| Cheap rolling/rate counts; old data self-expires \| More keys; window boundaries need care; cross-bucket reads sum many keys \| You need an exact lifetime total → keep a separate lifetime counter \|
66	\| Aggregate-on-read + cache \| Cheap reads of a heavy-write count \| Served total lags writes by the refresh interval \| Reads must be fresh-to-the-write → read shards live (eat the N-sum) \|
67
68	## Behavior under stress
69	A counter is a tiny thing

People who liked this also viewed

6 picks

Type

sharded-counters

44installs1forks

View on GitHub

How to install sharded-counters?

proyecto26/system-design-skills/sharded-counters

$npx -y skills add proyecto26/system-design-skills --skill sharded-counters

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Sharded counters
2
3	Count a thing that is incremented far faster than a single row, key, or
4	partition can serialize writes — likes, views, votes, rate tallies, inventory
5	decrements. The trap is the hot counter: every writer contends on one
6	record, so latency climbs and throughput plateaus no matter how big the box is.
7	Getting it wrong turns a trivial `+1` into the bottleneck of the whole feature.
8
9	## When to reach for this
10	Concurrent increments to a single logical count exceed what one row/key can
11	absorb — a viral post's like count, a live-event view counter, a global
12	rate tally. The symptom is write contention (lock waits, CAS retries, partition
13	hot-spotting) on one record while the rest of the store is idle. Reaching for
14	this means the write side is the problem, and an exact-to-the-millisecond
15	total is not required.
16
17	## When NOT to
18	Low write rate (a single atomic `INCR` handles thousands/sec — don't shard a
19	counter nobody is hammering; YAGNI). Counts that must be transactionally exact
20	and read-after-write consistent at every instant (bank balances, seat
21	inventory at sell-out) — that's a transactional decrement, see
22	`consistency-coordination`, not a fan-out tally. Counting distinct items
23	exactly (unique visitors) where you also need the member list — that's a set in
24	the store, not a counter. If reads dominate and writes are cheap, you need a
25	cached aggregate, not sharding.
26
27	## Clarify first
28	- Write rate to the hottest single count — peak increments/sec on one
29	logical counter, not the aggregate (→ `back-of-the-envelope`).
30	- Exact or approximate — is an off-by-a-few total acceptable, and for how
31	long may shards disagree (eventual)? Drives shard count and read path.
32	- Counting occurrences or distinct items — a running total vs. unique-count
33	(likes vs. unique viewers) decides plain shards vs. HyperLogLog.
34	- Read rate and freshness — how often is the total read, and how stale may
35	the served number be (sub-second? minutes?).
36	- Time-windowed or lifetime — "views in the last hour" needs bucketed keys
37	and expiry; a lifetime total does not.
38
39	## The options
40	- Single atomic counter — one row/key with atomic `INCR`/`UPDATE +1`. Use
41	when peak write rate on the hottest count is well within one node's serialized
42	write throughput. The default; don't outgrow it prematurely.
43	- Write-sharded (striped) counter — split one logical count into N physical
44	shards (`counter:{id}:shard:{0..N-1}`); each write increments a random/hashed
45	shard, reads sum all N. Use when single-key contention is the bottleneck
46	and the total may be eventually consistent.
47	- Approximate distinct count (HyperLogLog) — a fixed-size probabilistic
48	sketch (~12 KB) that counts unique items with ~2% error. Use for uniques at
49	scale where exact membership isn't needed (unique visitors, distinct search
50	terms).
51	- Time-windowed (bucketed) counters — key the counter by time bucket
52	(`views:{id}:2026-06-01T14`), increment the current bucket, sum recent buckets
53	on read, expire old ones. Use for "last N minutes/hours" rate-style counts.
54	- Aggregate-on-read + cached total — sum shards (or roll up) periodically and
55	serve the cached number. Use when reads vastly outnumber writes and a slightly
56	stale total is fine (pairs with `caching`).
57
58	## Trade-offs
59
60	\| Option \| What it solves \| What it worsens \| Change it when \|
61	\|---\|---\|---\|---\|
62	\| Single atomic counter \| Simplest; exact; read-after-write trivial \| One hot record caps write throughput; contention under spikes \| Increments on one count exceed one node → shard the writes \|
63	\| Write-sharded counter \| Spreads write load N-way; removes the hot spot \| Reads cost N lookups + sum; total is eventually consistent; pick N up front \| Read cost of summing N grows painful → cache the aggregate / roll up \|
64	\| HyperLogLog \| Counts uniques in fixed tiny memory at huge scale \| ~2% error; can't list members or do exact counts \| Exact uniques or the member set is required → use a stored set \|
65	\| Time-windowed buckets \| Cheap rolling/rate counts; old data self-expires \| More keys; window boundaries need care; cross-bucket reads sum many keys \| You need an exact lifetime total → keep a separate lifetime counter \|
66	\| Aggregate-on-read + cache \| Cheap reads of a heavy-write count \| Served total lags writes by the refresh interval \| Reads must be fresh-to-the-write → read shards live (eat the N-sum) \|
67
68	## Behavior under stress
69	A counter is a tiny thing

People who liked this also viewed

6 picks

Type