task-scheduling

Name: task-scheduling
Author: proyecto26

47installs1forks

How to install task-scheduling?

proyecto26/system-design-skills/task-scheduling

$npx -y skills add proyecto26/system-design-skills --skill task-scheduling

Installs into the current project.

Summary

This skill should be used when the user designs a "task scheduler", "job scheduler", "job queue", "cron at scale", "distributed cron", "delayed / scheduled / recurring tasks", a "worker pool", reaches for "Celery / Sidekiq / Airflow", or wrestles with "task leasing", visibility t

Files · 1

View on GitHub

SKILL.md

1	# Task Scheduling
2
3	Decide when work runs and which worker runs it: fire jobs on a schedule
4	(cron/delayed/recurring), hand each job to exactly one worker via a lease, and
5	make sure it completes once despite crashes and retries. This sits on top of
6	`messaging-streaming` queues — the queue is the transport; this skill adds the
7	scheduling, leasing, priorities, and task-level idempotency. Getting it wrong
8	shows up as jobs that never run, run twice (double charge, double email), or
9	pile up until a worker fleet falls permanently behind.
10
11	## When to reach for this
12	Work must run later (send a reminder in 24h), on a schedule (nightly
13	rollups, hourly cron), or repeatedly (poll every 5 min); a slow operation is
14	already off the request path (→ `messaging-streaming`) and now needs reliable
15	allocation to a pool of workers; jobs need priorities (paid before free) or
16	fairness (no single tenant starves others); or a job must complete **exactly
17	once** even though the worker holding it can crash mid-flight.
18
19	## When NOT to
20	The caller needs the result inline — that's a synchronous call, not a scheduled
21	job. A single fire-and-forget async step with no schedule, priority, or
22	exactly-once need — a plain queue + idempotent consumer (`messaging-streaming`)
23	is simpler; don't add a scheduler on top. One periodic job on one box —
24	OS `cron` is fine until you have multiple schedulers or need history and
25	retries. A long-running multi-step saga with rollback — reach for a durable
26	workflow engine instead of hand-rolling state across jobs. Don't stand up
27	Airflow/Celery "because we'll have batch jobs eventually" (YAGNI): it's a
28	stateful control plane to operate and monitor.
29
30	## Clarify first
31	- Trigger type — scheduled (cron/at a time), delayed (run after N seconds),
32	recurring (every N), or event-driven (a queue message arrives)? This decides
33	whether a scheduler is even in scope.
34	- Exactly-once vs at-least-once — is a duplicate run harmful (money, email)
35	or harmless (idempotent recompute)? Drives the leasing + dedup design.
36	- Latency budget vs throughput — must a delayed job fire within seconds of
37	its time, or is "within a few minutes" fine? Tight timing is more expensive.
38	- Priority / fairness — do some jobs jump the line, and must one tenant or
39	job class be prevented from starving the rest? (→ `back-of-the-envelope` for
40	arrival vs. service rate.)
41	- Job duration & variance — seconds or hours? Sets the visibility-timeout /
42	lease length and whether long jobs need heartbeats.
43	- Idempotency key — what identifies a task as the same task on retry?
44	(The key contract is owned by `api-design`.)
45
46	## The options
47
48	Scheduling trigger
49	- OS cron / single scheduler — one process fires jobs on a crontab. *Use
50	when* one node, a handful of jobs, no HA requirement.
51	- Distributed scheduler (HA cron) — a leader-elected scheduler enqueues due
52	jobs into a queue; followers stand by. Use when the schedule must survive a
53	node loss and must not double-fire.
54	- Delay queue / timer — jobs carry a "not before" time; the queue holds them
55	until due (delivery delay, sorted-set scoring, or a timer wheel). Use when
56	per-job delays vary and you don't want a cron tick.
57	- Workflow/orchestration DAG — declared task dependencies with backfill and
58	history (Airflow-style). Use when batch pipelines have dependencies and you
59	need a run history and reruns.
60
61	Worker allocation
62	- Pull (worker leasing) — workers poll the queue, lease a job for a
63	visibility timeout, and ack/delete on success. Use when you want
64	back-pressure for free and elastic, self-balancing workers. The default.
65	- Push (dispatcher assigns) — a coordinator routes jobs to specific workers.
66	Use when affinity/locality matters (a job must run where its data is).
67
68	Priority & fairness
69	- Priority queues — separate high/low queues drained in order. Use when
70	some classes must run first.
71	- Weighted / fair scheduling — round-robin or weighted draw across per-tenant
72	queues. Use when one tenant's burst must not starve others.
73
74	## Trade-offs
75
76	\| Option \| What it solves \| What it worsens \| Change it when \|
77	\|---\|---\|---\|---\|
78	\| OS cron / single scheduler \| Trivial; zero infra \| SPOF — node dies, schedule stops; no retry/history \| You need HA or missed-run recovery → distributed scheduler \|
79	\| Dist

People who liked this also viewed

6 picks

Type

task-scheduling

47installs1forks

View on GitHub

How to install task-scheduling?

proyecto26/system-design-skills/task-scheduling

$npx -y skills add proyecto26/system-design-skills --skill task-scheduling

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Task Scheduling
2
3	Decide when work runs and which worker runs it: fire jobs on a schedule
4	(cron/delayed/recurring), hand each job to exactly one worker via a lease, and
5	make sure it completes once despite crashes and retries. This sits on top of
6	`messaging-streaming` queues — the queue is the transport; this skill adds the
7	scheduling, leasing, priorities, and task-level idempotency. Getting it wrong
8	shows up as jobs that never run, run twice (double charge, double email), or
9	pile up until a worker fleet falls permanently behind.
10
11	## When to reach for this
12	Work must run later (send a reminder in 24h), on a schedule (nightly
13	rollups, hourly cron), or repeatedly (poll every 5 min); a slow operation is
14	already off the request path (→ `messaging-streaming`) and now needs reliable
15	allocation to a pool of workers; jobs need priorities (paid before free) or
16	fairness (no single tenant starves others); or a job must complete **exactly
17	once** even though the worker holding it can crash mid-flight.
18
19	## When NOT to
20	The caller needs the result inline — that's a synchronous call, not a scheduled
21	job. A single fire-and-forget async step with no schedule, priority, or
22	exactly-once need — a plain queue + idempotent consumer (`messaging-streaming`)
23	is simpler; don't add a scheduler on top. One periodic job on one box —
24	OS `cron` is fine until you have multiple schedulers or need history and
25	retries. A long-running multi-step saga with rollback — reach for a durable
26	workflow engine instead of hand-rolling state across jobs. Don't stand up
27	Airflow/Celery "because we'll have batch jobs eventually" (YAGNI): it's a
28	stateful control plane to operate and monitor.
29
30	## Clarify first
31	- Trigger type — scheduled (cron/at a time), delayed (run after N seconds),
32	recurring (every N), or event-driven (a queue message arrives)? This decides
33	whether a scheduler is even in scope.
34	- Exactly-once vs at-least-once — is a duplicate run harmful (money, email)
35	or harmless (idempotent recompute)? Drives the leasing + dedup design.
36	- Latency budget vs throughput — must a delayed job fire within seconds of
37	its time, or is "within a few minutes" fine? Tight timing is more expensive.
38	- Priority / fairness — do some jobs jump the line, and must one tenant or
39	job class be prevented from starving the rest? (→ `back-of-the-envelope` for
40	arrival vs. service rate.)
41	- Job duration & variance — seconds or hours? Sets the visibility-timeout /
42	lease length and whether long jobs need heartbeats.
43	- Idempotency key — what identifies a task as the same task on retry?
44	(The key contract is owned by `api-design`.)
45
46	## The options
47
48	Scheduling trigger
49	- OS cron / single scheduler — one process fires jobs on a crontab. *Use
50	when* one node, a handful of jobs, no HA requirement.
51	- Distributed scheduler (HA cron) — a leader-elected scheduler enqueues due
52	jobs into a queue; followers stand by. Use when the schedule must survive a
53	node loss and must not double-fire.
54	- Delay queue / timer — jobs carry a "not before" time; the queue holds them
55	until due (delivery delay, sorted-set scoring, or a timer wheel). Use when
56	per-job delays vary and you don't want a cron tick.
57	- Workflow/orchestration DAG — declared task dependencies with backfill and
58	history (Airflow-style). Use when batch pipelines have dependencies and you
59	need a run history and reruns.
60
61	Worker allocation
62	- Pull (worker leasing) — workers poll the queue, lease a job for a
63	visibility timeout, and ack/delete on success. Use when you want
64	back-pressure for free and elastic, self-balancing workers. The default.
65	- Push (dispatcher assigns) — a coordinator routes jobs to specific workers.
66	Use when affinity/locality matters (a job must run where its data is).
67
68	Priority & fairness
69	- Priority queues — separate high/low queues drained in order. Use when
70	some classes must run first.
71	- Weighted / fair scheduling — round-robin or weighted draw across per-tenant
72	queues. Use when one tenant's burst must not starve others.
73
74	## Trade-offs
75
76	\| Option \| What it solves \| What it worsens \| Change it when \|
77	\|---\|---\|---\|---\|
78	\| OS cron / single scheduler \| Trivial; zero infra \| SPOF — node dies, schedule stops; no retry/history \| You need HA or missed-run recovery → distributed scheduler \|
79	\| Dist

People who liked this also viewed

6 picks

Type