policyengine-data

Name: policyengine-data
Author: PolicyEngine

31stars5forks

How to install policyengine-data?

policyengine/policyengine-claude/policyengine-data

$npx -y skills add PolicyEngine/policyengine-claude --skill policyengine-data

Installs into the current project.

Summary

Load for PolicyEngine's data layer — how the microdata behind population microsimulations is built, calibrated, versioned, and named. Covers the current Populace stack (Frame kernel, populace-fit conditional models, populace-calibrate weights with L0 sparsity, build/release gates

Files · 1

View on GitHub

SKILL.md

1	# PolicyEngine data
2
3	How the microdata behind PolicyEngine population runs is built, calibrated, versioned, and
4	named. For using datasets in a simulation, see the `policyengine` skill (this skill is about
5	where the data comes from). For diagnosing why one score disagrees with a benchmark, see
6	`policyengine-calibration-diagnostics`.
7
8	The current data stack is Populace (repo `PolicyEngine/populace`, local mirror
9	`~/PolicyEngine/populace`; read its `README.md` + `DESIGN.md`). It replaced the technique-named
10	packages of the previous stack (microdf / microimpute / microcalibrate / L0 /
11	policyengine-us-data), which shared no datatype and had their worst bugs at the seams between
12	flat DataFrames.
13
14	## The Populace architecture
15
16	Populace is one kernel datatype — the `Frame` — with packages as operators on it. It is a
17	PEP 420 namespace (`populace.*`) shipped as independently-installable shard distributions, so an
18	analyst doing imputation never has to install torch and vice versa. A `populace` metapackage
19	pins the constellation.
20
21	\| Package \| Import \| Role \| Succeeds \|
22	\|---\|---\|---\|---\|
23	\| `populace-frame` \| `populace.frame` \| the kernel: `Frame`, typed weights, strata, links, weighted accounting, unit structure, the RulesEngine protocol \| microdf, microunit \|
24	\| `populace-fit` \| `populace.fit` \| conditional models (weight-aware by construction) \| ad-hoc imputation scripts \|
25	\| `populace-calibrate` \| `populace.calibrate` \| targets → calibrated weights (APG / L0) \| microcalibrate \|
26	\| `populace-build` \| `populace.build` \| build plans, donor graphs, release gates, country build stages \| one-off build drivers \|
27	\| `populace-data` \| `populace.data` \| published population registry + lazy engine loaders \| country-specific data packages \|
28
29	Key design facts (from `DESIGN.md`) that change how you reason about the data:
30
31	- The `Frame` is a weighted sampling frame of entity tables. Person + group-entity tables
32	with explicit `person_<group>_id` linkage established once at assembly — no operator
33	re-derives person↔unit attachment from a flat frame. It carries typed weights
34	(`design \| importance \| calibrated`, one vector per weighted entity) with conservation
35	invariants the kernel enforces (strata mass sums; no silent zeroing; no NaN/negative), and
36	strata giving every record explicit provenance (`cps_passthrough`,
37	`synthetic_conditional`, `tail_verbatim`, ...). Generation owns support (oversample where it
38	is scarce); calibration owns representation.
39	- The rules engine is an adapter, not a dependency. `populace.frame.rules.RulesEngine` is a
40	Protocol (`variable_entity`, `variable_dtype`, `entity_schema`, `materialize`,
41	`export_contract`, `write_dataset`). Today's adapter is `policyengine_us`; the Axiom
42	`rulespec-us` adapter is written against the same protocol so the swap is a new adapter, not a
43	migration.
44	- `populace-fit` is weight-aware by construction — fits read the frame's typed weights;
45	there is no unweighted default. Canonical model: regime-gated, chained quantile forests with
46	weights materialized by weighted bootstrap.
47	- `populace-calibrate` is the only place calibrated weights are produced. Sparse
48	target-matrix compilation + APG / L0 pruning is the core, not an option — "generate big then
49	prune" is the intended design (300k → 3M → 30M candidate pools pruned to a compact frame). Its
50	longitudinal rule: one weight per trajectory (multi-period targets stack as
51	`(target, period)` constraint rows over one weight vector).
52	- Process rules are as binding as the architecture: behavioral contract tests in CI from day
53	one (weighted fits shift draws toward the weighted truth; calibration conserves declared mass;
54	unit assignment partitions exactly); constellation versioning (consumers pin the constellation,
55	not git SHAs); artifacts embed a certificate of the rules-engine + package versio

People who liked this also viewed

6 picks

Type

policyengine-data

31stars5forks

View on GitHub

How to install policyengine-data?

policyengine/policyengine-claude/policyengine-data

$npx -y skills add PolicyEngine/policyengine-claude --skill policyengine-data

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# PolicyEngine data
2
3	How the microdata behind PolicyEngine population runs is built, calibrated, versioned, and
4	named. For using datasets in a simulation, see the `policyengine` skill (this skill is about
5	where the data comes from). For diagnosing why one score disagrees with a benchmark, see
6	`policyengine-calibration-diagnostics`.
7
8	The current data stack is Populace (repo `PolicyEngine/populace`, local mirror
9	`~/PolicyEngine/populace`; read its `README.md` + `DESIGN.md`). It replaced the technique-named
10	packages of the previous stack (microdf / microimpute / microcalibrate / L0 /
11	policyengine-us-data), which shared no datatype and had their worst bugs at the seams between
12	flat DataFrames.
13
14	## The Populace architecture
15
16	Populace is one kernel datatype — the `Frame` — with packages as operators on it. It is a
17	PEP 420 namespace (`populace.*`) shipped as independently-installable shard distributions, so an
18	analyst doing imputation never has to install torch and vice versa. A `populace` metapackage
19	pins the constellation.
20
21	\| Package \| Import \| Role \| Succeeds \|
22	\|---\|---\|---\|---\|
23	\| `populace-frame` \| `populace.frame` \| the kernel: `Frame`, typed weights, strata, links, weighted accounting, unit structure, the RulesEngine protocol \| microdf, microunit \|
24	\| `populace-fit` \| `populace.fit` \| conditional models (weight-aware by construction) \| ad-hoc imputation scripts \|
25	\| `populace-calibrate` \| `populace.calibrate` \| targets → calibrated weights (APG / L0) \| microcalibrate \|
26	\| `populace-build` \| `populace.build` \| build plans, donor graphs, release gates, country build stages \| one-off build drivers \|
27	\| `populace-data` \| `populace.data` \| published population registry + lazy engine loaders \| country-specific data packages \|
28
29	Key design facts (from `DESIGN.md`) that change how you reason about the data:
30
31	- The `Frame` is a weighted sampling frame of entity tables. Person + group-entity tables
32	with explicit `person_<group>_id` linkage established once at assembly — no operator
33	re-derives person↔unit attachment from a flat frame. It carries typed weights
34	(`design \| importance \| calibrated`, one vector per weighted entity) with conservation
35	invariants the kernel enforces (strata mass sums; no silent zeroing; no NaN/negative), and
36	strata giving every record explicit provenance (`cps_passthrough`,
37	`synthetic_conditional`, `tail_verbatim`, ...). Generation owns support (oversample where it
38	is scarce); calibration owns representation.
39	- The rules engine is an adapter, not a dependency. `populace.frame.rules.RulesEngine` is a
40	Protocol (`variable_entity`, `variable_dtype`, `entity_schema`, `materialize`,
41	`export_contract`, `write_dataset`). Today's adapter is `policyengine_us`; the Axiom
42	`rulespec-us` adapter is written against the same protocol so the swap is a new adapter, not a
43	migration.
44	- `populace-fit` is weight-aware by construction — fits read the frame's typed weights;
45	there is no unweighted default. Canonical model: regime-gated, chained quantile forests with
46	weights materialized by weighted bootstrap.
47	- `populace-calibrate` is the only place calibrated weights are produced. Sparse
48	target-matrix compilation + APG / L0 pruning is the core, not an option — "generate big then
49	prune" is the intended design (300k → 3M → 30M candidate pools pruned to a compact frame). Its
50	longitudinal rule: one weight per trajectory (multi-period targets stack as
51	`(target, period)` constraint rows over one weight vector).
52	- Process rules are as binding as the architecture: behavioral contract tests in CI from day
53	one (weighted fits shift draws toward the weighted truth; calibration conserves declared mass;
54	unit assignment partitions exactly); constellation versioning (consumers pin the constellation,
55	not git SHAs); artifacts embed a certificate of the rules-engine + package versio

People who liked this also viewed

6 picks

Type