policyengine-calibration-diagnostics

Name: policyengine-calibration-diagnostics
Author: PolicyEngine

31stars5forks

How to install policyengine-calibration-diagnostics?

policyengine/policyengine-claude/policyengine-calibration-diagnostics

$npx -y skills add PolicyEngine/policyengine-claude --skill policyengine-calibration-diagnostics

Installs into the current project.

Summary

Sensitivity registry for PolicyEngine microsim results — maps {program x deviation signature} to the calibration target or imputed variable most likely driving a mismatch, and reads the live per-target diagnostics for the current Populace release. The knowledge base behind the ca

Files · 1

View on GitHub

SKILL.md

1	# PolicyEngine calibration diagnostics
2
3	Converts tribal "I'd check the takeup rate first" knowledge into a structured sensitivity
4	registry, and pairs it with the live per-target calibration API so hypotheses are ranked against
5	real `relative_error` numbers rather than assumptions. When an `/analyze-policy` comparison
6	returns INVESTIGATE, this skill supplies the ranked candidate causes.
7
8	The calibrated microdata is now a Populace build (see the `policyengine-data` skill for how
9	targets, weights, and L0 sparsity work). Calibration targets live in the populace build's target
10	set — not in a hand-maintained loss file — and their fit is queryable per release from the
11	dashboard API below.
12
13	## When to use
14
15	- Stage 5.6 / Stage 6 of `/analyze-policy` — invoked by the `calibration-diagnostics` agent.
16	- Code review of microsim PRs where the headline number differs from priors.
17	- Designing or auditing a populace calibration target.
18	- Debugging why a state-level run looks volatile.
19
20	## Top-level architecture
21
22	PolicyEngine microsim results depend on three layers:
23
24	1. Country model logic (policyengine-us, policyengine-uk, policyengine-canada) — formulas,
25	parameters.
26	2. Calibrated microdata (Populace) — survey weights + imputations matched to administrative
27	targets.
28	3. Behavioral assumptions (takeup rates, labor-supply elasticities) — usually parameters but
29	easy to overlook.
30
31	A magnitude mismatch is almost always rooted in layer 2 or 3, not layer 1 (layer-1 mismatches
32	show up as outright simulation errors, not magnitude drift).
33
34	## Live calibration API (check this first)
35
36	Per-target fit for the current populace release, no auth, reads the release from Hugging Face:
37
38	```
39	BASE = https://calibration-diagnostics.vercel.app/calibration/dashboard/api/populace
40	GET {BASE}/target-diagnostics?source=<source> # every target for a source, with relative_error
41	GET {BASE}/target-investigation?target=<id> # full investigation packet for one target
42	GET {BASE}/releases # release ids for pinning
43	```
44
45	Sources map to reform domains: Social Security → `ssa`; Medicaid/ACA/Medicare →
46	`cms_medicaid` / `cms_aca` / `cms_medicare`; TANF → `hhs_acf_tanf`; income tax / credits →
47	`irs_soi`, `jct`, `cbo`; state income tax → `state_income_*`. A target already >10% off in the
48	release diagnostics is a stronger hypothesis than any prior. (Dashboard UI:
49	`calibration-diagnostics.vercel.app`.)
50
51	### The three-ring reading method (align with /analyze-policy Stage 5.6)
52
53	Checking only the reform's primary marginal is the classic miss — a reform's cost usually depends
54	on the joint distribution of its variable with other income, and the primary marginal can
55	calibrate perfectly while the interaction is off. Check three rings outward:
56
57	1. Ring 1 — the primary variable's marginals. Map the reform's domain to a calibration
58	source and fetch its targets.
59	2. Ring 2 — the mechanism's other inputs. Every variable that enters the formula the reform
60	changes. SS-benefit taxation depends on combined income, so dividends, taxable interest, and
61	pensions (`irs_soi`) are load-bearing; a CTC phase-in depends on earnings; a SNAP change on
62	rents and deductions.
63	3. Ring 3 — the interacted quantity itself (best single check). Search for the downstream
64	quantity the reform directly reprices — `taxable_social_security_amount` for SS-taxation,
65	`eitc_amount` for EITC, itemizer counts for deduction caps. When such a target exists it
66	validates the joint distribution end-to-end through the actual formula, and its
67	`relative_error` bounds the data-side bias of the score. Weight it above every marginal. When
68	no ring-3 target exists, say so — "marginals within tolerance; joint distribution untargeted"
69	is materially weaker evidence.
70
71	Worked example (HR 904, SS-taxation reform): `ssa` total benefits calibrate to −0.2% (ring 1 fine)
72	while `irs_soi ... taxable_social_security_amount` runs −10.9% (ring 3) — the score's operative
73	base is ~11% low in the data even though the headli

People who liked this also viewed

6 picks

Type

policyengine-calibration-diagnostics

31stars5forks

View on GitHub

How to install policyengine-calibration-diagnostics?

policyengine/policyengine-claude/policyengine-calibration-diagnostics

$npx -y skills add PolicyEngine/policyengine-claude --skill policyengine-calibration-diagnostics

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# PolicyEngine calibration diagnostics
2
3	Converts tribal "I'd check the takeup rate first" knowledge into a structured sensitivity
4	registry, and pairs it with the live per-target calibration API so hypotheses are ranked against
5	real `relative_error` numbers rather than assumptions. When an `/analyze-policy` comparison
6	returns INVESTIGATE, this skill supplies the ranked candidate causes.
7
8	The calibrated microdata is now a Populace build (see the `policyengine-data` skill for how
9	targets, weights, and L0 sparsity work). Calibration targets live in the populace build's target
10	set — not in a hand-maintained loss file — and their fit is queryable per release from the
11	dashboard API below.
12
13	## When to use
14
15	- Stage 5.6 / Stage 6 of `/analyze-policy` — invoked by the `calibration-diagnostics` agent.
16	- Code review of microsim PRs where the headline number differs from priors.
17	- Designing or auditing a populace calibration target.
18	- Debugging why a state-level run looks volatile.
19
20	## Top-level architecture
21
22	PolicyEngine microsim results depend on three layers:
23
24	1. Country model logic (policyengine-us, policyengine-uk, policyengine-canada) — formulas,
25	parameters.
26	2. Calibrated microdata (Populace) — survey weights + imputations matched to administrative
27	targets.
28	3. Behavioral assumptions (takeup rates, labor-supply elasticities) — usually parameters but
29	easy to overlook.
30
31	A magnitude mismatch is almost always rooted in layer 2 or 3, not layer 1 (layer-1 mismatches
32	show up as outright simulation errors, not magnitude drift).
33
34	## Live calibration API (check this first)
35
36	Per-target fit for the current populace release, no auth, reads the release from Hugging Face:
37
38	```
39	BASE = https://calibration-diagnostics.vercel.app/calibration/dashboard/api/populace
40	GET {BASE}/target-diagnostics?source=<source> # every target for a source, with relative_error
41	GET {BASE}/target-investigation?target=<id> # full investigation packet for one target
42	GET {BASE}/releases # release ids for pinning
43	```
44
45	Sources map to reform domains: Social Security → `ssa`; Medicaid/ACA/Medicare →
46	`cms_medicaid` / `cms_aca` / `cms_medicare`; TANF → `hhs_acf_tanf`; income tax / credits →
47	`irs_soi`, `jct`, `cbo`; state income tax → `state_income_*`. A target already >10% off in the
48	release diagnostics is a stronger hypothesis than any prior. (Dashboard UI:
49	`calibration-diagnostics.vercel.app`.)
50
51	### The three-ring reading method (align with /analyze-policy Stage 5.6)
52
53	Checking only the reform's primary marginal is the classic miss — a reform's cost usually depends
54	on the joint distribution of its variable with other income, and the primary marginal can
55	calibrate perfectly while the interaction is off. Check three rings outward:
56
57	1. Ring 1 — the primary variable's marginals. Map the reform's domain to a calibration
58	source and fetch its targets.
59	2. Ring 2 — the mechanism's other inputs. Every variable that enters the formula the reform
60	changes. SS-benefit taxation depends on combined income, so dividends, taxable interest, and
61	pensions (`irs_soi`) are load-bearing; a CTC phase-in depends on earnings; a SNAP change on
62	rents and deductions.
63	3. Ring 3 — the interacted quantity itself (best single check). Search for the downstream
64	quantity the reform directly reprices — `taxable_social_security_amount` for SS-taxation,
65	`eitc_amount` for EITC, itemizer counts for deduction caps. When such a target exists it
66	validates the joint distribution end-to-end through the actual formula, and its
67	`relative_error` bounds the data-side bias of the score. Weight it above every marginal. When
68	no ring-3 target exists, say so — "marginals within tolerance; joint distribution untargeted"
69	is materially weaker evidence.
70
71	Worked example (HR 904, SS-taxation reform): `ssa` total benefits calibrate to −0.2% (ring 1 fine)
72	while `irs_soi ... taxable_social_security_amount` runs −10.9% (ring 3) — the score's operative
73	base is ~11% low in the data even though the headli

People who liked this also viewed

6 picks

Type