model-evaluation

Name: model-evaluation
Author: pymc-labs

71stars10forks

How to install model-evaluation?

pymc-labs/python-analytics-skills/model-evaluation

$npx -y skills add pymc-labs/python-analytics-skills --skill model-evaluation

Installs into the current project.

Summary

Load when the user is comparing Bayesian models, computing LOO-CV / ELPD, calling arviz_stats.loo or arviz_stats.compare, doing model stacking/averaging, or computing Bayes factors. Covers the ArviZ 1.1 LOO/ELPD/stacking APIs exclusively (no waic). Triggers include: model compari

Files · 1

View on GitHub

SKILL.md

1	# Model Evaluation and Comparison (ArviZ 1.1)
2
3	CRITICAL: PyMC 6 returns xarray DataTree objects by default, and ArviZ 1.1 stats/plots are DataTree-first while still accepting idata-like inputs. `az.waic` is removed entirely — use PSIS-LOO-CV exclusively. Default credible intervals are 0.89 ETI, controlled via `ci_prob=` and `ci_kind=` for summaries/plots; low-level `hdi()` uses `prob=`.
4
5	For model building context, prior selection, and convergence diagnostics, see the [pymc-modeling skill](../pymc-modeling/SKILL.md).
6
7	## LOO-CV with ArviZ 1.1
8
9	Leave-one-out cross-validation via Pareto-smoothed importance sampling (PSIS).
10
11	```python
12	import arviz_stats as azs
13	import arviz_plots as azp
14
15	# dt is a DataTree from pm.sample()
16	loo_result = azs.loo(dt)
17	print(loo_result)
18	# Returns: ELPDData with elpd, se, p, n_data_points, pareto_k
19
20	# Equivalent via the xarray accessor when arviz_stats is imported:
21	loo_result = dt.azstats.loo()
22	```
23
24	### Pareto k Diagnostics
25
26	Pareto k values indicate reliability of PSIS approximation for each observation:
27
28	\| k value \| Interpretation \| Action \|
29	\|---\|---\|---\|
30	\| k < 0.5 \| Good \| LOO estimate reliable \|
31	\| 0.5 < k < 0.7 \| Marginal \| Results usable but less accurate \|
32	\| 0.7 < k < 1.0 \| Bad \| Estimate unreliable — use moment matching or k-fold \|
33	\| k > 1.0 \| Very bad \| PSIS fails entirely — must use k-fold CV \|
34
35	```python
36	# Check Pareto k values
37	print(loo_result.pareto_k)
38
39	# Plot Pareto k diagnostics
40	azp.plot_khat(loo_result)
41
42	# Count problematic observations
43	import numpy as np
44	k_values = loo_result.pareto_k.values
45	print(f"k > 0.7: {np.sum(k_values > 0.7)} observations")
46	```
47
48	### What to Do When k > 0.7
49
50	1. Try moment matching first (fast, automatic)
51	2. If still bad, use k-fold cross-validation
52	3. Check if problematic observations are outliers — consider robust likelihood
53	4. Re-examine the model — high k often signals model misspecification
54
55	## Moment Matching
56
57	Automatically refit problematic observations using moment matching:
58
59	```python
60	# Requires log_likelihood in the DataTree
61	loo_mm = azs.loo_moment_match(dt)
62	```
63
64	This importance-weights the posterior for each problematic observation, improving the PSIS approximation without refitting the model. Much faster than k-fold.
65
66	## K-Fold Cross-Validation
67
68	When LOO is unreliable for many observations, use exact k-fold CV:
69
70	```python
71	# Perform 10-fold cross-validation
72	kfold_result = azs.loo_kfold(dt, K=10)
73	print(kfold_result)
74	```
75
76	This refits the model K times, so it is K times slower than LOO. Use only when LOO diagnostics indicate problems.
77
78	## compare() — Full Workflow
79
80	Compare multiple models on predictive accuracy:
81
82	```python
83	# dt1, dt2, dt3 are DataTree objects from pm.sample()
84	comparison = azs.compare(
85	{"linear": dt1, "quadratic": dt2, "spline": dt3},
86	)
87	print(comparison)
88	```
89
90	Note: `compare` in ArviZ 1.1 only supports LOO, so the old `ic=` and `scale=` arguments have been dropped.
91
92	### Interpreting the Comparison Table
93
94	\| Column \| Meaning \|
95	\|---\|---\|
96	\| `rank` \| Model rank (0 = best) \|
97	\| `elpd` \| Expected log pointwise predictive density \|
98	\| `p` \| Effective number of parameters \|
99	\| `elpd_diff` \| Difference in ELPD from the reference model \|
100	\| `weight` \| Stacking weight (sums to 1) \|
101	\| `se` \| Standard error of ELPD \|
102	\| `dse` \| Standard error of the ELPD difference \|
103	\| `diag_elpd` \| Pareto-k diagnostic issues for each model's ELPD \|
104	\| `diag_diff` \| Small-data or practically-equivalent-difference diagnostics \|
105
106	### Decision Rules
107
108	- `elpd_diff` = 0: reference/best model
109	- `\|elpd_diff\| < 4`: models are practically indistinguishable — prefer simpler one
110	- `\|elpd_diff\| > 4` and `\|elpd_diff / dse\| > 2`: meaningful difference in predictive accuracy
111	- Non-empty `diag_elpd`: LOO unreliable for this model — investigate Pareto k values
112
113	```python
114	# Visualize comparison
115	azp.plot_compare(comparison)
116
117	# Detailed forest plot of ELPD differences
118	azp.plot_elpd({"linear": dt1, "quadratic": dt2, "spline": dt3})
119	```
120
121	See `references/model_comparison.md` for detailed usage.
122
123	## Model Averaging
124
125	### Stacking Weights (Default)
126
127	Stacking minimizes KL divergence from the true predictive distribution to the weighted mixture. This is the recommended default.
128
129	```python
130	comparison = azs.compare({"m1": dt1, "m2": dt2, "m3": dt3})
131	# Stacking weights are in the "weight" column by default
132	print(comparison["weight"])
133	```
134
135	### Pseudo-BMA+ Weights
136
137	Alternative weighting based on Bayesian bootstrap of ELPD:
138
139	```python
140	comparison = azs.compare(
141	{"m1": dt1, "m2": dt2, "m3": dt3},

People who liked this also viewed

6 picks

Type

model-evaluation

71stars10forks

View on GitHub

How to install model-evaluation?

pymc-labs/python-analytics-skills/model-evaluation

$npx -y skills add pymc-labs/python-analytics-skills --skill model-evaluation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Model Evaluation and Comparison (ArviZ 1.1)
2
3	CRITICAL: PyMC 6 returns xarray DataTree objects by default, and ArviZ 1.1 stats/plots are DataTree-first while still accepting idata-like inputs. `az.waic` is removed entirely — use PSIS-LOO-CV exclusively. Default credible intervals are 0.89 ETI, controlled via `ci_prob=` and `ci_kind=` for summaries/plots; low-level `hdi()` uses `prob=`.
4
5	For model building context, prior selection, and convergence diagnostics, see the [pymc-modeling skill](../pymc-modeling/SKILL.md).
6
7	## LOO-CV with ArviZ 1.1
8
9	Leave-one-out cross-validation via Pareto-smoothed importance sampling (PSIS).
10
11	```python
12	import arviz_stats as azs
13	import arviz_plots as azp
14
15	# dt is a DataTree from pm.sample()
16	loo_result = azs.loo(dt)
17	print(loo_result)
18	# Returns: ELPDData with elpd, se, p, n_data_points, pareto_k
19
20	# Equivalent via the xarray accessor when arviz_stats is imported:
21	loo_result = dt.azstats.loo()
22	```
23
24	### Pareto k Diagnostics
25
26	Pareto k values indicate reliability of PSIS approximation for each observation:
27
28	\| k value \| Interpretation \| Action \|
29	\|---\|---\|---\|
30	\| k < 0.5 \| Good \| LOO estimate reliable \|
31	\| 0.5 < k < 0.7 \| Marginal \| Results usable but less accurate \|
32	\| 0.7 < k < 1.0 \| Bad \| Estimate unreliable — use moment matching or k-fold \|
33	\| k > 1.0 \| Very bad \| PSIS fails entirely — must use k-fold CV \|
34
35	```python
36	# Check Pareto k values
37	print(loo_result.pareto_k)
38
39	# Plot Pareto k diagnostics
40	azp.plot_khat(loo_result)
41
42	# Count problematic observations
43	import numpy as np
44	k_values = loo_result.pareto_k.values
45	print(f"k > 0.7: {np.sum(k_values > 0.7)} observations")
46	```
47
48	### What to Do When k > 0.7
49
50	1. Try moment matching first (fast, automatic)
51	2. If still bad, use k-fold cross-validation
52	3. Check if problematic observations are outliers — consider robust likelihood
53	4. Re-examine the model — high k often signals model misspecification
54
55	## Moment Matching
56
57	Automatically refit problematic observations using moment matching:
58
59	```python
60	# Requires log_likelihood in the DataTree
61	loo_mm = azs.loo_moment_match(dt)
62	```
63
64	This importance-weights the posterior for each problematic observation, improving the PSIS approximation without refitting the model. Much faster than k-fold.
65
66	## K-Fold Cross-Validation
67
68	When LOO is unreliable for many observations, use exact k-fold CV:
69
70	```python
71	# Perform 10-fold cross-validation
72	kfold_result = azs.loo_kfold(dt, K=10)
73	print(kfold_result)
74	```
75
76	This refits the model K times, so it is K times slower than LOO. Use only when LOO diagnostics indicate problems.
77
78	## compare() — Full Workflow
79
80	Compare multiple models on predictive accuracy:
81
82	```python
83	# dt1, dt2, dt3 are DataTree objects from pm.sample()
84	comparison = azs.compare(
85	{"linear": dt1, "quadratic": dt2, "spline": dt3},
86	)
87	print(comparison)
88	```
89
90	Note: `compare` in ArviZ 1.1 only supports LOO, so the old `ic=` and `scale=` arguments have been dropped.
91
92	### Interpreting the Comparison Table
93
94	\| Column \| Meaning \|
95	\|---\|---\|
96	\| `rank` \| Model rank (0 = best) \|
97	\| `elpd` \| Expected log pointwise predictive density \|
98	\| `p` \| Effective number of parameters \|
99	\| `elpd_diff` \| Difference in ELPD from the reference model \|
100	\| `weight` \| Stacking weight (sums to 1) \|
101	\| `se` \| Standard error of ELPD \|
102	\| `dse` \| Standard error of the ELPD difference \|
103	\| `diag_elpd` \| Pareto-k diagnostic issues for each model's ELPD \|
104	\| `diag_diff` \| Small-data or practically-equivalent-difference diagnostics \|
105
106	### Decision Rules
107
108	- `elpd_diff` = 0: reference/best model
109	- `\|elpd_diff\| < 4`: models are practically indistinguishable — prefer simpler one
110	- `\|elpd_diff\| > 4` and `\|elpd_diff / dse\| > 2`: meaningful difference in predictive accuracy
111	- Non-empty `diag_elpd`: LOO unreliable for this model — investigate Pareto k values
112
113	```python
114	# Visualize comparison
115	azp.plot_compare(comparison)
116
117	# Detailed forest plot of ELPD differences
118	azp.plot_elpd({"linear": dt1, "quadratic": dt2, "spline": dt3})
119	```
120
121	See `references/model_comparison.md` for detailed usage.
122
123	## Model Averaging
124
125	### Stacking Weights (Default)
126
127	Stacking minimizes KL divergence from the true predictive distribution to the weighted mixture. This is the recommended default.
128
129	```python
130	comparison = azs.compare({"m1": dt1, "m2": dt2, "m3": dt3})
131	# Stacking weights are in the "weight" column by default
132	print(comparison["weight"])
133	```
134
135	### Pseudo-BMA+ Weights
136
137	Alternative weighting based on Bayesian bootstrap of ELPD:
138
139	```python
140	comparison = azs.compare(
141	{"m1": dt1, "m2": dt2, "m3": dt3},

People who liked this also viewed

6 picks

Type