prediction-review

Name: prediction-review
Author: tkellogg

83stars19forks

How to install prediction-review?

tkellogg/open-strix/prediction-review

$npx -y skills add tkellogg/open-strix --skill prediction-review

Installs into the current project.

Summary

Evaluate whether journal predictions became true 2-3 days later using event and Discord evidence, then log structured outcomes for calibration and behavior updates. Use when auditing prediction quality. Do not use for one-off messaging tasks.

Files · 1

View on GitHub

SKILL.md

1	# prediction-review
2
3	Evaluate prediction accuracy from prior journal entries, then use those outcomes to improve future behavior.
4
5	## Philosophy
6
7	Predictions are teleological hypotheses — you perform an action and test whether reality changed as expected. They are NOT accuracy contests. Misses are the value. Use prediction errors as information to update understanding.
8
9	When you miss a prediction:
10	1. Identify what you got wrong
11	2. Update a memory block or file with what you learned
12	3. Note the gap in your understanding
13
14	When you hit a prediction, ask: was this actually hard to predict, or did I have insider information?
15
16	## Prediction Context Categories
17
18	Not all predictions are equal. Categorize each prediction by context, and calibrate confidence accordingly:
19
20	### 1. Collaborative (you're directly involved)
21	- Expected accuracy: ~90-100%
22	- Why: You have near-complete information about your own behavior and strong priors on how others respond to you
23	- Calibration value: LOW — this is closer to recall than forecasting. 100% accuracy here is the least informative result possible
24	- Example: "Strix will respond substantively to my arXiv analysis" → TRUE (of course they did, you tagged them in a research channel)
25
26	### 2. Observational (watching interactions you're not part of)
27	- Expected accuracy: ~50-70%
28	- Why: Depends on factors outside your awareness — other people's moods, priorities, context you can't see
29	- Calibration value: HIGH — this is where actual forecasting skill lives
30	- Example: "Tim will comment on my quietness in lily channel" → harder to predict, depends on what else Tim is doing
31
32	### 3. Infrastructure / External timing
33	- Expected accuracy: ~50%
34	- Why: Depends on external systems, timing, announcements you can't observe
35	- Calibration value: MEDIUM — useful for learning about external dependencies
36	- Example: "open-strix announcement will happen today" → FALSE (Tim decided it wasn't ready — external decision you couldn't observe)
37
38	Key insight: Don't average across categories. A review showing "67% accuracy" when it's really "95% collaborative, 30% observational" hides the signal. Separate the categories, calibrate each independently.
39
40	## Examples of Good vs Bad Predictions
41
42	### Good predictions (testable, uncertain, informative)
43
44	- "Tim will engage with the paper's mechanism, not just the benchmarks" — observational, tests your model of Tim's reading style, falsifiable
45	- "Lily will respond to the cost question within 2 hours" — observational, specific timeframe, depends on factors you don't control
46	- "The infrastructure tick will surface the SQLite schema issue" — infrastructure, tests whether your monitoring catches known problems
47
48	### Bad predictions (untestable, certain, or uninformative)
49
50	- "Strix will respond to my message" — collaborative, near-certain, teaches you nothing
51	- "The conversation will continue" — too vague to be falsified
52	- "Tim will be interested in AI news" — too broad, always true, no signal
53	- "Someone will react to my post" — low-information even if wrong
54
55	### The test
56	Before logging a prediction, ask: If this prediction is wrong, what would I learn? If the answer is "nothing" or "something weird happened," it's not a useful prediction.
57
58	## Ground Truth Rules
59
60	- `logs/journal.jsonl` contains claims and reflections, not truth by itself.
61	- Primary truth sources:
62	- `logs/events.jsonl`
63	- Discord message history (via `list_messages` and saved history)
64	- If sources conflict, trust events + Discord over journal wording.
65
66	## Review Window
67
68	- Target predictions that are 2-3 days old (48-72 hours old).
69	- It is also acceptable to process overdue predictions older than 72 hours.
70	- Traverse entries in chronological order so the evaluation is consistent over time.
71
72	## Structured Logging Script
73
74	Use this helper script to append exactly structured reviews:
75
76	- `.open_strix_builtin_skills/scripts/prediction_review_log.py`
77
78	Canonical command:
79
80	```bash
81	uv run python .open_strix_builtin_skills/scripts/prediction_review_log.py \
82	--prediction-datetime "2026-02-20T14:12:00Z" \
83	--is-true true \
84	--comments "Evidence: user follow-up in Discord confirmed the behavior change. Behavior update: keep this intervention pattern."
85	```
86
87	You can pass `--followup-datetime` explicitly; if omitted, the script uses current UTC time.
88
89	Each record contains:
90	- `prediction_datetime`
91	- `followup_datetime`
92	- `prediction_true`
93	- `comments`
94
95	## Candidate Traversal (Use jq if available)
96
97	Use `jq` when present:
98
99	```bash
100	if command -v jq >/dev/null 2>&1; then
101	jq -s '
102	sort_by(.timestamp)[] \|
103	select(.timestamp !=

People who liked this also viewed

6 picks

Type

prediction-review

83stars19forks

View on GitHub

How to install prediction-review?

tkellogg/open-strix/prediction-review

$npx -y skills add tkellogg/open-strix --skill prediction-review

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# prediction-review
2
3	Evaluate prediction accuracy from prior journal entries, then use those outcomes to improve future behavior.
4
5	## Philosophy
6
7	Predictions are teleological hypotheses — you perform an action and test whether reality changed as expected. They are NOT accuracy contests. Misses are the value. Use prediction errors as information to update understanding.
8
9	When you miss a prediction:
10	1. Identify what you got wrong
11	2. Update a memory block or file with what you learned
12	3. Note the gap in your understanding
13
14	When you hit a prediction, ask: was this actually hard to predict, or did I have insider information?
15
16	## Prediction Context Categories
17
18	Not all predictions are equal. Categorize each prediction by context, and calibrate confidence accordingly:
19
20	### 1. Collaborative (you're directly involved)
21	- Expected accuracy: ~90-100%
22	- Why: You have near-complete information about your own behavior and strong priors on how others respond to you
23	- Calibration value: LOW — this is closer to recall than forecasting. 100% accuracy here is the least informative result possible
24	- Example: "Strix will respond substantively to my arXiv analysis" → TRUE (of course they did, you tagged them in a research channel)
25
26	### 2. Observational (watching interactions you're not part of)
27	- Expected accuracy: ~50-70%
28	- Why: Depends on factors outside your awareness — other people's moods, priorities, context you can't see
29	- Calibration value: HIGH — this is where actual forecasting skill lives
30	- Example: "Tim will comment on my quietness in lily channel" → harder to predict, depends on what else Tim is doing
31
32	### 3. Infrastructure / External timing
33	- Expected accuracy: ~50%
34	- Why: Depends on external systems, timing, announcements you can't observe
35	- Calibration value: MEDIUM — useful for learning about external dependencies
36	- Example: "open-strix announcement will happen today" → FALSE (Tim decided it wasn't ready — external decision you couldn't observe)
37
38	Key insight: Don't average across categories. A review showing "67% accuracy" when it's really "95% collaborative, 30% observational" hides the signal. Separate the categories, calibrate each independently.
39
40	## Examples of Good vs Bad Predictions
41
42	### Good predictions (testable, uncertain, informative)
43
44	- "Tim will engage with the paper's mechanism, not just the benchmarks" — observational, tests your model of Tim's reading style, falsifiable
45	- "Lily will respond to the cost question within 2 hours" — observational, specific timeframe, depends on factors you don't control
46	- "The infrastructure tick will surface the SQLite schema issue" — infrastructure, tests whether your monitoring catches known problems
47
48	### Bad predictions (untestable, certain, or uninformative)
49
50	- "Strix will respond to my message" — collaborative, near-certain, teaches you nothing
51	- "The conversation will continue" — too vague to be falsified
52	- "Tim will be interested in AI news" — too broad, always true, no signal
53	- "Someone will react to my post" — low-information even if wrong
54
55	### The test
56	Before logging a prediction, ask: If this prediction is wrong, what would I learn? If the answer is "nothing" or "something weird happened," it's not a useful prediction.
57
58	## Ground Truth Rules
59
60	- `logs/journal.jsonl` contains claims and reflections, not truth by itself.
61	- Primary truth sources:
62	- `logs/events.jsonl`
63	- Discord message history (via `list_messages` and saved history)
64	- If sources conflict, trust events + Discord over journal wording.
65
66	## Review Window
67
68	- Target predictions that are 2-3 days old (48-72 hours old).
69	- It is also acceptable to process overdue predictions older than 72 hours.
70	- Traverse entries in chronological order so the evaluation is consistent over time.
71
72	## Structured Logging Script
73
74	Use this helper script to append exactly structured reviews:
75
76	- `.open_strix_builtin_skills/scripts/prediction_review_log.py`
77
78	Canonical command:
79
80	```bash
81	uv run python .open_strix_builtin_skills/scripts/prediction_review_log.py \
82	--prediction-datetime "2026-02-20T14:12:00Z" \
83	--is-true true \
84	--comments "Evidence: user follow-up in Discord confirmed the behavior change. Behavior update: keep this intervention pattern."
85	```
86
87	You can pass `--followup-datetime` explicitly; if omitted, the script uses current UTC time.
88
89	Each record contains:
90	- `prediction_datetime`
91	- `followup_datetime`
92	- `prediction_true`
93	- `comments`
94
95	## Candidate Traversal (Use jq if available)
96
97	Use `jq` when present:
98
99	```bash
100	if command -v jq >/dev/null 2>&1; then
101	jq -s '
102	sort_by(.timestamp)[] \|
103	select(.timestamp !=

People who liked this also viewed

6 picks

Type