llm-calibration-logprobs

Name: llm-calibration-logprobs
Author: scdenney

34stars0forks

How to install llm-calibration-logprobs?

scdenney/open-science-skills/llm-calibration-logprobs

$npx -y skills add scdenney/open-science-skills --skill llm-calibration-logprobs

Installs into the current project.

Summary

Analyze LLM token logprobs and calibration. Use for per-decision confidence, ECE, Brier scores, reliability diagrams, and low-confidence triage.

Files · 1

View on GitHub

SKILL.md

1	# Reading a Model's Own Uncertainty from Token Log-Probabilities
2
3	## Instructions
4
5	This skill covers within-model confidence: how sure a single model is about each decision it makes, read off the token log-probabilities it emits, and whether that internal confidence is calibrated against ground truth. It pairs with the `model-council-voting` skill, which handles the complement — between-coder agreement across several independent models. Use both: one model's high self-reported confidence on an item, and three models independently agreeing on that item, are different kinds of evidence, and a careful pipeline reports both. This skill does not cover codebook design or human-validation statistics (κ, F1) — those live in the `text-classification` skill; cross-reference it rather than re-deriving them here.
6
7	### 1. What a Token Log-Probability Is, and When to Use It
8
9	- A generative LLM produces each output token by sampling from a probability distribution over its vocabulary. The log-probability (logprob) of the token it actually emitted is `log P(token \| context)`; exponentiating gives a probability in `[0, 1]`. A logprob of `0.0` means probability 1.0 (the model treated that token as certain); `-2.30` means probability ≈ 0.10. This is the model's own assessment of how likely that token was, conditional on everything before it.
10	- Use logprobs when you need a per-item, per-decision confidence signal that is nearly free (it falls out of the same forward pass that produced the label) and continuous. It answers "how sure was the model about this classification?" — a question a hard 0/1 label cannot.
11	- Do not treat a logprob as a probability of correctness. It is the model's subjective probability under its own learned distribution, which can be systematically wrong. Section 4 is the entire reason this distinction matters: a model can assign 0.95 to tokens that are right only 70% of the time. The logprob is a claim about confidence; calibration assessment is how you check whether the claim holds.
12	- This is a long-standing idea in NLP — token probabilities are the native uncertainty signal of a language model — but it is under-used in applied social science, where the convention is to take the final label at face value and discard the distribution it came from. Recovering and using that distribution is the contribution of this skill (see also §6's disciplinary note).
13	- Sequence-likelihood self-evaluation (asking the model itself to rate confidence, or reading off the probability it assigns to its own answer) is studied directly in the calibration literature (Kadavath et al. 2022; Tian et al. 2023). Prefer the emitted-token logprob over a prompted "rate your confidence HIGH/MEDIUM/LOW" string when both are available: the verbalized rating is itself generated text subject to the same biases, whereas the logprob is the underlying mechanism.
14
15	### 2. Collecting Logprobs and Aggregating Multi-Token Labels
16
17	- Turn on logprob return at the API/server level. OpenAI chat completions accept `logprobs=True` and `top_logprobs=K` (K ≤ 20 at time of writing); the response carries a per-token list with `.token`, `.logprob`, and a `.top_logprobs` list of the K most likely alternatives at that position. vLLM exposes the same fields through its OpenAI-compatible server (`logprobs=K`). Ollama returns per-token logprobs through its API as well. `analysis/classify_openai.py` in the open-text surveys project is a worked example: it sets `api_kwargs["logprobs"] = True; api_kwargs["top_logprobs"] = 5` and stashes `resp.choices[0].logprobs.content` for every response.
18	- Fix the decoding so the logprobs are reproducible. Set `temperature=0` and a fixed `seed` (that script uses `temperature=0, seed=42`). Temperature 0 makes the label deterministic on most backends, but note that it does not by itself make logprob values bit-identical across hosted-API calls — see §6. Record both settings.
19	- A label usually spans several tokens, so aggregate. A code like `civic_commitment` tokenizes into multiple sub-word pieces (`civic`, `_comm`, `itment`, …). You must combine their logprobs into one per-item confidence. The robust way to find the right tokens is to reconstruct the full output string, locate the label inside it (e.g., the value after `"code":`), build a character-offset map for each token, and select exactly the tokens overlapping the label's character span — this is what `extract_code_logprobs()` does in `classify_openai.py` rather than guessing token indices. Aggregation choices:
20	- Sum of label-token logprobs = `log P(whole label string)` = the joint sequence probability. This is the principled "how likely was this exact label" number but is length-confounded: longer labels score lowe

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

llm-calibration-logprobs

34stars0forks

View on GitHub

How to install llm-calibration-logprobs?

scdenney/open-science-skills/llm-calibration-logprobs

$npx -y skills add scdenney/open-science-skills --skill llm-calibration-logprobs

Installs into the current project.

Summary

Analyze LLM token logprobs and calibration. Use for per-decision confidence, ECE, Brier scores, reliability diagrams, and low-confidence triage.

Files · 1

View on GitHub

SKILL.md

1	# Reading a Model's Own Uncertainty from Token Log-Probabilities
2
3	## Instructions
4
5	This skill covers within-model confidence: how sure a single model is about each decision it makes, read off the token log-probabilities it emits, and whether that internal confidence is calibrated against ground truth. It pairs with the `model-council-voting` skill, which handles the complement — between-coder agreement across several independent models. Use both: one model's high self-reported confidence on an item, and three models independently agreeing on that item, are different kinds of evidence, and a careful pipeline reports both. This skill does not cover codebook design or human-validation statistics (κ, F1) — those live in the `text-classification` skill; cross-reference it rather than re-deriving them here.
6
7	### 1. What a Token Log-Probability Is, and When to Use It
8
9	- A generative LLM produces each output token by sampling from a probability distribution over its vocabulary. The log-probability (logprob) of the token it actually emitted is `log P(token \| context)`; exponentiating gives a probability in `[0, 1]`. A logprob of `0.0` means probability 1.0 (the model treated that token as certain); `-2.30` means probability ≈ 0.10. This is the model's own assessment of how likely that token was, conditional on everything before it.
10	- Use logprobs when you need a per-item, per-decision confidence signal that is nearly free (it falls out of the same forward pass that produced the label) and continuous. It answers "how sure was the model about this classification?" — a question a hard 0/1 label cannot.
11	- Do not treat a logprob as a probability of correctness. It is the model's subjective probability under its own learned distribution, which can be systematically wrong. Section 4 is the entire reason this distinction matters: a model can assign 0.95 to tokens that are right only 70% of the time. The logprob is a claim about confidence; calibration assessment is how you check whether the claim holds.
12	- This is a long-standing idea in NLP — token probabilities are the native uncertainty signal of a language model — but it is under-used in applied social science, where the convention is to take the final label at face value and discard the distribution it came from. Recovering and using that distribution is the contribution of this skill (see also §6's disciplinary note).
13	- Sequence-likelihood self-evaluation (asking the model itself to rate confidence, or reading off the probability it assigns to its own answer) is studied directly in the calibration literature (Kadavath et al. 2022; Tian et al. 2023). Prefer the emitted-token logprob over a prompted "rate your confidence HIGH/MEDIUM/LOW" string when both are available: the verbalized rating is itself generated text subject to the same biases, whereas the logprob is the underlying mechanism.
14
15	### 2. Collecting Logprobs and Aggregating Multi-Token Labels
16
17	- Turn on logprob return at the API/server level. OpenAI chat completions accept `logprobs=True` and `top_logprobs=K` (K ≤ 20 at time of writing); the response carries a per-token list with `.token`, `.logprob`, and a `.top_logprobs` list of the K most likely alternatives at that position. vLLM exposes the same fields through its OpenAI-compatible server (`logprobs=K`). Ollama returns per-token logprobs through its API as well. `analysis/classify_openai.py` in the open-text surveys project is a worked example: it sets `api_kwargs["logprobs"] = True; api_kwargs["top_logprobs"] = 5` and stashes `resp.choices[0].logprobs.content` for every response.
18	- Fix the decoding so the logprobs are reproducible. Set `temperature=0` and a fixed `seed` (that script uses `temperature=0, seed=42`). Temperature 0 makes the label deterministic on most backends, but note that it does not by itself make logprob values bit-identical across hosted-API calls — see §6. Record both settings.
19	- A label usually spans several tokens, so aggregate. A code like `civic_commitment` tokenizes into multiple sub-word pieces (`civic`, `_comm`, `itment`, …). You must combine their logprobs into one per-item confidence. The robust way to find the right tokens is to reconstruct the full output string, locate the label inside it (e.g., the value after `"code":`), build a character-offset map for each token, and select exactly the tokens overlapping the label's character span — this is what `extract_code_logprobs()` does in `classify_openai.py` rather than guessing token indices. Aggregation choices:
20	- Sum of label-token logprobs = `log P(whole label string)` = the joint sequence probability. This is the principled "how likely was this exact label" number but is length-confounded: longer labels score lowe

People who liked this also viewed

6 picks

Type