model-council-voting

Name: model-council-voting
Author: scdenney

34stars0forks

How to install model-council-voting?

scdenney/open-science-skills/model-council-voting

$npx -y skills add scdenney/open-science-skills --skill model-council-voting

Installs into the current project.

Summary

LLM council/panel voting: multi-model coders, consensus rules, inter-rater agreement (kappa, alpha), correlated-error diagnostics.

Files · 1

View on GitHub

SKILL.md

1	# Model Council Voting: Panels of Language Models as Independent Coders
2
3	## Instructions
4
5	A "council" runs the same labeling, scoring, or term-discovery task through several language models independently and reads their (dis)agreement as data. This skill sits on top of the single-model codebook-and-validation workflow in the `text-classification` skill: build and validate the codebook there first, then escalate to a council only when one model is not enough. It is a companion to `topic-modeling` (an independent, non-LLM method for cross-checking what a council finds), to `llm-calibration-logprobs` (per-item confidence from one model's token probabilities, a different signal than cross-model agreement), and to `methods-reporting` (the reporting standards the final write-up must meet).
6
7	### 1. When a Council Beats a Single Model
8
9	- Use a council when the labeling decision is contested or ambiguous — categories with fuzzy boundaries, stance or frame coding, or constructs where reasonable coders disagree. The council's disagreement rate is itself a measurable property of the task, not noise to be averaged away.
10	- Use a council for corpus-driven discovery where the output set is not fixed in advance — e.g., extracting which identity terms a corpus foregrounds. The user's `04_term_discovery/discovery/ollama_discover_terms.py` runs a zero-shot extraction prompt over sampled text windows; requiring several model families to independently surface the same term (see §4) is what separates a real corpus signal from one model's idiosyncrasy.
11	- Use a council for robustness reporting when results must survive a skeptical reader. Showing a finding holds across models from different training traditions answers the "would this replicate with a different model?" objection directly. Open-weight panels run locally also have lower and more predictable cross-run variance than proprietary APIs (Barrie, Palmer & Spirling 2025).
12	- A council is overkill when the task is unambiguous and a single validated model already agrees with humans at the level your downstream analysis needs (validate first via the `text-classification` workflow). For high-volume, well-defined coding, the cost of N model passes plus the agreement bookkeeping rarely buys anything. Reserve the council for the genuinely contested decisions and the discovery steps.
13	- A council is not a substitute for human validation (§7). N models agreeing tells you the label is reliable, not that it is correct (§5). Decide up front which role the council plays — reliability evidence, robustness check, or discovery filter — and say so in the write-up.
14
15	### 2. Assembling a Diverse Panel
16
17	- Diversify training families and origins to decorrelate errors. The threat a council guards against is a shared blind spot: models trained on overlapping data or by the same lab tend to make the same mistakes, so they vote together for the wrong reason. The user's term-discovery ensemble deliberately spans four origins — EXAONE-Deep (LG AI Research, Korea), Aya Expanse (Cohere, Canada), Qwen2.5 (Alibaba, China), and Gemma 3 (Google, US) — precisely because "none shares training data or architecture with the others in any direct way, which makes cross-family agreement a conservative test" (`appendix_a.tex`). Picking four checkpoints from one family is a near-useless council.
18	- Aim for 3–6 jurors. Below 3 you cannot compute many-rater agreement (§5) or run a meaningful k-of-N rule (§4). Above ~6 the marginal decorrelation falls off and the bookkeeping grows. These bounds are house defaults, not a cited optimum; the binding constraint is family diversity, not raw count (§6).
19	- Prefer open-weight models for reproducibility. Open weights can be pinned to an exact revision and re-run locally; proprietary APIs change underneath you and show high, unpredictable run-to-run variance even at temperature 0 (Barrie, Palmer & Spirling 2025). If a proprietary model is in the council, record its exact dated identifier and treat its votes as the least reproducible (`vlm-ocr-pipeline` makes the same point for OCR).
20	- Set decoding to be as deterministic as the stack allows. Temperature 0 (greedy) is the default for coding tasks so a juror's vote does not wobble between runs. The user's discovery pipeline runs at temperature 0.3 with a fixed window-sampling seed and notes that "because decoding at non-zero temperature is not bit-for-bit deterministic, reproducibility is enforced at the level of the term set rather than the raw generations" (`appendix_de.tex`). That is the right move when you want sampling diversity within a window but still need a reproducible final set: pin the seed, fix the inputs, and let the consensus rule (not the raw generation) be the stability criterion.
21	- **Consi

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

model-council-voting

34stars0forks

View on GitHub

How to install model-council-voting?

scdenney/open-science-skills/model-council-voting

$npx -y skills add scdenney/open-science-skills --skill model-council-voting

Installs into the current project.

Summary

LLM council/panel voting: multi-model coders, consensus rules, inter-rater agreement (kappa, alpha), correlated-error diagnostics.

Files · 1

View on GitHub

SKILL.md

1	# Model Council Voting: Panels of Language Models as Independent Coders
2
3	## Instructions
4
5	A "council" runs the same labeling, scoring, or term-discovery task through several language models independently and reads their (dis)agreement as data. This skill sits on top of the single-model codebook-and-validation workflow in the `text-classification` skill: build and validate the codebook there first, then escalate to a council only when one model is not enough. It is a companion to `topic-modeling` (an independent, non-LLM method for cross-checking what a council finds), to `llm-calibration-logprobs` (per-item confidence from one model's token probabilities, a different signal than cross-model agreement), and to `methods-reporting` (the reporting standards the final write-up must meet).
6
7	### 1. When a Council Beats a Single Model
8
9	- Use a council when the labeling decision is contested or ambiguous — categories with fuzzy boundaries, stance or frame coding, or constructs where reasonable coders disagree. The council's disagreement rate is itself a measurable property of the task, not noise to be averaged away.
10	- Use a council for corpus-driven discovery where the output set is not fixed in advance — e.g., extracting which identity terms a corpus foregrounds. The user's `04_term_discovery/discovery/ollama_discover_terms.py` runs a zero-shot extraction prompt over sampled text windows; requiring several model families to independently surface the same term (see §4) is what separates a real corpus signal from one model's idiosyncrasy.
11	- Use a council for robustness reporting when results must survive a skeptical reader. Showing a finding holds across models from different training traditions answers the "would this replicate with a different model?" objection directly. Open-weight panels run locally also have lower and more predictable cross-run variance than proprietary APIs (Barrie, Palmer & Spirling 2025).
12	- A council is overkill when the task is unambiguous and a single validated model already agrees with humans at the level your downstream analysis needs (validate first via the `text-classification` workflow). For high-volume, well-defined coding, the cost of N model passes plus the agreement bookkeeping rarely buys anything. Reserve the council for the genuinely contested decisions and the discovery steps.
13	- A council is not a substitute for human validation (§7). N models agreeing tells you the label is reliable, not that it is correct (§5). Decide up front which role the council plays — reliability evidence, robustness check, or discovery filter — and say so in the write-up.
14
15	### 2. Assembling a Diverse Panel
16
17	- Diversify training families and origins to decorrelate errors. The threat a council guards against is a shared blind spot: models trained on overlapping data or by the same lab tend to make the same mistakes, so they vote together for the wrong reason. The user's term-discovery ensemble deliberately spans four origins — EXAONE-Deep (LG AI Research, Korea), Aya Expanse (Cohere, Canada), Qwen2.5 (Alibaba, China), and Gemma 3 (Google, US) — precisely because "none shares training data or architecture with the others in any direct way, which makes cross-family agreement a conservative test" (`appendix_a.tex`). Picking four checkpoints from one family is a near-useless council.
18	- Aim for 3–6 jurors. Below 3 you cannot compute many-rater agreement (§5) or run a meaningful k-of-N rule (§4). Above ~6 the marginal decorrelation falls off and the bookkeeping grows. These bounds are house defaults, not a cited optimum; the binding constraint is family diversity, not raw count (§6).
19	- Prefer open-weight models for reproducibility. Open weights can be pinned to an exact revision and re-run locally; proprietary APIs change underneath you and show high, unpredictable run-to-run variance even at temperature 0 (Barrie, Palmer & Spirling 2025). If a proprietary model is in the council, record its exact dated identifier and treat its votes as the least reproducible (`vlm-ocr-pipeline` makes the same point for OCR).
20	- Set decoding to be as deterministic as the stack allows. Temperature 0 (greedy) is the default for coding tasks so a juror's vote does not wobble between runs. The user's discovery pipeline runs at temperature 0.3 with a fixed window-sampling seed and notes that "because decoding at non-zero temperature is not bit-for-bit deterministic, reproducibility is enforced at the level of the term set rather than the raw generations" (`appendix_de.tex`). That is the right move when you want sampling diversity within a window but still need a reproducible final set: pin the seed, fix the inputs, and let the consensus rule (not the raw generation) be the stability criterion.
21	- **Consi

People who liked this also viewed

6 picks

Type