text-classification

Name: text-classification
Author: scdenney

34stars0forks

How to install text-classification?

scdenney/open-science-skills/text-classification

$npx -y skills add scdenney/open-science-skills --skill text-classification

Installs into the current project.

Summary

Design and validate LLM-based text classification. Use for codebooks, prompts, validation samples, agreement statistics, and reporting model-coded data.

Files · 1

View on GitHub

SKILL.md

1	# LLM-Based Text Classification for Social Science Research
2
3	## Instructions
4
5	### 1. Codebook Design
6
7	- Before drafting the codebook, specify the population, sampling frame, and (for experimental data) the treatment condition each response is drawn from. These constrain which categories can plausibly exist and which demographic subgroups any bias assessment must cover. LLM classification extends, rather than replaces, the longer open-ended coding tradition in survey methodology (Geer 1988; Lupia 2018).
8	- Treat codebook design as the most consequential decision in the classification pipeline. LLMs struggle with loose instructions and revert to general-purpose definitions rather than following researcher-specific operationalizations (Halterman & Keith 2025).
9	- Structure each code with the following components (adapted from Halterman & Keith 2025):
10	- Label: The exact output string the model should return
11	- Definition: A single-sentence operationalization of the construct
12	- Clarification: What IS included — boundary cases that belong in this category
13	- Negative clarification: What is NOT included — common confusions and adjacent categories
14	- Examples: 2-3 positive examples (correctly classified) and 2-3 negative examples (common misclassifications)
15	- Keep the number of codes small (3-6) for initial classification. Larger coding schemes increase ambiguity and reduce inter-annotator agreement for both humans and LLMs (Chae & Davidson 2025).
16	- Allow multi-label assignment when responses may reflect more than one construct. Specify this explicitly in the prompt — models default to single-label output unless instructed otherwise.
17	- Include a residual category (e.g., `none_of_above` or `uncodeable`) for responses that are too vague, too short, or off-topic. Define this category as precisely as the substantive codes (Halterman & Keith 2025).
18	- Iterate the codebook through pilot testing. Examine disagreements between LLM output and hand-coding to identify ambiguous definitions, then revise. Most codebook problems are definition problems, not model problems (Halterman & Keith 2025).
19	- For a fully-worked example of a codebook with all five components filled in for a realistic three-category classification task, plus a matching system prompt that operationalizes it, see `reference/example-codebook-and-prompt.md`.
20
21	### 2. Choosing a Learning Regime
22
23	- Follow the decision framework from Chae & Davidson (2025), which maps document characteristics and available resources to the appropriate approach:
24
25	Zero-shot prompting: Use when classifying short documents with a large decoder model (GPT-4o, Llama3-70B+) and no labeled training data. Best for rapid prototyping and tasks where constructs are well-defined. GPT-4o achieves the best zero-shot performance across tasks (Chae & Davidson 2025).
26
27	Few-shot prompting: Add labeled examples to the prompt. Results are inconsistent — adding examples helps some models but degrades others (Chae & Davidson 2025). Always compare few-shot against zero-shot on a held-out sample before committing. Select diverse examples covering edge cases, not just prototypical instances.
28
29	Fine-tuning: Train a model on labeled data. Effective with as few as 100 hand-coded examples for smaller models (Chae & Davidson 2025). Fine-tuned smaller models (Llama3-8B, GPT-3 Davinci) can match GPT-4o zero-shot performance. Prefer this when you have labeled data and need cost-effective classification at scale.
30
31	Instruction-tuning: Combine detailed prompting with fine-tuning on paired instruction-output examples. Most powerful regime for complex tasks — instruction-tuned Llama3-70B surpasses GPT-4o zero-shot on stance detection (Chae & Davidson 2025). Requires more technical infrastructure but yields the highest accuracy.
32
33	Encoder-only fine-tuning: A distinct fourth regime often omitted from generative-LLM discussions. Fine-tuning a smaller encoder-only model (BERT, DeBERTa, SBERT; ~86–110M parameters, personal-computer hardware) on modest labeled data can match or exceed zero-shot generative LLMs on many classification tasks at a fraction of the cost and with fully reproducible (deterministic) output (Chae & Davidson 2025, Table 1; Ziems et al. 2024 find fine-tuned RoBERTa rarely under-performs larger generative models across 20 tasks). Prefer encoder fine-tuning when the label set is fixed, labeled data exists, and reproducibility matters more than generative flexibility.
34
35	- When resources permit, test multiple regimes on the same pilot sample and select based on empirical performance, not assumptions.
36
37	### 3. Model Selection and Reproducibility
38
39	- Prefer open-weight models (Llama 3, Gemma, Mistral) for publishable research. Open-weight models run locally produc

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

text-classification

34stars0forks

View on GitHub

How to install text-classification?

scdenney/open-science-skills/text-classification

$npx -y skills add scdenney/open-science-skills --skill text-classification

Installs into the current project.

Summary

Design and validate LLM-based text classification. Use for codebooks, prompts, validation samples, agreement statistics, and reporting model-coded data.

Files · 1

View on GitHub

SKILL.md

1	# LLM-Based Text Classification for Social Science Research
2
3	## Instructions
4
5	### 1. Codebook Design
6
7	- Before drafting the codebook, specify the population, sampling frame, and (for experimental data) the treatment condition each response is drawn from. These constrain which categories can plausibly exist and which demographic subgroups any bias assessment must cover. LLM classification extends, rather than replaces, the longer open-ended coding tradition in survey methodology (Geer 1988; Lupia 2018).
8	- Treat codebook design as the most consequential decision in the classification pipeline. LLMs struggle with loose instructions and revert to general-purpose definitions rather than following researcher-specific operationalizations (Halterman & Keith 2025).
9	- Structure each code with the following components (adapted from Halterman & Keith 2025):
10	- Label: The exact output string the model should return
11	- Definition: A single-sentence operationalization of the construct
12	- Clarification: What IS included — boundary cases that belong in this category
13	- Negative clarification: What is NOT included — common confusions and adjacent categories
14	- Examples: 2-3 positive examples (correctly classified) and 2-3 negative examples (common misclassifications)
15	- Keep the number of codes small (3-6) for initial classification. Larger coding schemes increase ambiguity and reduce inter-annotator agreement for both humans and LLMs (Chae & Davidson 2025).
16	- Allow multi-label assignment when responses may reflect more than one construct. Specify this explicitly in the prompt — models default to single-label output unless instructed otherwise.
17	- Include a residual category (e.g., `none_of_above` or `uncodeable`) for responses that are too vague, too short, or off-topic. Define this category as precisely as the substantive codes (Halterman & Keith 2025).
18	- Iterate the codebook through pilot testing. Examine disagreements between LLM output and hand-coding to identify ambiguous definitions, then revise. Most codebook problems are definition problems, not model problems (Halterman & Keith 2025).
19	- For a fully-worked example of a codebook with all five components filled in for a realistic three-category classification task, plus a matching system prompt that operationalizes it, see `reference/example-codebook-and-prompt.md`.
20
21	### 2. Choosing a Learning Regime
22
23	- Follow the decision framework from Chae & Davidson (2025), which maps document characteristics and available resources to the appropriate approach:
24
25	Zero-shot prompting: Use when classifying short documents with a large decoder model (GPT-4o, Llama3-70B+) and no labeled training data. Best for rapid prototyping and tasks where constructs are well-defined. GPT-4o achieves the best zero-shot performance across tasks (Chae & Davidson 2025).
26
27	Few-shot prompting: Add labeled examples to the prompt. Results are inconsistent — adding examples helps some models but degrades others (Chae & Davidson 2025). Always compare few-shot against zero-shot on a held-out sample before committing. Select diverse examples covering edge cases, not just prototypical instances.
28
29	Fine-tuning: Train a model on labeled data. Effective with as few as 100 hand-coded examples for smaller models (Chae & Davidson 2025). Fine-tuned smaller models (Llama3-8B, GPT-3 Davinci) can match GPT-4o zero-shot performance. Prefer this when you have labeled data and need cost-effective classification at scale.
30
31	Instruction-tuning: Combine detailed prompting with fine-tuning on paired instruction-output examples. Most powerful regime for complex tasks — instruction-tuned Llama3-70B surpasses GPT-4o zero-shot on stance detection (Chae & Davidson 2025). Requires more technical infrastructure but yields the highest accuracy.
32
33	Encoder-only fine-tuning: A distinct fourth regime often omitted from generative-LLM discussions. Fine-tuning a smaller encoder-only model (BERT, DeBERTa, SBERT; ~86–110M parameters, personal-computer hardware) on modest labeled data can match or exceed zero-shot generative LLMs on many classification tasks at a fraction of the cost and with fully reproducible (deterministic) output (Chae & Davidson 2025, Table 1; Ziems et al. 2024 find fine-tuned RoBERTa rarely under-performs larger generative models across 20 tasks). Prefer encoder fine-tuning when the label set is fixed, labeled data exists, and reproducibility matters more than generative flexibility.
34
35	- When resources permit, test multiple regimes on the same pilot sample and select based on empirical performance, not assumptions.
36
37	### 3. Model Selection and Reproducibility
38
39	- Prefer open-weight models (Llama 3, Gemma, Mistral) for publishable research. Open-weight models run locally produc

People who liked this also viewed

6 picks

Type