list-experiment

Name: list-experiment
Author: scdenney

34stars0forks

How to install list-experiment?

scdenney/open-science-skills/list-experiment

$npx -y skills add scdenney/open-science-skills --skill list-experiment

Installs into the current project.

Summary

Design and diagnose list experiments (item count technique).

Files · 1

View on GitHub

SKILL.md

1	# List Experiment Designer
2
3	Related skills: Use alongside `hypothesis-building` (state π and a SESOI before design choices), `survey-design` (mode effects, question ordering, and pre-testing of control items), and `methods-reporting` (deposit list wording, randomization seed, `list` package version, and `ict.test` / `ict.hausman.test` / `ictreg()` output).
4
5	## Instructions
6
7	### 1. Pre-Design: Is a List Experiment Warranted?
8
9	- Assess sensitivity bias first: Before committing to a list experiment, consult domain-specific evidence on sensitivity bias. Blair, Coppock, and Moor's (2020) meta-analysis of 30 years of list experiments shows that sensitivity biases are typically smaller than 10 percentage points. A list experiment is not automatically the right choice for any sensitive topic.
10	- Social reference theory: Sensitivity bias is largest when (a) the social norm on the topic is strong, (b) the norm is clear and widely shared, and (c) respondents believe others can infer their true attitude from their response (Blair et al. 2020). Evaluate all three conditions before deciding.
11	- Precision cost: List experiments are approximately 14 times noisier than a direct question, so equivalent precision requires roughly 14 times the sample. The trade-off is only favorable when the expected sensitivity bias exceeds the precision loss (Blair et al. 2020). If the topic is sensitive but the expected bias is small (< 5pp), a direct question with neutral framing is often preferable.
12	- Empirical benchmarks by domain: Voter turnout (~5–15pp overreport, wide confidence intervals), clientelism and vote-buying (~5–15pp underreport), racial prejudice (near-zero sensitivity bias — Blair et al. 2020 find little evidence respondents conceal prejudice on direct questions), authoritarian regime support (highly context-dependent and often dominated by artificial deflation rather than preference falsification). Use these as priors when no domain-specific estimates exist.
13
14	### 2. Basic Design
15
16	- Core logic: Assign respondents randomly to a control group (receives N baseline items) or a treatment group (receives N+1 items, including the sensitive item). Estimate prevalence as the difference in mean counts between treatment and control. This provides plausible deniability because no individual response can be traced to the sensitive item.
17	- Number of control items: Use 3–5 control items. Fewer items reduce plausible deniability (too easy to infer the sensitive item from a high count). More items increase cognitive load and floor/ceiling risk, and reduce statistical efficiency.
18	- Randomize item order: Randomize the order of all items within each respondent's list to prevent position effects from inflating or deflating specific items.
19	- Wording parity: Frame all items, including the sensitive item, in the same grammatical form and at the same abstraction level. Stylistic inconsistency makes the sensitive item stand out, undermining the design.
20
21	### 3. Control List Design
22
23	- The floor/ceiling constraint: Select control items so that virtually no respondent would endorse all N control items (ceiling) or zero control items (floor) when assigned to the treatment group. A respondent at the ceiling cannot truthfully report N+1 even if they hold the sensitive attitude; one at the floor cannot hide a "1" count. Both violations bias estimates downward (artificial deflation) and compromise identification (Blair & Imai 2012).
24	- Target prevalence range for control items: Each control item should have expected prevalence between 20% and 80% in the population. The sum of expected control item endorsements should have low variance — ideally each respondent endorses roughly 1–3 of N control items.
25	- Item independence: Control items should be uncorrelated with the sensitive item. If control items tap dimensions that predict the sensitive attitude, the no-design-effect assumption is threatened.
26	- Conventional vs. placebo vs. mixed control list design: Three design options exist for managing measurement error from inattentive respondents (Agerberg & Tannenberg 2021):
27	- Conventional: Standard baseline items, no placebo. Biased if nonstrategic error inflates or deflates counts.
28	- Placebo: Replace one control item with a universally false statement (all should answer "no") to equate list length in treatment and control. Riambau & Ostwald (2021) show this reduces mechanical inflation — the tendency for treatment group respondents to report more true items simply due to list length, especially among low-education respondents. However, Agerberg & Tannenberg (2021) show placebo items do not universally reduce bias and can increase it under some conditions.
29	- Mixed control list: Combines conventional and placebo items. Preferred when the expected direction and magnitude of sensitivity bias is uncertain

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

list-experiment

34stars0forks

View on GitHub

How to install list-experiment?

scdenney/open-science-skills/list-experiment

$npx -y skills add scdenney/open-science-skills --skill list-experiment

Installs into the current project.

Summary

Design and diagnose list experiments (item count technique).

Files · 1

View on GitHub

SKILL.md

1	# List Experiment Designer
2
3	Related skills: Use alongside `hypothesis-building` (state π and a SESOI before design choices), `survey-design` (mode effects, question ordering, and pre-testing of control items), and `methods-reporting` (deposit list wording, randomization seed, `list` package version, and `ict.test` / `ict.hausman.test` / `ictreg()` output).
4
5	## Instructions
6
7	### 1. Pre-Design: Is a List Experiment Warranted?
8
9	- Assess sensitivity bias first: Before committing to a list experiment, consult domain-specific evidence on sensitivity bias. Blair, Coppock, and Moor's (2020) meta-analysis of 30 years of list experiments shows that sensitivity biases are typically smaller than 10 percentage points. A list experiment is not automatically the right choice for any sensitive topic.
10	- Social reference theory: Sensitivity bias is largest when (a) the social norm on the topic is strong, (b) the norm is clear and widely shared, and (c) respondents believe others can infer their true attitude from their response (Blair et al. 2020). Evaluate all three conditions before deciding.
11	- Precision cost: List experiments are approximately 14 times noisier than a direct question, so equivalent precision requires roughly 14 times the sample. The trade-off is only favorable when the expected sensitivity bias exceeds the precision loss (Blair et al. 2020). If the topic is sensitive but the expected bias is small (< 5pp), a direct question with neutral framing is often preferable.
12	- Empirical benchmarks by domain: Voter turnout (~5–15pp overreport, wide confidence intervals), clientelism and vote-buying (~5–15pp underreport), racial prejudice (near-zero sensitivity bias — Blair et al. 2020 find little evidence respondents conceal prejudice on direct questions), authoritarian regime support (highly context-dependent and often dominated by artificial deflation rather than preference falsification). Use these as priors when no domain-specific estimates exist.
13
14	### 2. Basic Design
15
16	- Core logic: Assign respondents randomly to a control group (receives N baseline items) or a treatment group (receives N+1 items, including the sensitive item). Estimate prevalence as the difference in mean counts between treatment and control. This provides plausible deniability because no individual response can be traced to the sensitive item.
17	- Number of control items: Use 3–5 control items. Fewer items reduce plausible deniability (too easy to infer the sensitive item from a high count). More items increase cognitive load and floor/ceiling risk, and reduce statistical efficiency.
18	- Randomize item order: Randomize the order of all items within each respondent's list to prevent position effects from inflating or deflating specific items.
19	- Wording parity: Frame all items, including the sensitive item, in the same grammatical form and at the same abstraction level. Stylistic inconsistency makes the sensitive item stand out, undermining the design.
20
21	### 3. Control List Design
22
23	- The floor/ceiling constraint: Select control items so that virtually no respondent would endorse all N control items (ceiling) or zero control items (floor) when assigned to the treatment group. A respondent at the ceiling cannot truthfully report N+1 even if they hold the sensitive attitude; one at the floor cannot hide a "1" count. Both violations bias estimates downward (artificial deflation) and compromise identification (Blair & Imai 2012).
24	- Target prevalence range for control items: Each control item should have expected prevalence between 20% and 80% in the population. The sum of expected control item endorsements should have low variance — ideally each respondent endorses roughly 1–3 of N control items.
25	- Item independence: Control items should be uncorrelated with the sensitive item. If control items tap dimensions that predict the sensitive attitude, the no-design-effect assumption is threatened.
26	- Conventional vs. placebo vs. mixed control list design: Three design options exist for managing measurement error from inattentive respondents (Agerberg & Tannenberg 2021):
27	- Conventional: Standard baseline items, no placebo. Biased if nonstrategic error inflates or deflates counts.
28	- Placebo: Replace one control item with a universally false statement (all should answer "no") to equate list length in treatment and control. Riambau & Ostwald (2021) show this reduces mechanical inflation — the tendency for treatment group respondents to report more true items simply due to list length, especially among low-education respondents. However, Agerberg & Tannenberg (2021) show placebo items do not universally reduce bias and can increase it under some conditions.
29	- Mixed control list: Combines conventional and placebo items. Preferred when the expected direction and magnitude of sensitivity bias is uncertain

People who liked this also viewed

6 picks

Type