conjoint-diagnostics

Name: conjoint-diagnostics
Author: scdenney

34stars0forks

How to install conjoint-diagnostics?

scdenney/open-science-skills/conjoint-diagnostics

$npx -y skills add scdenney/open-science-skills --skill conjoint-diagnostics

Installs into the current project.

Summary

Diagnose conjoint design integrity, estimation choices, and validity.

Files · 1

View on GitHub

SKILL.md

1	# Conjoint Experiment Diagnostics
2
3	## Instructions
4
5	Work through each section below for the conjoint study under review. Assess whether the study addresses each item adequately, partially, or not at all. Flag items that pose threats to inference and prioritize recommendations by severity.
6
7	Branch on input:
8	- If a paper or manuscript is provided, proceed through Sections 1–5 sequentially and produce a verdict per section.
9	- If analysis code or data is provided, verify the actual implementation rather than just what the paper claims: (a) confirm clustering specification, (b) confirm the estimand matches the reported quantity, (c) if IRR is unmeasured, compute within-respondent task-pair agreement as a function of attribute-level differences (Clayton et al. 2023 §3.3 method 2 / `projoint`).
10
11	For neighboring concerns, invoke sibling skills: `conjoint-design` (design choices), `conjoint-cleaning` (Qualtrics exports → long format), `hypothesis-building` (linking estimands to "If-Then" predictions), `methods-reporting` (full JARS/DA-RT compliance and replication archive), `cross-national-design` (multi-country / multilingual conjoints).
12
13	---
14
15	## 1. Design Diagnostics
16
17	### 1.1 Attribute and Level Selection
18	- Are attributes conceptually distinct and non-overlapping?
19	- Are levels realistic and mutually exclusive within each attribute?
20	- Is the number of attributes justified? (Bansak et al. 2021 PSRM "Beyond the Breaking Point": response quality is generally robust even at 35 filler attributes, with detectable but modest satisficing — use as a ceiling, not an endorsement of 35 attributes as optimal. Distinct from Bansak et al. 2018 on task counts below.)
21	- For cross-national or multilingual designs, verify construct equivalence of attribute labels across languages (see `cross-national-design`).
22	- Are there "dominant" attributes that might crowd out attention to others?
23	- Do the attribute levels span a meaningful range of the construct of interest?
24
25	### 1.2 Profile Restrictions
26	- Are implausible or contradictory attribute combinations allowed?
27	- If restrictions are imposed, are they documented and justified?
28	- If no restrictions, does the paper acknowledge potential odd profiles? (Bansak & Jenke 2025: odd profiles have minimal impact on first-order inferences, but should be acknowledged)
29
30	### 1.3 Number of Tasks and Satisficing
31	- How many tasks per respondent? (Bansak et al. 2018: up to 30 tasks with limited satisficing)
32	- Are attention checks embedded? Are pass rates reported by task position?
33	- Is there evidence of survey fatigue or satisficing in later tasks? Look for: (a) response-time distributions by task position (not just means), (b) always-same-side / always-left or always-right rates, (c) random-choice / trembling-hand rates relative to Bansak et al. 2021 baselines, (d) attention-check failure rates by task position.
34	- Have profile-order, carryover, and fatigue assumptions been tested empirically? (Ham, Imai & Janson 2024 §3.5: `CRTConjoint` provides conditional randomization tests for all three of Hainmueller-Hopkins-Yamamoto 2014's identifying assumptions.)
35
36	### 1.4 Randomization
37	- Is attribute-level assignment fully randomized (uniform distribution)?
38	- If non-uniform, is this justified and documented?
39	- Are profile-pair attributes independent or dependent? (Clayton et al. 2023 distinguish independent, dependent, and pair-level attributes)
40
41	### 1.5 Sample and Power
42	- Is the sample size justified with a power analysis? (Schuessler & Freitag 2020 cjpowR; Stefanelli & Lukac 2020)
43	- What is the effective sample size (N respondents x T tasks)?
44	- Are Type S (sign) and Type M (magnitude/exaggeration) errors considered?
45	- For subgroup analyses, is there adequate power within each subgroup?
46
47	---
48
49	## 2. Estimation Diagnostics
50
51	### 2.1 Estimand Clarity
52	- Verify the estimand is clearly defined (AMCE, MM, AMIE, or pAMCE) and the choice is justified.
53	- Check whether AMCEs are interpreted with awareness that they depend on the attribute distribution used for averaging (Hainmueller et al. 2014).
54	- Check whether MMs are used where reference-category-free comparisons are needed (Leeper et al. 2020).
55
56	### 2.2 Reference Levels
57	- Are reference levels clearly specified and substantively meaningful?
58	- Are AMCEs interpreted relative to the correct reference category?
59	- For subgroup comparisons, is the reference category consistent across groups? (Leeper et al. 2020: conditional AMCEs are sensitive to reference choice)
60
61	### 2.3 Subgroup Analysis
62	- Are subgroups defined by pre-treatment characteristics (not post-treatment)?
63	- Use marginal means and diff-in-MMs for subgroup comparisons, not conditional AMCEs. (Leeper, Hobolt & Tilley 2020: conditional AMCEs are reference-category dependent and can yield arbitrary subgroup differences)
64	- Are diff-in-MMs presented alongside MMs for interpretability?
65	- Is he

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

conjoint-diagnostics

34stars0forks

View on GitHub

How to install conjoint-diagnostics?

scdenney/open-science-skills/conjoint-diagnostics

$npx -y skills add scdenney/open-science-skills --skill conjoint-diagnostics

Installs into the current project.

Summary

Diagnose conjoint design integrity, estimation choices, and validity.

Files · 1

View on GitHub

SKILL.md

1	# Conjoint Experiment Diagnostics
2
3	## Instructions
4
5	Work through each section below for the conjoint study under review. Assess whether the study addresses each item adequately, partially, or not at all. Flag items that pose threats to inference and prioritize recommendations by severity.
6
7	Branch on input:
8	- If a paper or manuscript is provided, proceed through Sections 1–5 sequentially and produce a verdict per section.
9	- If analysis code or data is provided, verify the actual implementation rather than just what the paper claims: (a) confirm clustering specification, (b) confirm the estimand matches the reported quantity, (c) if IRR is unmeasured, compute within-respondent task-pair agreement as a function of attribute-level differences (Clayton et al. 2023 §3.3 method 2 / `projoint`).
10
11	For neighboring concerns, invoke sibling skills: `conjoint-design` (design choices), `conjoint-cleaning` (Qualtrics exports → long format), `hypothesis-building` (linking estimands to "If-Then" predictions), `methods-reporting` (full JARS/DA-RT compliance and replication archive), `cross-national-design` (multi-country / multilingual conjoints).
12
13	---
14
15	## 1. Design Diagnostics
16
17	### 1.1 Attribute and Level Selection
18	- Are attributes conceptually distinct and non-overlapping?
19	- Are levels realistic and mutually exclusive within each attribute?
20	- Is the number of attributes justified? (Bansak et al. 2021 PSRM "Beyond the Breaking Point": response quality is generally robust even at 35 filler attributes, with detectable but modest satisficing — use as a ceiling, not an endorsement of 35 attributes as optimal. Distinct from Bansak et al. 2018 on task counts below.)
21	- For cross-national or multilingual designs, verify construct equivalence of attribute labels across languages (see `cross-national-design`).
22	- Are there "dominant" attributes that might crowd out attention to others?
23	- Do the attribute levels span a meaningful range of the construct of interest?
24
25	### 1.2 Profile Restrictions
26	- Are implausible or contradictory attribute combinations allowed?
27	- If restrictions are imposed, are they documented and justified?
28	- If no restrictions, does the paper acknowledge potential odd profiles? (Bansak & Jenke 2025: odd profiles have minimal impact on first-order inferences, but should be acknowledged)
29
30	### 1.3 Number of Tasks and Satisficing
31	- How many tasks per respondent? (Bansak et al. 2018: up to 30 tasks with limited satisficing)
32	- Are attention checks embedded? Are pass rates reported by task position?
33	- Is there evidence of survey fatigue or satisficing in later tasks? Look for: (a) response-time distributions by task position (not just means), (b) always-same-side / always-left or always-right rates, (c) random-choice / trembling-hand rates relative to Bansak et al. 2021 baselines, (d) attention-check failure rates by task position.
34	- Have profile-order, carryover, and fatigue assumptions been tested empirically? (Ham, Imai & Janson 2024 §3.5: `CRTConjoint` provides conditional randomization tests for all three of Hainmueller-Hopkins-Yamamoto 2014's identifying assumptions.)
35
36	### 1.4 Randomization
37	- Is attribute-level assignment fully randomized (uniform distribution)?
38	- If non-uniform, is this justified and documented?
39	- Are profile-pair attributes independent or dependent? (Clayton et al. 2023 distinguish independent, dependent, and pair-level attributes)
40
41	### 1.5 Sample and Power
42	- Is the sample size justified with a power analysis? (Schuessler & Freitag 2020 cjpowR; Stefanelli & Lukac 2020)
43	- What is the effective sample size (N respondents x T tasks)?
44	- Are Type S (sign) and Type M (magnitude/exaggeration) errors considered?
45	- For subgroup analyses, is there adequate power within each subgroup?
46
47	---
48
49	## 2. Estimation Diagnostics
50
51	### 2.1 Estimand Clarity
52	- Verify the estimand is clearly defined (AMCE, MM, AMIE, or pAMCE) and the choice is justified.
53	- Check whether AMCEs are interpreted with awareness that they depend on the attribute distribution used for averaging (Hainmueller et al. 2014).
54	- Check whether MMs are used where reference-category-free comparisons are needed (Leeper et al. 2020).
55
56	### 2.2 Reference Levels
57	- Are reference levels clearly specified and substantively meaningful?
58	- Are AMCEs interpreted relative to the correct reference category?
59	- For subgroup comparisons, is the reference category consistent across groups? (Leeper et al. 2020: conditional AMCEs are sensitive to reference choice)
60
61	### 2.3 Subgroup Analysis
62	- Are subgroups defined by pre-treatment characteristics (not post-treatment)?
63	- Use marginal means and diff-in-MMs for subgroup comparisons, not conditional AMCEs. (Leeper, Hobolt & Tilley 2020: conditional AMCEs are reference-category dependent and can yield arbitrary subgroup differences)
64	- Are diff-in-MMs presented alongside MMs for interpretability?
65	- Is he

People who liked this also viewed

6 picks

Type