methods-reporting

Name: methods-reporting
Author: scdenney

34stars0forks

How to install methods-reporting?

scdenney/open-science-skills/methods-reporting

$npx -y skills add scdenney/open-science-skills --skill methods-reporting

Installs into the current project.

Summary

Check methods reporting against CONSORT, JARS, DA-RT standards.

Files · 1

View on GitHub

SKILL.md

1	# Transparent Methods Reporter
2
3	## Instructions
4
5	### 1. Design Documentation and Pre-Registration
6	This section covers upstream documentation that makes transparent reporting possible. A well-documented bad design is still a bad design; transparency tools are necessary but not sufficient (Druckman 2022).
7
8	- Design Document: Before data collection, create a comprehensive document recording all decisions and their rationale -- motivation, stimuli, outcome measures, predictions, analysis plans, and logistics. This "design document" is the upstream practice that enables downstream transparent reporting (Druckman 2022, Ch. 5).
9	- Pre-Registration vs. Pre-Analysis Plan: Distinguish between study registration (basic: recording the study's existence, hypotheses, and population in a public repository) and a pre-analysis plan (detailed: specifying exact statistical tests, evidence thresholds, and contingency plans). Require the latter for confirmatory experiments. Name the repository used (OSF, EGAP, or AsPredicted) and provide the registration ID. For PAP structure, cross-reference the `pre-registration-writing` skill; for estimand, SESOI, and primary/secondary/exploratory classification, cross-reference `hypothesis-building`.
10	- Six Core Pre-Registration Elements (Lakens's JARS-aligned summary): For preregistered experiments, ensure the following six elements are specified, following Lakens's (2025) summary of the APA Journal Article Reporting Standards for quantitative research (JARS-Quant; Appelbaum et al. 2018): (1) randomization procedure, (2) inclusion/exclusion criteria, (3) sampling procedures and expected participation rate, (4) sample size justification with power analysis or precision rationale, (5) data diagnostics (exclusion criteria, missing data handling, outlier definitions, assumption checks), and (6) analytic strategy organized into primary, secondary, and exploratory tiers. The primary JARS-Quant tables (Appelbaum et al. 2018, American Psychologist; incorporated into APA 2020, Publication Manual, 7th ed., ch. 3) contain a longer branching item set covering observational, clinical-trial, longitudinal, replication, and N-of-1 designs; the six elements above are Lakens's pedagogical condensation, not JARS-Quant in full. For the broader reproducibility context in which JARS sits, see Munafò et al. (2017, "A Manifesto for Reproducible Science").
11	- Minimal vs. Complete Pre-Registration. Waldron and Allen (2022) show that minimal pre-registrations (hypotheses only, or under-specified analysis plans) re-admit exploratory researcher degrees of freedom under a confirmatory banner. A pre-registration that lists hypotheses but leaves variables, exclusion rules, or model specifications open is closer to a public hypothesis than a confirmatory PAP. Require complete specification (items 1--6 above) when a study will be reported as confirmatory.
12	- Analysis Code: The gold standard is to preregister analysis code that runs on a simulated dataset, eliminating ambiguity about all analytical decisions (Lakens 2025). For conjoint analyses, provide the regression specification in code form (e.g., R `lm()` or `feols()` call).
13	- Pilot Documentation: Document all pilot studies, including not just manipulation check results but also response rate data, recruitment language testing, and any modifications made as a result (Druckman 2022, Ch. 5). For conjoints, pilot whether respondents attend to all attributes, find combinations plausible, and process the display as intended.
14
15	### 2. Subjects, Recruitment, and Setting
16	- Eligibility: Explicitly state who was eligible to participate and the criteria for subject selection.
17	- Target Population: State the target population to which inference is intended. This includes the units, contexts, measures, and outcomes (Druckman 2022, Ch. 6). Distinguish between the sampling frame (who could be reached) and the target population (who the theory applies to).
18	- Timeline: Report the exact dates of the recruitment period and when the experiments were conducted, including any repeated measurements or follow-ups. For "firehouse studies" conducted in response to real-world events, document the lag between the event and data collection (Mutz 2011).
19	- Provider Details: For survey experiments, identify the survey firm used and describe their recruitment methods if they are not universally known. Note whether the sample is probability-based or nonprobability (quota-matched online panel) and the implications for generalizability.
20	- Response Rates: Provide the response rate and specify the exact formula used for its calculation.
21	- Survey Error Pipeline: Report how each of three sequential error sources was addressed: (1) coverage error (does the sampling frame reach the target population?), (2) sampling error (does the sample represent t

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

methods-reporting

34stars0forks

View on GitHub

How to install methods-reporting?

scdenney/open-science-skills/methods-reporting

$npx -y skills add scdenney/open-science-skills --skill methods-reporting

Installs into the current project.

Summary

Check methods reporting against CONSORT, JARS, DA-RT standards.

Files · 1

View on GitHub

SKILL.md

1	# Transparent Methods Reporter
2
3	## Instructions
4
5	### 1. Design Documentation and Pre-Registration
6	This section covers upstream documentation that makes transparent reporting possible. A well-documented bad design is still a bad design; transparency tools are necessary but not sufficient (Druckman 2022).
7
8	- Design Document: Before data collection, create a comprehensive document recording all decisions and their rationale -- motivation, stimuli, outcome measures, predictions, analysis plans, and logistics. This "design document" is the upstream practice that enables downstream transparent reporting (Druckman 2022, Ch. 5).
9	- Pre-Registration vs. Pre-Analysis Plan: Distinguish between study registration (basic: recording the study's existence, hypotheses, and population in a public repository) and a pre-analysis plan (detailed: specifying exact statistical tests, evidence thresholds, and contingency plans). Require the latter for confirmatory experiments. Name the repository used (OSF, EGAP, or AsPredicted) and provide the registration ID. For PAP structure, cross-reference the `pre-registration-writing` skill; for estimand, SESOI, and primary/secondary/exploratory classification, cross-reference `hypothesis-building`.
10	- Six Core Pre-Registration Elements (Lakens's JARS-aligned summary): For preregistered experiments, ensure the following six elements are specified, following Lakens's (2025) summary of the APA Journal Article Reporting Standards for quantitative research (JARS-Quant; Appelbaum et al. 2018): (1) randomization procedure, (2) inclusion/exclusion criteria, (3) sampling procedures and expected participation rate, (4) sample size justification with power analysis or precision rationale, (5) data diagnostics (exclusion criteria, missing data handling, outlier definitions, assumption checks), and (6) analytic strategy organized into primary, secondary, and exploratory tiers. The primary JARS-Quant tables (Appelbaum et al. 2018, American Psychologist; incorporated into APA 2020, Publication Manual, 7th ed., ch. 3) contain a longer branching item set covering observational, clinical-trial, longitudinal, replication, and N-of-1 designs; the six elements above are Lakens's pedagogical condensation, not JARS-Quant in full. For the broader reproducibility context in which JARS sits, see Munafò et al. (2017, "A Manifesto for Reproducible Science").
11	- Minimal vs. Complete Pre-Registration. Waldron and Allen (2022) show that minimal pre-registrations (hypotheses only, or under-specified analysis plans) re-admit exploratory researcher degrees of freedom under a confirmatory banner. A pre-registration that lists hypotheses but leaves variables, exclusion rules, or model specifications open is closer to a public hypothesis than a confirmatory PAP. Require complete specification (items 1--6 above) when a study will be reported as confirmatory.
12	- Analysis Code: The gold standard is to preregister analysis code that runs on a simulated dataset, eliminating ambiguity about all analytical decisions (Lakens 2025). For conjoint analyses, provide the regression specification in code form (e.g., R `lm()` or `feols()` call).
13	- Pilot Documentation: Document all pilot studies, including not just manipulation check results but also response rate data, recruitment language testing, and any modifications made as a result (Druckman 2022, Ch. 5). For conjoints, pilot whether respondents attend to all attributes, find combinations plausible, and process the display as intended.
14
15	### 2. Subjects, Recruitment, and Setting
16	- Eligibility: Explicitly state who was eligible to participate and the criteria for subject selection.
17	- Target Population: State the target population to which inference is intended. This includes the units, contexts, measures, and outcomes (Druckman 2022, Ch. 6). Distinguish between the sampling frame (who could be reached) and the target population (who the theory applies to).
18	- Timeline: Report the exact dates of the recruitment period and when the experiments were conducted, including any repeated measurements or follow-ups. For "firehouse studies" conducted in response to real-world events, document the lag between the event and data collection (Mutz 2011).
19	- Provider Details: For survey experiments, identify the survey firm used and describe their recruitment methods if they are not universally known. Note whether the sample is probability-based or nonprobability (quota-matched online panel) and the implications for generalizability.
20	- Response Rates: Provide the response rate and specify the exact formula used for its calculation.
21	- Survey Error Pipeline: Report how each of three sequential error sources was addressed: (1) coverage error (does the sampling frame reach the target population?), (2) sampling error (does the sample represent t

People who liked this also viewed

6 picks

Type