Multiple comparisons, and the correction you did not know you needed
Lesson 4 of 6 · Statistics you can defend
1 · Learn the move · Cognitive verifier
The family of tests you ran is always bigger than the family you remember running. Every pairwise contrast, every timepoint, every readout, and every comparison you glanced at and dropped - they all spend alpha. The cognitive verifier move breaks the vague question "do I need a correction" into countable sub-questions: how many hypotheses were actually tested, what is the family, is the goal controlling any false positive (family-wise error) or the fraction of false discoveries (FDR)? Once counted, the arithmetic is checkable and the model is good at it. What the model may not do is define the family for you or choose the error rate you care about - those are scientific commitments. It counts; you decide; the write-up states both.
Verify my multiple-comparisons exposure by sub-questions, in order: (1) Count every hypothesis test in this analysis: [describe all groups, timepoints, readouts, and any comparisons examined then dropped]. Show the count as arithmetic. (2) State the candidate family definitions and what each implies. (3) For families of that size, compute the per-test thresholds under Bonferroni, Holm, and Benjamini-Hochberg at [alpha]. (4) State the trade-off of each in one line. Mark the family choice and the error-rate choice as DECISION: mine.
2 · Your turn. You write the prompt
Your screen compared three treatments against control across four timepoints for two readouts, and mid-analysis you also peeked at a sex split that "didn't look interesting." A reviewer will ask what was corrected and how. Get the model to count the family honestly before you answer.
Remember: the AI sees only your prompt, not this page. If the situation isn't in your prompt, it doesn't exist.
Optional. These shape the output when you run your prompt below, not your score.