A p-value answers the question: “If the null hypothesis were true, how probable is a result at least this extreme?” It does not answer: “How large is the effect?” or “Is this effect scientifically meaningful?” In large samples, trivially small effects produce highly significant p-values; in small samples, practically important effects may not reach significance. Utts’s 1999 JSE paper explicitly draws this distinction, noting that statistical methods are designed to detect and measure relationships in situations where results cannot be identically replicated due to natural variability, and that the appropriate question is whether the cumulative effect size across studies is consistent and meaningful, not whether any individual study crossed a p < .05 threshold.¹

When Daryl Bem published precognition experiments in 2011, critics argued that the results, while nominally significant, reflected the inadequacy of frequentist significance testing rather than genuine evidence for precognition. Utts, Bem, and Johnson co-authored a reply in the Journal of Personality and Social Psychology arguing that Bayesian and likelihood-ratio approaches, which directly quantify the relative support for competing hypotheses, provide more informative summaries of the evidence than p-values alone.² The original paper by Bem, Utts, and Johnson (2011) also addressed the question of whether psychologists should change their analytic practices, arguing that the choice of statistical framework materially affects the conclusions drawn from the same data.⁴

Utts, Norris, Suess, and Johnson (2010) examined the relationship between prior beliefs and the interpretation of statistical evidence, asking whether scientists behave as Bayesians in practice even when they use frequentist methods formally. The paper explored how strong prior skepticism about psi can lead researchers to dismiss statistically compelling evidence that would be accepted in other domains, a form of motivated reasoning that the authors argued is inconsistent with the stated norms of scientific inference.⁵

In her 1999 JSE paper, Utts compared the effect sizes from ganzfeld and remote viewing meta-analyses against those from conventional medical research, specifically the antiplatelet therapy trials used as a benchmark for cumulative evidence in medicine. She noted that the effect sizes from mind-matter meta-analyses were in a range she described as consistent with the small-to-medium effect-size band by Cohen’s conventions, expressed equivalently as hit-rate elevations above chance for forced-choice paradigms. The comparison was methodological: Utts was not claiming psi effects are as large as drug effects, but that the inferential standard, cumulative meta-analytic evidence with consistent effect direction, should be applied uniformly.¹

In a 1996 book chapter on model selection in parapsychology, Utts examined how statistical model-selection criteria, tools designed to choose among competing explanatory models, apply to the question of whether observed anomalous data are better described by psychokinesis, precognition, or chance variation. The chapter appeared in a volume honoring statistician Seymour Geisser and engaged with formal model-comparison frameworks, illustrating that the question of which psi model best fits the data is a tractable statistical problem, not merely a philosophical one.⁶

The antiplatelet therapy example in Utts’s 1999 JSE paper is methodologically instructive. The medical evidence for antiplatelet drugs reducing vascular disease was accepted by the scientific community on the basis of cumulative meta-analytic evidence showing a consistent, modest effect across multiple randomized controlled trials, not on the basis of any single decisive experiment. Utts argued that the same logic applied to ganzfeld and remote viewing meta-analyses: the question is whether the cumulative evidence, evaluated by standard meta-analytic methods, shows a consistent effect of meaningful magnitude. She concluded that it did, while explicitly noting that this statistical conclusion does not resolve the question of mechanism or causal explanation.¹

Hurlbert, Levine, and Utts (2019), published in The American Statistician as part of a special issue on statistical inference, argued that the phrase “statistically significant” should be permanently retired. The paper’s title, “Coup de Grâce for a Tough Old Bull”, signals its polemical intent. The authors argued that the binary significant/non-significant framing: (1) encourages researchers to treat p = .049 and p = .051 as categorically different outcomes; (2) suppresses reporting of effect sizes and confidence intervals; (3) creates incentives for p-hacking and optional stopping; and (4) leads to systematic misinterpretation of null results as evidence of no effect. These are precisely the inferential errors Utts had identified in critiques of psi research, and in defenses of psi research against dismissal based on underpowered null replications.³

In the 2011 reply to critics of Bem’s precognition research, Utts and colleagues argued that Bayesian and likelihood-ratio approaches directly address the question that significance testing obscures: not “Is this result unlikely under the null?” but “How much more strongly do the data support the alternative hypothesis than the null?” The reply engaged specifically with critics who had argued that Bem’s results, while nominally significant, reflected the inadequacy of frequentist methods rather than genuine evidence. Utts and colleagues’ position was that applying Bayesian methods to the same data yielded Bayes factors that constituted meaningful evidence, though the magnitude and interpretation of those Bayes factors remained contested in the exchange.²