Overview

Statistical and file-drawer critiques of parapsychology are not unique to the field; they form a subset of the broader methodological reform movement that produced the replication crisis in psychology and the rise of pre-registration as standard practice. What distinguishes the psi case is the combination of small but persistent effect sizes, decades of cumulative meta-analyses, and the question of whether the cumulative record reflects a real anomaly, a residual artifact of publication bias and analytic flexibility, or some combination of both.

This page covers the principal statistical-methodology critiques: the file-drawer problem (Sterling 1959; Hyman 1985), optional stopping and multiple comparisons (Alcock 1987), effect-size interpretation, Bayesian reanalysis (Wagenmakers et al. 2011; Rouder & Morey 2011), and the extraordinary-claims standard. Each critique is presented with the response published by parapsychologists in the same literature, and with a neutral synthesis of what the published exchange establishes and what remains open.

The File-Drawer Problem in Psi Research

Sterling (1959) first documented that journals in psychology overwhelmingly published statistically significant results, creating systematic distortion in any literature relying on aggregated published findings.^[9] The “file drawer” metaphor — for the hypothetical drawer of unpublished null studies — became central to meta-analytic methodology by the 1980s.

Hyman (1985) applied the file-drawer concern directly to the ganzfeld database in his critical appraisal of Honorton’s meta-analytic claims.^[1] Honorton (1985) responded with a fail-safe N calculation using Rosenthal’s framework: the number of unpublished null studies that would need to exist to reduce the observed effect to non-significance. For the early ganzfeld database, Honorton calculated a fail-safe N in the hundreds, arguing that a file drawer of that magnitude was implausible given the small community of psi researchers and the field’s practice of submitting both positive and null results to parapsychology journals.^[2]

Rosenthal (1986), commenting on the same exchange, contributed methodological refinements to fail-safe N calculation and noted that the published parapsychology record showed a pattern of submitted null results inconsistent with severe file-drawer suppression at the level skeptics had hypothesized.^[10]

Optional Stopping and Multiple Comparisons

Alcock (1987) catalogued optional stopping — terminating data collection when results reach significance — as a structural concern in pre-1990 parapsychology designs that lacked explicit stopping rules.^[3] Multiple-comparisons concerns extended to post-hoc subgroup analyses (gender, target type, dynamic vs. static targets in ganzfeld) that, if not pre-specified, inflate the family-wise Type I error rate.

The methodological response in parapsychology took two forms: tighter protocol specification (the autoganzfeld procedure introduced by Honorton in the mid-1980s removed several sources of analytic flexibility), and adoption of pre-registration once the infrastructure became available. The Koestler Parapsychology Unit registry provides public time-stamped protocol records; Bem, Tressoldi, Rabeyron, & Duggan (2015) reported a meta-analysis of 90 precognition experiments that included a subset of pre-registered replications.^[11]

Effect Sizes in Context

Meta-analytic effect sizes in psi research are small. Storm, Tressoldi, & Di Risio (2010) reported a free-response meta-analysis covering 1992-2008 with an overall hit rate of 31.5% against a chance baseline of 25%, corresponding to a small Cohen’s d in the range typical of psi meta-analyses.^[5] Bem, Tressoldi, Rabeyron, & Duggan (2015) reported a comparable effect size across 90 precognition experiments.^[11]

The Open Science Collaboration (2015) reproducibility project reported that mainstream social-psychology effects, when subjected to direct replication, often shrank to effect sizes in the same small range — and in many cases failed to reach significance in the replication studies.^[12] This finding reframes the effect-size critique: the question is not whether psi effects are smaller than typical psychology effects (in many cases they are comparable) but whether the cumulative meta-analytic record in psi reflects a real signal that the methodological reforms of the 2010s would preserve.

Milton & Wiseman (1999) reported a non-significant ganzfeld meta-analysis over a study set partially overlapping with later analyses; subsequent re-analyses by parapsychologists disputed the study-inclusion criteria, with the exchange continuing in Psychological Bulletin through follow-up commentaries.^[4]

Bayesian Reanalyses and Prior Probabilities

Wagenmakers, Wetzels, Borsboom, & van der Maas (2011) responded to Bem’s (2011) Journal of Personality and Social Psychology precognition paper by arguing that default Bayes factor analysis with a Cauchy prior centered on zero would not support Bem’s findings against the null.^[7][6] Their critique extended beyond psi to a broader methodological argument that p-value-based inference systematically overstates evidence for novel effects.

Rouder & Morey (2011) computed a Bayes factor meta-analysis of Bem’s nine experiments using a default JZS prior and reported a Bayes factor of approximately 40 in favor of an effect, while explicitly noting that Bayes factors are prior-dependent and that a skeptic with a stronger prior against psi would draw a different conclusion from the same data.^[8] The Rouder & Morey reanalysis is sometimes cited as evidence for Bem’s findings and sometimes cited as illustrating the prior-dependence problem in Bayesian inference — both characterizations appear in the published literature.

The Bayesian-reanalysis exchange illustrated a structural feature of the dispute: when the prior probability assigned to psi is very low, Bayes factor calculations require correspondingly large effect sizes or sample sizes to shift posterior belief, even when the data would convince a frequentist analyst.

The Extraordinary-Claims Standard

Hyman (1985) and Alcock (1987) both invoked the principle that claims contradicting established physics require evidentiary thresholds higher than those applied to ordinary empirical claims in psychology.^[1][3] The principle is widely associated with Carl Sagan (“extraordinary claims require extraordinary evidence”) and traces further back to Hume’s argument on miracles.

Parapsychologists have responded that the standard, while reasonable in principle, has been applied asymmetrically: the claim that decades of psi data reflect pure chance also requires evidence, and the burden of demonstrating that a multi-decade meta-analytic record with consistent small effects across independent laboratories arose from artifact alone has not been formally discharged. The dispute over what counts as “extraordinary evidence” — replication count, effect size, mechanism specification, or some combination — is itself a methodological question with multiple published positions.

Pre-Registration as Response

The methodological-reform response to optional-stopping and multiple-comparisons critiques is pre-registration: public time-stamped specification of hypothesis, analysis plan, sample size, and stopping rule before data collection. A study registry associated with the Koestler Parapsychology Unit predates the widespread adoption of pre-registration in mainstream psychology

Bem, Tressoldi, Rabeyron, & Duggan (2015) reported a meta-analysis of 90 precognition experiments that included pre-registered replications as a separately analyzable subset; the authors reported that effect sizes in the pre-registered subset were comparable to the full set.^[11] Whether pre-registered replications in psi research will continue to show consistent small effects as the practice becomes more widespread is an empirical question that subsequent published replication attempts will address.

Counterarguments and Debate