Overview: replication as a contested concept

Replication in parapsychology has never been a settled concept. Effect sizes in psi experiments are characteristically small, which means that any single replication attempt typically lacks the statistical power to detect the predicted effect even if it is real. Alcock (1987), reviewing the field for Behavioral and Brain Sciences, argued that absence of routine independent replication was a structural problem^[11]. Honorton (1985), responding to Hyman’s earlier critiques of ganzfeld research, argued that meta-analytic pooling across many small studies was the appropriate response when individual studies lack power^[2].

The decades-long exchange that followed has turned on three intertwined questions: what statistical apparatus correctly summarizes a heterogeneous corpus of small studies, whether meta-analytic positive results survive specification choices about which studies to include, and whether direct replications by independent labs cohere with the meta-analytic record. Each of these questions has been answered differently by different authors. See the sibling spoke Ganzfeld Debate for the parallel exchange specific to ganzfeld telepathy studies, and the methods page Replication for the broader methodological framing.

The Milton-Wiseman 1999 inflection

Milton and Wiseman (1999) published a meta-analysis in Psychological Bulletin covering 30 ganzfeld studies conducted between 1987 and 1997 (i.e., the studies completed after Honorton’s earlier autoganzfeld series)^[1]. They reported a mean effect size statistically indistinguishable from chance, and concluded that the ganzfeld effect had not replicated in the post-autoganzfeld corpus.

The paper became the most-cited skeptical reference in the ganzfeld literature and is frequently treated as the methodological inflection point for the debate. Subsequent reanalyses, including Storm, Tressoldi, and Di Risio (2010), proposed that Milton and Wiseman’s pooling decisions, particularly their treatment of trial-count weighting and inclusion of methodologically heterogeneous studies, materially affected the reported effect size^[3].

Post-1999 ganzfeld meta-analyses

Storm, Tressoldi, and Di Risio (2010) reanalyzed free-response psi studies published from 1992 through 2008 using updated inclusion criteria and reported a small but statistically significant cumulative effect across 29 ganzfeld studies, with a Stouffer Z ≈ 5.48 and effect sizes in the range typically reported for similarly-sized social-psychology effects^[3]. They argued that updating the corpus through 2008 and applying contemporary meta-analytic practice produced a different summary than Milton and Wiseman’s 1997 cutoff.

Storm, Tressoldi, and Utts (2013) returned to the corpus after a Bayesian critique and reported both frequentist and Bayesian analyses, arguing that the choice of prior distribution materially affects the Bayesian summary and that under more diffuse priors the data still favor a non-null effect^[4]. The exchange illustrates a recurring pattern: the same data, analyzed with different defensible statistical conventions, produce divergent summary conclusions.

The Bem 2011 episode

Daryl Bem’s 2011 paper “Feeling the Future,” published in the Journal of Personality and Social Psychology, reported nine experiments using standard social-psychology paradigms (priming, recall, habituation) run in time-reversed form^[5]. Eight of the nine reported statistically significant results in the predicted direction. Publication in a top mainstream journal, with positive results across a standard battery, made the paper a focal point for methodological discussion in psychology more broadly.

Wagenmakers, Wetzels, Borsboom, and van der Maas (2011) published an accompanying critique using default Bayes factors with the symmetric Cauchy prior recommended by Rouder and colleagues; they reported that the data provided substantial evidence for the null hypothesis under that prior, and argued that the apparent significance of Bem’s results reflected limitations of null-hypothesis significance testing rather than evidence of psi^[6]. Rouder and Morey (2011) independently published a Bayes-factor meta-analysis of Bem’s experiments reaching a similar conclusion^[7]. Galak, LeBoeuf, Nelson, and Simmons (2012) reported seven direct replication attempts of Bem’s retroactive-priming paradigm and found no evidence of the predicted effect across approximately 3,000 participants^[8].

Multi-lab replication of Bem 2011

Bem, Tressoldi, Rabeyron, and Duggan (2015), published in F1000Research, compiled a meta-analysis of 90 replication attempts of the Bem (2011) paradigms drawn from 33 laboratories in 14 countries, including both successful and failed replication attempts identified through systematic search^[9]. They reported a small overall positive effect across the pooled corpus.

The Bem et al. 2015 meta-analysis itself drew methodological discussion. The included corpus mixed independent direct replications with conceptual and methodological variants, and the file-drawer correction relied on assumptions about unreported attempts. Galak et al. (2012) provide one external benchmark: seven independent attempts at the retroactive-priming paradigm, the most studied of Bem’s nine experiments, reported null results^[8].

The wider reproducibility-crisis context

The Open Science Collaboration (2015) Reproducibility Project: Psychology attempted direct replications of 100 mainstream studies drawn from three high-impact psychology journals, including the journal that published Bem (2011); 36 of 100 replications produced statistically significant results in the original direction, and average effect sizes in replications were roughly half those of original studies^[10]. This benchmark situates the Bem 2011 replication exchange within a broader methodological reassessment of social and cognitive psychology, not as an isolated parapsychology dispute.

Bem (2011) is frequently cited as a proximate trigger for the post-2011 methodological tightening in psychology, alongside the Stapel fraud case and the Simmons, Nelson, and Simonsohn “false-positive psychology” paper from the same year. The substantive question of whether psi effects replicate is therefore inseparable from the methodological question of what replication standards apply across psychology as a whole.

The pre-registration era

Pre-registration, in which a study’s hypotheses, sample size, exclusion criteria, and analysis plan are publicly recorded before data collection, became one of the central methodological responses to the post-2011 reassessment. Bem et al. (2015) report pre-registration status as an inclusion variable in their meta-analysis of psi replication attempts^[9].

Pre-registered psi studies remain a small subset of the overall corpus. The question of whether pre-registered direct replications by labs without prior commitment to psi research yield positive effect sizes comparable to the Storm et al. (2010) and Bem et al. (2015) meta-analytic summaries is one specific empirical question on which independent published evidence is still accumulating^[3][9].

Counterarguments and Debate