Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project

Kekecs, Z., Palfi, B., Szaszi, B., Szecsi, P., Zrubka, M., Kovacs, M., … Aczel, B. (2023). Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project. Royal Society Open Science, 10(2), 191375. https://doi.org/10.1098/rsos.191375

AI Assessment

A 10-laboratory, preregistered registered-report replication of Bem’s (2011) precognition experiment, built from the ground up as a credibility showcase — born-open data, tamper-evident software, formal external audit, and a design co-authored by a consensus panel of ESP proponents and opponents. Across 37,836 trials from 2,115 participants it found no above-chance guessing (49.89% versus a 50% chance baseline); all four pre-specified statistical tests supported the no-effect model at the minimum planned sample size. Methodologically it is among the most rigorous studies in this literature; the honest limits it states are that a design powered for a 51% effect cannot exclude a still-smaller one, and that one well-run replication of a single experiment cannot settle the field. This audit describes what the paper reports and how it was run; it takes no position on whether precognition exists.

Provenance

DOI. 10.1098/rsos.191375 · Royal Society Open Science 10(2):191375 (2023) · gold open access (published by the Royal Society under a Creative Commons licence).

Study type. A preregistered Stage-2 Registered Report: a multi-laboratory (10 laboratories, 9 countries) direct replication of Experiment 1 of Bem (2011), paired with a demonstration of a set of research-transparency tools.

Authors. Thirty co-authors led by Zoltan Kekecs, with Balazs Aczel as senior author (Eotvos Lorand University, Budapest, and collaborating sites). The consensus design panel that fixed the protocol comprised 29 experts — 15 who considered ESP plausible and 14 who did not.

Funding. The project was funded by the Bial Foundation (grant no. 122/16); individual co-authors report additional support in the article’s funding statement.

Ethics. Testing was conducted with the approval of the Eotvos Lorand University Faculty of Education and Psychology, and each site’s local requirements.

Data availability. Born-open: raw data were deposited directly and streamed live to a public GitHub repository throughout collection; all data, analysis code, and materials are archived on the Open Science Framework (preregistration osf.io/a6ew3), and the server-side software was version-controlled in a GitLab repository made public after data collection ended.

Source basis. Every figure below was confirmed against the primary article’s own stored full text (the born-digital Royal Society Open Science PDF; DOI DOI-verified against Crossref). The text layer is clean; no extraction defects affect the audited statistics.

What the paper reports

The study asked whether Bem’s (2011) headline precognition finding — that people guess the future random position of an erotic image slightly better than chance — survives when the known sources of bias in psychological research are removed by design. Participants made a two-alternative forced-choice guess (left or right curtain); only after the guess did a random process determine which side actually hid the image.¹ Following Bem, the confirmatory hypothesis test was restricted to the erotic trials, and the models were one-sided (M1: success rate above chance; M0: not above chance). Across 2,115 participants and 37,836 erotic trials the observed success rate was 49.89%, against Bem’s originally reported 53.07%, and all four pre-specified tests supported M0 — no evidence of precognition.²

We found 49.89% successful guesses, while Bem reported 53.07% success rate, with the chance level being 50%… All four tests used for the primary hypothesis testing supported M0, thus the stopping rule was triggered with our minimum sample size achieved.

How it was run

• Design. A preregistered, multi-site direct replication. The protocol, analysis plan, and the wording of every possible conclusion (including the abstract) were fixed in a Stage-1 Registered Report and accepted in principle before any data were collected.
• Consensus design. The protocol was co-designed by a 29-expert panel balanced between proponents (15) and opponents (14) of ESP, so that neither camp could later dismiss the design as stacked against its position.
• Apparatus and fraud protection. All sites ran identical software hosted on a central cloud server (participants interacted through a web browser). The server code was version-controlled in git; an IT auditor could verify at any time that it was unaltered, and the raw data were deposited directly and mirrored live to GitHub, creating a tamper-evident audit trail.
• Participants. 2,115 participants contributed valid data (from 2,220 who began); most (92.62%) were aged 18–29, 67.52% identified as women. Belief in ESP and sensation-seeking were measured; site leaders’ and experimenters’ paranormal belief was scored on the Australian Sheep-Goat Scale.
• Procedure. Each participant completed 36 trials (18 erotic, 18 non-erotic; non-erotic trials retained only to match Bem’s protocol). The target side was randomly determined after each guess.
• Analysis plan. Four tests were run simultaneously — a frequentist mixed-effects logistic regression plus three Bayesian proportion tests with different priors (uniform, a knowledge-based “BUJ” prior, and a replication prior built from Bem’s own data). A conclusion was drawn only if all four agreed; otherwise the result would be declared inconclusive.
• Sequential sampling. Five pre-specified analysis points were set from a minimum of 37,836 erotic trials up to a maximum of 136,080. To make the analysis immune to premature-stopping bias, the Bayesian tests pooled successes across all trials regardless of which participant contributed them, and every completed erotic trial was included whether or not the session finished.
• External audit. Two research auditors (Stephen Baumgart, UC Santa Barbara; Michelangelo Vianello, University of Padova) and one IT auditor (Luca Semenzato, Padova), none of them authors, evaluated protocol delivery, data integrity, and software integrity, and published their reports on OSF.

Results, as reported

Values are reproduced from the article’s Results section and text. A minor internal inconsistency in the paper: Bem’s original erotic-trial count is given as 1,560 (consistent with its own 828 successes + 732 failures) in two places but as “1,650 trials” in the sample-size section; the internally consistent 1,560 is used here. Bem’s rate appears as both 53.1% (rounded) and 53.07%.

Eleven-dimension audit

Pre-registration

Exemplary. This is a Stage-2 Registered Report: the design and analysis plan were preregistered on OSF (osf.io/a6ew3) after in-principle acceptance and before any data collection, and the Stage-1 protocol pre-specified the conclusion text for every possible outcome, including the abstract. Confirmatory analyses were locked in advance and separated explicitly from exploratory ones. This is the strongest form of preregistration available.

Randomization

The target side was randomly determined by the software after each participant’s guess, so at the moment of the guess no target existed to be inferred. There is no participant-allocation randomization to balance (a single within-participant guessing task), and the randomization method was fixed in the preregistered protocol and discussed as an ESP-specific consideration in the supplement.

Sensory leakage

Classical sensory leakage is impossible by design: because the target is generated after the guess, there is no contemporaneous target state that a cue could reveal. The analogous threat in a multi-site online study is manipulation of the software or data, which the tamper-evident architecture (central version-controlled server, born-open data with a GitHub audit trail, independent IT audit) was specifically built to foreclose.

Blinding

No party could know the future-random target, so the outcome-relevant blind is structural. Experimenter and site-leader belief in the paranormal was not merely controlled but measured (ASGS) and tested as a moderator; it showed no effect on participant performance, addressing the “psi experimenter effect” directly. Trial sessions were video-recorded to verify experimenter conduct (the videos were withheld from public release only to protect participant identity).

Optional stopping

A particular strength. The sequential plan had five pre-specified analysis points, and the analysis was deliberately engineered so that stopping could not bias the result: the Bayesian proportion tests pool successes across all trials irrespective of participant, and all completed erotic trials were retained regardless of whether a session finished. Data collection stopped at the first (minimum) analysis point because all four tests already agreed — a stopping rule fixed in advance, not chosen after seeing the data.

Outcome measure

Pre-stated and singular: the success rate on erotic trials only, one-tailed, matching the exact quantity in which Bem reported his original effect. Non-erotic trials were collected to preserve the original protocol but excluded from the confirmatory test by prior rule, removing an obvious researcher degree of freedom.

Effect size

Essentially zero: 49.89% success against a 50% baseline, versus Bem’s 53.07%. The 99.75% confidence interval (49.11%–50.67%) excludes the 51% effect the study was powered to detect. The stated limitation is symmetric and honest: simulations show the design has >95% power to detect a true rate of 51% or higher, but its sensitivity falls away below about 50.7%, and a true rate of 50.2% or lower would still yield strong (false) support for M0 in most experiments — so a very small effect cannot be excluded, only a Bem-sized one.

Multiple comparisons

Handled conservatively. Rather than one test, four had to agree before any conclusion could be drawn, and the sequential confidence intervals were Bonferroni-widened across the five analysis points (99.75% at the first point, widening thereafter). Exploratory analyses were reported separately and explicitly barred from influencing the confirmatory conclusion or abstract.

Internal replication

Built into the design: 10 independent laboratories in 9 countries collected data on the same central software, so the result is a within-study multi-site aggregate rather than a single lab’s finding. Site leaders ranged widely in paranormal belief (ASGS 0–23), reducing the chance that one believer- or skeptic-heavy lab drove the outcome.

External replication

This study is the external replication — a high-powered, adversarially-designed direct replication of Bem (2011) Experiment 1. It did not reproduce the original above-chance effect. The authors are careful that a clean replication of one experiment does not adjudicate the wider precognition literature, which includes a contested meta-analysis reporting small positive effects.⁴

Transparency

The paper’s central subject and its strongest dimension. Beyond preregistration, it demonstrates a stack of credibility tools rarely seen together in psychology: born-open data streamed live to GitHub during collection, preregistered and public analysis code, cloud-hosted tamper-evident software with a git audit trail, and a formal external audit by independent research and IT auditors who published their own reports. The authors also disclose the tools’ limits candidly — a potential conflict of interest (two of the three auditors work at the University of Padova, where one collaborating laboratory was located, and the IT auditor has co-published with that site’s lead researcher), and the withholding of trial-session videos to protect participant identity.

The adversarial record

• Lineage. The target study, Bem’s 2011 “Feeling the Future,” is the paper that helped ignite psychology’s reproducibility crisis: a mainstream-journal report of nine experiments claiming precognition.² The transparent Psi project was conceived as the definitive, bias-controlled test of its Experiment 1.
• Contested literature. Bem’s claim was immediately challenged on statistical grounds — most influentially by Wagenmakers and colleagues, who argued that his analyses exploited researcher degrees of freedom — and later defended by a proponent meta-analysis (Bem, Tressoldi, Rabeyron & Duggan, 2015).³⁴ This registered report was engineered specifically to remove the flexibility the critics named: fixed hypotheses, pooled-trial analysis, a locked stopping rule, and a proponent-plus-opponent consensus design.
• Reading against the study. The limits are the ones the authors themselves state, not hidden flaws. The design cannot exclude a genuinely tiny effect (below roughly 50.7% success), so a committed proponent can argue the true effect is simply smaller than Bem’s and below this study’s resolution. The auditor conflict-of-interest at Padova is disclosed rather than concealed, and the withheld trial videos (for privacy) mean the strongest fraud-protection layer rests partly on trust in the audit. And, as the paper notes, one clean replication of a single experiment “cannot address the research question about the literature as a whole.”