Hocus Pocus: using comics to promote skepticism about the paranormal

Wiseman, R., Collver, J., Worth, R., & Watt, C. (2021). Hocus Pocus: using comics to promote skepticism about the paranormal. Journal of Science Communication, 20(02), A04. https://doi.org/10.22323/2.20020204

AI Assessment

A science-communication experiment, from the skeptical side of the field, testing whether comics are better than plain text at making people more skeptical about the paranormal. With 517 online participants randomly given a skeptical account in one of three formats (text, a comic, or a comic containing an interactive magic trick), the main prediction largely failed: there were no significant differences between formats in skepticism shift or recall, and the plain text was actually rated more interesting and entertaining than the comics. The positive findings are secondary and correlational, people who already liked comics engaged more with the comic versions and, in the comic condition, shifted more toward skepticism. The design is clean (matched materials, an a-priori speed exclusion, analyses pre-planned to mirror a prior study), but the sample was an opportunistic 24-hour social-media recruitment and the moderator finding is partly post hoc. Auditing skeptic-side research to the same standard as proponent research is deliberate: ESP-Nexus takes no side. This audit reports what the study found and how.

Provenance

DOI. 10.22323/2.20020204 · Journal of Science Communication 2021, 20(02), A04. Submitted 22 July 2020, published 8 March 2021.

Study type. A randomized, three-condition online experiment on attitude change. It tests a science-communication method, not a psi effect, and is written from a skeptical standpoint.

Authors. Richard Wiseman and Caroline Watt (both established researchers who approach the paranormal skeptically) with comic creators Jordan Collver and Rik Worth. Ethics approval was granted by the University of Edinburgh PPLS Research Ethics Committee.

Why it is here. ESP-Nexus audits research from every side of the field to the same standard. Including a skeptic-authored intervention study alongside proponent studies is part of that neutrality.

Source basis. Every figure below is taken from the article’s own Abstract, Methods, and Results.

What the paper reports

Comics have a track record in science education, so the authors asked whether they could also promote skepticism about the paranormal, a novel topic for the comics-in-education literature.¹ Participants rated their prior interest in comics, read the same skeptical account of alleged paranormal phenomena in one of three formats, and then rated their engagement, their shift in skepticism, and their recall. The headline conclusion is cautious: comics may help for certain audiences, but not in general.

The text was rated as more interesting and entertaining than the comics, and participants’ prior interest in comics positively correlated with engagement and shift in skepticism. This suggests that for certain cohorts, comics may be an effective way to promote engagement and attitude change.

How it was run

• Sample. 517 participants, recruited opportunistically via social media over a 24-hour window. Seventeen participants who finished in under three minutes were excluded in advance as too fast to have engaged with the material.
• Conditions. Three stimulus materials, matched on narrative content: a plain Text version, a Comic version, and a Magic Comic that additionally included an interactive mathematical mind-reading trick.
• Measures. Prior interest in comics (before reading), then engagement, shift in skepticism, and recall (after reading).
• Analysis. Pre-planned to mirror the authors’ earlier study (Wiseman, Houstoun, & Watt, 2020), comparing the three conditions and examining the relationship between prior comic interest and the outcomes.

Results, as reported

Values are reproduced from the article’s Abstract and Results. The overall pattern is a null result on the main comparison (format did not change skepticism or recall), with the positive story carried by a correlational moderator: comics helped the people who already liked comics.

Eleven-dimension audit

Pre-registration

Not preregistered, but the analyses were pre-planned to replicate the structure of the authors’ prior 2020 study, which lends the main comparisons an a-priori character. The high-versus-low comic-interest split, however, is described as a follow-up, so the strongest positive statement rests partly on a post-hoc contrast.

Randomization

Participants were allocated across the three matched-content conditions, so between-format differences are interpretable as effects of format rather than content. This is the appropriate experimental control for the question asked.

Sensory leakage

Not applicable; this is an attitude-change study with no anomalous-perception component. The relevant threats are sampling and self-report bias, addressed below.

Blinding

The study is self-administered online, and outcomes are self-reported (engagement, skepticism, recall). There is no rater to blind, but participants were necessarily aware of which format they received, and demand characteristics on self-reported skepticism cannot be excluded.

Optional stopping

Recruitment ran for a fixed 24-hour window rather than stopping when results looked favourable, and the under-three-minute exclusion was set in advance, so optional stopping is not a concern.

Outcome measure

Three self-report outcomes (engagement, skepticism shift, recall). They are reasonable for the question, but they are subjective ratings collected immediately after reading, so they capture short-term self-reported change rather than durable attitude or knowledge shift.

Effect size

The central between-condition effects are null. The reported effects are correlational (prior comic interest with engagement and, in one condition, with skepticism shift) and a post-hoc group difference. These are modest, conditional effects rather than a demonstration that comics outperform text.

Multiple comparisons

Several outcomes were tested across three conditions, along with correlational and post-hoc moderator analyses. The main comparisons are pre-planned, but the positive moderator result emerges from a larger set of contrasts, so it should be read as suggestive rather than confirmatory.

Internal replication

The study mirrors the design of the authors’ 2020 work, giving it a family resemblance to a prior result, but it contains no internal replication sample of its own.

External replication

This is, by the authors’ account, the first empirical test of comics for promoting paranormal skepticism, so it establishes rather than replicates a finding. Independent replication, ideally preregistered and with a general-population sample, is the natural next step.

Transparency

Reasonable: the stimulus materials are provided in an appendix, the pre-planned analysis structure and the speed exclusion are stated, and ethics approval is documented. The honesty of reporting the null main effect and the counter-hypothesis finding (text rated more interesting than the comics) is a credit. The limits are the absence of a preregistration and of a full open-data deposit.

The adversarial record

• The main hypothesis largely failed. Comics did not produce more skepticism shift or better recall than plain text, and the text was actually rated more interesting and entertaining. A fair reading is that the study did not find the effect it set out to find.
• The positive story is conditional and partly post hoc. The supportive conclusion (“for certain cohorts”) rests on a correlation between pre-existing comic interest and outcomes, and on a high-versus-low comic-interest split described as a follow-up. That is a real but weak and self-selected effect.
• Sample. An opportunistic 24-hour social-media recruitment, likely skewed toward the authors’ own audience, limits generalizability, and a sample already inclined to skepticism may leave little room for a skepticism shift to be detected.
• What is done well. Content-matched conditions, an a-priori speed exclusion, analyses pre-planned to mirror prior work, documented ethics approval, and candid reporting of a mostly-null result are all methodological credits. The same standards that make a proponent study credible make this skeptic-side study credible, and by those standards it is an honestly reported near-null.