Promoting the Benefits and Clarifying Misconceptions about Preregistration, Preprints, and Open Science for the Cognitive Science of Religion


  • Christopher Kavanagh University of Oxford
  • Rohan Kapitany Keele University



open science, preregistration, preprint, methodology, prepublication, social science, psychology, cognitive science of religion


Adopting newly proposed “open science” reforms to improve transparency and increase rigor is hard and can make us, as researchers, feel vulnerable. Nonetheless, these reforms are vital to improving the overall quality and confidence of our collective research. We have, through our own experiences, found that preregistration and detailed analysis plans can help to identify, and potentially avoid, errors. Prepublication has similarly helped us to collaborate and receive feedback on manuscripts, particularly during prolonged periods of peer-review. The Cognitive Science of Religion (CSR) field is constituted by a diversity of scholars and disciplines, and thus faces somewhat unique challenges in its efforts to establish common practices, standards, and research terminology. In this paper we offer an introduction to the open science reforms of preregistration and prepublication specifically orientated towards the CSR field. We discuss common concerns around these practices, highlighting areas of misunderstanding while conceding and discussing genuine limitations. We conclude with voluntary, low-investment recommendations for best-practices with regards to preregistration and preprints for the field of the Cognitive Science of Religion.

Author Biography

Rohan Kapitany, Keele University

Lecturer in Psychology


Abdill, Richard J. and Ran Blekhman. 2019. “Tracking the Popularity and Outcomes of All BioRxiv Preprints.” BioRxiv, 515643: 1–21. DOI:

Allen, Christopher and David M. A. Mehler. 2019. “Open Science Challenges, Benefits and Tips in Early Career and Beyond.” PLOS Biology 17 (5): e3000246. DOI:

Arjó, Gemma, Manuel Portero, Carme Piñol, Juan Viñas, Xavier Matias-Guiu, Teresa Capell, Andrew Bartholomaeus, Wayne Parrott, and Paul Christou. 2013. “Plurality of Opinion, Scientific Discourse and Pseudoscience: An In-Depth Analysis of the Séralini et al. Study Claiming That RoundupTM Ready Corn or the Herbicide RoundupTM Cause Cancer in Rats.” Transgenic Research 22(2): 255–267. DOI:

Atkinson, Quentin D., Andrew J. Latham, and Joseph A. Watts. 2015. “Are Big Gods a Big Deal in the Emergence of Big Groups?” Religion, Brain & Behavior 5(4): 266–274. DOI:

Baumard, Nicolas, Alexandre Hyafil, Ian Morris, and Pascal Boyer. 2014. “Increased Affluence Explains the Emergence of Ascetic Wisdoms and Moralizing Religions.” Current Biology 25(1): 10–15. DOI:

Begley, C. Glenn and Lee M. Ellis. 2012. “Drug Development: Raise Standards for Preclinical Cancer Research.” Nature 483(7391): 531–533. DOI:

Beheim, Bret, Quentin Atkinson, Joseph Bulbulia, Will Gervais, Russell Gray, Joseph Henrich, Martin Lang et al. 2019. “Corrected Analyses Show That Moralizing Gods Precede Complex Societies but Serious Data Concerns Remain.” Preprint. PsyArXiv, jwa2n: 1–25. DOI:

Bourne, Philip E., Jessica K. Polka, Ronald D. Vale, and Robert Kiley. 2017. “Ten Simple Rules to Consider Regarding Preprint Submission.” PLoS Computational Biology 13(5): e1005473. DOI:

Bulbulia, Joseph, Michael L. Spezio, Richard Sosis, and Wesley J. Wildman. 2016. “Standards for Publishing in Religion, Brain & Behavior.” Religion, Brain & Behavior 6(4): 275–277. DOI:

Button, Katherine S., John P. A. Ioannidis, Claire Mokrysz, Brian A. Nosek, Jonathan Flint, Emma S. J. Robinson, and Marcus R. Munafo. 2013. “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience.” Nature Review Neuroscience 14(5): 365–376. DOI:

Camerer, Colin F., Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler et al. 2016. “Evaluating Replicability of Laboratory Experiments in Economics.” Science 351(6280): 1433–1436. DOI:

Camerer, Colin F., Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A Nosek, and Thomas Pfeiffer. 2018. “Evaluating the Replicability of Social Science Experiments in Nature and Science between 2010 and 2015.” Nature Human Behaviour 2(9): 637. DOI:

Carlson, Jedidiah and Kelley Harris. 2020. “Quantifying and Contextualizing the Impact of BioRxiv Preprints through Social Media Audience Segmentation.” BioRxiv, 2020.03.06.981589v1: 1–45. DOI:

Carneiro, Clarissa F. D., Victor G. S. Queiroz, Thiago C. Moulin, Carlos A. M. Carvalho, Clarissa B. Haas, Danielle Rayêe, David E. Henshall, Evandro A. De-Souza, Felippe Espinelli, and Flávia Z. Boos. 2019. “Comparing Quality of Reporting between Preprints and Peer-Reviewed Articles in the Biomedical Literature.” BioRxiv, 581892v3: 1–15. DOI:

Chambers, Christopher D., Eva Feredoes, Suresh Daniel Muthukumaraswamy, and Peter Etchells. 2014. “Instead of ‘Playing the Game’ It Is Time to Change the Rules: Registered Reports at AIMS Neuroscience and Beyond.” AIMS Neuroscience 1(1): 4–17. DOI:

Charles, Sarah J., James E. Bartlett, Kyle J. Messick, Thomas J. Coleman III, and Alex Uzdavines. 2019. “Researcher Degrees of Freedom in the Psychology of Religion.” The International Journal for the Psychology of Religion 29(4): 230–245. DOI:

Christensen, Garret, Zenan Wang, Elizabeth Levy Paluck, Nicholas Swanson, David J. Birke, Edward Miguel, and Rebecca Littman. 2019. “Open Science Practices Are on the Rise: The State of Social Science (3S) Survey.” Preprint. MetaArXiv, 5rksu: 1–64. DOI:

Claesen, Aline, Sara Lucia Brazuna Tavares Gomes, Francis Tuerlinckx, and Wolf Vanpaemel. 2019. “Preregistration: Comparing Dream to Reality.” Preprint. PsyArXiv, d8wex: 1–24. DOI:

Cohen, Jacob. 1994. “The Earth Is Round (ptextless.05).” American Psychologist 49(12): 997–1003. DOI:

Complexity Science Hub Vienna. 2019. “Complex Societies Gave Birth to Big Gods, Not the Other Way Around.” Science Daily. 20 March.

Errington, Timothy M., Elizabeth Iorns, William Gunn, Fraser Elisabeth Tan, Joelle Lomax, and Brian A. Nosek. 2014. “Science Forum: An Open Investigation of the Reproducibility of Cancer Biology Research.” Elife 3: e04333. DOI:

Fanelli, Daniele. 2012. “Negative Results Are Disappearing from Most Disciplines and Countries.” Scientometrics 90(3): 891–904. DOI:

Fecher, Benedikt and Sascha Friesike. 2014. “Open Science: One Term, Five Schools of Thought.” In Opening Science, edited by S. Bartling and S. Friesike, 17–47. Basel: Springer. DOI:

Ferguson, Christopher J. and Moritz Heene. 2012. “A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science’s Aversion to the Null.” Perspectives on Psychological Science 7(6): 555–561. DOI:

Fiedler, Klaus and Norbert Schwarz. 2016. “Questionable Research Practices Revisited.” Social Psychological and Personality Science 7(1): 45–52. Fiske, Susan T. 2016. “A Call to Change Science’s Culture of Shaming.” APS Observer 29(9). DOI:

FOSTER Open Science. 2020. “About FOSTER.” Fosteropenscience.Eu. 2020.

Gelman, Andrew and Eric Loken. 2013. “The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No ‘Fishing Expedition’ or ‘p-Hacking’ and the Research Hypothesis Was Posited Ahead of Time.” Department of Statistics, Columbia University.

Gervais, Will. 2017. “Post Publication Peer Review.” WillGervais.Com . 2 March.

Gervais, Will and Ara Norenzayan. 2012. “Analytic Thinking Promotes Religious Disbelief.” Science 336(6080): 493–496. DOI:

———. 2018. “Analytic Atheism Revisited.” Nature Human Behaviour 2(9): 609. DOI:

Giles, Jim. 2003. “Preprint Server Seeks Way to Halt Plagiarists.” Nature 426(6962): 7. DOI:

Ginsparg, Paul. 2011. “ArXiv at 20.” Nature 476(August): 145–147. DOI:

Goldin-Meadow, Susan. 2016. “Why Preregistration Makes Me Nervous.” APS Observer 29 (7).

Gomes, Cristina M. and Michael E. McCullough. 2015. “The Effects of Implicit Religious Primes on Dictator Game Allocations: A Preregistered Replication Experiment.” Journal of Experimental Psychology: General 144(6): e94. DOI:

Gregory, Justin P. and Tyler S. Greenway. 2017. “The Mnemonic of Intuitive Ontology Violation Is Not the Distinctiveness Effect: Evidence from a Broad Age Spectrum of Persons in the UK and China during a Free-Recall Task.” Journal of Cognition and Culture 17(1–2): 169–197. DOI:

Hagger, Martin S., Nikos L. D. Chatzisarantis, Hugo Alberts, Calvin O. Anggono, Cedric Batailler, Angela R. Birt, Ralf Brand, Mark J. Brandt, Gene Brewer, and Sabrina Bruyneel. 2016. “A Multilab Preregistered Replication of the Ego-Depletion Effect.” Perspectives on Psychological Science 11(4): 546–573. DOI:

Hoogeveen, S., E. J. Wagenmakers, A. C. Kay, and M. van Elk. 2018. “Compensatory Control and Religious Beliefs: A Registered Replication Report across Two Countries.” Comprehensive Results in Social Psychology 3(3): 240–265. DOI:

Hoogeveen, Suzanne, Lukas Snoek, and M. van Elk. 2020. “Religious Belief and Cognitive Conflict Sensitivity: A Preregistered FMRI Study.” Cortex 129(August): 247–265. DOI:

Ioannidis, John P. A. 2005a. “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research.” Journal of the American Medical Association 294(2): 218–228. DOI:

———. 2005b. “Why Most Published Research Findings Are False.” PLoS Medicine 2(8): e124: 0696-0701. DOI:

Jerome, F. 1989. “Science by Press Conference.” Technology Review 42(5): 72–73.

John, Leslie K., George Loewenstein, and Drazen Prelec. 2012. “Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling.” Psychological Science 23(5): 524–532. DOI:

Kaplan, Robert M. and Veronica L. Irvin. 2015. “Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time.” PLoS ONE 10(8): e0132382. DOI:

Kerr, Norbert L. 1998. “HARKing: Hypothesizing after the Results Are Known.” Personality and Social Psychology Review 2(3): 196–217. DOI:

Kidwell, Mallory C., Ljiljana B. Lazarevi?, Erica Baranski, Tom E. Hardwicke, Sarah Piechowski, Lina-Sophia Falkenberg, Curtis Kennett, Agnieszka Slowik, Carina Sonnleitner, and Chelsey Hess-Holden. 2016. “Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency.” PLoS Biology 14(5): e1002456. DOI:

Kirkham, Jamie J., Naomi Penfold, Fiona Murphy, Isabelle Boutron, John PA Ioannidis, Jessica K. Polka, and David Moher. 2020. “A Systematic Examination of Preprint Platforms for Use in the Medical and Biomedical Sciences Setting.” BioRxiv, April, 2020.04.27.063578. DOI:

Klein, Richard A., Michelangelo Vianello, Fred Hasselman, Byron G. Adams, Reginald B. Adams, Sinan Alper, Mark Aveyard et al. 2018. “Many Labs 2: Investigating Variation in Replicability Across Samples and Settings.” Advances in Methods and Practices in Psychological Science 1(4): 443–490. DOI:

Koole, Sander L. and Daniël Lakens. 2012. “Rewarding Replications: A Sure and Simple Way to Improve Psychological Science.” Perspectives on Psychological Science 7(6): 608–614. DOI:

Kunda, Ziva. 1987. “Motivated Inference: Self-Serving Generation and Evaluation of Causal Theories.” Journal of Personality and Social Psychology 53(4): 636–647. DOI:

Laakso, Mikael. 2014. “Green Open Access Policies of Scholarly Journal Publishers: A Study of What, When, and Where Self-Archiving Is Allowed.” Scientometrics 99(2): 475–494. DOI:

Lakens, Daniel. 2019. “The Value of Preregistration for Psychological Science: A Conceptual Analysis.” Japanese Psychological Review 62(3): 221–230. DOI:

Levy Paluck, Betsy. 2018. “Open Science Practices Are on the Rise across Four Social Science Disciplines.” Presented at the 2018 Berkeley Initiative for Transparency in the Social Sciences, David Brower Center, Berkeley, 30 November.

Martin, Luther H. and Donald Wiebe. 2017. Religion Explained?: The Cognitive Science of Religion after Twenty-Five Years. London, UK: Bloomsbury.

Matthews, Robert. 2000. “Storks Deliver Babies (P=0.008).” Teaching Statistics 22(2): 36–38. DOI:

Meehl, Paul E. 1967. “Theory-Testing in Psychology and Physics: A Methodological Paradox.” Philosophy of Science 34(2): 103–115. DOI:

Miyatake, Sanae and Masataka Higuchi. 2017. “Does Religious Priming Increase the Prosocial Behaviour of a Japanese Sample in an Anonymous Economic Game?” Asian Journal of Social Psychology 20(1): 54–59. DOI:

Moore, Andrew. 2006. “Bad Science in the Headlines: Who Takes Responsibility When Science Is Distorted in the Mass Media?” EMBO Reports 7(12): 1193–1196. DOI:

Nickerson, Raymond S. 1998. “Confirmation Bias: A Ubiquitous Phenomenon in Many Guises.” Review of General Psychology 2(2): 175–220. DOI:

Norenzayan, Ara. 2013. Big Gods: How Religion Transformed Cooperation and Conflict. Princeton, NJ: Princeton University Press. DOI:

Nosek, Brian A., George Alter, George C. Banks, Denny Borsboom, Sara D. Bowman, Steven J. Breckler, Stuart Buck, Christopher D. Chambers, Gilbert Chin, and Garret Christensen. 2015. “Promoting an Open Research Culture.” Science 348(6242): 1422–1425. DOI:

———. 2016. “Transparency and Openness Promotion (TOP) Guidelines.” Center for Open Science.

Nosek, Brian A., Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. “The Preregistration Revolution.” Proceedings of the National Academy of Sciences 115(11): 2600–2606. DOI:

Nosek, Brian A. and Daniël Lakens. 2014. “Registered Reports.” Social Psychology 45(3): 137–141. DOI:

Nosek, Brian A., Jeffrey R. Spies, and Matt Motyl. 2012. “Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth over Publishability.” Perspectives on Psychological Science 7(6): 615–631. DOI:

Open Science Collaboration. 2012. “An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science.” Perspectives on Psychological Science 7(6): 657–660. DOI:

Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349(6251): aac4716. DOI:

Pfungst, Oskar. 1911. Clever Hans: (The Horse of Mr. Von Osten.) a Contribution to Experimental Animal and Human Psychology. New York: Holt, Rinehart and Winston. DOI:

Powell, Kendall. 2016. “Does It Take Too Long to Publish Research?” Nature News 530(7589): 148–151. DOI:

Purzycki, Benjamin Grant and Aiyana K. Willard. 2015. “Accounting for Variation and Stability in Religious Cognition.” Religion, Brain & Behavior, April, 60–68. DOI:

Quintana, Daniel S. and Nhat Trung Doan. 2016. “Twitter Article Mentions and Citations: An Exploratory Analysis of Publications in the American Journal of Psychiatry.” American Journal of Psychiatry 173(2): 194. DOI:

Ranehill, Eva, Anna Dreber, Magnus Johannesson, Susanne Leiberg, Sunhae Sul, and Roberto A Weber. 2015. “Assessing the Robustness of Power Posing: No Effect on Hormones and Risk Tolerance in a Large Sample of Men and Women.” Psychological Science 26(5): 653–656. DOI:

Retraction Watch. 2020. “Retracted Coronavirus (COVID-19) Papers.” Retraction Watch.

Ritchie, Stuart. 2020. Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth. London: The Bodley Head.

Rödder, Simone, Martina Franzen, and Peter Weingart. 2011. The Sciences’ Media Connection–Public Communication and Its Repercussions. Berlin: Springer Science and Business Media. DOI:

Rosenthal, Robert. 1979. “The File Drawer Problem and Tolerance for Null Results.” Psychological Bulletin 86(3): 638. DOI:

Sanchez, Clinton, Brian Sundermeier, Kenneth Gray, and Robert J. Calin-Jageman. 2017. “Direct Replication of Gervais & Norenzayan (2012): No Evidence That Analytic Thinking Decreases Religious Belief.” PLoS ONE 12(2): e0172636. DOI:

Savage, Patrick. 2019. “Additional Robustness Analyses Confirm That Complex Societies Precede Moralizing Gods throughout World History.” Nature Ecology & Evolution Community (blog). 5 May.

Savage, Patrick E., Harvey Whitehouse, Pieter François, Thomas E. Currie, Kevin C. Feeney, Enrico Cioni, Rosalind Purcell et al. 2019. “Reply to Beheim et al.: Reanalyses confirm robustness of original analyses.” SocArXiv, xjryt: V1 & V2. DOI:

Scheel, Anne M., Mitchell Schijen, and Daniël Lakens. 2020. “An Excess of Positive Results: Comparing the Standard Psychology Literature with Registered Reports.” PsyArXiv p6e9c: 1–14. DOI:

Schijen, M., A. Scheel, and D. Lakens. 2019, March. Positive Result Rates in Psychology: Registered Reports Compared to the Conventional Literature. Presented at the Open Science 2019, Trier, Germany.

Schiltz, Marc. 2018. “Science without Publication Paywalls: COAlition S for the Realisation of Full and Immediate Open Access.” PLoS Medicine 15(9): e1002663 DOI:

Serghiou, Stylianos and John P. A. Ioannidis. 2018. “Altmetric Scores, Citations, and Publication of Studies Posted as Preprints.” Jama 319(4): 402–404. DOI:

Shariff, Azim F., Aiyana K. Willard, Teresa Andersen and Ara Norenzayan. 2015. “Religious Priming: A Meta-Analysis with a Focus on Prosociality.” Personality and Social Psychology Review 20(1): 1–22. DOI:

Shuai, Xin, Alberto Pepe, and Johan Bollen. 2012. “How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations.” PloS One 7(11): e47523: 1–8. DOI:

Simmons, Joseph P., Leif D. Nelson, and Uri Simonsohn. 2011. “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science 22(11): 1359–1366. DOI:

Simonsohn, Uri, Leif D. Nelson, and Joseph P. Simmons. 2014. “P-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results.” Perspectives on Psychological Science 9(6): 666–681. DOI:

Slingerland, Edward, M. Willis Monroe, Rachel Spicer, and Michael Muthukrishna. 2019. “Historians Respond to Whitehouse et al. (2019), ‘Complex Societies Precede Moralizing Gods throughout World History’.” PsyArXiv 2amjz: 1–33. DOI:

Stroebe, W. and Fritz Strack. 2014. “The Alleged Crisis and the Illusion of Exact Replication.” Perspectives on Psychological Science 9(1): 59–71. DOI:

Turchin, Peter. 2019. “They Call It ‘Coding Errors’?” Cliodynamica. 4 May.

Tversky, Amos and Daniel Kahneman. 1971. “Belief in the Law of Small Numbers.” Psychological Bulletin 76(2): 105. DOI:

Vale, R. D. and A. A. Hyman. 2016. “Point of View: Priority of Discovery in the Life Sciences.” Elife 5, e16931: 1–5. DOI:

van Elk, Michiel. 2019. “Replication and Open Science in the Psychology of Religion: Editorial to the Special Issue.” The International Journal for the Psychology of Religion 29(4): 227–229. DOI:

van Elk, Michiel. 2020. “What’s Hidden in My Filedrawer and What’s in Yours? Disclosing Non-Published Findings in the Cognitive Science of Religion.” Religion, Brain & Behavior, 1–12. DOI:

van Elk, Michiel, Dora Matzke, Quentin F. Gronau, Maime Guan, Joachim Vandekerckhove, and Eric-Jan Wagenmakers. 2015. “Meta-Analyses Are No Substitute for Registered Replications: A Skeptical Perspective on Religious Priming.” Frontiers in Psychology 6(1365): 1–7. DOI:

Wade, Lizzie. 2019. “Did Judgmental Gods Help Societies Grow?” ScienceMag.Org. 20 March.

Walker, Richard and Pascal Rocha da Silva. 2015. “Emerging Trends in Peer Review—a Survey.” Frontiers in Neuroscience 9(169): 1–18. DOI:

Watts, Joseph A., S. J. Greenhill, Quentin D. Atkinson, Thomas E. Currie, Joseph A. Bulbulia, and R. D. Gray. 2015. “Broad Supernatural Punishment but Not Moralizing High Gods Precede the Evolution of Political Complexity in Austronesia.” Proceedings of the Royal Society B: Biological Sciences 282(1804): 1–7. DOI:

Weingart, Peter. 2012. “The Lure of the Mass Media and Its Repercussions on Science.” In The Sciences’ Media Connection–Public Communication and Its Repercussions, Sociology of the Sciences Yearbook, Volume 28, edited by S. Rödder, M. Franzen and P. Weingart, 17–32. Dordrecht: Springer. DOI:

Whitehouse, Harvey and Pieter Francois. 2019. “A Bad Time for (Some) Theories but a Good Time for History?” Cliodynamica. 5 May.

Whitehouse, Harvey, Pieter François, Patrick Savage, Thomas E Currie, Kevin C Feeney, Enrico Cioni, Rosalind Purcell, Robert M Ross, Jennifer Larson and John Baines. 2019. “Complex Societies Precede Moralizing Gods throughout World History.” Nature 568(7751): 226–229. DOI:

Woloshin, Steven, Lisa M. Schwartz, Samuel L. Casella, Abigail T. Kennedy, and Robin J. Larson. 2009. “Press Releases by Academic Medical Centers: Not So Academic? Press Releases by Academic Medical Centers.” Annals of Internal Medicine 150(9): 613–618. DOI:



How to Cite

Kavanagh, C., & Kapitany, R. (2021). Promoting the Benefits and Clarifying Misconceptions about Preregistration, Preprints, and Open Science for the Cognitive Science of Religion. Journal for the Cognitive Science of Religion, 6(1-2), 130–157.