Advancing the Cognitive Science of Religion through Replication and Open Science

Authors

  • Suzanne Hoogeveen University of Amsterdam
  • Michiel van Elk Leiden University

DOI:

https://doi.org/10.1558/jcsr.39039

Keywords:

open science, replication, cognitive science of religion

Abstract

The Cognitive Science of Religion (CSR) is a relatively young but prolific field that has offered compelling insights into religious minds and practices. However, many empirical findings within this field are still preliminary and their reliability remains to be determined. In this paper, we first argue that it is crucial to critically evaluate the CSR literature and adopt  open science practices and replication research in particular to move the field forward. Second, we highlight the outcomes of previous replications and make suggestions for future replication studies in the CSR, with a particular focus on neuroscience, developmental psychology, and qualitative research. Finally, we provide a “replication script” with advice on how to select, conduct, and organize replication research. Our approach is illustrated with a “glimpse behind the scenes” of the recently launched Cross-Cultural Religious Replication Project, in the hope of inspiring scholars of religion to embrace open science and replication in their own research.

Author Biographies

Suzanne Hoogeveen, University of Amsterdam

Suzanne Hoogeveen, Department of Social Psychology, PhD Candidate, University of Amsterdam, The Netherlands.

Michiel van Elk, Leiden University

Michiel van Elk obtained degrees in philosophy (MA), biological psychology (MSc) and the psychology of religion (MSc) in Utrecht, Amsterdam and Nijmegen. He completed his PhD in Cognitive Neuroscience at the Donders Institute in Nijmegen (cum laude). He worked as a visiting researcher at the University of California Santa Barbara (2010), as a Marie Curie post-dotoral fellow at the École Polytechnique Fédérale de Lausanne in Switzerland (2010-2012), as a Fulbright Scholar at Stanford University (2017), as a Research Fellow at the Netherlands Institute for Advanced Studies (NIAS) in Amsterdam (2019-2010) and as a researcher at the University of Amsterdam (2013-2020). Since 2020 Michiel is affiliated to the University of Leiden and supported by grants from the Templeton Foundation, NWO and the BIAL Foundation he supervises the Religion Cognition and Behavior Lab. Michiel has published several popular science books on such different topics as the Babybrain, the Evolution of Religion, Ecstatic Experiences and Psychedelics.

References

Anczyk, A., H. Grzymala-Moszczynska, A. Krzysztof-Swiderska, and J. Prusak. 2019. "The Replication Crisis and Qualitative Research in the Psychology of Religion." The International Journal for the Psychology of Religion 29: 278-291. https://doi.org/10.1080/10508619.2019.1687197

Anderson, M. S., B. C. Martinson and R. De Vries. 2007. "Normative Dissonance in Science: Results from a National Survey of U.S. Scientists." Journal of Empirical Research on Human Research Ethics 2: 3-14. https://doi.org/10.1525/jer.2007.2.4.3

Baker, M. 2016. "Is There a Reproducibility Crisis?" Nature 533: 452-454.

Bakker, M., C. H. J. Hartgerink, J. M. Wicherts, and H. L. J. van der Maas. 2016. "Researchers' Intuitions about Power in Psychological Research." Psychological Science 27: 1069-1077. https://doi.org/10.1177/0956797616647519

Banerjee, K. and P. Bloom. 2013. "Would Tarzan Believe in God? Conditions for the Emergence of Religious Belief." Trends in Cognitive Sciences 17: 7-8. https://doi.org/10. 1016/j.tics.2012.11.005

Barch, D. M. and T. Yarkoni. 2013. "Introduction to the Special Issue on Reliability and Replication in Cognitive and Affective Neuroscience Research." Cognitive, Affective, and Behavioral Neuroscience 13: 687-689. https://doi.org/10.3758/s13415-013-0201-7

Barrett, J. L. 1998. "Cognitive Constraints on Hindu Concepts of the Divine." Journal for the Scientific Study of Religion 37(4): 608-619. https://doi.org/10.2307/1388144

------. 2000. "Exploring the Natural Foundations of Religion." Trends in Cognitive Sciences 4: 29-34. https://doi.org/10.1016/S1364-6613(99)01419-9

------. 2018. "Some Common Misunderstandings about Cognitive Approaches to the Study of Religion: A Reply to Sterelny." Religion, Brain & Behavior 8: 425-428. https: //doi.org/10.1080/2153599X.2017.1323787

Barrett, J. L., R. M. Newman, and R. Richert. 2003. "When Seeing Is Not Believing: Children's Understanding of Humans' and Non-Humans' Use of Background Knowledge in Interpreting Visual Displays." Journal of Cognition and Culture 3: 91-108. https: //doi.org/10.1163/156853703321598590

Barrett, J. L., R. A. Richert, and A. Driesenga. 2001. "God's Beliefs Versus Mother's: The Development of Nonhuman Agent Concepts." Child Development 72: 50-65. https: //doi.org/10.1111/1467-8624.00265

Bem, D. J. 2011. "Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect." Journal of Personality and Social Psychology 100: 407-425. https://doi.org/10.1037/a0021524

Bering, J. M. 2006. "The Folk Psychology of Souls." Behavioral and Brain Sciences 29: 453- 462. https://doi.org/10.1017/S0140525X06009101

Bishop, L. 2009. "Ethical Sharing and Reuse of Qualitative Data." Australian Journal of Social Issues 44: 255-272. https://doi.org/10.1002/j.1839-4655.2009.tb00145.x

Bloom, P. 2005. Descartes' Baby: How the Science of Child Development Explains What Makes Us Human. New York: Random House.

------. 2007. "Religion Is Natural." Developmental Science 10: 147-151. https://doi.org/ 10.1111/j.1467-7687.2007.00577.x

Boekel, W., E. Wagenmakers, L. Belay, A. J. Verhagen, S. D. Brown, and B. Forstmann. 2015. "A Purely Confirmatory Replication Study of Structural Brain-Behavior Correlations." Cortex 66: 115-133. https://doi.org/10.1016/j.cortex.2014.11.019

Bonett, D. G. 2012. "Replication-Extension Studies." Current Directions in Psychological Science 21: 409-412. https://doi.org/10.1177/0963721412459512

Botvinik-Nezer, R., F. Holzmeister, C. F. Camerer, A. Dreber, J. Huber, M. Johannesson, M. Kirchler, R. Iwanir, J. A. Mumford, R. A. Adcock, P. Avesani, B. M. Baczkowski, A. Bajracharya, L. Bakst, S. Ball, M. Barilari, N. Bault, D. Beaton, J. Beitner, and T. Schonberg. 2020. "Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams." Nature 582: 84-88. https://doi.org/10.1038/s41586-0202314-9

Botvinik-Nezer, R., R. Iwanir, F. Holzmeister, J. Huber, M. Johannesson, M. Kirchler, A. Dreber, C. F. Camerer, R. A. Poldrack, and T. Schonberg. 2019. "fMRI Data of Mixed Gambles from the Neuroimaging Analysis Replication and Prediction Study." Scientific Data 6: 106. https://doi.org/10.1038/s41597-019-0113-7

Bulbulia, J. 2013. "The Arts Transform the Cognitive Science of Religion." Journal for the Cognitive Science of Religion 1(2): 141-160. https://doi.org/10.1558/jcsr.v1i2.141

Bulbulia, J. and E. Slingerland. 2012. "Religious Studies as a Life Science." Numen. International Review for the History of Religions 59: 564-613. https://doi.org/10. 1163/15685276-12341240

Button, K. S., J. P. A. Ioannidis, C. Mokrysz, B. A. Nosek, J. Flint, E. S. J. Robinson, and M. R. Munafo. 2013. "Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience." Nature Reviews Neuroscience 14: 365-376. https://doi.org/10.1038/nrn3475

Camerer, C. F., A. Dreber, F. Holzmeister, T. Ho, J. Huber, M. Johannesson, M. Kirchler, G. Nave, B. Nosek, T. Pfeiffer, A. Altmejd, N. Buttrick, T. Chan, Y. Chen, E. Forsell, E. Heikensten, L. Hummer, T. Imai, S. Isaksson, and H. Wu. 2018. "Evaluating Replicability of Social Science Experiments in Nature and Science." Nature Human Behaviour 2: 637-644.

Camerer, C. F., A. Dreber, T.-H. Ho, J. Huber, M. Johannesson, M. Kirchler, J. Almenberg, A. Altmejd, T. Chan, E. Heikensten, F. Holzmeister, T. Imai, S. Isaksson, G. Nave, T. Pfeiffer, and M. Razen. 2016. "Evaluating Replicability of Laboratory Experiments in Economics." Science 351: 1433-1436. https://doi.org/10.1126/science.aaf0918

Cesario, J. 2014. "Priming, Replication, and the Hardest Science." Perspectives on Psychological Science 9: 40-48. https://doi.org/10.1177/1745691613513470

Chambers, C. D. 2013. "Registered Reports: A New Publishing Initiative at Cortex." Cortex 49: 609-610. https://doi.org/10.1016/j.cortex.2012.12.016

Charles, S. J., J. E. Bartlett, K. J. Messick, T. J. Coleman III, and A. Uzdavines. 2019. "Researcher Degrees of Freedom in the Psychology of Religion." The International Journal for the Psychology of Religion 29: 230-245. https://doi.org/10.1080/10508619.2019.1660573

Chartier, C., M. Kline, R. McCarthy, M. Nuijten, D. J. Dunleavy, and A. Ledgerwood. 2018. "The Cooperative Revolution Is Making Psychological Science Better." APS Observer (30 November). https://doi.org/10.31234/osf.io/zu7sj

Chatard, A., G. Hirschberger and T. Pyszczynski. 2020. "A Word of Caution about Many Labs 4: If You Fail to Follow Your Preregistered Plan, You May Fail to Find a Real Effect." (Preprint). PsyArXiv. https://doi.org/10.31234/osf.io/ejubn

Claesen, A., S. L. B. T. Gomes, F. Tuerlinckx, and W. Vanpaemel. 2019. "Preregistration: Comparing Dream to Reality." PsyArXiv. https://doi.org/10.31234/osf.io/d8wex

Consortium, T. M. 2020. "Quantifying Sources of Variability in Infancy Research Using the Infant-Directed-Speech Preference." Advances in Methods and Practices in Psychological Science 3(1): 24-52. https://doi.org/10.1177/2515245919900809

Corriveau, K. H., E. E. Chen, and P. L. Harris. 2015. "Judgments about Fact and Fiction by Children from Religious and Nonreligious Backgrounds." Cognitive Science 39: 353-382. https://doi.org/10.1111/cogs.12138

Cova, F., B. Strickland, A. Abatista, A. Allard, J. Andow, M. Attie, J. Beebe, R. Berniunas, J. Boudesseul, M. Colombo, F. Cushman, R. Diaz, N'D. N. van Dongen, V. Dranseika, B. D. Earp, A. G. Torres, I. Hannikainen, J. V. HernandezConde, W. Hu, and X. Zhou. 2018. "Estimating the Reproducibility of Experimental Philosophy." Review of Philosophy and Psychology (14 June): 1-36. https://doi.org/10.1007/s13164-018-0400-9

Cristia, A., A. Seidl, L. Singh, and D. Houston. 2016. "Test-Retest Reliability in Infant Speech Perception Tasks." Infancy 21: 648-667. https://doi.org/10.1111/infa.12127

Cruwell, S., J. van Doorn, A. Etz, M. C. Makel, H. Moshontz, J. C. Niebaum, A. Orben, S. Parsons, and M. Schulte-Mecklenbeck. 2019. "Seven Easy Steps to Open Science." Zeitschrift fur Psychologie 227: 237-248. https://doi.org/10.1027/2151-2604/ a000387

de Rijcke, S. and B. Penders. 2018. "Resist Calls for Replicability in the Humanities." Nature 560: 29. https://doi.org/10.1038/d41586-018-05845-z

DeHaven, A. C. 2017. Preregistration: A Plan, Not a Prison. 23 May. Center for Open Science. https://www.cos.io/blog/preregistration-plan-not-prison

Dienes, Z. 2014. "Using Bayes to Get the Most Out of Non-Significant Results." Frontiers in Psychology 5: 781 (1-17). https://doi.org/10.3389/fpsyg.2014.00781

Doyen, S., O. Klein, C. L. Pichon, and A. Cleeremans. 2012. "Behavioral Priming: It's All in the Mind, but Whose Mind?" PLoS ONE 7(1): 1-7. e29081. https://doi.org/10.1371/journal. pone.0029081

Dreber, A., T. Pfeiffer, J. Almenberg, S. Isaksson, B. Wilson, Y. Chen, B. A. Nosek, and M. Johannesson. 2015. "Using Prediction Markets to Estimate the Reproducibility of Scientific Research." Proceedings of the National Academy of Sciences of the United States of America 112: 15343-15347. https://doi.org/10.1073/pnas.1516179112

Dubois, J. and R. Adolphs. 2016. "Building a Science of Individual Differences from fMRI." Trends in Cognitive Sciences 20: 425-443. https://doi.org/10.1016/j.tics.2016.03. 014

Dutilh, G., A. Sarafoglou, and E.-J. Wagenmakers. 2019. "Flexible Yet Fair: Blinding Analyses in Experimental Psychology." Synthese (19 November): 1-28. https://doi.org/10.1007/s11229-01902456-7

Ebersole, C., O. Atherton, A. Belanger, H. Skulborstad, J. Allen, J. Banks, E. Baranski, M. Bernstein, D. Bonfiglio, L. Boucher, E. Brown, N. Budiman, A. Cairo, C. Capaldi, C. Chartier, J. Chung, D. Cicero, J. Coleman, J. Conway, and B. Nosek. 2016. "Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication." Journal of Experimental Social Psychology 67: 68-82.

Etz, A., Q. F. Gronau, F. Dablander, P. A. Edelsbrunner, and B. Baribault. 2018. "How to Become a Bayesian in Eight Easy Steps: An Annotated Reading List." Psychonomic Bulletin & Review 25: 219-234.

Evans, E. M. 2001. "Cognitive and Contextual Factors in the Emergence of Diverse Belief Systems: Creation versus Evolution." Cognitive Psychology 42(3): 217-266. https://doi.org/ 10.1006/cogp.2001.0749

Forsell, E., D. Viganola, T. Pfeiffer, J. Almenberg, B. Wilson, Y. Chen, B. A. Nosek, M. Johannesson, and A. Dreber. 2018. "Predicting Replication Outcomes in the Many Labs 2 Study." Journal of Economic Psychology 75: 1-16. https://doi.org/10.1016/j.joep.2018.10.009

Francis, G. 2013. "Replication, Statistical Consistency, and Publication Bias." Journal of Mathematical Psychology 57: 153-169. https://doi.org/10.1016/j.jmp.2013.02.003

Frank, M. C., E. Bergelson, C. Bergmann, A. Cristia, C. Floccia, J. Gervain, J. K. Hamlin, E. E. Hannon, M. Kline, C. Levelt, C. Lew-Williams, T. Nazzi, R. Panneton, H. Rabagliati, M. Soderstrom, J. Sullivan, S. Waxman, and D. Yurovsky. 2017. "A Collaborative Approach to Infant Research: Promoting Reproducibility, Best Practices, and Theory-Building." Infancy 22: 421-435. https://doi.org/10.1111/infa.12182

Gervais, W. M., S. E. McKee, and S. Malik. 2020. "Do Religious Primes Increase Risk Taking? Evidence against 'Anticipating Divine Protection' in Two Preregistered Direct Replications of Kupor, Laurin, and Levav (2015)." Psychological Science 31: 858-864. https://doi.org/10.1177/0956797620922477

Gervais, W. M. and A. Norenzayan. 2012. "Analytic Thinking Promotes Religious Disbelief." Science 336: 493-496. https://doi.org/10.1126/science.1215647

Gervais, W. M., A. K. Willard, A. Norenzayan, and J. Henrich. 2011. "The Cultural Transmission of Faith: Why Innate Intuitions Are Necessary, but Insufficient, to Explain Religious Belief." Religion 41: 389-410. https://doi.org/10.1080/0048721X.2011.604510

Gervais, W. M., D. Xygalatas, R. T. McKay, M. van Elk, E. E. Buchtel, M. Aveyard, S. R. Schiavone, I. Dar-Nimrod, A. M. Svedholm-Hakkinen, T. Riekki, E. Kundtova Klocova, J. E. Ramsay, and J. Bulbulia. 2017. "Global Evidence of Extreme Intuitive Moral Prejudice against Atheists." Nature Human Behaviour 1: 0151 (1-6). https://doi.org/10.1038/s41562-017-0151

Gewin, V. 2016. "Data Sharing: An Open Mind on Open Data." Nature 529: 117-119. https://doi.org/10.1038/nj7584-117a

Gilder, T. S. E. and E. A. Heerey. 2018. "The Role of Experimenter Belief in Social Priming." Psychological Science 29: 403-417. https://doi.org/10.1177/0956797617737128

Gilmore, R. O., J. L. Kennedy and K. E. Adolph. 2018. "Practical Solutions for Sharing Data and Materials from Psychological Research." Advances in Methods and Practices in Psychological Science 1: 121-130. https://doi.org/10.1177/2515245917746500

Gomes, C. M. and M. E. McCullough. 2015. "The Effects of Implicit Religious Primes on Dictator Game Allocations: A Preregistered Replication Experiment." Journal of Experimental Psychology: General 144: e94-e104. https://doi.org/10.1037/xge0000027

Haaf, J. M., S. Hoogeveen, S. Berkhout, Q. F. Gronau, and E.-J. Wagenmakers. 2020. "A Bayesian Multiverse Analysis of Many Labs 4: Quantifying the Evidence against Mortality Salience." (Preprint). PsyArXiv. https://doi.org/10.31234/osf.io/cb9er

Haith, M. M. 1998. "Who Put the Cog in Infant Cognition? Is Rich Interpretation Too Costly?" Infant Behavior and Development 21: 167-179. https://doi.org/10.1016/s01636383(98)90001-7

Hardwicke, T. E., M. B. Mathur, K. MacDonald, G. Nilsonne, G. C. Banks, M. C. Kidwell, A. Hofelich Mohr, E. Clayton, E. J. Yoon, M. Henry Tessler, and M. C. Frank. 2018. "Data Availability, Reusability, and Analytic Reproducibility: Evaluating the Impact of a Mandatory Open Data Policy at the journal Cognition." Royal Society Open Science 5: 180448 (1-18). https://doi.org/10.31222/osf.io/39cfb

Hardwicke, T. E., M. H. Tessler, B. Peloquin, and M. C. Frank. 2018. "A Bayesian Decision-Making Framework for Replication." Behavioral and Brain Sciences 41: e132. https: //doi.org/10.1017/S0140525X18000675

Hardy, A. 1981. The Spiritual Nature of Man: A Study of Contemporary Religious Experience. Oxford: Oxford University Press.

Harris, P. and M. Gimenez. 2005. "Children's Acceptance of Conflicting Testimony: The Case of Death." Journal of Cognition and Culture 5: 143-164. https://doi.org/10.1163/1568537054068606

Harris, S., J. T. Kaplan, A. Curiel, S. Y. Bookheimer, M. Iacoboni, and M. S. Cohen. 2009. "The Neural Correlates of Religious and Nonreligious Belief." PLoS ONE 4(10): e7272 (1-9). https://doi.org/10.1371/journal.pone.0007272

Hone, L. S. E. and M. E. McCullough. 2015. "Does Religious Cognition Really Down-Regulate Hand Grip Endurance in Men? A Failure to Replicate." Evolution and Human Behavior 36: 81-85. https://doi.org/10.1016/j.evolhumbehav.2014.08.007

Hoogeveen, S., A. Sarafoglou, and E.-J. Wagenmakers. 2020. "Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully." Advances in Methods and Practices in Psychological Science, 1-19. https://doi.org/10.1177/2515245920919667

Hoogeveen, S., L. Snoek and M. van Elk. 2020. "Religious Belief and Cognitive Conflict Sensitivity: A Preregistered fMRI Study." Cortex 129: 247-265. https://doi.org/10. 1016/j.cortex.2020.04.011

Hoogeveen, S., E.-J. Wagenmakers, A. C. Kay, and M. van Elk. 2018. "Compensatory Control and Religious Beliefs: A Registered Replication Report across Two Countries." Comprehensive Results in Social Psychology 3: 240-265. https://doi.org/10.1080/ 23743603.2019.1684821

Huber, D. E., K. W. Potter, and L. D. Huszar. 2019. "Less 'Story' and More 'Reliability' in Cognitive Neuroscience." Cortex 113: 347. https://doi.org/10.1016/j.cortex.2018.10.030

Inzlicht, M., I. McGregor, J. B. Hirsh, and K. Nash. 2009. "Neural Markers of Religious Conviction." Psychological Science 20: 385-392. https://doi.org/10.1111/j.14679280.2009.02305.x

Inzlicht, M., A. M. Tullett, and M. Good. 2011. "The Need to Believe: A Neuroscience Account of Religion as a Motivated Process." Religion, Brain & Behavior 1: 192-212. https://doi.org/10.1080/2153599x.2011.647849

Ioannidis, J. P. A. 2005. "Why Most Published Research Findings Are False." PLoS Medicine 2: 696-701. https://doi.org/10.1080/09332480.2005.10722754

JASP Team. 2019. JASP (version 0.11.1) [Computer software].

John, L. K., G. Loewenstein, and D. Prelec. 2012. "Measuring the Prevalence of Questionable Research Practices with Incentives for Truth-Telling." Psychological Science 23: 524-532. https://doi.org/10.1037/e632032012-001

Jong, J., J. Halberstadt, M. Bluemke, C. Kavanagh, and C. Jackson. 2019. "Death Anxiety, Exposure to Death, Mortuary Preferences, and Religiosity in Five Countries." Scientific Data 6: 154 (1-5). https://doi.org/10.1038/s41597-019-0163-x

Kavanagh, C. M., J. Jong, R. McKay, and H. Whitehouse. 2018. "Positive Experiences of High Arousal Martial Arts Rituals Are Linked to Identity Fusion and Costly Pro-Group Actions." European Journal of Social Psychology 49(3): 461-481. https://doi.org/10.1002/ejsp. 2514

Kavanagh, C. M. and R. Kapitany. 2017/2019. "Promoting the Benefits and Clarifying Misconceptions about Preregistration, Preprints, and Open Science for Cognitive Science of Religion." Journal for the Cognitive Science of Religion 5(2): 130-159.

Kay, A. C., D. Gaucher, J. L. Napier, M. J. Callan, and K. Laurin. 2008. "God and the Government: Testing a Compensatory Control Mechanism for the Support of External Systems." Journal of Personality and Social Psychology 95: 18-35. https://doi.org/10.1037/0022-3514.95.1.18

Kelemen, D. 2004. "Are Children 'Intuitive Theists'? Reasoning about Purpose and Design in Nature." Psychological Science 15: 295-301. https://doi.org/10.1111/j.09567976.2004.00672.x

Kidwell, M. C., L. B. Lazarevic, E. Baranski, T. E. Hardwicke, S. Piechowski, L. Falkenberg, C. Kennett, A. Slowik, C. Sonnleitner, C. Hess-Holden, T. M. Errington, S. Fiedler, and B. A. Nosek. 2016. "Badges to Acknowledge Open Practices: A Simple, Low Cost, Effective Method for Increasing Transparency." PLoS Biology 14: e1002456 (1-15). https://doi.org/10.1371/journal.pbio.1002456

Klein, O., T. E. Hardwicke, F. Aust, J. Breuer, H. Danielsson, A. Hofelich Mohr, H. IJzerman, G. Nilsonne, W. Vanpaemel, and M. C. Frank. 2018. "A Practical Guide for Transparency in Psychological Science." Collabra: Psychology 4: 1-15. https://doi.org/10.1525/collabra.158

Klein, R., C. L. Cook, C. R. Ebersole, C. A. Vitiello, B. A. Nosek, C. R. Chartier, C. D. Christopherson, S. Clay, B. Collisson, J. Crawford, R. Cromar, D. Vidamuerte, G. Gardiner, C. Gosnell, J. E. Grahe, C. Hall, J. A. Joy-Gaba, A. M. Legg, C. Levitan, and K. A. Ratliff. 2019. "Many Labs 4: Failure to Replicate Mortality Salience Effect with and without Original Author Involvement." PsyArXiv. https://doi.org/10.31234/osf.io/vef2c

Klein, R., K. Ratliff, M. Vianello, R. B. Adams, Jr., S. Bahnik, M. Bernstein, K. Bocian, M. Brandt, B. Brooks, C. Brumbaugh, Z. Cemalcilar, J. Chandler, W. Cheong, W. Davis, T. Devos, M. Eisner, N. Frankowska, D. Furrow, E. M. Galliani, and B. Nosek. 2014. "Investigating Variation in Replicability: A 'Many Labs' Replication Project." Social Psychology 45: 142-152. https://doi.org/10.1027/1864-9335/a000178

Klein, R., M. Vianello, F. Hasselman, B. Adams, R. Adams, S. Alper, M. Aveyard, J. Axt, S. Bahnik, R. Batra, M. Berkics, M. Bernstein, D. Berry, O. Bialobrzeska, E. Binan, K. Bocian, M. Brandt, R. Busching, A. Redei, and B. Nosek. 2018. "Many Labs 2: Investigating Variation in Replicability across Sample and Setting." Advances in Methods and Practices in Psychological Science 1: 443-490.

Konvalinka, I., D. Xygalatas, J. Bulbulia, U. Schjodt, E.-M. Jegindo, S. Wallot, G. Van Orden, and A. Roepstorff. 2011. "Synchronized Arousal between Performers and Related Spectators in a Fire-Walking Ritual." Proceedings of the National Academy of Sciences 108: 8514-8519. https://doi.org/10.1073/pnas.1016955108

Kupor, D. M., K. Laurin and J. Levav. 2015. "Anticipating Divine Protection? Reminders of God Can Increase Nonmoral Risk Taking." Psychological Science 26: 374-384. https://doi.org/10.1177/0956797614563108

Leshinskaya, A., J. M. Contreras, A. Caramazza, and J. P. Mitchell. 2017. "Neural Representations of Belief Concepts: A Representational Similarity Approach to Social Semantics." Cerebral Cortex 27(1): 344-357. https://doi.org/10.1093/cercor/bhw401

Luhrmann, T. M. 2005. "The Art of Hearing God: Absorption, Dissociation, and Contemporary American Spirituality." Spiritus: A Journal of Christian Spirituality 5: 133-157. https://doi.org/10.1353/scs.2006.0014

------. 2012. When God Talks Back: Understanding the American Evangelical Relationship with God. New York: Vintage.

Luhrmann, T. M., R. Padmavati, H. Tharoor, and A. Osei. 2015. "Hearing Voices in Different Cultures: A Social Kindling Hypothesis." Topics in Cognitive Science 7: 646-663. https://doi.org/10.1111/tops.12158

MacCoun, R. and S. Perlmutter. 2015. "Hide Results to Seek the Truth." Nature 526: 187-189.

Makel, M. C., J. A. Plucker, and B. Hegarty. 2012. "Replications in Psychology Research: How Often Do They Really Occur?" Perspectives on Psychological Science 7: 537- 542. https://doi.org/10.1177/1745691612460688

Makris, N. and D. Pnevmatikos. 2007. "Children's Understanding of Human and Supernatural Mind." Cognitive Development 22: 365-375. https://doi.org/10.1016/j. cogdev.2006.12.003

McCullough, M. E., E. C. Carter, C. N. DeWall, and C. M. Corrales. 2012. "Religious Cognition Down-Regulates Sexually Selected, Characteristically Male Behaviors in Men, but Not in Women." Evolution and Human Behavior 33: 562-568. https://doi.org/10.1016/j.evolhumbehav.2012.02.004

McPhetres, J. 2018. "Religiosity and Confirmation Bias or: How I Learned to Stop Worrying and Love Preregistered Direct Replications." OSF Preprints. https://doi.org/10.31219/ osf.io/g7apd

Moshontz, H., L. Campbell, C. R. Ebersole, H. IJzerman, H. L. Urry, P. S. Forscher, J. E. Grahe, R. J. McCarthy, E. D. Musser, J. Antfolk, C. M. Castille, T. R. Evans, S. Fiedler, J. K. Flake, D. A. Forero, S. M. J. Janssen, J. R. Keene, J. Protzko, B. Aczel, and C. R. Chartier. 2018. "The Psychological Science Accelerator: Advancing Psychology through a Distributed Collaborative Network." Advances in Methods and Practices in Psychological Science 1: 501-515. https://doi.org/10.1177/2515245918797607

Munafo, M. R. and G. D. Smith. 2018. "Robust Research Needs Many Lines of Evidence." Nature 553: 399-401. https://doi.org/10.1038/d41586-018-01023-3

Nakagawa, S. and M. E. Hauber. 2011. "Great Challenges with Few Subjects: Statistical Strategies for Neuroscientists." Neuroscience & Biobehavioral Reviews 35: 462-473. https://doi.org/10.1016/j.neubiorev.2010.06.003

Neubauer, R. L. 2014. "Prayer as an Interpersonal Relationship: A Neuroimaging Study." Religion, Brain & Behavior 4: 92-103. https://doi.org/10.1080/2153599X.2013. 768288

Norenzayan, A., A. F. Shariff, W. M. Gervais, A. K. Willard, R. A. McNamara, E. Slingerland and J. Henrich. 2016. "The Cultural Evolution of Prosocial Religions." Behavioral and Brain Sciences 39 (1-65). https://doi.org/10.1017/S0140525X14001356

Nosek, B. A., G. Alter, G. C. Banks, D. Borsboom, S. D. Bowman, S. J. Breckler, S. Buck, C. D. Chambers, G. Chin, G. Christensen, M. Contestabile, A. Dafoe, E. Eich, J. Freese, R. Glennerster, D. Goroff, D. P. Green, B. Hesse, M. Humphreys, and T. Yarkoni. 2015. "Promoting an Open Research Culture." Science 348: 1422-1425. https://doi.org/10.1126/science.aab2374

Nosek, B. A., C. R. Ebersole, A. C. DeHaven and D. T. Mellor. 2018. "The Preregistration Revolution." Proceedings of the National Academy of Sciences 115: 2600-2606. https: //doi.org/10.1073/pnas.1708274114

Nosek, B. A. and T. M. Errington. 2017. "Making Sense of Replications." eLife 6: e23383. https://doi.org/10.7554/eLife.23383

------. 2020. "What Is Replication?" PLoS Biology 18: e3000691. https://doi.org/10.1371/journal.pbio.3000691

Nosek, B. A. and D. S. Lindsay. 2018. "Preregistration Becoming the Norm in Psychological Science." APS Observer 31.

Open Science Collaboration, T. 2012. "An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science." Perspectives on Psychological Science 7: 657-660. https://doi.org/10.1177/1745691612462588

Open Science Collaboration, T. 2015. "Estimating the Reproducibility of Psychological Science." Science 349(6251): aac4716. https://doi.org/10.1126/science.aac4716

Pan, D. 2017. "Is Chinese Culture Dualist? An Answer to Edward Slingerland from a Medical Philosophical Viewpoint." Journal of the American Academy of Religion 85: 1017-1031. https://doi.org/10.1093/jaarel/lfx028

Pashler, H., D. Rohrer and C. R. Harris. 2013. "Can the Goal of Honesty Be Primed?" Journal of Experimental Social Psychology 49: 959-964. https://doi.org/10.1016/j.jesp. 2013.05.011

Payne, G. and M. Williams. 2005. "Generalization in Qualitative Research." Sociology - British Sociological Association 39: 295-314. https://doi.org/10.1177/0038038505050540

Peels, R. and L. Bouter. 2018a. "The Possibility and Desirability of Replication in the Humanities." Palgrave Communications 4: 95 (1-4). https://doi.org/10.1057/s41599-018-0149-x

------. 2018b. "Replication in Humanities Just as Desirable as in Sciences." LSE Impact Blog. https://blogs.lse.ac.uk/impactofsocialsciences/2018/10/01/replication-is-both-possible-and-desirable-in-the-humanities-just-as-it-is-in-thesciences/

Potter, K. W., L. D. Huszar and D. E. Huber. 2018. "Does Inhibition Cause Forgetting after Selective Retrieval? A Reanalysis and Failure to Replicate." Cortex 104: 26-45. https://doi.org/10.1016/j.cortex.2018.03.026

Power, E. A. 2017. "Discerning Devotion: Testing the Signaling Theory of Religion." Evolution and Human Behavior 38: 82-91. https://doi.org/10.1016/j.evolhumbehav.2016.07. 003

Purzycki, B. G., C. T. Ross, C. Apicella, Q. D. Atkinson, E. Cohen, R. A. McNamara, A. K. Willard, D. Xygalatas, A. Norenzayan and J. Henrich. 2018. "Material Security, Life History, and Moralistic Religions: A Cross-Cultural Examination." PLoS ONE 13: e0193856 (1-14). https://doi.org/10.1371/journal.pone.0193856

Rouder, J. N. 2014. "Optional Stopping: No Problem for Bayesians." Psychonomic Bulletin & Review 21: 301-308. https://doi.org/10.3758/s13423-014-0595-4

Rowhani-Farid, A., A. Aldcroft and A. G. Barnett. 2020. "Did Awarding Badges Increase Data Sharing in BMJ Open? A Randomized Controlled Trial." Royal Society Open Science 7: 191818 (1-12). https://doi.org/10.1098/rsos.191818

Sanchez, C., B. Sundermeier, K. Gray and R. J. Calin-Jageman. 2017. "Direct Replication of Gervais and Norenzayan (2012): No Evidence That Analytic Thinking Decreases Religious Belief." PLoS ONE 12: e0172636 (1-8). https://doi.org/10.1371/journal.pone.0172636

Schjoedt, U. and J. Bulbulia. 2011. "The Need to Believe in Conflicting Propositions." Religion, Brain & Behavior 1: 236-239. https://doi.org/10.1080/2153599X.2011.647857

Schjoedt, U., J. Sorensen, K. L. Nielbo, D. Xygalatas, P. Mitkidis and J. Bulbulia. 2013. "Cognitive Resource Depletion in Religious Interactions." Religion, Brain & Behavior 3: 39-55. https://doi.org/10.1080/2153599X.2012.736714

Schjoedt, U., H. Stodkilde-Jorgensen, A. W. Geertz, T. E. Lund and A. Roepstorff. 2011. "The Power of Charisma - Perceived Charisma Inhibits the Frontal Executive Network of Believers in Intercessory Prayer." Social Cognitive and Affective Neuroscience 6: 119-127. https://doi.org/10.1093/scan/nsq023

Schjoedt, U., H. Stodkilde-Jorgensen, A. W. Geertz and A. Roepstorff. 2009. "Highly Religious Participants Recruit Areas of Social Cognition in Personal Prayer." Social Cognitive and Affective Neuroscience 4: 199-207. https://doi.org/10.1093/scan/nsn050

Schjoedt, U. and M. van Elk. 2019. "The Neuroscience of Religion." In Oxford Handbook of the Cognitive Science of Religion, edited by J. Barrett, 358-378. Oxford: Oxford University Press.

Schmidt, S. 2009. "Shall We Really Do It Again? The Powerful Concept of Replication Is Neglected in the Social Sciences." Review of General Psychology 13: 90-100. https://doi.org/10.1037/a0015108

Schonbrodt, F. D. and M. Perugini. 2013. "At What Sample Size Do Correlations Stabilize?" Journal of Research in Personality 47: 609-612. https://doi.org/10.1016/j.jrp.2013.05.009

Schott, E., M. Rhemtulla and K. Byers-Heinlein. 2018. "Should I Test More Babies? Solutions for Transparent Data Peeking." Infant Behavior and Development 54: 166-176. https://doi.org/10.31234/osf.io/gxfaj

Shanks, D. R., B. R. Newell, E. H. Lee, D. Balakrishnan, L. Ekelund, Z. Cenac, F. Kavvadia and C. Moore. 2013. "Priming Intelligent Behavior: An Elusive Phenomenon." PLoS ONE 8(4): e56515 (1-10). https://doi.org/10.1371/journal.pone.0056515

Shariff, A. F. and A. Norenzayan. 2007. "God Is Watching You: Priming God Concepts Increases Prosocial Behavior in an Anonymous Economic Game." Psychological Science 18: 803-809. https://doi.org/10.1111/j.1467-9280.2007.01983.x

Shariff, A. F., A. K. Willard, T. Andersen and A. Norenzayan. 2016. "Religious Priming: A Meta-Analysis with a Focus on Prosociality." Personality and Social Psychology Review 20: 27-48. https://doi.org/10.1177/1088868314568811

Slingerland, E. and M. Chudek. 2011. "The Prevalence of Mind-Body Dualism in Early China." Cognitive Science 35: 997-1007. https://doi.org/10.1111/j.1551-6709.2011.01186.x

Slingerland, E. and M. Collard. 2011. Creating Consilience: Integrating the Sciences and the Humanities. Oxford: Oxford University Press. https://doi.org/10.1093/acprof:oso/ 9780199794393.001.0001

Smith, S. M., C. F. Beckmann, J. Andersson, E. J. Auerbach, J. Bijsterbosch, G. Douaud, E. Duff, D. A. Feinberg, L. Griffanti, M. P. Harms, M. Kelly, T. Laumann, K. L. Miller, S. Moeller, S. Petersen, J. Power, G. Salimi-Khorshidi, A. Z. Snyder, A. T. Vu, and M. F. Glasser. 2013. "Resting-State fMRI in the Human Connectome Project." NeuroImage 80: 144-168. https://doi.org/10.1016/j.neuroimage.2013.05. 039

Snoek, L., M. M. van der Miesen, T. Beemsterboer, A. van der Leij, A. Eigenhuis, and H. S. Scholte. 2020. "The Amsterdam Open MRI Collection: A Set of Multimodal MRI Datasets for Individual Difference Analyses." bioRxiv. https://doi.org/10.1101/2020.06.16.155317

Spellman, B. A. 2015. "A Short (Personal) Future History of Revolution 2.0." Perspectives on Psychological Science 10: 886-899. https://doi.org/10.1177/1745691615609918

Sterelny, K. 2018. "Religion Re-Explained." Religion, Brain & Behavior 8: 406-425. https://doi.org/10.1080/2153599X.2017.1323779

Szucs, D. and J. P. A. Ioannidis. 2017. "Empirical Assessment of Published Effect Sizes and Power in the Recent Cognitive Neuroscience and Psychology Literature." PLoS Biology 15: e2000797 (1-18). https://doi.org/10.1371/journal.pbio.2000797

Tamminen, K. A. and Z. A. Poucher. 2018. "Open Science in Sport and Exercise Psychology: Review of Current Approaches and Considerations for Qualitative Inquiry." Psychology of Sport and Exercise 36: 17-28. https://doi.org/10.1016/j.psychsport.2017.12.010

Taves, A. 1999. Fits, Trances, & Visions: Experiencing Religion and Explaining Experience from Wesley to James. Princeton: Princeton University Press. https://doi.org/10.2307/2674486

------. 2010. "No Field Is an Island: Fostering Collaboration between the Academic Study of Religion and the Sciences." Method & Theory in the Study of Religion 22: 170-188. https://doi.org/10.1163/157006810X512356

Tierney, W., J. Hardy III, C. Ebersole, D. Viganola, E. Clemente, E. Gordon, S. Hoogveveen, J. M. Haaf, A. Dreber, M. Johannesson, T. Pfeiffer, H. Chapman, A. Gantman, M. Vanaman, K. DeMarree, E. Igou, J. Wylie, J. Storbeck, M. Andreychik, and E. L. Uhlmann. (In press). "A Creative Destruction Approach to Replication: Implicit Work and Sex Morality across Cultures." Journal of Experimental Social Psychology.

Uhlmann, E. L., T. A. Poehlman and J. A. Bargh. 2009. "American Moral Exceptionalism." In Social and Psychological Bases of Ideology and System Justification, edited by J. T. Jost, A. C. Kay and H. Thorisdottir, 27-52. New York, NY, Oxford University Press.

Uhlmann, E. L., T. A. Poehlman, D. Tannenbaum, and J. A. Bargh. 2011. "Implicit Puritanism in American Moral Cognition." Journal of Experimental Social Psychology 47: 312-320. https://doi.org/10.1016/j.jesp.2010.10.013

van Doorn, J., D. van den Bergh, U. Bohm, F. Dablander, K. Derks, T Draws, and E. Wagenmakers. 2019. "The JASP Guidelines for Conducting and Reporting a Bayesian Analysis." PsyArXiv. https://doi.org/10.31234/osf.io/yqxfr

van Elk, M. 2019. "Replication and Open Science in the Psychology of Religion: Editorial to the Special Issue." The International Journal for the Psychology of Religion 29: 227-229. https://doi.org/10.1080/10508619.2019.1687189

van Elk, M., B. T. Rutjens, J. van der Pligt, and F. van Harreveld. 2016. "Priming of Supernatural Agent Concepts and Agency Detection." Religion, Brain & Behavior 6: 4-33.

van Elk, M. and L. Snoek. 2020. "The Relationship between Individual Differences in Grey Matter Volume and Religiosity and Mystical Experiences: A Pre-Registered Voxel-based Morphometry Study." European Journal of Neuroscience 51: 850-865. https://doi.org/10.1111/ejn.14563

van Elk, M. and E.-J. Wagenmakers. 2017. "Can the Experimental Study of Religion Be Advanced Using a Bayesian Predictive Framework?" Religion, Brain & Behavior 7: 331-334. https://doi.org/10.1080/2153599X.2016.1249915

van 't Veer, A. E. and R. Giner-Sorolla. 2016. "Pre-Registration in Social Psychology - A Discussion and Suggested Template." Journal of Experimental Social Psychology 67: 2-12. https://doi.org/10.31234/osf.io/4frms

Vazire, S. 2018. "Implications of the Credibility Revolution for Productivity, Creativity, and Progress." Perspectives on Psychological Science 13: 411-417. https://doi.org/10.31234/osf.io/2yphf

Wagenmakers, E.-J., Q. F. Gronau and J. Vandekerckhove. 2019. "Five Bayesian Intuitions for the Stopping Rule Principle." PsyArXiv. https://doi.org/10.31234/osf.io/5ntkd

Wagenmakers, E.-J., J. Love, M. Marsman, T. Jamil, A. Ly, J. Verhagen, R. Selker, Q. F. Gronau, D. Dropmann, B. Boutin, F. Meerhoff, P. Knight, A. Raj, E.-J. van Kesteren, J. van Doorn, M. Smira, S. Epskamp, A. Etz, D. Matzke, and R. D. Morey. 2018. "Bayesian Inference for Psychology. Part II: Example Applications with JASP." Psychonomic Bulletin & Review 25: 58-76. https://doi.org/10.3758/s13423-017-1323-7

Wagenmakers, E.-J., M. Marsman, T. Jamil, A. Ly, J. Verhagen, J. Love, R. Selker, Q. F. Gronau, M. Smira, S. Epskamp, D. Matzke, J. N. Rouder and R. D. Morey. 2018. "Bayesian Inference for Psychology. Part I: Theoretical Advantages and Practical Ramifications." Psychonomic Bulletin & Review 25: 35-57. https://doi.org/10. 3758/s13423-017-1343-3

Wagenmakers, E.-J., R. Wetzels, D. Borsboom and H. L. J. van der Maas. 2011. "Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi: Comment on Bem." Journal of Personality and Social Psychology 100: 426-432. https://doi.org/10.1037/a0022790

Wagenmakers, E.-J., R. Wetzels, D. Borsboom, H. L. J. van der Maas, and R. A. Kievit. 2012. "An Agenda for Purely Confirmatory Research." Perspectives on Psychological Science 7: 627-633. https://doi.org/10.1177/1745691612463078

Wagge, J. R., M. J. Brandt, L. B. Lazarevic, N. Legate, C. Christopherson, B. Wiggins and J. E. Grahe. 2019. "Publishing Research with Undergraduate Students via Replication Work: The Collaborative Replications and Education Project." Frontiers in Psychology 10(13 Feburary). https://doi.org/10.3389/fpsyg.2019.00247

White, C. J. M., A. Norenzayan and M. Schaller. 2018. "The Content and Correlates of Belief in Karma across Cultures." Personality and Social Psychology Bulletin: 1-18. https://doi.org/10.1177/0146167218808502

Whitehouse, H. and J. A. Lanman. 2014. "The Ties That Bind Us: Ritual, Fusion, and Identification." Current Anthropology 55: 674-695. https://doi.org/10.1086/678698

Wilkinson, M. D., M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J. Boiten, L. B. da Silva Santos, P. E. Bourne, J. Bouwman, A. J. Brookes, T. Clark, M. Crosas, I. Dillo, O. Dumon, S. Edmunds, C. T. Evelo, R. Finkers, and B. Mons. 2016. "The FAIR Guiding Principles for Scientific Data Management and Stewardship." Scientific Data 3 (1-9). https://doi.org/10.1038/sdata.2016.18

Wimber, M., A. Alink, I. Charest, N. Kriegeskorte and M. C. Anderson. 2015. "Retrieval Induces Adaptive Forgetting of Competing Memories via Cortical Pattern Suppression." Nature Neuroscience 18: 582-589. https://doi.org/10.1038/nn.3973

Wright, R. 2010. The Evolution of God: The Origins of Our Beliefs. Hachette UK: Abacus.

Yeung, A. W. K. 2017. "Do Neuroscience Journals Accept Replications? A Survey of Literature." Frontiers in Human Neuroscience 11: 468 (1-6). https://doi.org/10.3389/fnhum.2017.00468

Yong, E. 2012. "A Failed Replication Draws a Scathing Personal Attack from a Psychology Professor." Discover Magazine (10 March). https://www.discovermagazine.com/mind/a-failed-replication-draws-a-scathing-personal-attack-from-a-psychology-professo

Published

2021-02-06

How to Cite

Hoogeveen, S., & van Elk, M. (2021). Advancing the Cognitive Science of Religion through Replication and Open Science. Journal for the Cognitive Science of Religion, 6(1-2), 158–190. https://doi.org/10.1558/jcsr.39039

Issue

Section

Articles