Analyzing Processing Time Data in Applied Linguistics and Second Language Research
A Multivariate Mixed-effects Approach
In this paper, I introduce the use of an extension of the popular linear mixed-effects models: multivariate mixed models which can handle multiple outcome variables. This technique is especially useful for applied linguists and second language researchers who use processing time data (e.g., accuracy and reaction times) obtained from timed decision tasks, text-based eye tracking, and self-paced reading. It can address the long-standing issue of multiple comparisons as a result of having multiple outcomes (e.g., first fixation durations and total reading times in text-based eye tracking, and time spent in different regions in self-paced reading). This technique also provides exciting opportunities for researchers to ask new questions that could not be addressed in a straightforward manner with traditional statistics. With this technique, researchers are able to investigate differential effects of a predictor on different outcomes. Through a demonstration in R using published, open eye-tracking data, I contextualize my discussion of the technique, offering also practical, step-by-step, and annotated guidelines for interested researchers.
Baayen, R. H., Davidson, D. J., and Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59 (4): 390-412. https://doi.org/10.1016/j.jml.2007.12.005
Baldwin, S. A., Imel, Z. E., Braithwaite, S. R., and Atkins, D. C. (2014). Analyzing multiple outcomes in clinical research using multivariate multilevel models. Journal of Consulting and Clinical Psychology 82 (5), 920-930. https://doi.org/10.1037/a0035628
Barr, D. J., Levy, R., Scheepers, C., and Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 68 (3): 255-278. https://doi.org/10.1016/j.jml.2012.11.001
Cunnings, I. (2012). An overview of mixed-effects statistical models for second language researchers. Second Language Research Analyzing multiple outcomes in clinical research using multivariate multilevel models 28 (3): 369-382. https://doi.org/10.1177/0267658312443651
Cunnings, I. and Finlayson, I. (2015). Mixed effects modeling and longitudinal data analysis. In L. Plonsky (Ed.), Advancing Quantitative Methods in Second Language Research, 159-181. New York: Routledge. https://doi.org/10.4324/9781315870908-8
Draheim, C., Hicks, K. L., and Engle, R. W. (2016). Combining reaction time and accuracy: The relationship between working memory capacity and task switching as a case example. Perspectives on Psychological Science 11: 133-155. https://doi.org/10.1177/1745691615596990
Draheim, C., Mashburn, C. A., Martin, J. D., and Engle, R. W. (2019). Reaction time in differential and developmental research: A review and commentary on the problems and alternatives. Psychological Bulletin 145 (5): 508-535. https://doi.org/10.1037/bul0000192
Elgort, I., Brysbaert, M., Stevens, M., and Van Assche, E. (2018). Contextual word learning during reading in a second language: An eye-movement study. Studies in Second Language Acquisition 40 (2): 341-366. https://doi.org/10.1017/S0272263117000109
Godfroid, A. (2019). Eye Tracking in Second Language Acquisition and Bilingualism: A Research Synthesis and Methodological Guide. New York: Routledge.
Godfroid, A. and Winke, P. (2015). Investigating implicit and explicit processing using https://doi.org/10.1075/sibil.48.14god
L2 learners' eye-movement data. In P. Rebuschat (Ed.) Implicit and Explicit Learning of Languages, 325-348. Amsterdam: John Benjamins.
Gries, S. Th. (2015). The most under-used statistical method in corpus linguistics: Multi-level (and mixed-effects) models. Corpora 10 (1): 95-125. https://doi.org/10.3366/cor.2015.0068
Hadfield, J. D. (2010). MCMC methods for multi-response generalized linear mixed models: The MCMCglmm R package. Journal of Statistical Software 33 (2): 1-22. https://doi.org/10.18637/jss.v033.i02
Hamaker, E. L. and Klugkist, I. (2011). Bayesian estimation of multilevel models. In J. J. Hox and J. K. Roberts (Eds), Handbook of Advanced Multilevel Analysis, 137-161. New York: Routledge.
Hox, J. J., Moerbeek, M., and Van de Schoot, R. (2018). Multilevel Analysis: Techniques and Applications. New York: Routledge. https://doi.org/10.4324/9781315650982
Indrarathne, B., Ratajczak, M., and Kormos, J. (2018). Modelling changes in the cognitive processing of grammar in implicit and explicit learning conditions: Insights from an eye‐tracking study. Language Learning 68 (3): 669-708. https://doi.org/10.1111/lang.12290
Jeon, E. H. (2015). Multiple regression. In L. Plonsky (Ed.), Advancing Quantitative Methods in Second Language Research, 131-158. New York: Routledge. https://doi.org/10.4324/9781315870908-7
Jiang, N. (2013). Conducting Reaction Time Research in Second Language Studies. New York: Routledge. https://doi.org/10.4324/9780203146255
Keating, G. and Jegerski, J. (2015). Experimental designs in sentence processing research. Studies in Second Language Acquisition 37: 1-32. https://doi.org/10.1017/S0272263114000187
Lagrou, E., Hartsuiker, R. J., and Duyck, W. (2011). Knowledge of a second language influences auditory word recognition in the native language. Journal of Experimental Psychology. Learning, Memory, and Cognition 37 (4): 952-965. https://doi.org/10.1037/a0023217
Larson-Hall, J. and Herrington, R. (2010). Improving data analysis in second language acquisition by utilizing modern developments in applied statistics. Applied Linguistics 31 (3): 368-390. https://doi.org/10.1093/applin/amp038
Linck, J. A. and Cunnings, I. (2015). The utility and application of mixed-effects models in second language research. Language Learning 65 (S1), 185-207. https://doi.org/10.1111/lang.12117
Marsden, E., Morgan-Short, K., Thompson, S., and Abugaber, D. (2018). Replication in second language research: Narrative and systematic reviews and recommendations for the field. Language Learning 68: 321-391. https://doi.org/10.1111/lang.12286
Marsden, E., Thompson, S., and Plonsky, L. (2018). A methodological synthesis of self-paced reading in second language research. Applied Psycholinguistics 39 (5): 861-904. https://doi.org/10.1017/S0142716418000036
Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., and Bates, D. (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language 94: 305-315. https://doi.org/10.1016/j.jml.2017.01.001
Norouzian, R., de Miranda, M., and Plonsky, L. (2018). The Bayesian revolution in second language research: An applied approach. Language Learning 68 (4): 1032-1075. https://doi.org/10.1111/lang.12310
Plonsky, L. (2014). Study quality in quantitative L2 research (1990-2010): A methodological synthesis and call for reform. The Modern Language Journal 98 (1): 450-470. https://doi.org/10.1111/j.1540-4781.2014.12058.x
Plonsky, L. and Oswald, F. L. (2017). Multiple regression as a flexible alternative to ANOVA in L2 research. Studies in Second Language Acquisition 39 (3): 579-592. https://doi.org/10.1017/S0272263116000231
Polio, C. and Gass, S. M. (1997). Replication and reporting: A commentary. Studies in Second Language Acquisition 19: 499-508. https://doi.org/10.1017/S027226319700404X
Porte, G. (2012). Replication Research in Applied Linguistics. Cambridge: Cambridge University Press.
Raudenbush, S. W. and Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed). London: Sage.
Rayner, K. (2009). Eye movements in reading: Models and data. Journal of Eye Movement Research 2: 1-10.
Rayner, K., Reichle, E. D., and Pollatsek, A. (2005). Eye movement control in reading and the E-Z Reader model. In G. Underwood (Ed.), Cognitive Processes in Eye Guidance, 131-162. Oxford: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780198566816.003.0006
Regev, S. and Meiran, N. (2014). Post-error slowing is influenced by cognitive control demand. Acta Psychologica 152: 10-18. https://doi.org/10.1016/j.actpsy.2014.07.006
Ross, S. J. and Mackey, B. (2015). Bayesian approaches to imputation, hypothesis testing, and parameter estimation. Language Learning 65 (S1), 208-227. https://doi.org/10.1111/lang.12118
Tabachnick, B. G. and Fidell, L. S. (2013). Using Multivariate Statistics (6th ed.) Boston, MA: Pearson.
Van de Schoot, R., Winter, S., Ryan, O., Zondervan-Zwijnenburg, M., and Depaoli, S. (2017). A systematic review of Bayesian papers in psychology: The last 25 years. Psychological Methods 22: 217-239. https://doi.org/10.1037/met0000100
Von der Malsburg, T. and Angele, B. (2017). False positives and other statistical errors in standard analyses of eye movements in reading. Journal of Memory and Language 94: 119-133. https://doi.org/10.1016/j.jml.2016.10.003
Whitford, V. and Titone, D. (2019). Lexical entrenchment and cross-language activation: Two sides of the same coin for bilingual reading across the adult lifespan. Bilingualism: Language and Cognition 22 (1): 58-77. https://doi.org/10.1017/S1366728917000554
Winke, P. M., Godfroid, A., and Gass, S. M. (2013). Introduction to the special issue: Eye-movement recordings in second language research. Studies in Second Language Acquisition 35 (2): 205-212. https://doi.org/10.1017/S027226311200085X
© Equinox Publishing Ltd.
For information regarding our Open Access policy, click here.