Effects of task type on morphosyntactic complexity across proficiency
evidence from a large learner corpus of A1 to C2 writings
Keywords:instructed second language acquisition, second language acquisition, corpus linguistics, natural language processing, IATBLT Best Article Award Winner 2021
This study investigates the effect of instructional design on (morpho)syntactic complexity in second language (L2) writing development. We operationalised instructional design in terms of task type and empirically based the investigation on a large subcorpus (669,876 writings by 119,960 learners from 128 tasks at all Common European Framework of Reference for Languages levels) of the EF-Cambridge Open Language Database (EFCAMDAT; Geertzen, Alexopoulou and Korhonen 2014).
First, the 128 task prompts were manually categorised for task type (e.g. argumentation, description). Next, developmental trajectories of syntactic complexity from A1 to C2 were established using a variety of global (e.g. mean length of clause) and specific (e.g. non-third person singular present tense verbs) measures extracted using natural language processing techniques. The effects of task type were analysed using the categorisation from the first step. Finally, tasks that showed atypical behaviour for a measure given their task type were explored qualitatively.
Our results partially confirm earlier experimental and corpus-based studies (e.g. subordination associated with argumentative tasks). Going beyond, our large-scale data-driven analysis made it possible to identify specific measures that were naturally prompted by instructional design (e.g. narrations eliciting wh-phrases). We discuss which measures typically align with certain task types and highlight how instructional design relates to L2 developmental trajectories over time.
Winner of the 2021 Best Article Award from the International Association for Task-Based Language Teaching (IATBLT).
Alexopoulou, T., Michel, M., Murakami, A. and Meurers, D. (2017) Task effects on linguistic complexity and accuracy: a large?scale learner corpus analysis employing natural language processing techniques. Language Learning 67(s1): 180–208. https://doi.org/10.1111/lang.12232
Baralt, M., Gilabert, R. and Robinson, P. (eds) (2014) Task Sequencing and Instructed Second Language Learning. London: Bloomsbury Publishing.
Benson, P. and Reinders, H. (eds) (2011) Beyond the Language Classroom. London: Palgrave Macmillan. https://doi.org/10.1057/9780230306790
Biber, D., Gray, B and Staples, S. (2014) Predicting patterns of grammatical complexity across language exam task types and proficiency levels. Applied Linguistics 37(5): 639–68. https://doi.org/10.1093/applin/amu059
Brezina, V. and Flowerdew, L. (eds) (2017) Learner Corpus Research: New Perspectives and Applications. London: Bloomsbury Academic.
Bulté, B. and Housen, A. (2012) Defining and operationalising L2 complexity. In A. Housen, F. Kuiken and I. Vedder (eds) Dimensions of L2 Performance and Proficiency 21–46. Amsterdam: John Benjamins Publishing Company. https://doi.org/10.1075/lllt.32.02bul
Bulté, B. and Housen, A. (2014) Conceptualizing and measuring short-term changes in L2 writing complexity. Journal of Second Language Writing 26: 42–65. https://doi.org/10.1016/j.jslw.2014.09.005
Bulté, B. and Housen, A. (2018) Syntactic complexity in L2 writing: individual pathways and emerging group trends. International Journal of Applied Linguistics 28(1): 147–64. https://doi.org/10.1111/ijal.12196
Byrnes, H., Maxim, H. H., & Norris, J. M. (eds) (2010) Realizing advanced foreign language writing development in collegiate education: curricular design, pedagogy, assessment [Special issue]. Modern Language Journal 94(s1). https://doi.org/10.1111/j.1540-4781.2010.01137.x
Chen, X. and Meurers, D. (2016) CTAP: a web-based tool supporting automatic complexity analysis. Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity 113–19. Osaka, Japan. http://aclweb.org/anthology/W16-4113
Connor-Linton, J. and Polio, C. (2014) Comparing perspectives on L2 writing: multiple analyses of a common corpus. Journal of Second Language Writing 26: 1–9. https://doi.org/10.1016/j.jslw.2014.09.002
Crossley, S. A. and McNamara, D. S. (2014) Does writing development equal writing quality? A computational investigation of syntactic complexity in L2 learners. Journal of Second Language Writing 26: 66–79. https://doi.org/10.1016/j.jslw.2014.09.006
Cumming, A. (1990) Expertise in evaluating second language compositions. Language Testing 7(1): 31–51. https://doi.org/10.1177/026553229000700104
Ferris, D. R. (1994) Lexical and syntactic features of ESL writing by students at different levels of L2 proficiency. TESOL Quarterly 28: 414–20. https://doi.org/10.2307/3587446
Foster, P. and Skehan, P. (1996) The influence of planning and task type on second language performance. Studies in Second Language Acquisition 18: 299–323. https://doi.org/10.1017/S0272263100015047
Frear, M. W. and Bitchener, J. (2015) The effects of cognitive task complexity on writing complexity. Journal of Second Language Writing 20: 45–57. https://doi.org/10.1016/j.jslw.2015.08.009
Gablasova, D., Brezina, V. and McEnery, T. (2017a) Exploring learner language through corpora: comparing and interpreting corpus frequency information. Language Learning 67(s1): 130–54. https://doi.org/10.1111/lang.12226
Gablasova, D., Brezina, V., McEnery, T. and Boyd, E. (2017b) Epistemic stance in spoken L2 English: the effect of task and speaker style. Applied Linguistics 38(5): 613–37. https://doi.org/10.1093/applin/amv055
Geertzen, J., Alexopoulou, T. and Korhonen, A. (2014) Automatic linguistic annotation of large scale L2 databases: the EF-Cambridge Open Language Database (EFCAMDAT). Proceedings of the 31st Second Language Research Forum (SLRF). Pittsburgh, PA: Cascadilla Press.
Graesser, A. C., McNamara, D. S., Louwerse, M. M. and Cai, Z. (2004) Coh-Metrix: analysis of text on cohesion and language. Behavior Research Methods, Instruments, and Computers 36: 193–202. https://doi.org/10.3758/BF03195564
Granger, S., Gilquin, G. and Meunier, F. (2015) The Cambridge Handbook of Learner Corpus Research. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139649414
Hinkel, E. (2009) The effects of essay topics on modal verb uses in L1 and L2 academic writing. Journal of Pragmatics 41(4): 667–83. https://doi.org/10.1016/j.pragma.2008.09.029
Hommel, G. (1988) A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75(2): 383–6. https://doi.org/10.1093/biomet/75.2.383
Kormos, J. (2011) Task complexity and linguistic and discourse features of narrative writing performance. Journal of Second Language Writing 20(2): 148–61. https://doi.org/10.1016/j.jslw.2011.02.001
Kuiken, F. and Vedder, I. (2008) Cognitive task complexity and written output in Italian and French as a foreign language. Journal of Second Language Writing, 17(1): 48–60. https://doi.org/10.1016/j.jslw.2007.08.003
Kyle, K. (2016) Measuring syntactic development in L2 writing: fine grained indices of syntactic complexity and usage-based indices of syntactic sophistication. Doctoral dissertation, Georgia State University. Retrieved on 19 September 2019 from http://scholarworks.gsu.edu/alesl_diss/35
Lee, C. and Polio, J. (2017) Written language learning. In S. Loewen and M. Sato (eds) The Routledge Handbook of Instructed Second Language Acquisition 299–318. New York: Routledge. https://doi.org/10.1080/15475441.2016.1180983
Loewen, S. (2015) Introduction to Instructed Second Language Acquisition. New York: Routledge.
Loewen, S. and Sato, M. (2017) The Routledge Handbook of Instructed Second Language Acquisition. New York: Routledge. https://doi.org/10.4324/9781315676968
Loschky, L. and Bley-Vroman, R. (1993) Grammar and task-based methodology. In G. Crookes and S. Gass (eds) Tasks and Language Learning: Integrating Theory and Practice 123–67. Bristol: Multilingual Matters.
Lu, X. (2010) Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics 15(4): 474–96. https://doi.org/10.1075/ijcl.15.4.02lu
Lu, X. (2011) A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers’ language development. TESOL Quarterly 45(1): 36–62. https://doi.org/10.5054/tq.2011.240859
Mancho?n, R. M. (2011) Learning-to-Write and Writing-to-Learn in an Additional Language. Amsterdam: John Benjamins Publishing Company. https://doi.org/10.1075/lllt.31
McCarthy, P. M. and Jarvis, S. (2010) MTLD, vocd-D, and HD-D: a validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods 42: 381–92. https://doi.org/10.3758/BRM.42.2.381
Meurers, D. (2015) Learner corpora and natural language processing. In S. Granger, G. Gilquin and F. Meunier (eds) The Cambridge Handbook of Learner Corpus Research 537–66. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139649414.024
Meurers, D. and Dickinson, M. (2017) Evidence and interpretation in language learning research: opportunities for collaboration with computational linguistics. Language Learning 67(2): 67–96. https://doi.org/10.1111/lang.12233
Michel, M. (2017) Complexity, accuracy and fluency in L2 production. In S. Loewen and M. Sato (eds) Routledge Handbook of Instructed Second Language Acquisition 50–68. New York: Routledge. https://doi.org/10.4324/9781315676968-4
Murakami, A. (2016) Modeling systematicity and individuality in nonlinear second language development: the case of English grammatical morphemes. Language Learning 66(4): 834–71. https://doi.org/10.1111/lang.12166
Murakami, A. and Alexopoulou, T. (2016) Longitudinal L2 development of the English article in individual learners. In A. Papafragou, D. Grodner, D. Mirman and J. Trueswell (eds) Proceedings of the 38th Annual Meeting of the Cognitive Science Society 1050–5. Austin, TX: Cognitive Science Society.
Nassaji, H. (2016) Interactional feedback in second language teaching and learning: a synthesis and analysis of current research. Language Teaching Research 20(4): 535–62. https://doi.org/10.1177/1362168816644940
Norris, J. M. and Ortega, L. (2009) Towards an organic approach to investigating CAF in instructed SLA: the case of complexity. Applied Linguistics 30(4): 555–78. https://doi.org/10.1093/applin/amp044
Ott, N., Ziai, R. and Meurers, D. (2012) Creation and analysis of a reading comprehension exercise corpus: towards evaluating meaning in context. In T. Schmidt and K. Wörner (eds) Multilingual Corpora and Multilingual Corpus Analysis 47–69. Hamburg Studies in Multilingualism (HSM). Amsterdam, Netherlands: John Benjamins. https://doi.org/10.1075/hsm.14.05ott
Pallotti, G. (2015) A simple view of linguistic complexity. Second Language Research 31(1): 117–34. https://doi.org/10.1177/0267658314536435
Pander Maat, H., Kraf, R., van den Bosch, A. P. J., Dekker, N., Gompel, M. V., Kleijn, S. D., ... and Sloot, K. (2014) T-Scan: a new tool for analyzing Dutch text. Computational Linguistics in the Netherlands (4):53–74. Retrieved on 19 September 2019 from http://hdl.handle.net/2066/134833
Polio, C. and Park, J.-H. (2016) Language development in second language writing. In R. M. Manchón and P. Matsuda (eds) Handbook of Second and Foreign Language Writing 287–306. New York: Routledge. https://doi.org/10.1515/9781614511335-016
Polio, C. and Yoon, H.-J. (2018) The reliability and validity of automated tools for examining variation in syntactic complexity across genres. International Journal of Applied Linguistics 28(1): 165–88. https://doi.org/10.1111/ijal.12200
Quixal, M. and Meurers, D. (2016) How can writing tasks be characterized in a way serving pedagogical goals and automatic analysis needs? CALICO Journal 33(1): 19–48. https://doi.org/10.1558/cj.v33i1.26543
Rebuschat, P., Meurers, D. and McEnery, T. (2017) Language learning research at the intersection of experimental, computational, and corpus-based approaches. Language Learning, 67(s1): 6–13. https://doi.org/10.1111/lang.12243
Robinson, P. (2001) Task complexity, task difficulty, and task production: exploring interactions in a componential framework. Applied Linguistics 22(1): 27–57. https://doi.org/10.1093/applin/22.1.27
Sato M. and Loewen, S. (2019) Evidence-based Second Language Pedagogy: A Collection of Instructed Second Language Acquisition Studies. New York: Routledge. https://doi.org/10.4324/9781351190558
Skehan, P. (2009) Modelling second language performance: integrating complexity, accuracy, fluency, and lexis. Applied Linguistics 30(4): 510–32. https://doi.org/10.1093/applin/amp047
Swain, M. and Lapkin, S. (2002) Talking it through: two French immersion learners’ response to reformulation. International Journal of Educational Research 37(3–4): 285–304. https://doi.org/10.1016/S0883-0355(03)00006-5
Tracy-Ventura, N. and Myles, F. (2015) The importance of task variability in the design of learner corpora for SLA research. International Journal of Learner Corpus Research 1(1): 58–95. https://doi.org/10.1075/ijlcr.1.1.03tra
Tracy-Ventura, N. and Paquot, M. (eds) (in preparation) The Routledge Handbook of Second Language Acquisition and Corpora. New York: Routledge.
Vajjala, S. and Meurers, D. (2012) On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the 7th Workshop on Innovative Use of NLP for Building Educational Applications (BEA) 163–73. Montréal, Canada: ACL. Retrieved on 19 September 2019 from http://aclweb.org/anthology/W12-2019.pdf
Vajjala, S. and Meurers, D. (2014) Readability assessment for text simplification: from analysing documents to identifying sentential simplifications. ITL – International Journal of Applied Linguistics 165(2): 194–222. https://doi.org/10.1075/itl.165.2.04vaj
Vyatkina, N. (2012) The development of second language writing complexity in groups and individuals: a longitudinal learner corpus study. Modern Language Journal 96: 576–98. https://doi.org/10.1111/j.1540-4781.2012.01401.x
Vyatkina, N., Hirschmann, H. and Golcher, F. (2015) Syntactic modification at early stages of L2 German writing development: a longitudinal learner corpus study. Journal of Second Language Writing 29: 28–50. https://doi.org/10.1016/j.jslw.2015.06.006
Way, D. P., Joiner, E. G. and Seaman, M. A. (2000) Writing in the secondary foreign language classroom: the effects of prompts and tasks on novice learners of French. Modern Language Journal 84(2): 171–84. https://doi.org/10.1111/0026-7902.00060
Weigle, S. C. (2002) Assessing Writing. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511732997
Wieling, M. (2018) Analyzing dynamic phonetic data using generalized additive mixed modeling: a tutorial focusing on articulatory differences between L1 and L2 speakers of English. Journal of Phonetics 70: 86–116. https://doi.org/10.1016/j.wocn.2018.03.002
Wieling, M., Montemagni, S., Nerbonne, J. and Baayen, R. H. (2014) Lexical differences between Tuscan dialects and standard Italian: accounting for geographic and socio-demographic variation using generalized additive mixed modeling. Language 90: 669–92. https://doi.org/10.1353/lan.2014.0064
Williams, J. (2012) The potential role(s) of writing in second language development. Journal of Second Language Writing 21(4): 321–31. https://doi.org/10.1016/j.jslw.2012.09.007
Wolfe-Quintero, K., Inagaki, S. and Kim, H.-Y. (1998) Second Language Development in Writing: Measures of Fluency, Accuracy, and Complexity. Manoa, Hawaii: Second Language Teaching and Curriculum Center, University of Hawaii at Manoa.
Wood, S. N. (2017) Generalized Additive Models: An Introduction with R (2nd edn). Boca Raton, FL: Chapman and Hall/CRC.
Yang, W., Lu, X. and Weigle, S. C. (2015) Different topics, different discourse: relationships among writing topic, measures of syntactic complexity, and judgments of writing quality. Journal of Second Language Writing 28: 53–67. https://doi.org/10.1016/j.jslw.2015.02.002
Yoon, H.-J. and Polio, C. (2017) The linguistic development of students of English as a second language in two written genres. TESOL Quarterly 51(2): 275–301. https://doi.org/10.1002/tesq.296