Potential of Automated Writing Evaluation Feedback
DOI:
https://doi.org/10.11139/cj.28.2.420-459Keywords:
automated writing evaluation (AWE), feedback, L2 academic writingAbstract
This paper presents an empirical evaluation of automated writing evaluation (AWE) feedback used for L2 academic writing teaching and learning. It introduces the Intelligent Academic Discourse Evaluator (IADE), a new web-based AWE program that analyzes the introduction section to research articles and generates immediate, individualized, and discipline-specific feedback. The purpose of the study was to investigate the potential of IADE’s feedback. A mixed-methods approach with a concurrent transformative strategy was employed. Quantitative data consisted of responses to Likert-scale, yes/no, and open-ended survey questions; automated and human scores for first and final drafts; and pre-/posttest scores. Qualitative data contained students’ first and final drafts as well as transcripts of think-aloud protocols and Camtasia computer screen recordings, observations, and semistructured interviews. The findings indicate that IADE’s colorcoded and numerical feedback possesses potential for facilitating language learning, a claim supported by evidence of focus on discourse form, noticing of negative evidence, improved rhetorical quality of writing, and increased learning gains.
References
Attali, Y. (2004). Exploring the feedback and revision features of the Criterion service. Paper presented at the National Council on Measurement in Education Annual Meeting, San Diego, CA. Retrieved from http://www.ets.org/Media/Research/pdf/erater_NCME_2004_Attali_B.pdf
Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater v.2. Journal of Technology, Learning, and Assessment, 4(3), 1-30.
Bates, L., Lane, J., & Lange, E. (1993). Writing clearly: Responding to ESL compositions. Boston: Heinle & Heinle.
Block, D. (2003). The social turn in second language acquisition. Edinburgh: Edinburgh University Press.
Burstein, J., Marcu, D., & Knight, K. (2003). Finding the WRITE stuff: Automatic identification of discourse structure in essays. IEEE Intelligent Systems, 18(1), 32-39.
Byrnes, H. (2005). Review of task-based language learning and teaching. The Modern Language Journal, 89, 297-298.
Carroll, S. (1999). Putting “input” in its proper place. Second Language Research, 15, 337-388.
Carroll, S. (2001). Input and evidence: The raw material of second language acquisition. Amsterdam: John Benjamins.
Chapelle, C. A. (1998). Multimedia CALL: Lessons to be learned from research on instructed SLA. Language Learning & Technology, 2(1), 22-34. Retrieved from http://llt.msu.edu/vol2num1/article1/index.html
Chapelle, C. A. (2001). Computer applications in second language acquisition. Cambridge: Cambridge University Press.
Chapelle, C. A. (2007). Technology and second language acquisition. Annual Review of Applied Linguistics, 27, 98–114.
Chen, C. F., & Cheng, W. Y. (2008). Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology, 12(2), 94-112. Retrieved from llt.msu.edu/vol12num2/chencheng.pdf
Chen, L., & Tokuda, N. (2003). A new template-template-enhanced ICALL system for a second language composition course. CALICO Journal, 20, 561-578. Retrieved from https://www.calico.org/memberBrowse.php?action=article&id=295
Cheville, J. (2004). Automated scoring technologies and the rising influence of error. English Journal, 93(4), 47-52.
Conference on College Composition and Communication (CCCC). (2006). Writing assessment: A position statement. Retrieved from http://www.ncte.org/cccc/resources/positions/writingassessment
Cook, V. J. (1991). Second language learning and language teaching. London: Edward Arnold.
Cortes, V. (2007). Exploring genre and corpora in the English for academic writing class. The ORTESOL Journal, 25, 8-14.
Cotos, E. (2009). Designing an intelligent discourse evaluation tool: Theoretical, empirical, and technological considerations. In C. A. Chapelle, H.-S. Jun, & I. Katz (Eds.), Developing and evaluating language learning materials (pp. 103-127). Ames, IA: Iowa State University.
Cotos, E. (2010). Automated writing evaluation for non-native speaker English academic writing: The case of IADE and its formative feedback (Unpublished dissertation). Iowa State University, Ames, Iowa.
Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches. London: SAGE Publishers.
Ellis, R. (1999). Learning a second language through interaction. Amsterdam: John Benjamins.
Ellis, R. (2009). A typology of written corrective feedback types. English Language Teaching Journal, 63, 97-107.
Ellis, R., Loewen, S., & Erlam, R. (2006). Implicit and explicit corrective feedback and the acquisition of L2 grammar. Studies in Second Language Acquisition, 28, 339-368.
Elliot, S., & Mikulas, C. (2004). The impact of MY Access! use on student writing performance: A technology overview and four studies. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.
Ericsson, K., & Simon, H. (1987). Verbal reports on thinking. In C. Faerch & G. Kasper (Eds.), Introspection in second language research (pp. 24-54). Clevedon, UK: Multilingual Matters.
Ericsson, P. (2006). The meaning of meaning: Is a paragraph more than an equation? In P. F. Ericsson & R. Haswell (Eds.), Machine scoring of student essays: Truth and consequences (pp. 28-37). Logan, UT: Utah State University Press.
Ericsson, P., & Haswell, R. (Eds.). (2006). Machine scoring of student essays: Truth and consequences. Logan, UT: Utah State University Press.
Ferris, D. (1995). Student reactions to teacher response in multiple-draft composition classrooms. TESOL Quarterly, 29, 33-53.
Ferris, D. (1999). The case for grammar correction in L2 writing classes: A response to Truscott (1996). Journal of Second Language Writing, 8, 1-10.
Ferris, D. (2002). Treatment of error in second language writing classes. Ann Arbor, MI: University of Michigan Press.
Ferris, D. (2003). Response to student writing: Implications for second language students. Mahwah, NJ: Lawrence Erlbaum.
Ferris, D. (2004). The “grammar correction” debate in L2 writing: Where are we, and where do we go from here? (and what do we do in the meantime…?) Journal of Second Language Writing, 13, 49-62.
Ferris, D. (2006). Does error feedback help student writers? New evidence on short- and long-term effects of written error correction. In K. Hyland & F. Hyland (Eds.), Feedback in second language writing: Contexts and issues (pp. 81-104). Cambridge: Cambridge University Press.
Foltz, P., Laham, D., & Landauer, T. (1999). The Intelligent Essay Assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Leaning, 1(2). Retrieved from http://imej.wfu.edu/articles/1999/2/04/index.asp
Garrett, N. (1987). A psycholinguistic perspective on grammar and CALL. In W. Flint Smith (Ed.), Modern media in foreign language education: Theory and implementation (pp. 169-196). Lincolnwood, IL: National Textbook.
Gass, S. (1997). Input, interaction, and the second language learner. Mahwah, NJ: Lawrence Erlbaum.
Gass, S., & Mackey, A. (2006). Input, interaction and output: An overview. AILA Review, 19, 3-17.
Gass, S., & Mackey, A. (2007). Input, interaction, and output in second language acquisition. In J. Williams & B. VanPatten (Eds.), Theories in second language acquisition: An introduction (pp. 175-199). Mahwah, NJ: Lawrence Erlbaum.
Gass, S., Mackey, A., & Ross-Feldman, L. (2005). Task-based interactions in classroom and laboratory settings. Language Learning, 55, 575-611.
Gor, K., & Long, M. H. (2009). Input and second language processing. In W. C. Ritchie & T. J. Bhatia (Eds.), Handbook of second language acquisition (pp. 445-472). New York: Academic Press.
Guenette, D. (2007). Is feedback pedagogically correct? Research design issues in studies of feedback on writing. Journal of Second Language Writing, 16, 40-53.
Heift, T. (2001). Error-specific and individualized feedback in a web-based language tutoring system: Do they read It? ReCALL, 13, 129-142.
Heift, T. (2002). Learner control and error correction in ICALL: Browsers, peekers and adamants. CALICO Journal, 19, 295-313. Retrieved from https://www.calico.org/memberBrowse.php?action=article&id=428
Heift, T. (2003). Multiple learner errors and meaningful feedback: A challenge for ICALL systems. CALICO Journal, 20, 533-549. Retrieved from https://www.calico.org/memberBrowse.php?action=article&id=293
Heift, T. (2004). Corrective feedback and learner uptake in CALL. ReCALL, 16, 416-431.
Heift, T. (2005). Inspectable learner reports for web-based language learning. ReCALL, 17, 32-46.
Heift, T. (2008). Corrective feedback in CALL: A longitudinal study of learner uptake. Paper presented at the annual meeting of the American Association of Applied Linguistics, Washington, DC.
Heift, T., & Rimrott, A. (2008). Learner responses to corrective feedback for spelling errors in CALL. System, 36, 1-18.
Heift, T., & Schulze, M. (2007). Errors and intelligence in computer-assisted language learning. Parsers and pedagogues. New York: Routledge.
Holland, M., Maisano, R., Alderks, C., & Martin, J. (1993). Parsers in tutors: What are they good for? CALICO Journal, 11, 28-46. Retrieved from https://www.calico.org/memberBrowse.php?action=article&id=562
Hyland, K., & Hyland, F. (Eds.). (2006). Feedback in second language writing: Contexts and issues. New York: Cambridge University Press.
Katz, J. (1983). A theory of qualitative methodology. In R. M. Emerson (Ed.), Contemporary field research. Prospect Heights, IL: Waveland.
Katz, J. (2001). Analytic induction. In N. Smelser & P. Baltes (Eds.), International Encyclopaedia of the Social and Behavioral Sciences (pp. 480-484). New York: Elsevier
Krashen, S. (1980). The input hypothesis. In J. Alatis (Ed.), Current issues in bilingual education (pp. 168-180). Washington, DC: Georgetown University Press.
Krashen, S. (1982). Principles and practice in second language acquisition. London: Pergamon.
Krashen, S. (1984). Writing: Research, theory, and application. Oxford: Pergamon Press.
Krashen, S. (1985). The input hypothesis: Issues and implications. New York: Longman.
Kroll, B. (1977). Combining ideas in written and spoken English: A look at subordination and coordination. In E. Ochs & T. Bennett (Eds.), Discourse across time and space (Southern California Occasional Papers in Linguistics, 5). Los Angeles, University of Southern California.
Labrie, G., & Singh, L. (1991). Parsing, error diagnostics, and instruction in a French tutor. CALICO Journal, 9, 9-25. Retrieved from https://www.calico.org/memberBrowse.php?action=article&id=521
Lalande, J. F. (1982). Reducing composition errors: An experiment. The Modern Language Journal, 66, 140-149.
Leah Rock, J. (2007). The impact of short-term use of CriterionSM on writing skills in ninth grade (Research report). Educational Testing Service, Princeton, NJ. Retrieved from http://www.ets.org/Media/Research/pdf/RR-07-07.pdf
Liou, H. C. (1991). Development of an English grammar checker: A progress report. CALICO Journal, 9, 57-70. Retrieved from https://www.calico.org/memberBrowse.php?action=article&id=523
Long, M. H. (1983). Native speaker/non-native speaker conversation and the negotiation of comprehensible input. Applied Linguistics, 4, 126-141.
Long, M. H. (1996). The role of the linguistic environment in second language acquisition. In W. Ritchie & T. K. Bhatia (Eds.), Handbook of language acquisition: Vol. 2. Second language acquisition (pp. 413-468). San Diego, CA: Academic Press.
Long, M. H. (2007). Problems in SLA. Mahwah, NJ: Lawrence Erlbaum.
Mackey, A. (2006). Feedback, noticing and instructed second language learning. Applied Linguistics, 27, 405-430.
Mackey, A., Gass, S., & McDonough, K. (2000). How do learners perceive interactional feedback? Studies in Second Language Acquisition, 22, 471-497.
Miles, M., & Huberman, A. (1994). Qualitative data analysis (2nd ed.). Newbury Park, CA: Sage.
Nagata, N. (1993). Intelligent computer feedback for second language instruction. The Modern Language Journal, 77, 330-339.
Nagata, N. (1995). An effective application of natural language processing in second language instruction. CALICO Journal, 13, 47-67. Retrieved from https://www.calico.org/memberBrowse.php?action=article&id=597
Nwongu, K. N. (1990). Discourse variation in medical texts: Schema, theme and cohesion in professional and journalistic accounts (Monographs in Systemic Linguistics, 2). Nottingham: University of Nottingham.
Oliver, R., & Mackey, A. (2003). Interactional context and feedback in child ESL classrooms. The Modern Language Journal, 87, 519-543.
Pearson Education, Inc. (2007). General overview of WriteToLearn™ and its components. Retrieved on December 8, 2008 from http://www.knowledge-technologies.com/papers/WTLOverview-040609.pdf
Pendar, N., & Cotos, E. (2008). Automatic identification of discourse moves in scientific article introductions. In The Proceedings of The 3rd Workshop on Innovative Use of NLP for Building Educational Applications (pp. 62-70). Columbus, OH: Ohio State University.
Polio, C. (1997). Measures of linguistic accuracy in second language writing research. Language Learning, 47, 101-143.
Ramirez, A. (2005). Review of the social turn in second language acquisition. The Modern Language Journal, 45, 292-293.
Reuer, V. (2003). Error recognition and feedback with lexical functional grammar. CALICO Journal, 20, 497-512. Retrieved from https://www.calico.org/memberBrowse.php?action=article&id=291
Russell, J., & Spada, N. (2006). The effectiveness of corrective feedback for second language acquisition: A meta-analysis of the research. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 133-164). Amsterdam: John Benjamins.
Shermis, M., & Burstein, J. (2003). Introduction. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective. (pp. xiii-xvi). Mahwah, NJ: Lawrence Erlbaum.
Schmidt, R., & Frota, S. (1986). Developing basic conversational ability in a second language. A case study of an adult learner of Portuguese. In R. Day (Ed.), Talking to learn: Conversation in second language acquisition (pp. 237-326). Rowley, MA: Newbury House.
Shaalan, K. (2005). An intelligent computer assisted language learning system for Arabic learners. Computer Assisted Language Learning, 18, 81-108.
Swain, M. (1985). Communicative competence: Some roles of comprehensible input and comprehensible output in its development. In S. Gass & C. Madden (Eds.), Input in second language acquisition (pp. 235-253). Rowley, MA: Newbury House.
Swain, M. (1995). Three functions of output in second language learning. In G. Cook & B. Seidlhofer (Eds.), Principle and practice in applied linguistics: Studies in honor of H. G. Widdowson (pp. 125-144). Oxford: Oxford University Press.
Swales, J. M. (1981). Aspects of articles introductions (Aston ESP Reports No. 1). Birmingham, England: The University of Aston in Birmingham.
Swales, J. M. (1990). Genre analysis: English in academic and research settings. Cambridge: Cambridge University Press.
Swales, J. M. (2004). Research genres: Exploration and applications. Cambridge: Cambridge University Press.
Thorndike, E. L. (1913). Educational psychology: Vol. I. The original nature of man. New York: Teachers College, Columbia University.
Tokuda, N., & Chen, L. (2001). An online tutoring system for language translation. IEEE Multimedia, 8(3), 46-55.
Tokuda, N., & Chen, L. (2004). A new KE-free online ICALL system featuring error contingent feedback. Computer Assisted Language Learning, 17, 177-201.
Truscott, J. (1996). The case against grammar correction in L2 writing classes. Language Learning, 4, 327-369.
Truscott, J. (2004). Evidence and conjecture on the effects of correction: A response to Chandler. Journal of Second Language Writing, 13, 337-343.
VanPatten, B., & Williams, J. (2007). Introduction: The nature of theories. In B. VanPatten & J. Williams (Eds.), Theories in second language acquisition: An introduction (pp. 1-16). Mahwah, NJ: Lawrence Erlbaum.
van der Linden, E. (1993). Does feedback enhance computer-assisted language learning? Computers & Education, 21, 61-65.
Vantage Learning. (2007). MY Access! Efficacy Report. Newtown, PA: Author. Retrieved from http://www.vantagelearning.com/school/research/myaccess.html
Ware, P. (2005). Automated writing evaluation as a pedagogical tool for writing assessment. In A. Pandian, G. Chakravarthy, P. Kell, & S. Kaur (Eds.), Strategies and practices for improving learning and literacy (pp. 174-184). Selangor, Malaysia: University Putra Malaysia Press.
Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10, 1-24.
Yang, N. (2004). Using My Access! in EFL writing. In The proceedings of 2004 International Conference and Workshop on TEFL & Applied Linguistics (pp. 550-564). Taipei, Taiwan: Ming Chuan University.
Yang, J. C., & Akahori, K. (1998). Error analysis in Japanese writing and its implementation in a computer assisted language learning system on the world wide web. CALICO Journal, 15, 47-66. Retrieved from https://www.calico.org/memberBrowse.php?action=article&id=445
Yang, J. C., & Akahori, K. (1999). An evaluation of Japanese CALL systems on the WWW comparing a freely input approach with multiple selection. Computer Assisted Language Learning, 12, 59-79.
