Generating Genre-Based Automatic Feedback on English for Research Publication Purposes

Authors

  • Stephanie Link Oklahoma State University
  • Robert Redmon Oklahoma State University
  • Yaser Shamsi Oklahoma State University
  • Martin Hagan Oklahoma State University

DOI:

https://doi.org/10.1558/cj.26273

Keywords:

genre-based feedback, automated writing evaluation, natural language processing, transformer network, writing for publication

Abstract

Artificial intelligence (AI) for supporting second language (L2) writing processes and practices has garnered increasing interest in recent years, establishing AI-mediated L2 writing as a new norm for many multilingual classrooms. As such, the emergence of AI-mediated technologies has challenged L2 writing instructors and their philosophies regarding computer-assisted language learning (CALL) and teaching. Technologies that can combine principled pedagogical practices and the benefits of AI can help to change the landscape of L2 writing instruction while maintaining the integrity of knowledge production that is so important to CALL instructors. To align L2 instructional practices and CALL technologies, we discuss the development of an AI-mediated L2 writing technology that leverages genre-based instruction (GBI) and large language models to provide L2 writers and instructors with tools to enhance English for research publication purposes. Our work reports on the accuracy, precision, and recall of our network classification, which surpass previously reported research in the field of genre-based automated writing evaluation by offering a faster network training approach with higher accuracy of feedback provision and new beginnings for genre-based learning systems. Implications for tool development and GBI are discussed.

Author Biographies

  • Stephanie Link, Oklahoma State University

    Stephanie Link is an Associate Professor of Applied Linguistics at Oklahoma State University. Her research focuses on automated writing evaluation tools and intelligent tutoring systems for second language writing and written scientific communication. She is the Editor of the Advances in CALL Research and Practice book series and Book Review Editor for the English for Specific Purposes Journal. Her work has been published in notable journals, such as CALICO Journal, Computer Assisted Language Learning, and Language Learning & Technology. Her latest project (Dissemity, funded through the National Science Foundation) integrates genre-based pedagogy and artificial intelligence to help developing writers disseminate scientific results.

  • Robert Redmon, Oklahoma State University

    Robert Redmon is a postdoctoral researcher at Oklahoma State University, working on the development of Dissemity, a genre-based writing instruction platform with AI-driven automated writing evaluation features. His research interests are in corpus linguistics, discourse analysis, and natural language processing. He served as entrepreneurial lead for a recent grant through the National Science Foundation Innovation Corps National Program to bring Dissemity into the commercial market.

  • Yaser Shamsi, Oklahoma State University

    Yaser Shamsi is a PhD student in TESOL and Applied Linguistics at Oklahoma State University. Currently serving as a graduate research associate, he is involved in developing Dissemity, an online writing tool designed to provide feedback using AI-driven technology. His research interests involve second language acquisition, automated writing evaluation, and genre analysis.

  • Martin Hagan, Oklahoma State University

    Martin Hagan is Professor Emeritus of Electrical and Computer Engineering at Oklahoma State University, Stillwater, where he has taught and conducted research in the areas of statistical modeling, neural networks, and dynamic systems since 1986. He is the author, with H. Demuth and M. Beale, of the textbook Neural Network Design (Boston: PWS, 1994, 2nd ed. 2014). He was also a co-author of the Neural Network Toolbox (now Deep Learning Toolbox) for MATLAB until 2015.

References

Anthony, L., & Lashkia, G. V. (2003). Mover: A machine learning tool to assist in the reading and writing of technical papers. IEEE Transactions on Professional Communication, 46(3), 185–193. https://doi.org/10.1109/TPC.2003.816789

Bahktin, M. M. (1986). Speech genres and other late essays. Austin: University of Texas Press.

Bhatia, V. K. (1993). Analysing genre: Language use in professional settings. Harlow: Longman.

Boulton, A., & Vyatkina, N. (2021). Thirty years of data-driven learning: Taking stock and charting new directions over time. Language Learning & Technology, 25(3), 66–89. http://hdl.handle.net/10125/73450

Burstein, J., Elliot, N., & Molloy, H. (2016). Informing automated writing evaluation using the lens of genre: Two studies. CALICO Journal, 33(1), 117–141. https://doi.org/10.1558/cj.v33i1.26374

Cheng, A. (2008). Analyzing genre exemplars in preparation for writing: The case of an L2 graduate student in the ESP genre-based instructional framework of academic literacy. Applied linguistics, 29(1), 50–71. https://doi.org/10.1093/applin/amm021

Cheng, A. (2011). Language features as the pathways to genre: Students’ attention to non-prototypical features and its implications. Journal of Second Language Writing, 20(1), 69–82. https://doi.org/10.1016/j.jslw.2010.12.002

Cheng, A. (2021). The place of language in the theoretical tenets, textbooks, and classroom practices in the ESP genre-based approach to teaching writing. English for Specific Purposes, 64, 26–36. https://doi.org/10.1016/j.esp.2021.07.001

Clark, K., Luong, M. T., Le, Q. V., & Manning, C. D. (2020). Electra: Pre-training text encoders as discriminators rather than generators. arXiv:2003.10555. https://doi.org/10.48550/arXiv.2003.10555

Cotos, E. (2014). Genre-based automated writing evaluation for L2 research writing: From design to evaluation and enhancement. London: Palgrave Macmillan. https://doi.org/10.1057/9781137333377

Cotos, E., & Huffman, S. (2013). Learner fit in scaling up automated writing evaluation. International Journal of Computer-Assisted Language Learning and Teaching, 3(3), 77–98. https://doi.org/10.4018/ijcallt.2013070105

Cotos, E., & Pendar, N. (2016). Discourse classification into rhetorical functions for AWE feedback. CALICO Journal, 33(1), 92–116. https://doi.org/10.1558/cj.v33i1.27047

Cotos, E., Link, S., & Huffman, S. (2016). Studying disciplinary corpora to teach the craft of Discussion. Writing and Pedagogy, 8(1), 33–64. https://doi.org/10.1558/wap.v8i1.27661

Devitt, A. J. (2004). Writing genres. Carbondale: Southern Illinois University Press.

Dizon, G., & Gayed, J. (2021). Examining the impact of Grammarly on the quality of mobile L2 writing. JALT CALL Journal, 17(2), 74–9. https://doi.org/10.29140/jaltcall.v17n2.336

Dudley-Evans, T., & St John, M. J. (1998). Developments in English for specific purposes: A multi-disciplinary approach. Cambridge: Cambridge University Press.

Feng, H.-H., & Chukharev-Hudilainen, E. (2022). Genre-based AWE system for engineering graduate writing: Development and evaluation. Language Learning & Technology, 26(2), 58–77. https://doi.org/10125/73479

Fiacco, J., Cotos, E., & Rose, C. (2019). Towards enabling feedback on rhetorical structure with neural sequence models. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 310–319). New York: ACM Press. https://doi.org/10.1145/3303772.3303808

Flowerdew, L. (2015). Corpus-based research and pedagogy in EAP: From lexis to genre. Language Teaching, 48(1), 99–116. https://doi.org/10.1017/S0261444813000037

Freedman, A. (1993). Show and tell? The role of explicit teaching in the learning of new genres. Research in the Teaching of English, 27(3), 222–251.

Gayed, J. M., Carlon, M. K. J., Oriola, A. M., & Cross, J. S. (2022). Exploring an AI-based writing assistant’s impact on English language learners. Computers and Education: Artificial Intelligence, 3, 100055. https://doi.org/10.1016/j.caeai.2022.100055

Habibie, P., & Starfield, S. (2023). AI-mediated English for research publication purposes: Are we there yet? Journal of English for Research Publication Purposes, 4(1), 1–4. https://doi.org/10.1075/jerpp.00013.hab

Halliday, M. A. K. (1994). An introduction to functional grammar (2nd ed.). London: Edward Arnold.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487

Howard, J., & Ruder, S. (2018). Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long papers, pp. 328–339). Melbourne: Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-1031

Huang, J. C. (2014). Learning to write for publication in English through genre-based pedagogy: A case in Taiwan. System, 45, 175–186. https://doi.org/10.1016/j.system.2014.05.010

Hyland, K. (2007). Genre pedagogy: Language, literacy and L2 writing instruction. Journal of Second Language Writing, 16(3), 148–164. https://doi.org/10.1016/j.jslw.2007.07.005

Hyon, S. (1996). Genre in three traditions: Implications for ESL. TESOL Quarterly, 30(4), 693–722. https://doi.org/10.2307/3587930

Johns, A. M. (Ed.) (2002). Genre in the classroom: Multiple perspectives. Mahwah: Lawrence Erlbaum. https://doi.org/10.4324/9781410604262

Kershaw, D., & Koeling, R. (2020). Elsevier CC-BY corpus (V3) [Dataset]. Elsevier Data Repository. https://doi.org/10.17632/zm33cdndxs.3

Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sándor, A., Kitto, K., Liu, M., Mogarkar, R. V., & Buckingham Shum, S. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12(1), 141–186. https://doi.org/10.17239/jowr-2020.12.01.06

Link, S., Mehrzad, M., & Rahimi, M. (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 35(4), 605–634. https://doi.org/10.1080/09588221.2020.1743323

McCarthy, K. S., Roscoe, R. D., Allen, L. K., Likens, A. D., & McNamara, D. S. (2022). Automated writing evaluation: Does spelling and grammar feedback support high-quality writing and revision? Assessing Writing, 52, 100608. https://doi.org/10.1016/j.asw.2022.100608

Paltridge, B. (2019). Focusing on language in second language writing classrooms: Rethinking the approach. Journal of Second Language Writing, 46, 100680. https://doi.org/10.1016/j.jslw.2019.100680

Park, L. A., & Read, J. (2019). A blended metric for multi-label optimisation and evaluation. In M. Berlingerio, F. Bonchi, T. Gärtner, N. Hurley, & G. Ifrim (Eds.), Machine learning and knowledge discovery in databases (pp. 719–734). ECML PKDD 2018. Lecture Notes in Computer Science, vol. 11051. Cham: Springer. https://doi.org/10.1007/978-3-030-10925-7_44

Ranalli, J., Link, S., & Chukharev-Hudilainen, E. (2017). Automated writing evaluation for formative assessment of second language writing: Investigating the accuracy and usefulness of feedback as part of argument-based validation. Educational Psychology, 37(1), 8–25. https://doi.org/10.1080/01443410.2015.1136407

Saeli, H., Rahmati, P., & Koltovskaia, S. (2023). Learner engagement with written corrective feedback: The case of automated writing evaluation. Journal of Response to Writing, 9(2), 1–39. https://doi.org/10.32038/frsl.2023.01.08

Singh, M. K. M. (2014). Challenges in academic reading and overcoming strategies in taught master programmes: A case study of international graduate students in Malaysia. Higher Education Studies, 4(4), 76–88. https://doi.org/10.5539/hes.v4n4p76

Swales, J. M. (1981). Aspects of article introductions. Birmingham: University of Aston.

Swales, J. M. (1990). Genre Analysis: English in academic and research settings. Cambridge: Cambridge University Press.

Swales, J. M. (2004). Research genres: Explorations and applications. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139524827

Swales, J. M., & Feak, C. B. (2012). Academic writing for graduate students: Essential tasks and skills (3rd ed.). Ann Arbor: University of Michigan Press. https://doi.org/10.3998/mpub.2173936

Tardy, C. M. (2006). Researching first and second language genre learning: A comparative review and a look ahead. Journal of Second Language Writing, 15(2), 79–101. https://doi.org/10.1016/j.jslw.2006.04.003

Tardy, C. M. (2009). Building genre knowledge. West Lafayette: Parlor Press.

Tardy, C. M. (2017). The challenge of genre in the academic writing classroom: Implications for L2 writing teacher education. In J. Bitchener, N. Storch, & R. Wette (Eds.), Teaching writing for academic purposes to multilingual students: Instructional approaches (pp. 69–83). New York: Routledge. https://doi.org/10.4324/9781315269665-5

Tardy, C. M., Sommer-Farias, B., & Gevers, J. (2020). Teaching and researching genre knowledge: Toward an enhanced theoretical framework. Written Communication, 37(3), 287–321. https://doi.org/10.1177/0741088320916554

Tardy, C. M., Caplan, N. A., & Johns, A. M. (2023). Genre explained: Frequently asked questions and answers about genre-based instruction. Ann Arbor: University of Michigan Press. https://doi.org/10.3998/mpub.11714330

Truscott, J. (1996). The case against grammar correction in L2 writing classes. Language Learning, 46(2), 327–369. https://doi.org/10.1111/j.1467-1770.1996.tb01238.x

Truscott, J. (1999). The case for “The case against grammar correction in L2 writing classes”: A response to Ferris. Journal of Second Language Writing, 8(2), 111–122. https://doi.org/10.1016/S1060-3743(99)80124-6

Williams, J. M., & Colomb, G. G. (1993). The case for explicit teaching: Why what you don’t know won’t help you. Research in the Teaching of English, 27(3), 252–264. https://www.jstor.org/stable/40171226

Yasuda, S. (2011). Genre-based tasks in foreign language writing: Developing writers’ genre awareness, linguistic knowledge, and writing competence. Journal of Second Language Writing, 20(2), 111–133. https://doi.org/10.1016/j.jslw.2011.03.001

Zhai, N., & Ma, X. (2022). The effectiveness of automated writing evaluation on writing quality: A meta-analysis. Journal of Educational Computing Research, 61(4), 875–900. https://doi.org/10.1177/07356331221127300

Zhang, Z. V. (2020). Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisions. Assessing Writing, 43, 100439. https://doi.org/10.1016/j.asw.2019.100439

Downloads

Published

2024-03-25

Issue

ADVANCE ACCESS TO FORTHCOMING ARTICLES

Section

Articles

How to Cite

Link, S., Redmon, R., Shamsi, Y., & Hagan, M. (2024). Generating Genre-Based Automatic Feedback on English for Research Publication Purposes. CALICO Journal. https://doi.org/10.1558/cj.26273