Identifying texts in the Warning Zone
Empirical foundation of a screening instrument to adapt early writing instruction
DOI:
https://doi.org/10.1558/wap.22095Keywords:
writing assessment, elementary school, standard setting, proficiency levelsAbstract
[Open Access CC BY-NC-ND]
This article addresses the basis for the development of the screening tool Norwegian Early Writers Signal (NEWS). The aim of the study was to develop a tool for teachers in grades 1–3 to identify student texts in ‘the Warning Zone’, i.e., texts that signal insufficient overall text quality associated with students in need of extra instructional support. Text norms were elicited from a panel of 14 experts in a standardsetting seminar. The standard-setting procedure was a benchmarking-like approach in which panelists chose texts that according to their judgement were in the Warning Zone. Additionally, in an online questionnaire, data on experts’ expectation growth pattern for eight text quality aspects in grades 1–3 were collected. Furthermore, student texts in the Warning Zone were marked and then included in the screening tool to concretize the norms, showing that texts in this zone can take several shapes. The article discusses what steps can be taken to further validate and implement the NEWS tool.
References
Bay, L. (2012). Developing achievement levels on the 2011 National Assessment of Educational Progress in grades 8 and 12 writing: Technical report. https://www.nagb.gov/content/dam/nagb/en/documents/publications/achievement/developing-achievement-levels-2011-naep-grade8-grade12-writing-technical-report.pdf
Berge, K. L., Evensen, L. S., & Thygesen, R. (2016). The wheel of writing: A model of the writing domain for the teaching and assessing of writing as a key competency. The Curriculum Journal, 27(2), 172–189. https://doi.org/10.1080/09585176.2015.1129980
Berge, K. L., Skar, G. B., Matre, S., Solheim, R., Evensen, L. S., Otnes, H., & Thygesen, R. (2019). Introducing teachers to new semiotic tools for writing instruction and writing assessment: Consequences for students’ writing proficiency. Assessment in Education: Principles, Policy and Practice, 26(1), 6–25. https://doi.org/10.1080/0969594X.2017.1330251
Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. SAGE Publications.
Eckes, T. (2011). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments (2nd ed.). Peter Lang.
Evensen, L. S. (2002). Convention from below: Negotiating interaction and culture in argumentative writing. Written Communication, 19(3), 382–413. https://doi.org/10.1177/074108802237750
Foorman, B. R., Francis, D. J., Shaywitz, S. E., Shaywitz, B. A., & Fletcher, J. M. (1997). The case for early reading intervention. In B. A. Blachman (Ed.), Foundations of reading acquisition and dyslexia (pp. 243–264). Lawrence Erlbaum Associates.
Graham, S. (2018). A revised writer(s)-within-community model of writing. Educational Psychologist, 53(4), 258–279. https://doi.org/10.1080/00461520.2018.1481406
Graham, S., & Harris, K. R. (2018). Evidence-based practices in writing. In S. Graham, C. A. MacArthur & M. Hebert (Eds.), Best practices in writing instruction (pp. 3–29). The Guilford Press.
Graham, S., & Rijlaarsdam, G. (2016). Writing education around the globe: Introduction and call for a new global analysis. Reading and Writing, 29, 781–792. https://doi.org/10.1007/s11145-016-9640-1
Graham, S., Skar, G. B., & Falk, D. Y. (2021). Teaching writing in the primary grades in Norway: A national survey. Reading and Writing, 34(2), 529–563. https://doi.org/10.1007/s11145-020-10080-y
Harsch, C., & Kanistra, V. P. (2020). Using an innovative standard-setting approach to align integrated and independent writing tasks to the CEFR. Language Assessment Quarterly, 17(3), 262–281. https://doi.org/10.1080/15434303.2020.1754828
Hooper, S. R., Roberts, J. E., Nelson, L., Zeisel, S., & Kasambira Fannin, D. (2010). Preschool predictors of narrative writing skills in elementary school children. School Psychology Quarterly, 25(1), 1–12. https://doi.org/10.1037/a0018329
Jeffery, J. V., Elf, N., Skar, G. B., & Wilcox, K. C. (2018). Writing development and education standards in cross-national perspective. Writing & Pedagogy, 10(3), 333–370. https://doi.org/10.1558/wap.34587
Jeffery, J. V., & Parr, J. M. (Eds.). (2021). International perspectives on writing curricula and development. Routledge. https://doi.org/10.4324/9781003051404
Juel, C. (1988). Learning to read and write: A longitudinal study of 54 children from first through fourth grades. Journal of Educational Psychology, 80(4), 437–447. https://doi.org/10.1037/0022-0663.80.4.437
Opplæringslova [The Education Act]. (1998). Lov om grunnskolen og den vidaregåande opplæringa ({LOV}-1998-07-17-61). Norwegian Ministry of Education and Research. https://lovdata.no/lov/1998-07-17-61
Purves, A. C. (1992). Conclusion. In A. C. Purves (Ed.), The IEA study of written composition II: Education and performance in fourteen countries (Vol. 2, pp. 199–203). Pergamon.
Rijlaarsdam, G., Jansen, T., Braaksma, M., Van Steendam, E., Van den Branden, K., & Verheyden, L. (2013). Learning and instruction in writing. In C. A. Stone, E. R. Silliman, B. J. Ehren & G. P. Wallach (Eds.), Handbook of language and literacy, Second Edition (pp. 545–566), Guilford Press.
Russell, D. R. (1997). Rethinking genre in school and society: An activity theory analysis. Written Communication, 14(4), 504–554. https://doi.org/10.1177/0741088397014004004
Schoonen, R. (2012). The validity and generalizability of writing scores: The effect of rater, task and language. In Measuring writing: Recent insights into theory, methodology and practice (pp. 1–22). Brill.
Skar, G. B. (2017). The Norwegian national sample-based writing test 2016: Technical report. Nasjonalt senter for skriveopplæring og skriveforsking. http://www.skrivesenteret.no/uploads/files/Skriveproven2017/NSBWT2017.pdf
Skar, G. B., Aasen, A. J., Kvistad, A. H., & Johansen, M. B. (2022). Audience awareness in elementary school students’ texts: Variations within and between grades 1–3. Writing & Pedagogy, 13(1–3), 155–180.
Skar, G. B., Aasen, A. J., & Jølle, L. (2020). Functional writing in the primary years: Protocol for a mixed-methods writing intervention study. Nordic Journal of Literacy Research, 6(1), 201–216. https://doi.org/10.23865/njlr.v6.2040
Skar, G. B., & Huebner, A. (2022). The predictability of first grade students’ writing proficiency [Submitted], Department of Teacher Education, Norwegian University of Science and Technology & Department of Applied and Computational Mathematics and Statistics, University of Notre Dame.
Skar, G. B., & Jølle, L. (2017). Teachers as raters: Investigation of a long-term writing assessment program. L1, 17(Open Issue), 1–30. https://doi.org/10.17239/L1ESLL-2017.17.01.06
Skar, G. B., Jølle, L., & Aasen, A. J. (2020). Establishing scales to assess writing proficiency development in young learners. Acta Didactica Norge, 14(1), 1–30. https://doi.org/10.5617/adno.7909
Skar, G. B., Lei, P-W., Graham, S., Aasen, A. J., Johansen, M. B., & Kvistad, A. H. (2022). Handwriting fluency and the quality of primary grade students’ writing. Reading and Writing, 35, 509–538. https://doi.org/10.1007/s11145-021-10185-y
Van den Bergh, H., De Maeyer, S., Van Weijen, D., & Tillema, M. (2012). Generalizability of text quality scores. In Measuring writing: Recent insights into theory, methodology and practice (pp. 23–32). Brill.
Verheyden, L. (2010). Achter de lijn. Vier empirische studies over ontluikende stelvaardigheid. [The story behind the line. Four empirical studies on writing by third and fourth graders of primary school.] Diss KU Leuven.
Verheyden, L., Van den Branden, K., Rijlaarsdam, G., Van den Bergh, H., & De Maeyer, S. (2010). Written narrations by 8- to 10-year-old Turkish pupils in Flemish primary education: A follow-up of seven text features. Journal of Research in Reading, 33(1), 20–38. https://doi.org/10.1111/j.1467-9817.2009.01430.x
Wilson, J. (2018). Universal screening with automated essay scoring: Evaluating classification accuracy in grades 3 and 4. Journal of School Psychology, 68, 19–37. https://doi.org/10.1016/j.jsp.2017.12.005
Wilson, J., Olinghouse, N. G., McCoach, D. B., Santangelo, T., & Andrada, G. N. (2016). Comparing the accuracy of different scoring methods for identifying sixth graders at risk of failing a state writing assessment. Assessing Writing, 27, 11–23. https://doi.org/10.1016/j.asw.2015.06.003
