Identifying texts in the Warning Zone

Empirical foundation of a screening instrument to adapt early writing instruction




writing assessment, elementary school, standard setting, proficiency levels


This article addresses the basis for the development of the screening tool Norwegian Early Writers Signal (NEWS). The aim of the study was to develop a tool for teachers in grades 1–3 to identify student texts in ‘the Warning Zone’, i.e., texts that signal insufficient overall text quality associated with students in need of extra instructional support. Text norms were elicited from a panel of 14 experts in a standardsetting seminar. The standard-setting procedure was a benchmarking-like approach in which panelists chose texts that according to their judgement were in the Warning Zone. Additionally, in an online questionnaire, data on experts’ expectation growth pattern for eight text quality aspects in grades 1–3 were collected. Furthermore, student texts in the Warning Zone were marked and then included in the screening tool to concretize the norms, showing that texts in this zone can take several shapes. The article discusses what steps can be taken to further validate and implement the NEWS tool.


Bay, L. (2012). Developing achievement levels on the 2011 National Assessment of Educational Progress in grades 8 and 12 writing: Technical report.

Berge, K. L., Evensen, L. S., & Thygesen, R. (2016). The wheel of writing: A model of the writing domain for the teaching and assessing of writing as a key competency. The Curriculum Journal, 27(2), 172–189.

Berge, K. L., Skar, G. B., Matre, S., Solheim, R., Evensen, L. S., Otnes, H., & Thygesen, R. (2019). Introducing teachers to new semiotic tools for writing instruction and writing assessment: Consequences for students’ writing proficiency. Assessment in Education: Principles, Policy and Practice, 26(1), 6–25.

Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. SAGE Publications.

Eckes, T. (2011). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments (2nd ed.). Peter Lang.

Evensen, L. S. (2002). Convention from below: Negotiating interaction and culture in argumentative writing. Written Communication, 19(3), 382–413.

Foorman, B. R., Francis, D. J., Shaywitz, S. E., Shaywitz, B. A., & Fletcher, J. M. (1997). The case for early reading intervention. In B. A. Blachman (Ed.), Foundations of reading acquisition and dyslexia (pp. 243–264). Lawrence Erlbaum Associates.

Graham, S. (2018). A revised writer(s)-within-community model of writing. Educational Psychologist, 53(4), 258–279.

Graham, S., & Harris, K. R. (2018). Evidence-based practices in writing. In S. Graham, C. A. MacArthur & M. Hebert (Eds.), Best practices in writing instruction (pp. 3–29). The Guilford Press.

Graham, S., & Rijlaarsdam, G. (2016). Writing education around the globe: Introduction and call for a new global analysis. Reading and Writing, 29, 781–792.

Graham, S., Skar, G. B., & Falk, D. Y. (2021). Teaching writing in the primary grades in Norway: A national survey. Reading and Writing, 34(2), 529–563.

Harsch, C., & Kanistra, V. P. (2020). Using an innovative standard-setting approach to align integrated and independent writing tasks to the CEFR. Language Assessment Quarterly, 17(3), 262–281.

Hooper, S. R., Roberts, J. E., Nelson, L., Zeisel, S., & Kasambira Fannin, D. (2010). Preschool predictors of narrative writing skills in elementary school children. School Psychology Quarterly, 25(1), 1–12.

Jeffery, J. V., Elf, N., Skar, G. B., & Wilcox, K. C. (2018). Writing development and education standards in cross-national perspective. Writing & Pedagogy, 10(3), 333–370.

Jeffery, J. V., & Parr, J. M. (Eds.). (2021). International perspectives on writing curricula and development. Routledge.

Juel, C. (1988). Learning to read and write: A longitudinal study of 54 children from first through fourth grades. Journal of Educational Psychology, 80(4), 437–447.

Opplæringslova [The Education Act]. (1998). Lov om grunnskolen og den vidaregåande opplæringa ({LOV}-1998-07-17-61). Norwegian Ministry of Education and Research.

Purves, A. C. (1992). Conclusion. In A. C. Purves (Ed.), The IEA study of written composition II: Education and performance in fourteen countries (Vol. 2, pp. 199–203). Pergamon.

Rijlaarsdam, G., Jansen, T., Braaksma, M., Van Steendam, E., Van den Branden, K., & Verheyden, L. (2013). Learning and instruction in writing. In C. A. Stone, E. R. Silliman, B. J. Ehren & G. P. Wallach (Eds.), Handbook of language and literacy, Second Edition (pp. 545–566), Guilford Press.

Russell, D. R. (1997). Rethinking genre in school and society: An activity theory analysis. Written Communication, 14(4), 504–554.

Schoonen, R. (2012). The validity and generalizability of writing scores: The effect of rater, task and language. In Measuring writing: Recent insights into theory, methodology and practice (pp. 1–22). Brill.

Skar, G. B. (2017). The Norwegian national sample-based writing test 2016: Technical report. Nasjonalt senter for skriveopplæring og skriveforsking.

Skar, G. B., Aasen, A. J., Kvistad, A. H., & Johansen, M. B. (2022). Audience awareness in elementary school students’ texts: Variations within and between grades 1–3. Writing & Pedagogy, 13(1–3), 155–180.

Skar, G. B., Aasen, A. J., & Jølle, L. (2020). Functional writing in the primary years: Protocol for a mixed-methods writing intervention study. Nordic Journal of Literacy Research, 6(1), 201–216.

Skar, G. B., & Huebner, A. (2022). The predictability of first grade students’ writing proficiency [Submitted], Department of Teacher Education, Norwegian University of Science and Technology & Department of Applied and Computational Mathematics and Statistics, University of Notre Dame.

Skar, G. B., & Jølle, L. (2017). Teachers as raters: Investigation of a long-term writing assessment program. L1, 17(Open Issue), 1–30.

Skar, G. B., Jølle, L., & Aasen, A. J. (2020). Establishing scales to assess writing proficiency development in young learners. Acta Didactica Norge, 14(1), 1–30.

Skar, G. B., Lei, P-W., Graham, S., Aasen, A. J., Johansen, M. B., & Kvistad, A. H. (2022). Handwriting fluency and the quality of primary grade students’ writing. Reading and Writing, 35, 509–538.

Van den Bergh, H., De Maeyer, S., Van Weijen, D., & Tillema, M. (2012). Generalizability of text quality scores. In Measuring writing: Recent insights into theory, methodology and practice (pp. 23–32). Brill.

Verheyden, L. (2010). Achter de lijn. Vier empirische studies over ontluikende stelvaardigheid. [The story behind the line. Four empirical studies on writing by third and fourth graders of primary school.] Diss KU Leuven.

Verheyden, L., Van den Branden, K., Rijlaarsdam, G., Van den Bergh, H., & De Maeyer, S. (2010). Written narrations by 8- to 10-year-old Turkish pupils in Flemish primary education: A follow-up of seven text features. Journal of Research in Reading, 33(1), 20–38.

Wilson, J. (2018). Universal screening with automated essay scoring: Evaluating classification accuracy in grades 3 and 4. Journal of School Psychology, 68, 19–37.

Wilson, J., Olinghouse, N. G., McCoach, D. B., Santangelo, T., & Andrada, G. N. (2016). Comparing the accuracy of different scoring methods for identifying sixth graders at risk of failing a state writing assessment. Assessing Writing, 27, 11–23.

CC BY-NC=-ND logo




How to Cite

Skar, G. B., Kvistad, A. H., Johansen, M. B., Rijlaarsdam, G., & Aasen, A. J. (2022). Identifying texts in the Warning Zone: Empirical foundation of a screening instrument to adapt early writing instruction. Writing and Pedagogy, 13(1-3), 267–311.



Assessment of Elementary Students’ Writing