Extending Preselected Items Evaluation (PIE) with the ATA (American Translators Association) error categories for objective and targeted translation evaluation


  • Amy Colman University of Malta
  • Winibert Segers KU Leuven
  • Heidi Verplaetse KU Leuven




ATA framework for standardized error marking, preselected items evaluation, professional translation, translation evaluation, translation quality assessment, translator training


This paper proposes to extend Preselected Items Evaluation (PIE), an item-based analytical method that seeks to provide greater objectivity in translation evaluation, with the error categories of the American Translators Association’s (ATA) Framework for Standardized Error Marking. The aim is to address a shortcoming of PIE, specifically its restriction to the identification of error numbers, disregarding error types. We describe the application of PIE and the ATA Framework for Standardized Error Marking and explain the advantages of combining PIE with the ATA error categories. We also outline the findings of a pilot study using this method and suggest proposals for further research.

Author Biographies

  • Amy Colman, University of Malta

    Amy Colman has over 15 years of experience as a professional translator, proofreader and copywriter. She is Assistant Lecturer and Coordinator of the Interpreting programme at the University of Malta and she is currently undertaking a PhD in Translation Studies at KU Leuven, Belgium. Her research focuses on translation evaluation and directionality.

  • Winibert Segers, KU Leuven

    Winibert Segers is affiliated with KU Leuven, Belgium. He conducts translation workshops on legal and medical translation, and gives lectures on translation theory. His research interests include translation didactics, translation evaluation and translation theory. 

  • Heidi Verplaetse, KU Leuven

    Heidi Verplaetse obtained her PhD in linguistics from Ghent University and is currently an associate professor at KU Leuven, Belgium. Her research interests include linguistic approaches to translation assessment and didactics, and translation in news flows. She conducts translation workshops on scientific, medical, business and journalistic translation, among the delivery of other courses.


Angelelli, Claudia (2009) Using a rubric to assess translation ability: Defining the construct. In Claudia V. Angelelli and Holly E. Jacobson (eds) Testing and Assessment in Translation and Interpreting Studies: A Call for Dialogue between Research and Practice, 13–47. Amsterdam: John Benjamins. https://doi.org/10.1075/ata.xiv.03ang

Angelelli, Claudia and Holly E. Jacobson (2009) Testing and Assessment in Translation and Interpreting Studies: A Call for Dialogue between Research and Practice. Amsterdam: John Benjamins. https://doi.org/10.1075/ata.xiv

Beeby, Allison, Mónica Fernández, Olivia Fox, Amparo Albir, Inna Kozlova, Anna Kuznik, Wilhelm Neunzig, Patricia Rodríguez et al. (2009) Results of the validation of the PACTE translation competence model: Acceptability and decision making. Across Languages and Cultures 10 (2): 207–230. https://doi.org/10.1556/Acr.10.2009.2.3

Colina, Sonia (2009) Further evidence for a functionalist approach to translation quality evaluation. Target: International Journal of Translation Studies 21 (2): 235–264. https://doi.org/10.1075/target.21.2.02col

Colman, Amy, Winibert Segers and Heidi Verplaetse (2021a) PIE & ATA: An innovative approach to translation evaluation. Conference session, Applied Linguistics and Professional Practice (ALAPP) 2021, Translational Data Analytics Institute, Ohio State University, 15–17 September. https://lirias.kuleuven.be/retrieve/626331

Colman, Amy, Winibert Segers and Heidi Verplaetse (2021b) The PIE method (Preselected Items Evaluation): An innovative approach to translation evaluation in translator training. Conference session, TRANSLATA IV, 4th International Conference on Translation and Interpreting Studies, Future Perspectives in Translation and Interpreting Studies, University of Innsbruck, Austria, 20–22 September. https://lirias.kuleuven.be/retrieve/628950

Colman, Amy, Winibert Segers and Heidi Verplaetse (2021c) Refining the PIE method (Preselected Items Evaluation) in translator training. Current Trends in Translation Teaching and Learning E 8: 236–276. https://doi.org/10.51287/cttle20218

Colman, Amy, Heidi Verplaetse and Winibert Segers (2022) Directionality in translator training: Contrastive evaluation of L1 and L2 translations using the PIE method (Preselected Items Evaluation) and the ATA Framework for Standardized Error Marking. Transletters: International Journal of Translation and Interpreting 6: 31–62.

Daems, Joke, Lieve Macken and Sonia Vandepitte (2014) On the origin of errors: A fine-grained analysis of MT and PE errors and their relationship. In Proceedings of the Ninth International Conference on Language Resources and Evaluation. Reykjavik, Iceland, 26–31 May 2014, 62–66. Luxembourg: European Language Resources Association (ELRA).

Doyle, Michael S. (2003) Translation pedagogy and assessment: Adopting ATA’s framework for Standard Error Marking. ATA Chronicle 2003 (November–December): 21–28.

Drugan, Joanna (2013) Quality in Professional Translation: Assessment and Improvement. London: Bloomsbury.

Ebel, Robert L. (1979) Essentials of Educational Measurement. Englewood Cliffs, NJ: Prentice Hall.

Eyckmans, June and Philippe Anckaert (2017) Item-based assessment of translation competence: Chimera of objectivity versus prospect of reliable measurement. Linguistica Antverpiensia (new series) 16: 40–56. https://doi.org/10.52034/lanstts.v16i0.436

Eyckmans, June, Philippe Anckaert and Winibert Segers (2009) The perks of norm-referenced translation evaluation. In Claudia Angelelli and Holly Jacobson (eds) Testing and Assessment in Translation and Interpreting Studies: A Call for Dialogue between Research and Practice, 73–93. Amsterdam: John Benjamins. https://doi.org/10.1075/ata.xiv.06eyc

Huertas Barros, Elsa and Juliet Vine (2018) Current trends on MA translation courses in the UK: Changing assessment practices on core translation modules. Interpreter and Translator Trainer 12 (1): 5–24. https://doi.org/10.1080/1750399X.2017.1400365

Koby, Geoffrey S. (2015) The ATA flowchart and framework as a differentiated error-marking scale in translation teaching. In Ying Cui and Wei Zhao (eds) Handbook of Research on Teaching Methods in Language Translation and Interpretation, 220–253. Hershey, PA: IGI Global. https://doi.org/10.4018/978-1-4666-6615-3.ch013

Koby, Geoffrey S. and Brian J. Baer (2005) From professional certification to the translator training classroom: Adapting the ATA error marking scale. Translation Watch Quarterly 1 (1): 33–45.

Koby, Geoffrey S., Paul Fields, Daryl R. Hague, Arle Lommel and Alan Melby (2014) Defining translation quality. Tradumàtica 12: 413–420. https://doi.org/10.5565/rev/tradumatica.76

Koby, Geoffrey S. and Isabel Lacruz (2017) The thorny problem of translation and inter­preting quality. Linguistica Antverpiensia (new series) 16: 1–12. https://doi.org/10.52034/lanstts.v16i0.487

Kockaert, Hendrik and Winibert Segers (2014) Evaluation de la traduction: La méthode PIE (Preselected Items Evaluation). Turjuman 23 (2): 232–250.

Kockaert, Hendrik and Winibert Segers (2017) Evaluation of legal translations: PIE method (Preselected Items Evaluation). Journal of Specialised Translation 27: 148–163.

Kübler, Natalie, Alexandra Mestivier and Mojca Pecman (2018) Teaching specialised translation through corpus linguistics: Translation quality assessment and methodology evaluation and enhancement by experimental approach. Meta 63 (3): 807–825. https://doi.org/10.7202/1060174ar

Mariana, Valerie, Troy Cox and Alan Melby (2015) The multidimensional quality metrics (MQM) framework: A new framework for translation quality assessment. Journal of Specialised Translation 23: 137–161.

Phelan, Mary (2017) Analytical assessment of legal translation: A case study using the American Translators Association framework. Journal of Specialised Translation 27: 189–210.

Segers, Winibert (2007) IJkpuntenmethode. In Chris Van de Poel and Winibert Segers (eds) Vertalingen objectief evalueren: Matrices en ijkpunten, 21–25. Leuven: Acco.

Segers, Winibert and Hendrik Kockaert (2016) Can subjectivity be avoided in translation evaluation? In Marcel Thelen, Gys-Walt van Egdom, Dirk Verbeeck and Barbara Lewandowska-Tomaszczyk (eds) Translation and Meaning: New Series, Volume 1, 69–78. Lódz Studies in Language 41. Frankfurt am Main: Peter Lang.

Segers, Winibert, Hendrik Kockaert and Bert Wylin (2018) Vertaalevaluatie en subjectiviteit. Tijdschrift N/f 13: 41–51.

Segers, Winibert and Gys-Walt van Egdom (2018) De kwaliteit van vertalingen: Een terminologie van de vertaalevaluatie. Kalmthout, Belgium: Pelckmans Pro.

Tijtgat, Evelien and Winibert Segers (2019) Wat is een goede vertaling? Vertaalevaluatie: methodes en technieken. In Gert De Sutter and Isabelle Delaere (eds) In balans. Een inleiding tot vertaal- en tolkwetenschap, 307–328. Leuven: Acco.

van Egdom, Gys-Walt, Heidi Verplaetse, Iris Schrijver, Hendrik Kockaert, Winibert Segers, Jasper Pauwels, Henri Bloemen and Bert Wylin (2018) How to put the translation test to the test? On Preselected Items Evaluation and perturbation. In Elsa Huertas Barros, Sonia Vandepitte and Emilia Iglesias-Fernández (eds) Quality Assurance and Assessment Practices in Translation and Interpreting, 26–56. Hershey, PA: IGI Global. https://doi.org/10.4018/978-1-5225-5225-3.ch002

Verplaetse, Heidi (2022) Translation quality in student specialized translation: The impact of corpus use. In Marie-Aude Lefer and Sylviane Granger (eds) Extending the Scope of Corpus-Based Translation Studies, 209–236. London: Bloomsbury Academic. https://doi.org/10.5040/9781350143289.0017

Verplaetse, Heidi and An Lambrechts (2022) Analytical translation quality assessment: Insights for translators’ training. In Said M. Shiyab (ed.) Research into Translation and Training in Arab Academic Institutions, 89–112. New York: Routledge. https://doi.org/10.4324/9781003034667-6

Waddington, Christopher (2004) Should student translations be assessed holistically or through error analysis? Lebende Sprachen 49 (1): 28–35. https://doi.org/10.1515/LES.2004.28






How to Cite

Colman, A., Segers, W., & Verplaetse, H. (2023). Extending Preselected Items Evaluation (PIE) with the ATA (American Translators Association) error categories for objective and targeted translation evaluation. Journal of Applied Linguistics and Professional Practice, 17(3), 274-293. https://doi.org/10.1558/jalpp.21681