A framework to accumulate validity evidence for intended and actual consequences of Korean language proficiency testing
DOI:
https://doi.org/10.1558/jmtp.19451Keywords:
Validation, critical language testing, assessment use argument, critical discourse analysis, TOPIK (Test of Proficiency in Korean)-speaking, Korean immigrantsAbstract
Korea is rapidly becoming a multicultural society. As the notion of purity in ethnic bloodline has long been a requisite for ‘Koreanness’, however, the cultural nationalist identity seems to be maintained on proficiency levels tested by the most powerful, government-instituted Korean language test, TOPIK (Test of Proficiency in Korean). Research on immigrants’ language practices, ideologies and (language) testing requirements has been bleak in Korean contexts, and this article proposes one such framework by combining critical discursive approaches and contemporary argument-based approaches to validation to evaluate a newly developed test, TOPIK-speaking, and related policy issues. Drawing on Shohamy’s critical language testing (CLT), Fairclough’s critical discourse analysis (CDA), and Bachman and Palmer’s assessment use argument (AUA), it illustrates the unique features of a validation framework and uses TOPIK-speaking as an example to collect and evaluate empirical evidence for its intended and actual consequences. The practice of testing consequences should be discursively analysed as a multilayered phenomenon, reinforced by discursive conflicts, such as represented in the media.
References
Bachman, L. F. and Palmer, A. S. (1996). Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford University Press.
Bachman, L. F. and Palmer, A. S. (2010). Language Assessment in Practice: Developing Language Assessments and Justifying Their Use in the Real World. Oxford University Press.
Barakos, E. (2016). Language policy and critical discourse studies: Toward a combined approach. In E. Barakos and J. Unger (Eds.), Discursive Approaches to Language Policy (pp. 23–49). Palgrave Macmillan. https://doi.org/10.1057/978-1-137-53134-6_2
Barakos, E. and Unger, W. J. (2016). Introduction: What are discursive approaches to language policy. In E. Barakos and J. Unger (Eds.), Discursive Approaches to Language Policy (pp. 1–10). Palgrave Macmillan. https://doi.org/10.1057/978-1-137-53134-6_1
Bartlett, T. (2010). Towards intervention in positive discourse analysis. In C. Coffin, K. O’Halloran and T. Illis. (Eds.), Applied Linguistics Methods: A Reader (pp. 133–147). Routledge. https://doi.org/10.4324/9781315739342-10
Baxter, J. (2010). Discourse-analytic approaches to text and talk. In L. Litosseliti (Ed.), Research Methods in Linguistics (pp. 117–137). Continuum.
Blackledge, A. (2005). Discourse and Power in a Multilingual World. John Benjamins. https://doi.org/10.1075/dapsac.15
Blackledge, A. (2009). As a country we do expect: The further extension of language testing regimes in the United Kingdom. Language Assessment Quarterly, 6, 6–16. https://doi.org/10.1080/15434300802606465
Borsboom, D., Mellenberg, G. J. and Van Herden, J. (2004). The concept of validity. Psychological Review, 111.4, 106–171. https://doi.org/10.1037/0033-295X.111.4.1061
Chapelle, C. A., Enright, M. K. and Jamieson, J. (Eds.). (2008). Building a Validity Argument for the Test of English as a Foreign Language. Routledge.
Cheng, L. (2014). Consequences, impact, and washback. In A. J. Kunnan (Ed.), The Companion to Language Assessment (pp. 1130–1146). John Wiley & Sons. https://doi.org/10.1002/9781118411360.wbcla071
Chouliaraki, L. and Fairclough, N. (1999). Discourse in Late Modernity: Rethinking Critical Discourse Analysis. Edinburgh University Press.
Colier, J.-K. (2020). Social consequences of testing for language-minoritized bilinguals in the United States. Language and Education, 35.1, 96–99. https://doi.org/10.1080/09500782.2020.1807564
Extra, G., Spotti, M. and Van Avermaet, P. (2009). Language Testing, Migration and Citizenship: Cross-National Perspectives on Integration Regimes. Continuum.
Fairclough, N. (1992). Discourse and Social Change. Blackwell.
Fairclough, N. (2001). Language and Power (2nd edition). Longman.
Fairclough, N. (2013). Critical discourse analysis and critical policy studies. Critical Policy Studies, 7.2, 177–197. https://doi.org/10.4324/9781315834368
Fulcher, G. (2009). Test use and political philosophy. Annual Review of Applied Linguistics, 29, 3–20. https://doi.org/10.1017/S0267190509090023
Fulcher, G. and Davidson, F. (2007). Language Testing and Assessment. Routledge.
Gee, J. P. (2014). An Introduction to Discourse Analysis: Theory and Practice (2nd edition). Routledge. https://doi.org/10.4324/9781315819679
Ghazarian, P. G. (2018). Multiculturalism in South Korea: Examining government aspirations through the second basic plan for immigration. Multicultural Education Review, 10.1, 18–34. https://doi.org/10.1080/2005615X.2018.1423538
Hogan-Brun, G., Mar-Molinero, C. and Stevenson, P. (Eds.). (2009). Discourses on Language and Integration: Critical Perspectives on Language Testing Regimes in Europe. John Benjamins. https://doi.org/10.1075/dapsac.33
Horner, K. (2009). Language, citizenship and Europeanization: Unpacking the discourse of integration. In G. Hogan-Brun, C. Mar-Molinero and P. Stevenson (Eds.), Discourses on Language and Integration (pp. 109–128). John Benjamins. https://doi.org/10.1075/dapsac.33.10hor
Hughes, J. M. (2018). Progressing positive discourse analysis and/in critical discourse studies: Reconstructing resistance through progressive discourse analysis. Review of Communication, 18.3, 193–211. https://doi.org/10.1080/15358593.2018.1479880
Hundt, D. (2016). Public opinion, social cohesion, and the politics of immigration in South Korea. Contemporary Politics, 22.4, 487–504. https://doi.org/10.1080/13569775.2016.1186346
Im, G.-H., Shin, D. and Cheng, L. (2019). Critical review of validation models in language testing: Their conceptual limitations and future directions for validation research. Language Testing in Asia, 9, 14. https://doi.org/10.1186/s40468-019-0089-4
Im, G.-H., Shin, D. and Park, S. (2021). Suggesting a policy-driven approach to validation in the context of Test of Proficiency in Korean (TOPIK). Current Issues in Language Planning. https://doi.org/10.1080/14664208.2021.1984674
Johnson, D. C. (2016). Introduction to part I: Theoretical foundations for discursive approaches to language policy. In E. Barakos and J. Unger (Eds.), Discursive Approaches to Language Policy (pp. 11–21). Palgrave Macmillan.
Kane, M. T. (2006). Validation. In R.L. Brennan (Ed.), Educational Measurement (4th edition) (pp. 17–64). American Council on Education and Praeger.
Karami, H. (2013). The quest for fairness in language testing. Educational Research and Evaluation, 19.2–3, 158–169. https://doi.org/10.1080/13803611.2013.767618
Khan, K. (2019). Becoming a Citizen: Linguistic Trials and Negotiations. Bloomsbury.
Kim, H. (20 January 2019). TOPIK to introduce speaking test from 2023. The Korea Times. Retrieved on 31 January 2021 from: https://koreatimes.co.kr/www/nation/2020/04/181_262337.html
Koch, M. J. and DeLuca, C. (2012). Rethinking validation in complex high-stakes assessment contexts. Assessment in Education: Principles, Policy & Practice, 19.1, 99–116. https://doi.org/10.1080/0969594X.2011.604023
Korea Immigration Service. (2009). Basic Plan for Immigration Policy. Korea Immigration Service, Ministry of Justice.
Korea Immigration Service. (2012). The Second Basic Plan for Immigration Policy. Korea Immigration Service, Ministry of Justice.
Korea Immigration Service. (2018). The Third Basic Plan for Immigration Policy. Korea Immigration Service, Ministry of Justice.
Lin, A. (2014). Critical discourse analysis in applied linguistics: A methodological review. Annual Review of Applied Linguistics, 34, 213–232. https://doi.org/10.1017/S0267190514000087
Lynch, B. K. (2001). Rethinking assessment from a critical perspective. Language Testing, 18, 351–372. https://doi.org/10.1191/026553201682430085
Macgilchrist, F. M. (2007). Positive discourse analysis: Contesting dominant discourses by reframing the issues. Critical Approaches to Discourse Analysis across Disciplines, 1.1, 74–94.
McCarthy, T. L. (Ed.). (2011). Ethnography and Language Policy. Routledge.
McNamara, T. (2010). The use of language tests in the service of policy: Issues of validity. Rev. franc. de linguistique appiquee, XV, 7–23. https://doi.org/10.3917/rfla.151.0007
McNamara, T. and Roever, C. (2006). Language Testing: The Social Dimension. Blackwell.
McNamara, T. and Ryan, K. (2011). Fairness versus justice in language testing: The place of English literacy in the Australian citizenship test. Language Assessment Quarterly, 8.2, 161–178. https://doi.org/10.1080/15434303.2011.565438
McNamara, T. and Shohamy, E. (2008). Language tests and human rights. International Journal of Applied Linguistics, 18.1, 89–95.
https://doi.org/10.1111/j.1473-4192.2008.00191.x
Mehrens, W. (1997). The consequences of consequential validity. Educational Measurement: Issues and Practice, 16.2, 16–18. https://doi.org/10.1111/j.1745-3992.1997.tb00588.x
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd edition) (pp. 13–103). American Council on Education and Macmillan.
Messick, S. (1996). Validity and washback in language testing. Language Testing, 13.3, 241–256. https://doi.org/10.1177/026553229601300302
Milani, T. (2008). Language testing and citizenship: A language ideological debate in Sweden. Language in Society, 37.1, 27–59. https://doi.org/10.1017/S0047404508080020
Moss, P. A. (1998). The role of consequences in validity theory. Educational Measurement: Issues and Practice, 17.2, 6–12. https://doi.org/10.1111/j.1745-3992.1998.tb00826.x
Popham, W. J. (1997). Consequential validity: Right concern – wrong concept. Educational Measurement: Issues and Practice, 16.2, 9–13.
https://doi.org/10.1111/j.1745-3992.1997.tb00586.x
Rogers, R. (2011). An Introduction to Critical Discourse Analysis in Education (2nd edition). Routledge. https://doi.org/10.4324/9780203836149
Ryan, K. (2002). Assessment validation in the context of high-stakes assessment. Educational Measurement: Issues and Practice, 211, 7–15. https://doi.org/10.1111/j.1745-3992.2002.tb00080.x
Schissel, J. L. (2019). Social Consequences of Testing for Language-Minoritized Bilinguals in the United States. Multilingual Matters. https://doi.org/10.21832/SCHISS2708
Shepard, L. A. (1993). Evaluating test validity. Review of Research in Education, 19, 405–450. https://doi.org/10.2307/1167347
Shepard, L. A. (1997). The centrality of test use and consequences for test validity. Educational Measurement: Issues and Practice, 16.2, 13–24. https://doi.org/10.1111/j.1745-3992.1997.tb00585.x
Shin, D. (2019). Analyzing media discourse on the development of the National English Ability Test (NEAT) in South Korea. Language Testing in Asia, 9, 4. https://doi.org/10.1186/s40468-019-0081-z
Shin, D. (2021). Inviting critical discourse analysis in the argument-based approach to validation: The inference of social impact in policy-driven testing. Unpublished manuscript.
Shin, D. and Cho, E. (2020). Discursive conflicts on National English Ability Test in news media: How was a government-led test of English suspended in Korea? Language Testing in Asia, 10.5. https://doi.org/10.1186/s40468-020-00100-7
Shin, D. and Cho, E. (2021). The National English Ability Test in Korea and its legitimising discourses. Journal of Multilingual and Multicultural Development, 42.6, 537–550. https://doi.org/10.1080/01434632.2020.1745818
Shohamy, E. (2001). The Power of Tests: A Critical Perspective on the Uses of Language Tests. Pearson.
Shohamy, E. (2006). Language Policy: Hidden Agendas and New Approaches. Routledge. https://doi.org/10.4324/9780203387962
Shohamy, E. (2007). Language tests as language policy tools. Assessment in Education, 14.1, 117–130. https://doi.org/10.1080/09695940701272948
Shohamy, E. (2009). Language tests for immigrants: Why languages? Why tests? Why citizenship? In G. Hogan-Brun, C. Mar-Molinero and P. Stevenson (Eds.), Discourses on Language and Integration (pp. 45–59). John Benjamins. https://doi.org/10.1075/dapsac.33.07sho
Shohamy, E. and McNamara, T. (2009). Language tests for citizenship, immigration and asylums. Language Assessment Quarterly, 6, 1–5. https://doi.org/10.1080/15434300802606440
Toulmin, S. (2003). The Uses of Argument (updated edition). Cambridge University Press. https://doi.org/10.1017/CBO9780511840005
Van Avermaet, P. (2009). Fortress Europe? Language policy regimes for immigration and citizenship. In G. Hogan-Brun, C. Mar-Molinero and P. Stevenson (Eds.), Discourse on Language and Integration (pp. 15–43). John Benjamins. https://doi.org/10.1075/dapsac.33.06ave
Wodak, R. (2001). What CDA is about – A summary of its history, important concepts and its developments. In R. Wodak and M. Meyer (eds.), Methods of Critical Discourse Analysis (pp. 1–14). Sage. https://doi.org/10.4135/9780857028020.n1
Wodak, R. and Boukala, S. (2015). European identities and the revival of nationalism in the European Union: A discourse-historical approach. Journal of Language and Politics, 14.1, 87–109. https://doi.org/10.1075/jlp.14.1.05wod
Wodak, R. and Meyer, M. (2015). Critical discourse studies: History, agenda, theory, and methodology. In R. Wodak and M. Meyer (Eds.), Methods of Critical Discourse Studies (3rd edition) (pp. 1–22). Sage.
Xi, X. (2010). How do we go about investigation test fairness? Language Testing, 27, 147–170. https://doi.org/10.1177/0265532209349465
