Translation and Adaptation of Tests: Lessons Learned and Recommendations for Countries Participating in timss, pisa and other International Comparisons
Keywords:
Educational tests, international tests, test translation, TIMSS, PISA.Abstract
In this paper we present a conceptual model and methodology for the review of translated tests in the context of such international comparisons as the Trends in International Mathematics and Science Study (TIMSS) and the Programme for International Student Assessment (PISA). We also present the results of an investigation into the quality of the Mexican translation of the TIMSS-1995 into the Spanish language. We identified translation errors in a significant percentage of the items, as well as relatively high correlations between the severity of translation errors and the items’ p-values. These findings indicate that our error-coding system is highly sensitive to test-translation error. The results underscore the need for improved translation and translation-review procedures in international comparisons. In our opinion, to implement the guidelines properly for test translation in international comparisons, each participating country needs to have internal procedures that would ensure a rigorous review of its own translations. The article concludes with four recommendations for countries participating in international comparisons. These recommendations relate to: (a) the characteristics of the individuals in charge of translating instruments; (b) the use of review, not simply at the end of the process, but during the process of test translation; (c) the minimum time needed for various translation review iterations to take place; and (d) the need for proper documentation of the entire process of test translation.Downloads
Download data is not yet available.
References
American Educational Research Association, American Psychological Association-National Council on Measurement in Education. (1999). Standards for Educational and Psychological Testing. Washington, DC: Autor.
American Translators Association. (2003). Framework for standard error marking and explanation. Consultado el 10 de octubre de 2003 en: http://www.atanet.org
Backhoff, E. y Solano-Flores, G. (2003). Tercer estudio internacional de matemáticas y ciencias naturales (TIMSS): resultados de México en 1995 y 2000. (Col. Cuadernos de Investigación No. 4). México: Instituto Nacional para la Evaluación de la Educación.
Behling, O. y Law, K. S. (2000). Translating questionnaires and other research instruments: Problems and solutions. Thousand Oaks, CA: Sage.
Ercikan, K. (1998). Translation effects in international assessment. International Journal of Educational Research, 29, 543-553.
Greenfield, P. M. (1997). You can't take it with you: Why ability assessments don't cross cultures. American Psychologist, 52 (10), 1115-1124.
Grisay, A. (2002). Translation and cultural appropriateness of the test and survey material. En R. Adams y M. Wu (Eds.), PISA 2000 Technical Report (pp. 42-54). París: Organisation for Economic Co-operation and Development
Hambleton, R.K. (1994). Guidelines for adapting educational and psychological tests: A progress report. European Journal of Psychological Assessment, 10 (3), 229-244.
Hambleton, R. K. (2005). Issues, designs, and technical guidelines for adapting tests into multiple languages and cultures. En R. K. Hambleton, P. Merenda y C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 3-38). Mahwah, NJ: Erlbaum.
Maxwell, B. (1996). Translation and cultural adaptation of the survey instruments. En M.O. Martin y D.L. Kelly (Eds), Third International Mathematics and Science Studdy (TIMSS) Technical Report, Volume 1: Design and Development. Chestnut HIll, MA: Boston College.
Mullis, I.V.S., Kelly, D.L., y Haley, K. (1996). Translation verification procedures. En M.O. Martin e I.V.S. Mullis (Eds.), Third International Mathematics and Science Study: Quality assurance in data collection. Chestnut Hill, MA: Boston College. Consultado el 10 de octubre de 2003 en: http://timss.bc.edu/timss1995i/TIMSSPDF/QACHP1.PDF
O’Connor, K. M. y Malak, B. (2000). Translation and cultural adaptation of the TIMSS instruments. En M. O. Martin, K. D.Gregory y S. E. Stemler, (Eds.), TIMSS 1999 technical report (pp. 89-100). Chestnut Hill, MA: International Study Center, Boston College.
Solano-Flores, G. (en prensa). Successive test development. En C.R. Reynolds, R.W. Kamphaus y C. DiStefano (Eds.), Encyclopedia of psychological and educational testing: Clinical and psychoeducational applications. New York: Oxford University Press.
Solano-Flores, G. y Backhoff-Escudero, E. (2003). La traducción de pruebas en las comparaciones internacionales: un estudio preliminar (Informe técnico para el Instituto Nacional para la Evaluación de la Educación). México, D.F.: Instituto Nacional para la Evaluación de la Educación.
Solano-Flores, G., Contreras-Niño, L. A. y Backhoff-Escudero, E. (2005, 12-14 de abril). The Mexican translation of TIMSS-95: Test translation lessons from a post-mortem study. Trabajo presentado en la Annual Meeting of the National Council on Measurement in Education. Montreal, Québec, Canadá.
Solano-Flores, G., Contreras-Niño, L. A., Backhoff-Escudero, E. y Andrade, E. (2006). Development and evaluation of software for test translation review sessions. Poster presented at the 5th Conference of the International Test Commission: Psychological and Educational Test Adaptation Across Languages and Cultures: Building Bridges Among People. Brussels, Belgium, July 6-8.
Solano-Flores, G. y Nelson-Barber, S. (2001). On the cultural validity of science assessments. Journal of Research in Science Teaching, 38 (5), 553-573.
Solano-Flores, G. y Trumbull, E. (2003). Examining language in context: The need for new research and practice paradigms in the testing of English-language learners. Educational Researcher, 32 (2), 3-13.
Solano-Flores, G., Trumbull, E. y Nelson-Barber, S. (2002). Concurrent development of dual language assessments: An alternative to translating tests for linguistic minorities. International Journal of Testing, 2 (2), 107-129.
Solano-Flores, G., Trumbull, E., y Kwon, M. (2003, 21-25 de abril). The metrics of linguistic complexity and the metrics of student performance in the testing of English language learners. Symposio presentado en 2003 Annual Meeting de la American Evaluation Research Association. Chicago, IL.
Vijver, F. van de y Hambleton, R. J. (1996). Translating tests: Some practical guidelines. European Psychologist 1, 89-99.
Vijver, F. van de y Poortinga, Y. H. (2005). Conceptual and methodological issues in adapting tests. En R. K. Hambleton, P. Merenda y C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 39-63). Mahwah, NJ: Erlbaum.
American Translators Association. (2003). Framework for standard error marking and explanation. Consultado el 10 de octubre de 2003 en: http://www.atanet.org
Backhoff, E. y Solano-Flores, G. (2003). Tercer estudio internacional de matemáticas y ciencias naturales (TIMSS): resultados de México en 1995 y 2000. (Col. Cuadernos de Investigación No. 4). México: Instituto Nacional para la Evaluación de la Educación.
Behling, O. y Law, K. S. (2000). Translating questionnaires and other research instruments: Problems and solutions. Thousand Oaks, CA: Sage.
Ercikan, K. (1998). Translation effects in international assessment. International Journal of Educational Research, 29, 543-553.
Greenfield, P. M. (1997). You can't take it with you: Why ability assessments don't cross cultures. American Psychologist, 52 (10), 1115-1124.
Grisay, A. (2002). Translation and cultural appropriateness of the test and survey material. En R. Adams y M. Wu (Eds.), PISA 2000 Technical Report (pp. 42-54). París: Organisation for Economic Co-operation and Development
Hambleton, R.K. (1994). Guidelines for adapting educational and psychological tests: A progress report. European Journal of Psychological Assessment, 10 (3), 229-244.
Hambleton, R. K. (2005). Issues, designs, and technical guidelines for adapting tests into multiple languages and cultures. En R. K. Hambleton, P. Merenda y C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 3-38). Mahwah, NJ: Erlbaum.
Maxwell, B. (1996). Translation and cultural adaptation of the survey instruments. En M.O. Martin y D.L. Kelly (Eds), Third International Mathematics and Science Studdy (TIMSS) Technical Report, Volume 1: Design and Development. Chestnut HIll, MA: Boston College.
Mullis, I.V.S., Kelly, D.L., y Haley, K. (1996). Translation verification procedures. En M.O. Martin e I.V.S. Mullis (Eds.), Third International Mathematics and Science Study: Quality assurance in data collection. Chestnut Hill, MA: Boston College. Consultado el 10 de octubre de 2003 en: http://timss.bc.edu/timss1995i/TIMSSPDF/QACHP1.PDF
O’Connor, K. M. y Malak, B. (2000). Translation and cultural adaptation of the TIMSS instruments. En M. O. Martin, K. D.Gregory y S. E. Stemler, (Eds.), TIMSS 1999 technical report (pp. 89-100). Chestnut Hill, MA: International Study Center, Boston College.
Solano-Flores, G. (en prensa). Successive test development. En C.R. Reynolds, R.W. Kamphaus y C. DiStefano (Eds.), Encyclopedia of psychological and educational testing: Clinical and psychoeducational applications. New York: Oxford University Press.
Solano-Flores, G. y Backhoff-Escudero, E. (2003). La traducción de pruebas en las comparaciones internacionales: un estudio preliminar (Informe técnico para el Instituto Nacional para la Evaluación de la Educación). México, D.F.: Instituto Nacional para la Evaluación de la Educación.
Solano-Flores, G., Contreras-Niño, L. A. y Backhoff-Escudero, E. (2005, 12-14 de abril). The Mexican translation of TIMSS-95: Test translation lessons from a post-mortem study. Trabajo presentado en la Annual Meeting of the National Council on Measurement in Education. Montreal, Québec, Canadá.
Solano-Flores, G., Contreras-Niño, L. A., Backhoff-Escudero, E. y Andrade, E. (2006). Development and evaluation of software for test translation review sessions. Poster presented at the 5th Conference of the International Test Commission: Psychological and Educational Test Adaptation Across Languages and Cultures: Building Bridges Among People. Brussels, Belgium, July 6-8.
Solano-Flores, G. y Nelson-Barber, S. (2001). On the cultural validity of science assessments. Journal of Research in Science Teaching, 38 (5), 553-573.
Solano-Flores, G. y Trumbull, E. (2003). Examining language in context: The need for new research and practice paradigms in the testing of English-language learners. Educational Researcher, 32 (2), 3-13.
Solano-Flores, G., Trumbull, E. y Nelson-Barber, S. (2002). Concurrent development of dual language assessments: An alternative to translating tests for linguistic minorities. International Journal of Testing, 2 (2), 107-129.
Solano-Flores, G., Trumbull, E., y Kwon, M. (2003, 21-25 de abril). The metrics of linguistic complexity and the metrics of student performance in the testing of English language learners. Symposio presentado en 2003 Annual Meeting de la American Evaluation Research Association. Chicago, IL.
Vijver, F. van de y Hambleton, R. J. (1996). Translating tests: Some practical guidelines. European Psychologist 1, 89-99.
Vijver, F. van de y Poortinga, Y. H. (2005). Conceptual and methodological issues in adapting tests. En R. K. Hambleton, P. Merenda y C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 39-63). Mahwah, NJ: Erlbaum.
Downloads
-
HTML
-
PDF
-
HTML
-
PDFENGLISH 558
Article abstract page views: 1197
Published
2006-11-01
