Using the Rasch Model to Calibrate Items for an English as a Second Language Reading Comprehension Computer Adaptive Placement Test

Using the Rasch Model to Calibrate Items for an English as a Second Language Reading Comprehension Computer Adaptive Placement Test

Tzemin Chung* tzemin_chung@CommonTown.com CommonTown (Singapore)
Mohd Zali Mohd Nor mohd.zali@my-newstar.com Newstar Agencies (Malaysia)
Richard Yan richard_yan@CommonTown.com CommonTown (Singapore)
Peing Ling Loo joel@CommonTown.com CommonTown (Singapore)
Summary: 
This study aimed to calibrate items aiming at measuring English reading comprehension ability in students who learn English as a second language. A total of 571 multiple-choice items were administered to 466 participants from 14 schools via a computer-adaptive test. These participants were mainly secondary school students. Data were analyzed using the Rasch model for dichotomous items. Results indicated that the instrument was sufficiently unidimensional and was quite well targeted at the students. It was able to measure the English abilities of secondary school L2 students. Item measures were also compared to the expert’s levelling of item difficulty levels. Based on the Pearson Correlation coefficient of 0.77, the items demonstrate a moderate shared variance, indicating a reasonably positive correlation with the expert’s levelling.
Keywords: 
Rasch model; English as a second language
reading comprehension
item calibration; computer adaptive placement test.
Refers: 

[1] Bond, T. & Fox, C. (2015). Applying the Rasch Model: Fundamental Measurement in the Human Sciences (3rd ed.). New York: Routledge

[2] Brown, H. D. (2015). Principles of language learning and teaching (6th ed.). Pearson Education

[3] Chapelle, C. A., & Hegelheimer, V. (2004). Theorizing and testing technology-mediated second language learning: Issues and hypotheses. Annual Review of Applied Linguistics, 24, 223- 247.

[4] Cho, K. S. & Krashen, S. (2016). What does it take to develop a long-term pleasure reading habit? Turkish Online Journal of English Language Teaching (TOJELT), 1(1), 1-9. http://www.sdkrashen.com/content/articles/2016_ cho_and_krashen_long-term_reading.pdf

[5] Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. Harper Perennial.

[6] Cummins, J. (2000). Language, power, and pedagogy: Bilingual children in the crossfire. Multilingual Matters.

[7] Gardner, D. (2013). Vocabulary recycling in children’s authentic reading materials: A corpus-based investigation of narrow reading. Reading in a Foreign Language, 25(2), 234-253.

[8] Granger, C. (2008). Rasch Analysis is important to understand and use for measurement. Rasch Measurement Transactions, 21(3) p. 1122-3. Retrieved from https://www.rasch.org/rmt/rmt213d.htm

[9] Kelley T., Ebel R., & Linacre, J. M. (2002). Item discrimination indices. Rasch Measurement Transactions, (16)3, p.883-4.

[10] Krashen, S. D. (1981). Second language acquisition and second language learning. Oxford: Pergamon

[11] Krashen, S. D. (1982). Principles and practice in second language acquisition. Oxford: Pergamon

[12] Krashen, S. D. (1985). The input hypothesis: Issues and implications. New York: Longman

[13] Krashen, S. D. (2009). Principles and Practice in Second Language Acquisition. http://www.sdkrashen.com/ content/books/principles_and_practice.pdf

[14] Krashen, S. D. (2011). The Compelling (not just interesting) Input Hypothesis. The English Connection (KOTESOL). A Publication of KOTESOL, 15(3). http://www.sdkrashen.com/content/articles/the_ compelling_input_hypothesis.pdf

[15] Lee, S. Y. (2018). The Power of Story in SLA: Insights from Research [monograph]. Reconceptualizing English Language Teaching and Learning in the 21st Century A Special Monograph in Memory of Professor Kaichong Cheung. http://www.sdkrashen.com/content/ articles/35-sy-ying_lee_(final).pdf

[16] Lewis, R. (2020, November 4). What Is Comprehensible Input and Why Does It Matter for Language Learning? https://www.leonardoenglish.com/blog/ comprehensible-input

[17] Linacre, J.M. (2000). Redundant items, overfit and measure bias. Rasch Measurement Transactions, 14(3), 755. Retrieved from https://www.rasch.org/rmt/rmt143a. htm

[18] Linacre, J. M. (2002). What do infit and outfit, meansquare and standardized mean? Rasch Measurement Transactions, 16(2), p.878. Retrieved from http:// www.rasch.org/rmt/rmt162f.htm

[19] Linacre, J. M. (2014a). Correlations: Point-biserial, point-measure, residual. Help for Winsteps Rasch Measurement Software: www.winsteps.com. Retrieved from http://www.winsteps.com/winman/ correlations.htm

[20] Linacre, J. M. (2014b). Dimensionality: Contrasts & variances. Help for Winsteps Rasch Measurement Software: www.winsteps.com. Retrieved from http:// www.winsteps.com/winman/principalcomponents.htm

[21] Linacre, J. M. (2014c). Dimensionality: When is a test multidimensional? Help for Winsteps Rasch Measurement Software: www.winsteps.com. Retrieved from http://www.winsteps.com/winman/ reliability.htm

[22] Linacre, J. M. (2014d).Fit diagnosis: Infit outfit mean-square standardized. Help for Winsteps Rasch Measurement Software: www.winsteps.com. Retrieved from http:// www.winsteps.com/winman/misfitdiagnosis.htm

[23] Linacre, J. M. (2014e). Reliability and separation of measures. Help for Winsteps Rasch Measurement Software: www.winsteps.com. Retrieved from http:// www.winsteps.com/winman/reliability.htm

[24] Linacre, J. M. (2014f). Table 23.0 variance components for items. Help for Winsteps Rasch Measurement Software: www.winsteps.com. Retrieved from http:// www.winsteps.com/winman/reliability.htm

[25] Linacre, J. M. (2014g). Winsteps Rasch measurement software [Computer software]. Retrieved from www. winsteps.com

[26] Linacre, J. M. (2014h). Item discrimination or slope estimation. Help for Winsteps Rasch Measurement Software: www.winsteps.com. Retrieved from https://www.winsteps.com/winman/ discriminationestimation.htm

[27] Lunz, M. (2010). Using The very useful Wright Map. Measurement Research Associates Test Insights. https://www.rasch.org/mra/mra-01- 10.htm#:~:text=The%20Wright%20Map%20 provides%20a,how%20appropriately%20the%20 test%20measured.

[28] Marais, I. (2015). Implications of Removing Random Guessing from Rasch Item Estimates in Vertical Scaling. Journal of Applied Measurement, 16(2), 113- 128.

[29] Mason, B., & Krashen, S. (1997). Extensive reading in English as a foreign language. System, 25(1), 91-102.

[30] Oppland, R. (2016). Flow in the foreign language classroom. The Language Learning Journal, 44(4), 418-428.

[31] Renandya, W., Krashen, S., & Jacobs, G. (2018). The Potential of Series Books: How Narrow Reading Leads to Advanced L2 Proficiency. RELC Journal, 49(2), 199-211. doi: 10.1177/0033688218778549

[32] Salzberger, T. (2003). Item Information: When Gaps Can Be Bridged. Rasch Measurement Transactions, 17(1), 910-911

[33] Smith, A.B., Rush, R., Fallowfield, L.J. et al. (2008). Rasch fit statistics and sample size considerations for polytomous data. BMC Med Res Methodol 8(33). https://doi.org/10.1186/1471-2288-8-33

[34] Tennant A., & Pallant J.F. (2006). Unidimensionality matters! (A tale of two Smiths?). Rasch Measurement Transactions, 20(1), p. 1048-51. Retrieved from https://www.rasch.org/rmt/rmt201c.htm

[35] Wright, B.D., Huber, M., O’Neill, T., & Linacre, J.M. (2000). The problem of measure invariance. Rasch Measurement Transactions, 14(2), 745. Retrieved from https://www.rasch.org/rmt/rmt142d.htm

[36] Wright, B.D., & Linacre, J.M. (1994). Reasonable meansquare fit values. Rasch Measurement Transactions, 8(3), 370. Retrieved from https://www.rasch.org/rmt/ rmt83b.htm

[37] Wright, B. D., & Stone, M. (1999). Measurement Essentials (2nd ed.). Wide Range, Inc

Articles in Issue