Examining the Effect of Item Difficulty and Rater Leniency on Iranian Test Takers’ Performance on WDCT and DSAT: A Comparative Study

Shahi, Reza; Ravand, Hamdollah; Rohani, Gholam Reza

doi:10.22034/ijlt.2024.454478.1341

Examining the Effect of Item Difficulty and Rater Leniency on Iranian Test Takers’ Performance on WDCT and DSAT: A Comparative Study

Document Type : Original Research Article

Authors

Reza Shahi ¹

Hamdollah Ravand ²

Gholam Reza Rohani ³

¹ English language Teaching, Ilam University, Ilam, Iran

² Associate professor, Vali-e-Asr University of Rafsanjan, Kerman

³ Assistant professor, Shahid Bahonar University of Kerman

10.22034/ijlt.2024.454478.1341

Abstract

The current paper intends to exploit the Many Facet Rasch Model to investigate and compare the impact of situations (items) and raters on test takers' performance on the Written Discourse Completion Test (WDCT) and Discourse Self-Assessment Tests (DSAT). In this study, the participants were 110 English as a Foreign Language (EFL) students at Vali-e-Asr University of Rafsanjan in Iran. The students were asked to complete the WDCT and rate themselves on that test. Four raters scored the WDCT tests. According to the FACETS results, there were significant differences in students' performance between the two methods. The stable fit statistics and differing levels of difficulty measures for each test method indicated that each test had a unique way of differentiating the test taker's pragmatic ability. Based on the results, both DSAT and WDCT are acceptable measures for pragmatic ability; however, there are some fitness problems in DSAT. This shows the unpredictable pattern of ratings in the DSAT. It is recommended to have rater training to obtain more accurate results from the DSAT. Finally, the implications were discussed.

Keywords

Pragmatic test

Many Facet Rasch Model

WDCT

DSAT