Tabaran Institute of Higher Education
International Journal of Language Testing
2476-5880
3
1
2013
03
01
Validation of an Analytic Rating Scale for Writing: A Rasch Modeling Approach
1
10
EN
Susan
Tan
National University of Singapore, Centre for English Language Communication.
Writing assessments often make use of analytic rating scales to describe the criteria for different performance levels. However, the use of such rating scales requires a level of interpretation by raters and if several raters are involved, the reliability of examinee scores can be significantly affected (Engelhard, 1992; McNamara 1996). Variability between raters is partly managed by rater training in the use of the rating scale and this necessarily means that the rating scale itself should be well constructed and can be accurately applied to discriminate examinee performance consistently. This paper reports on the use of the Many-facets Rasch model (MFRM, Linacre, 1989) to assess the validity of a proposed analytic rating scale. The MFRM is widely used to study examinee performance and rater behavior and is useful in rating scale validation to analyze sources of variation in tests (Schaeffer, 2008). Bias analysis allows systematic subpatterns of interactions between raters and the rating scale to be examined. In this paper, scores from a set of essays rated by a team using a revised analytic descriptor were analyzed and the indices for rater severity, rater consistency, rater bias, criteria difficulty and scale functionality were studied. The findings indicate that raters were able to use the revised rating scale to discriminate performances in a consistent manner. The MFRM can contribute to improvements in rater training and rating scale development.
Performance Assessments,Rating Scales,Many-Facets Rasch Model,Reliability,Validity
https://www.ijlt.ir/article_114381.html
https://www.ijlt.ir/article_114381_f1740a379103b65ecce453ca302b5998.pdf
Tabaran Institute of Higher Education
International Journal of Language Testing
2476-5880
3
1
2013
03
01
Exploring Diagnostic Capacity of a High Stakes Reading Comprehension Test: A Pedagogical Demonstration
11
37
EN
Hamdollah
Ravand
English Department, University of Isfahan, Iran
Hossein
Barati
English Department, University of Isfahan, Iran.
Wahyu
Widhiarso
English Department, University of Isfahan, Iran.
Traditional analysis of test results with Item Response Theory (IRT) models, and classical test theory (CTT) provide unidimensional ability values, which allow for ranking of students according to their abilities. Cognitive diagnostic assessment (CDA) is aimed at providing formative diagnostic feedback through a fine-grained reporting of learners’ skill mastery. The present study, through application of deterministic input, noisy ‘‘and’’ gate [de la Torre & Douglas, 2004 (DINA) model], investigated the degree to which the items of a high stakes reading comprehension test can provide diagnostically useful information. Through expert rating, a Q-matrix including five subskills was developed. Using DINA model, student’s skill mastery profiles were produced and issues related to the diagnostic capacity of the reading test were discussed. Meanwhile, this study pedagogically demonstrates the application of the DINA model employing R software.
attribute,Cognitive Diagnostic Model (CDM),DINA,skill,Q-matrix
https://www.ijlt.ir/article_114382.html
https://www.ijlt.ir/article_114382_51590e43d05173c5d4661deebb90fb82.pdf