Examining the Impacts of Rater Effects in Performance Assessments

Loading...
Thumbnail Image
Date
2019
Journal Title
Journal ISSN
Volume Title
Publisher
Sage
Abstract

Rater effects such as severity, centrality, and misfit are recurrent concerns in performance assessments. Despite their persistence in operational assessment settings and frequent discussion in research, researchers have not fully explored the impacts of rater effects as they relate to estimates of student achievement. The purpose of this study is to explore the impacts of rater severity, centrality, and misfit on student achievement estimates and on classification decisions. The results suggest that these three types of rater effects have substantial impacts on estimates of student achievement and on classification decisions that impact the fairness of rater-mediated assessments. Accordingly, it is essential that researchers and practitioners evaluate ratings across all stages of rater-mediated assessment procedures, including rater training and operational scoring.

Description
Keywords
rater effects, performance assessment, Rasch measurement theory, PRACTICAL CONSEQUENCES, MODEL MISFIT, QUALITY, RATINGS, Social Sciences, Mathematical Methods, Psychology, Mathematical
Citation
Wind, S. A. (2018). Examining the Impacts of Rater Effects in Performance Assessments. In Applied Psychological Measurement (Vol. 43, Issue 2, pp. 159–171). SAGE Publications. https://doi.org/10.1177/0146621618789391