The Use of Multivariate Generalizability Theory to Evaluate the Quality of Subscores

Simple item page
dc.contributor.authorJiang, Zhehan
dc.contributor.authorRaymond, Mark
dc.contributor.otherUniversity of Alabama Tuscaloosa
dc.date.accessioned2023-09-28T19:34:01Z
dc.date.available2023-09-28T19:34:01Z
dc.date.issued2018
dc.description.abstractConventional methods for evaluating the utility of subscores rely on reliability and correlation coefficients. However, correlations can overlook a notable source of variability: variation in subtest means/difficulties. Brennan introduced a reliability index for score profiles based on multivariate generalizability theory, designated as G, which is sensitive to variation in subtest difficulty. However, there has been little, if any, research evaluating the properties of this index. A series of simulation experiments, as well as analyses of real data, were conducted to investigate G under various conditions of subtest reliability, subtest correlations, and variability in subtest means. Three pilot studies evaluated G in the context of a single group of examinees. Results of the pilots indicated that G indices were typically low; across the 108 experimental conditions, G ranged from .23 to .86, with an overall mean of 0.63. The findings were consistent with previous research, indicating that subscores often do not have interpretive value. Importantly, there were many conditions for which the correlation-based method known as proportion reduction in mean-square error (PRMSE; Haberman, 2006) indicated that subscores were worth reporting, but for which values of G fell into the .50s, .60s, and .70s. The main study investigated G within the context of score profiles for examinee subgroups. Again, not only G indices were generally low, but it was also found that G can be sensitive to subgroup differences when PRMSE is not. Analyses of real data and subsequent discussion address how G can supplement PRMSE for characterizing the quality of subscores.en_US
dc.format.mediumelectronic
dc.format.mimetypeapplication/pdf
dc.identifier.citationJiang, Z., & Raymond, M. (2018). The Use of Multivariate Generalizability Theory to Evaluate the Quality of Subscores. In Applied Psychological Measurement (Vol. 42, Issue 8, pp. 595–612). SAGE Publications. https://doi.org/10.1177/0146621618758698
dc.identifier.doi10.1177/0146621618758698
dc.identifier.orcidhttps://orcid.org/0000-0002-1376-9439
dc.identifier.urihttps://ir.ua.edu/handle/123456789/11447
dc.languageEnglish
dc.language.isoen_US
dc.publisherSage
dc.subjectsubscores
dc.subjectscore profiles
dc.subjectgeneralizability theory
dc.subjectreliability
dc.subjectdimensionality
dc.subjectsimulation
dc.subjectMULTIPLE-CHOICE
dc.subjectSCORES
dc.subjectSocial Sciences, Mathematical Methods
dc.subjectPsychology, Mathematical
dc.titleThe Use of Multivariate Generalizability Theory to Evaluate the Quality of Subscoresen_US
dc.typeArticle
dc.typetext

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
10.11770146621618758698.pdf
Size:
653.46 KB
Format:
Adobe Portable Document Format
Download

Collections

Research and Publications - Department of Gender and Race Studies