TY - JOUR
T1 - Uncertainty about Rater Variance and Small Dimension Effects Impact Reliability in Supervisor Ratings
AU - Jackson, Duncan
AU - Michaelides, George
AU - Dewberry, Chris
AU - Jones, Amanda
AU - Toms, Simon
AU - Schwenke, Benjamin
AU - Yang, Wei-Ning
N1 - Publisher Copyright:
© 2022 The Author(s). Published with license by Taylor & Francis Group, LLC.
PY - 2022/8/19
Y1 - 2022/8/19
N2 - We modeled the effects commonly described as defining the measurement structure of supervisor performance ratings. In doing so, we contribute to different theoretical perspectives, including components of the multifactor and mediated models of performance ratings. Across two reanalyzed samples (Sample 1, N
ratees = 392, N
raters = 244; Sample 2, N
ratees = 342, N
raters = 397), we found a structure primarily reflective of general (>27% of variance explained) and rater-related (>49%) effects, with relatively small performance dimension effects (between 1% and 11%). We drew on findings from the assessment center literature to approximate the proportion of rater variance that might theoretically contribute to reliability in performance ratings. We found that even moderate contributions of rater-related variance to reliability resulted in a sizable impact on reliability estimates, drawing them closer to accepted criteria.
AB - We modeled the effects commonly described as defining the measurement structure of supervisor performance ratings. In doing so, we contribute to different theoretical perspectives, including components of the multifactor and mediated models of performance ratings. Across two reanalyzed samples (Sample 1, N
ratees = 392, N
raters = 244; Sample 2, N
ratees = 342, N
raters = 397), we found a structure primarily reflective of general (>27% of variance explained) and rater-related (>49%) effects, with relatively small performance dimension effects (between 1% and 11%). We drew on findings from the assessment center literature to approximate the proportion of rater variance that might theoretically contribute to reliability in performance ratings. We found that even moderate contributions of rater-related variance to reliability resulted in a sizable impact on reliability estimates, drawing them closer to accepted criteria.
UR - http://www.scopus.com/inward/record.url?scp=85136467589&partnerID=8YFLogxK
U2 - 10.1080/08959285.2022.2111433
DO - 10.1080/08959285.2022.2111433
M3 - Article
SN - 0895-9285
VL - 35
SP - 278
EP - 301
JO - HUMAN PERFORMANCE
JF - HUMAN PERFORMANCE
IS - 3-4
ER -