King's College London

Research portal

Reliability in Assessment Centers Depends on Measurement Intentions

Research output: Contribution to journalArticlepeer-review

Duncan Jackson, George Michaelides, Chris Dewberry, Jo Nelson, Catherine Stephens

Original languageEnglish
JournalJOURNAL OF OCCUPATIONAL AND ORGANIZATIONAL PSYCHOLOGY
Accepted/In press23 May 2022

King's Authors

Abstract

In two recent studies, the median reliability of assessment centers (ACs) was estimated at .90 (range = .23, Jackson et al., 2016; Putka & Hoffman, 2013). However, these studies, among many others (e.g., Lance et al., 2004; Sackett & Dreher, 1982) indicate that the dimensions which ACs are designed to measure contribute very little to their reliability. This raises a fundamental question: how can ACs be reliable when the dimensions they are designed to assess do not contribute to reliable measurement in ACs? Using evidence from 10 samples, we resolve this issue by showing that the reliability of ACs greatly depends on the intentions of the researcher or practitioner. When the intent was to measure dimensions, we found evidence that AC reliability was unacceptably low (mean reliability = .38, SD = .15). However, when the intent was to include the measurement of exercise scores, we found evidence that AC reliability exceeded acceptable criteria (mean reliability = .91, SD = .09). We additionally found evidence that, at least in ACs that follow professional design guidelines, dimension effects and assessor effects do not generally make an appreciable difference to AC reliability.

View graph of relations

© 2020 King's College London | Strand | London WC2R 2LS | England | United Kingdom | Tel +44 (0)20 7836 5454