When effects are confounded, they cannot be interpreted: A study of confounding in assessment centre ratings

Jackson, Duncan and Michaelides, George and Dewberry, Chris (2016) When effects are confounded, they cannot be interpreted: A study of confounding in assessment centre ratings. In: European Academy of Occupational Health Psychology, 11-13 Apr 2016, Athens, Greece. (Unpublished)

Abstract

Background. Dimensions (e.g., communication skills, health awareness) measured through multifaceted measures hold a key position across a vast array of disciplines relevant to health and organisations. However, the measurement properties of dimensions have long been affected by confounds. The outcome of confounding in the context of multifaceted measurement is that the extent to which reliable variance in ratings reflects dimensions is difficult or impossible to establish. Nowhere have dimensions been scrutinized more closely than in the literature on assessment centres (ACs). In recent years, dimensions have enjoyed a revival in the AC literature and now feature as important, meaningful, and impactful variables of interest in a range of different studies published in leading journals (e.g., Journal of Applied Psychology, Personnel Psychology). However, none of these studies have fully accounted for confounding. As a consequence, the extent to which the effects found in these studies are attributable to dimensions, or to a multitude of other variables relevant to ACs, is unclear. This issue has implications for any context in which the aim is to measure dimensions using multiple samples of behaviour. Study Aim. Our aim is to disentangle the different effects in AC ratings in order to establish what role dimensions have in AC ratings as distinct from other effects in AC ratings (i.e., different samples, rating items, assessors, and exercises). Doing so would work towards informing the discipline about the "real" basis for AC ratings and, more widely, about the role of dimensions in multifaceted measures. Method. Ratings were collected from an operational AC used in a public institution. The AC under study was developed according to international guidelines. Data were subjected to a Bayesian random effects analysis. Results. Results suggested that when confounding is (properly) taken into account, AC ratings have little to do with dimensions. The lion's share of variance in AC ratings is accounted for by (a) a general performance factor and (b) exercise-related effects. Discussion. The findings of the current study call into question the interpretation of much of the "dimension revival" literature on ACs and the role of dimensions in organization-related studies. They suggest that if dimensions are the intended focus for assessment in ACs, an AC does not allow sufficient high fidelity behavioural sampling for such variables to manifest. In the light of these findings, we call for an alternative direction for AC research and practice.

Metadata

Statistics

DownloadsShow export options

Export as XMLJSONCSV

Activity Overview

6 month trend

125Downloads

6 month trend

232Hits

Additional statistics are available via IRStats2.

Archive Staff Only (login required)

Edit/View Item