The psychometric structure of a game-based assessment

Close, Liam Kevin (2022) The psychometric structure of a game-based assessment. PhD thesis, Birkbeck, University of London.

Abstract

This thesis examines the psychometric properties of a game-based assessment (GBA) by utilizing generalizability theory to estimate the reliability and the effect of aggregation on GBA results. The results of this study provide insight into the theoretical standpoints of trait theory, situational theory and the interactionist perspective, and explores the extent to which the reliability of a GBA can be impacted by a chosen theoretical standpoint. Secondary data were used within this study, compiled of three cohorts of candidates that completed a psychometric GBA as part of the selection process. The largest contribution to variance in the GBA was the Person ×Level ×Dimension effect (14.35% –15.26%), with little variance accounted for by the Person ×Level (0.67% –11.04%) or Person ×Dimension effects (<0.01% –4.09%). These findings support the argument that dimensions have little effect within GBAs. Aggregating scores to dimensions were found to result in poor reliability estimates (0.36–0.66) but when aggregating to an overall score, the G coefficient approached acceptable levels (0.63–0.86). However, aggregating to an overall score raises concerns about the interpretability of these scores. Attention within the literature has focused on the validity of dimension-related scores within a GBA, so further research is required to understand the validity of an overall score generated from a GBA. With a negligible amount of variance associated with the Person ×Dimension effect, this suggests that dimensions do not behave in a stable and consistent nature, which contradicts the theoretical underpinnings of trait theory. Little level-related variance was also noted within this study, which suggests that the situation has less overall effect than outlined by traditional situational theorists. However, the Person ×Level ×Dimension interaction supports the interactionist theoretical perspective which posits that behaviour is determined by an interaction between the trait and the situation. This study highlights the need for further understanding of the measurement structure of GBAs. Like other multifaceted measures, dimensions were found to contribute little variance, and the impact of the situation (or level within this study) on scores suggests that there is little or no evidence here that dimensions are cross-situationally stable. Aggregated overall scores offer a more reliable measure of behaviour, but the validity of these scores needs additional investigation to help understand what overall scores measure from a psychological perspective. Further implications for research and practice are discussed.

Metadata

Statistics

DownloadsShow export options

Export as XMLJSONCSV

Activity Overview

6 month trend

195Downloads

6 month trend

184Hits

Additional statistics are available via IRStats2.

Archive Staff Only (login required)

Edit/View Item