BIROn - Birkbeck Institutional Research Online

    The psychometric structure of a game-based assessment

    Close, Liam Kevin (2022) The psychometric structure of a game-based assessment. PhD thesis, Birkbeck, University of London.

    [img]
    Preview
    Text
    Liam Close Post_Correction_Thesis_v2 (1).pdf - Full Version

    Download (2MB) | Preview

    Abstract

    This thesis examines the psychometric properties of a game-based assessment (GBA) by utilizing generalizability theory to estimate the reliability and the effect of aggregation on GBA results. The results of this study provide insight into the theoretical standpoints of trait theory, situational theory and the interactionist perspective, and explores the extent to which the reliability of a GBA can be impacted by a chosen theoretical standpoint. Secondary data were used within this study, compiled of three cohorts of candidates that completed a psychometric GBA as part of the selection process. The largest contribution to variance in the GBA was the Person ×Level ×Dimension effect (14.35% –15.26%), with little variance accounted for by the Person ×Level (0.67% –11.04%) or Person ×Dimension effects (<0.01% –4.09%). These findings support the argument that dimensions have little effect within GBAs. Aggregating scores to dimensions were found to result in poor reliability estimates (0.36–0.66) but when aggregating to an overall score, the G coefficient approached acceptable levels (0.63–0.86). However, aggregating to an overall score raises concerns about the interpretability of these scores. Attention within the literature has focused on the validity of dimension-related scores within a GBA, so further research is required to understand the validity of an overall score generated from a GBA. With a negligible amount of variance associated with the Person ×Dimension effect, this suggests that dimensions do not behave in a stable and consistent nature, which contradicts the theoretical underpinnings of trait theory. Little level-related variance was also noted within this study, which suggests that the situation has less overall effect than outlined by traditional situational theorists. However, the Person ×Level ×Dimension interaction supports the interactionist theoretical perspective which posits that behaviour is determined by an interaction between the trait and the situation. This study highlights the need for further understanding of the measurement structure of GBAs. Like other multifaceted measures, dimensions were found to contribute little variance, and the impact of the situation (or level within this study) on scores suggests that there is little or no evidence here that dimensions are cross-situationally stable. Aggregated overall scores offer a more reliable measure of behaviour, but the validity of these scores needs additional investigation to help understand what overall scores measure from a psychological perspective. Further implications for research and practice are discussed.

    Metadata

    Item Type: Thesis
    Copyright Holders: The copyright of this thesis rests with the author, who asserts his/her right to be known as such according to the Copyright Designs and Patents Act 1988. No dealing with the thesis contrary to the copyright or moral rights of the author is permitted.
    Depositing User: Acquisitions And Metadata
    Date Deposited: 14 Nov 2022 13:51
    Last Modified: 01 Nov 2023 15:50
    URI: https://eprints.bbk.ac.uk/id/eprint/49831
    DOI: https://doi.org/10.18743/PUB.00049831

    Statistics

    Activity Overview
    6 month trend
    278Downloads
    6 month trend
    263Hits

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item
    Edit/View Item