BIROn - Birkbeck Institutional Research Online

    The meaning of significant mean-group differences for biomarker discovery

    Loth, E. and Ahmad, J. and Chatham, C. and Lopez, B. and Carter, B. and Crawley, D. and Oakley, B. and Hayward, H. and Cooke, J. and San Jose Caceres, A. and Bzdok, D. and Jones, Emily J.H. and Charman, T. and Beckmann, C. and Bougeron, T. and Toro, R. and Buitelaar, J. and Murphy, D. and Duman, G. (2021) The meaning of significant mean-group differences for biomarker discovery. PLoS Computational Biology 17 (11), e1009477. ISSN 1553-7358.

    [img]
    Preview
    Text
    46138a.pdf - Published Version of Record
    Available under License Creative Commons Attribution.

    Download (1MB) | Preview

    Abstract

    Over the past decade, biomarker discovery has become a key goal in psychiatry to aid in the more reliable diagnosis and prognosis of heterogeneous psychiatric conditions and the development of tailored therapies. Nevertheless, the prevailing statistical approach is still the mean group comparison between “cases” and “controls,” which tends to ignore within-group variability. In this educational article, we used empirical data simulations to investigate how effect size, sample size, and the shape of distributions impact the interpretation of mean group differences for biomarker discovery. We then applied these statistical criteria to evaluate biomarker discovery in one area of psychiatric research—autism research. Across the most influential areas of autism research, effect size estimates ranged from small (d = 0.21, anatomical structure) to medium (d = 0.36 electrophysiology, d = 0.5, eye-tracking) to large (d = 1.1 theory of mind). We show that in normal distributions, this translates to approximately 45% to 63% of cases performing within 1 standard deviation (SD) of the typical range, i.e., they do not have a deficit/atypicality in a statistical sense. For a measure to have diagnostic utility as defined by 80% sensitivity and 80% specificity, Cohen’s d of 1.66 is required, with still 40% of cases falling within 1 SD. However, in both normal and nonnormal distributions, 1 (skewness) or 2 (platykurtic, bimodal) biologically plausible subgroups may exist despite small or even nonsignificant mean group differences. This conclusion drastically contrasts the way mean group differences are frequently reported. Over 95% of studies omitted the “on average” when summarising their findings in their abstracts (“autistic people have deficits in X”), which can be misleading as it implies that the group-level difference applies to all individuals in that group. We outline practical approaches and steps for researchers to explore mean group comparisons for the discovery of stratification biomarkers.

    Metadata

    Item Type: Article
    School: Birkbeck Faculties and Schools > Faculty of Science > School of Psychological Sciences
    Research Centres and Institutes: Brain and Cognitive Development, Centre for (CBCD)
    Depositing User: Emily Jones
    Date Deposited: 27 Jan 2022 14:20
    Last Modified: 02 Aug 2023 18:13
    URI: https://eprints.bbk.ac.uk/id/eprint/46138

    Statistics

    Activity Overview
    6 month trend
    58Downloads
    6 month trend
    86Hits

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item