Michael L Thomas1,2, Gregory G Brown1,3, Ruben C Gur4, Tyler M Moore4, Virginie M Patt1,5, Victoria B Risbrough1,2, Dewleen G Baker1,2. 1. a Department of Psychiatry , University of California San Diego , La Jolla , CA , USA. 2. b VA Center of Excellence for Stress and Mental Health (CESAMH) , San Diego , CA , USA. 3. c VISN-22 Mental Illness, Research, Education and Clinical Center (MIRECC) , VA San Diego Healthcare System , San Diego , CA , USA. 4. d Department of Psychiatry, Perelman School of Medicine , University of Pennsylvania , Philadelphia , PA , USA. 5. e Department of Psychology , San Diego State University/University of California , San Diego , CA , USA.
Abstract
INTRODUCTION: Models from signal detection theory are commonly used to score neuropsychological test data, especially tests of recognition memory. Here we show that certain item response theory models can be formulated as signal detection theory models, thus linking two complementary but distinct methodologies. We then use the approach to evaluate the validity (construct representation) of commonly used research measures, demonstrate the impact of conditional error on neuropsychological outcomes, and evaluate measurement bias. METHOD: Signal detection-item response theory (SD-IRT) models were fitted to recognition memory data for words, faces, and objects. The sample consisted of U.S. Infantry Marines and Navy Corpsmen participating in the Marine Resiliency Study. Data comprised item responses to the Penn Face Memory Test (PFMT; N = 1,338), Penn Word Memory Test (PWMT; N = 1,331), and Visual Object Learning Test (VOLT; N = 1,249), and self-report of past head injury with loss of consciousness. RESULTS: SD-IRT models adequately fitted recognition memory item data across all modalities. Error varied systematically with ability estimates, and distributions of residuals from the regression of memory discrimination onto self-report of past head injury were positively skewed towards regions of larger measurement error. Analyses of differential item functioning revealed little evidence of systematic bias by level of education. CONCLUSIONS: SD-IRT models benefit from the measurement rigor of item response theory-which permits the modeling of item difficulty and examinee ability-and from signal detection theory-which provides an interpretive framework encompassing the experimentally validated constructs of memory discrimination and response bias. We used this approach to validate the construct representation of commonly used research measures and to demonstrate how nonoptimized item parameters can lead to erroneous conclusions when interpreting neuropsychological test data. Future work might include the development of computerized adaptive tests and integration with mixture and random-effects models.
INTRODUCTION: Models from signal detection theory are commonly used to score neuropsychological test data, especially tests of recognition memory. Here we show that certain item response theory models can be formulated as signal detection theory models, thus linking two complementary but distinct methodologies. We then use the approach to evaluate the validity (construct representation) of commonly used research measures, demonstrate the impact of conditional error on neuropsychological outcomes, and evaluate measurement bias. METHOD: Signal detection-item response theory (SD-IRT) models were fitted to recognition memory data for words, faces, and objects. The sample consisted of U.S. Infantry Marines and Navy Corpsmen participating in the Marine Resiliency Study. Data comprised item responses to the Penn Face Memory Test (PFMT; N = 1,338), Penn Word Memory Test (PWMT; N = 1,331), and Visual Object Learning Test (VOLT; N = 1,249), and self-report of past head injury with loss of consciousness. RESULTS: SD-IRT models adequately fitted recognition memory item data across all modalities. Error varied systematically with ability estimates, and distributions of residuals from the regression of memory discrimination onto self-report of past head injury were positively skewed towards regions of larger measurement error. Analyses of differential item functioning revealed little evidence of systematic bias by level of education. CONCLUSIONS: SD-IRT models benefit from the measurement rigor of item response theory-which permits the modeling of item difficulty and examinee ability-and from signal detection theory-which provides an interpretive framework encompassing the experimentally validated constructs of memory discrimination and response bias. We used this approach to validate the construct representation of commonly used research measures and to demonstrate how nonoptimized item parameters can lead to erroneous conclusions when interpreting neuropsychological test data. Future work might include the development of computerized adaptive tests and integration with mixture and random-effects models.
Authors: Michael L Thomas; Virginie M Patt; Andrew Bismark; Joyce Sprock; Melissa Tarasenko; Gregory A Light; Gregory G Brown Journal: J Abnorm Psychol Date: 2017-03-09
Authors: R C Gur; J D Ragland; P J Moberg; T H Turner; W B Bilker; C Kohler; S J Siegel; R E Gur Journal: Neuropsychopharmacology Date: 2001-11 Impact factor: 7.853
Authors: Farzin Irani; Colleen M Brensinger; Jan Richard; Monica E Calkins; Paul J Moberg; Waren Bilker; Raquel E Gur; Ruben C Gur Journal: Am J Geriatr Psychiatry Date: 2012-01 Impact factor: 4.105
Authors: Michael L Thomas; Gregory G Brown; Ruben C Gur; John A Hansen; Matthew K Nock; Steven Heeringa; Robert J Ursano; Murray B Stein Journal: J Clin Exp Neuropsychol Date: 2013-02-05 Impact factor: 2.475
Authors: Kelly M Kenzik; I-Chan Huang; Tara M Brinkman; Brandon Baughman; Kirsten K Ness; Elizabeth A Shenkman; Melissa M Hudson; Leslie L Robison; Kevin R Krull Journal: Neuropsychology Date: 2014-06-16 Impact factor: 3.295
Authors: Richard C Gershon; Karon F Cook; Dan Mungas; Jennifer J Manly; Jerry Slotkin; Jennifer L Beaumont; Sandra Weintraub Journal: J Int Neuropsychol Soc Date: 2014-06-24 Impact factor: 2.892
Authors: Mathias Basner; Adam Savitt; Tyler M Moore; Allison M Port; Sarah McGuire; Adrian J Ecker; Jad Nasrini; Daniel J Mollicone; Christopher M Mott; Thom McCann; David F Dinges; Ruben C Gur Journal: Aerosp Med Hum Perform Date: 2015-11 Impact factor: 1.053
Authors: Daniel Zarate; Lana Fullwood; Maria Prokofieva; Mark D Griffiths; Vasileios Stavropoulos Journal: Int J Ment Health Addict Date: 2022-06-20 Impact factor: 11.555
Authors: Michael L Thomas; John R Duffy; Neal Swerdlow; Gregory A Light; Gregory G Brown Journal: J Int Neuropsychol Soc Date: 2021-05-05 Impact factor: 2.892
Authors: Rebecca W Gelding; Peter M C Harrison; Sebastian Silas; Blake W Johnson; William F Thompson; Daniel Müllensiefen Journal: Psychol Res Date: 2020-04-30