Kenneth R Paap1, Oliver Sawi2. 1. Department of Psychology, EP301, San Francisco State University, 1600 Holloway Avenue, San Francisco, CA 94132, USA. Electronic address: kenp@sfsu.edu. 2. Department of Psychology, University of Connecticut, Storrs, CT, USA.
Abstract
BACKGROUND: Studies testing for individual or group differences in executive functioning can be compromised by unknown test-retest reliability. NEW METHOD: Test-retest reliabilities across an interval of about one week were obtained from performance in the antisaccade, flanker, Simon, and color-shape switching tasks. There is a general trade-off between the greater reliability of single mean RT measures, and the greater process purity of measures based on contrasts between mean RTs in two conditions. The individual differences in RT model recently developed by Miller and Ulrich was used to evaluate the trade-off. RESULTS: Test-retest reliability was statistically significant for 11 of the 12 measures, but was of moderate size, at best, for the difference scores. The test-retest reliabilities for the Simon and flanker interference scores were lower than those for switching costs. COMPARISON WITH EXISTING METHODS: Standard practice evaluates the reliability of executive-functioning measures using split-half methods based on data obtained in a single day. Our test-retest measures of reliability are lower, especially for difference scores. These reliability measures must also take into account possible day effects that classical test theory assumes do not occur. CONCLUSIONS: Measures based on single mean RTs tend to have acceptable levels of reliability and convergent validity, but are "impure" measures of specific executive functions. The individual differences in RT model shows that the impurity problem is worse than typically assumed. However, the "purer" measures based on difference scores have low convergent validity that is partly caused by deficiencies in test-retest reliability.
BACKGROUND: Studies testing for individual or group differences in executive functioning can be compromised by unknown test-retest reliability. NEW METHOD: Test-retest reliabilities across an interval of about one week were obtained from performance in the antisaccade, flanker, Simon, and color-shape switching tasks. There is a general trade-off between the greater reliability of single mean RT measures, and the greater process purity of measures based on contrasts between mean RTs in two conditions. The individual differences in RT model recently developed by Miller and Ulrich was used to evaluate the trade-off. RESULTS: Test-retest reliability was statistically significant for 11 of the 12 measures, but was of moderate size, at best, for the difference scores. The test-retest reliabilities for the Simon and flanker interference scores were lower than those for switching costs. COMPARISON WITH EXISTING METHODS: Standard practice evaluates the reliability of executive-functioning measures using split-half methods based on data obtained in a single day. Our test-retest measures of reliability are lower, especially for difference scores. These reliability measures must also take into account possible day effects that classical test theory assumes do not occur. CONCLUSIONS: Measures based on single mean RTs tend to have acceptable levels of reliability and convergent validity, but are "impure" measures of specific executive functions. The individual differences in RT model shows that the impurity problem is worse than typically assumed. However, the "purer" measures based on difference scores have low convergent validity that is partly caused by deficiencies in test-retest reliability.
Authors: A Zeynep Enkavi; Ian W Eisenberg; Patrick G Bissett; Gina L Mazza; David P MacKinnon; Lisa A Marsch; Russell A Poldrack Journal: Proc Natl Acad Sci U S A Date: 2019-03-06 Impact factor: 11.205
Authors: Jason S Tsukahara; Tyler L Harrison; Christopher Draheim; Jessie D Martin; Randall W Engle Journal: Atten Percept Psychophys Date: 2020-10 Impact factor: 2.199
Authors: Sarah V Clark; Eric S Semmel; Holly A Aleksonis; Stephanie N Steinberg; Tricia Z King Journal: Neuropsychol Rev Date: 2021-01-29 Impact factor: 7.444
Authors: Senne Braem; Julie M Bugg; James R Schmidt; Matthew J C Crump; Daniel H Weissman; Wim Notebaert; Tobias Egner Journal: Trends Cogn Sci Date: 2019-07-19 Impact factor: 20.229