Joseph C Cappelleri1, J Jason Lundy2, Ron D Hays3. 1. Pfizer Inc, Groton, Connecticut. Electronic address: joseph.c.cappelleri@pfizer.com. 2. Critical Path Institute, Tucson, Arizona. 3. Division of General Internal Medicine & Health Services Research, University of California at Los Angeles, Los Angeles, California.
Abstract
BACKGROUND: The US Food and Drug Administration's guidance for industry document on patient-reported outcomes (PRO) defines content validity as "the extent to which the instrument measures the concept of interest" (FDA, 2009, p. 12). According to Strauss and Smith (2009), construct validity "is now generally viewed as a unifying form of validity for psychological measurements, subsuming both content and criterion validity" (p. 7). Hence, both qualitative and quantitative information are essential in evaluating the validity of measures. METHODS: We review classical test theory and item response theory (IRT) approaches to evaluating PRO measures, including frequency of responses to each category of the items in a multi-item scale, the distribution of scale scores, floor and ceiling effects, the relationship between item response options and the total score, and the extent to which hypothesized "difficulty" (severity) order of items is represented by observed responses. RESULTS: If a researcher has few qualitative data and wants to get preliminary information about the content validity of the instrument, then descriptive assessments using classical test theory should be the first step. As the sample size grows during subsequent stages of instrument development, confidence in the numerical estimates from Rasch and other IRT models (as well as those of classical test theory) would also grow. CONCLUSION: Classical test theory and IRT can be useful in providing a quantitative assessment of items and scales during the content-validity phase of PRO-measure development. Depending on the particular type of measure and the specific circumstances, the classical test theory and/or the IRT should be considered to help maximize the content validity of PRO measures.
BACKGROUND: The US Food and Drug Administration's guidance for industry document on patient-reported outcomes (PRO) defines content validity as "the extent to which the instrument measures the concept of interest" (FDA, 2009, p. 12). According to Strauss and Smith (2009), construct validity "is now generally viewed as a unifying form of validity for psychological measurements, subsuming both content and criterion validity" (p. 7). Hence, both qualitative and quantitative information are essential in evaluating the validity of measures. METHODS: We review classical test theory and item response theory (IRT) approaches to evaluating PRO measures, including frequency of responses to each category of the items in a multi-item scale, the distribution of scale scores, floor and ceiling effects, the relationship between item response options and the total score, and the extent to which hypothesized "difficulty" (severity) order of items is represented by observed responses. RESULTS: If a researcher has few qualitative data and wants to get preliminary information about the content validity of the instrument, then descriptive assessments using classical test theory should be the first step. As the sample size grows during subsequent stages of instrument development, confidence in the numerical estimates from Rasch and other IRT models (as well as those of classical test theory) would also grow. CONCLUSION: Classical test theory and IRT can be useful in providing a quantitative assessment of items and scales during the content-validity phase of PRO-measure development. Depending on the particular type of measure and the specific circumstances, the classical test theory and/or the IRT should be considered to help maximize the content validity of PRO measures.
Authors: Donald L Patrick; Laurie B Burke; Chad J Gwaltney; Nancy Kline Leidy; Mona L Martin; Elizabeth Molsen; Lena Ring Journal: Value Health Date: 2011-10-10 Impact factor: 5.725
Authors: Donald L Patrick; Laurie B Burke; Chad J Gwaltney; Nancy Kline Leidy; Mona L Martin; Elizabeth Molsen; Lena Ring Journal: Value Health Date: 2011-10-13 Impact factor: 5.725
Authors: Wen-Hung Chen; William Lenderking; Ying Jin; Kathleen W Wyrwich; Heather Gelhorn; Dennis A Revicki Journal: Qual Life Res Date: 2013-08-03 Impact factor: 4.147
Authors: Michael A Kallen; Karon F Cook; Dagmar Amtmann; Elizabeth Knowlton; Richard C Gershon Journal: Qual Life Res Date: 2018-05-05 Impact factor: 4.147
Authors: Noelle E Carlozzi; Michael A Kallen; Angelle M Sander; Tracey A Brickell; Rael T Lange; Louis M French; Phillip A Ianni; Jennifer A Miner; Robin Hanks Journal: Arch Phys Med Rehabil Date: 2018-06-26 Impact factor: 3.966
Authors: Noelle E Carlozzi; Michael A Kallen; Phillip A Ianni; Angelle M Sander; Elizabeth A Hahn; Rael T Lange; Tracey A Brickell; Louis M French; Jennifer A Miner; Robin Hanks Journal: Arch Phys Med Rehabil Date: 2018-06-26 Impact factor: 3.966
Authors: Marisa Toups; Thomas Carmody; Madhukar H Trivedi; A John Rush; S Susan Hedayati Journal: Gen Hosp Psychiatry Date: 2016-07-25 Impact factor: 3.238
Authors: Chelsea L Reighard; Manju R Pillai; Sujani Shroff; George L Spaeth; Stephen G Schilling; Sheryl S Wizov; Joshua D Stein; Alan L Robin; Vidya Raja; Joshua R Ehrlich Journal: Ophthalmol Glaucoma Date: 2019-06-18
Authors: Meghan E Morean; Suchitra Krishnan-Sarin; Steve Sussman; Jonathan Foulds; Howard Fishbein; Rachel Grana; Stephanie S O'Malley Journal: Nicotine Tob Res Date: 2019-10-26 Impact factor: 4.244
Authors: Ellen M Lavoie Smith; Noah Zanville; Grace Kanzawa-Lee; Clare Donohoe; Celia Bridges; Charles Loprinzi; Jennifer Le-Rademacher; James J Yang Journal: Support Care Cancer Date: 2018-11-20 Impact factor: 3.603
Authors: Yan Wang; Ron Hays; Marvin Marcus; Carl Maida; Jie Shen; Di Xiong; Steve Lee; Vladimir Spolsky; Ian Coulter; James Crall; Honghu Liu Journal: Int J Paediatr Dent Date: 2019-01-24 Impact factor: 3.455