Chantelle Bailey1,2, Mary P Tully3, Maria Pampaka4, Jonathan Cooke3,5. 1. Manchester Pharmacy School, 1st Floor, Stopford Building, University of Manchester, Manchester M13 9PT, UK chantelle.bailey2@gmail.com. 2. Centre for Clinical Epidemiology and Evaluation, 7th Floor, 828 West 10th Avenue, Research Pavilion, Vancouver, British Columbia, Canada V5Z 1M9. 3. Manchester Pharmacy School, 1st Floor, Stopford Building, University of Manchester, Manchester M13 9PT, UK. 4. Manchester Institute for Education, Ellen Wilkinson Building, University of Manchester, Manchester M13 9PT, UK. 5. Centre for Infection Prevention and Management, Division of Medicine, Imperial College, Hammersmith Campus, London W12 0NN, UK.
Abstract
OBJECTIVES: The Antimicrobial Self-Assessment Toolkit for National Health Service (NHS) Trusts (ASAT) was developed to evaluate hospital-based antimicrobial stewardship programmes. Iterative validity investigations of the ASAT were used to produce a 91-item ASAT v17 utilizing qualitative methodology. Rasch analysis was used to generate question (item) behaviour estimates and to investigate the validity of ASAT v17. METHODS: In 2012, the partial credit model (PCM) was used to analyse ASAT responses from 33 NHS Trusts within England. WINSTEPS® outputs such as fit statistics and respondent/item maps were examined to determine unidimensionality, item discrimination and item hierarchy. Ordinary least squares regression modelling was used to determine the predictive validity using NHS Trust ability estimates generated from the PCM and corresponding Clostridium difficile rates. RESULTS: Each domain contained items that were misfitting the PCM (with INFIT MNSQ <0.7 or >1.3), except Domain 3. Subsequent iterative item removal had a negligible effect on the fit indices within most ASAT domains. Scale analysis demonstrated that most items were productive for measurement (n = 81). Respondent/item maps showed ceiling effects (n = 3) and floor effects (n = 1) within ASAT domains. Ordinary least squares regression modelling identified that there was limited predictive validity due to the small positive correlation between the predictor and outcome variables for participating hospitals (ρ = 0.146; P = 0.418). CONCLUSIONS: Rasch analysis was an effective measurement technique for evaluating the validity of ASAT v17 by providing evidence that each sub-scale and the overall scale demonstrated unidimensionality (construct validity). Improved item targeting may be required to improve item discrimination within the toolkit.
OBJECTIVES: The Antimicrobial Self-Assessment Toolkit for National Health Service (NHS) Trusts (ASAT) was developed to evaluate hospital-based antimicrobial stewardship programmes. Iterative validity investigations of the ASAT were used to produce a 91-item ASAT v17 utilizing qualitative methodology. Rasch analysis was used to generate question (item) behaviour estimates and to investigate the validity of ASAT v17. METHODS: In 2012, the partial credit model (PCM) was used to analyse ASAT responses from 33 NHS Trusts within England. WINSTEPS® outputs such as fit statistics and respondent/item maps were examined to determine unidimensionality, item discrimination and item hierarchy. Ordinary least squares regression modelling was used to determine the predictive validity using NHS Trust ability estimates generated from the PCM and corresponding Clostridium difficile rates. RESULTS: Each domain contained items that were misfitting the PCM (with INFIT MNSQ <0.7 or >1.3), except Domain 3. Subsequent iterative item removal had a negligible effect on the fit indices within most ASAT domains. Scale analysis demonstrated that most items were productive for measurement (n = 81). Respondent/item maps showed ceiling effects (n = 3) and floor effects (n = 1) within ASAT domains. Ordinary least squares regression modelling identified that there was limited predictive validity due to the small positive correlation between the predictor and outcome variables for participating hospitals (ρ = 0.146; P = 0.418). CONCLUSIONS: Rasch analysis was an effective measurement technique for evaluating the validity of ASAT v17 by providing evidence that each sub-scale and the overall scale demonstrated unidimensionality (construct validity). Improved item targeting may be required to improve item discrimination within the toolkit.