C Bailey1, M Tully, J Cooke. 1. Manchester Pharmacy School, University of Manchester, Manchester, UK; Centre for Clinical Epidemiology and Evaluation, Research Pavilion, Vancouver, BC, Canada.
Abstract
WHAT IS KNOWN AND OBJECTIVE: The Antimicrobial Self-Assessment Toolkit for NHS Trusts (ASAT) was developed to evaluate the organizational strategies used to implement hospital-based antimicrobial stewardship programmes. An iterative approach was used to develop ASAT v15a, which has been previously investigated for face validity; however, further investigation into other types of validity was required. Therefore, the aim of this study was to investigate the content validity of ASAT v15a and hence modify and improve the content validity of the toolkit. METHODS: A purposive sample of eight antimicrobial pharmacists was interviewed using cognitive interviewing techniques from within the former North-west Strategic Health Authority in England. Respondents were asked to 'think aloud' and to verbally express their thought processes as they generated responses to each question with the ASAT. RESULTS: There were no cognitive difficulties reported by respondents in response to 26/83 (31·3%) questions within the ASAT. However, cognitive difficulties were reported by respondents at each stage of the cognitive processing pathway in response to 57/83 (68·7%) questions. These difficulties were comprehension/interpretation in 27/83 (32·5%) questions, information retrieval in 10/83 (12%) questions, judgment/decision in 6/83 (7·2%) questions and response generation/formatting in 13/83 (15·7%) questions. Other findings included disagreement with the weightings applied to 13/83 (15·7%) questions. Respondents recommended that these questions should be modified to reflect their impact on hospital-based antimicrobial stewardship programmes (ASPs). Based on these findings, modifications were made to ASAT v15a to produce the next iteration (ASAT v16). Furthermore, respondents indicated that the role of clinical microbiologists was underrepresented in the current version of the toolkit; therefore, seven proposed questions were drafted, based on a literature review. WHAT IS NEW AND CONCLUSION: Cognitive interviews were effectively able to detect problems encountered by respondents along the cognitive processing pathway by identifying words and/or phrases that required further modifications. Also, this method highlighted that there was a disparity between the respondents' interpretation and the ASAT developers' intent within some questions. Although respondents indicated that the toolkit evaluated the most pertinent components of ASPs, further modifications and testing would be required to improve its validity. These results highlight the importance of the inclusion of end-users in the development of reporting and/or evaluation tools or questionnaires.
WHAT IS KNOWN AND OBJECTIVE: The Antimicrobial Self-Assessment Toolkit for NHS Trusts (ASAT) was developed to evaluate the organizational strategies used to implement hospital-based antimicrobial stewardship programmes. An iterative approach was used to develop ASAT v15a, which has been previously investigated for face validity; however, further investigation into other types of validity was required. Therefore, the aim of this study was to investigate the content validity of ASAT v15a and hence modify and improve the content validity of the toolkit. METHODS: A purposive sample of eight antimicrobial pharmacists was interviewed using cognitive interviewing techniques from within the former North-west Strategic Health Authority in England. Respondents were asked to 'think aloud' and to verbally express their thought processes as they generated responses to each question with the ASAT. RESULTS: There were no cognitive difficulties reported by respondents in response to 26/83 (31·3%) questions within the ASAT. However, cognitive difficulties were reported by respondents at each stage of the cognitive processing pathway in response to 57/83 (68·7%) questions. These difficulties were comprehension/interpretation in 27/83 (32·5%) questions, information retrieval in 10/83 (12%) questions, judgment/decision in 6/83 (7·2%) questions and response generation/formatting in 13/83 (15·7%) questions. Other findings included disagreement with the weightings applied to 13/83 (15·7%) questions. Respondents recommended that these questions should be modified to reflect their impact on hospital-based antimicrobial stewardship programmes (ASPs). Based on these findings, modifications were made to ASAT v15a to produce the next iteration (ASAT v16). Furthermore, respondents indicated that the role of clinical microbiologists was underrepresented in the current version of the toolkit; therefore, seven proposed questions were drafted, based on a literature review. WHAT IS NEW AND CONCLUSION: Cognitive interviews were effectively able to detect problems encountered by respondents along the cognitive processing pathway by identifying words and/or phrases that required further modifications. Also, this method highlighted that there was a disparity between the respondents' interpretation and the ASAT developers' intent within some questions. Although respondents indicated that the toolkit evaluated the most pertinent components of ASPs, further modifications and testing would be required to improve its validity. These results highlight the importance of the inclusion of end-users in the development of reporting and/or evaluation tools or questionnaires.