Macquarie Surgical Innovation Identification Tool (MSIIT): a study protocol for a usability and pilot test.

INTRODUCTION: Medicine relies on innovation to continually improve. However, innovation is potentially risky, and not all innovations are successful. Therefore, it is important to identify innovations prospectively and provide support, to make innovation as safe and effective as possible. The Macquarie Surgical Innovation Identification Tool (MSIIT) is a simple checklist designed as a practical tool for hospitals to identify planned surgical innovations. This project aims to test the usability and pilot the use of the MSIIT in a surgical setting. METHODS AND ANALYSIS: The project will run in two phases at two Australian hospitals, one public and one private. Phase I will involve interviews, focus groups and a survey of hospital administrators and surgical teams to assess the usability and system requirements for the use of the MSIIT. Current practice regarding surgical innovation within participating hospitals will be mapped, and the best implementation strategy for MSIIT completion will be established. Phase II will involve trialling the MSIIT for each surgery within the trial period by various surgical personnel. Follow-up interviews, focus groups and a survey will be conducted with trial participants to collect feedback on their experience of using the MSIIT during the trial period. Comparative data on rates of surgical innovation during the trial period will also be gathered from existing hospital systems and compared to the rates identified by the MSIIT. ETHICS AND DISSEMINATION: Ethical approval has been obtained. The results of this study will be presented to interested health services and other stakeholders, presented at conferences and published in a peer-reviewed MEDLINE-indexed journal. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

This pilot project will potentially provide a reliable mechanism for identifying planned surgical innovations, indicate the current prevalence of surgical innovation and establish the track record of surgical innovation for participating hospitals.

Phase II of this pilot study will lead to a large-scale project to implement a potentially refined Macquarie Surgical Innovation Identification Tool (MSIIT) on a national and international scale.

Depending on the timing of MSIIT completion, it may not identify unplanned innovations or those arising during surgery.

This is a usability study and is not designed to determine possible effects of the MSIIT on surgical outcomes.

As there is no gold standard measure of surgical innovation available, the reliability of the MSIIT to identify surgical innovation will rely on comparison to existing hospital processes and user ratings.

Introduction

Surgical innovation has resulted in many significant improvements in patient care. Progress in the field of surgery, in fact, relies on innovations ranging from continuous incremental improvement to revolutionary techniques or technologies.1–3 Therefore, it is important to encourage surgical innovation and to disseminate its results so that the maximum number of patients can reap its benefits. However, along with these potential benefits, surgical innovation brings with it many risks of harm.1 These risks arise partly because much innovation occurs without the kind of oversight that applies to research.4 In Australia, for instance, it is often up to the discretion of the clinician as to whether or not innovation constitutes research and requires ethical review.5 6 However, innovation does by nature offer novel treatment with potentially variable results. Therefore, it is necessary to ensure surgical innovations occur in such a way as to extend benefits and reduce risk.

There are several initiatives around the world aimed at improving the safety of innovative medical practices. For example, the IDEAL (Idea—Development—Exploration—Assessment—Long-term studies) Collaboration is “[a]n international group of surgeons, researchers, journal editors, methodologists, statisticians, and other people who are committed to producing, disseminating, and evaluating quality research in surgery.”7 The IDEAL Collaboration's work towards developing guidelines to support innovative surgical procedures is well recognised.

However, thus far, there is a lack of consensus on the definition of innovative surgery.6 The lack of clarity surrounding what constitutes surgical innovations makes them difficult to identify, particularly when they do not involve the purchase of new equipment or tools.8 Currently, there is no gold standard for identifying innovation in surgery, which means hospitals may not have adequate processes for its management.6 Identifying surgical innovation is central to ensuring its safety. Reliably flagging innovation prospectively means the untested procedure can be managed to support the best possible successful outcome. Further, more reliable identification of any success of surgical innovation is necessary to ensure rapid dissemination of effective innovation. Hence, there is a need for mechanisms to identify innovation that would trigger the use of guidelines such as the IDEAL framework, enabling the prevention of harm and advancing improvements.

To help address this critical missing element, researchers at Macquarie University devised a practical, universal and clear definition of innovative surgery through the conceptual analysis of interviews with surgeons and healthcare professionals.6 8 This definition led to the development of the Macquarie Surgical Innovation Identification Tool (MSIIT), which is designed as an easy checklist to identify when procedures fall into the category of ‘innovative’ (figure 1). The checklist is intended to capture instances of both true innovation, in which the procedure has never been performed before, and introduction, in which the procedure has been performed before but not by the surgeon or not at the hospital in question. Instances of the latter are included to ensure the checklist identifies all cases in which the novelty of the procedure may impact patient safety. The purpose of the MSIIT is not to determine definitively whether or not a particular procedure is innovative, but is to flag procedures that may require extra support to manage innovation.6 Although the MSIIT is intended for use in a surgical setting, it has not yet been tested in practice. This pilot project will thus be the first to assess the usability of this tool and to pilot its use by surgical staff in hospitals.

Figure 1

The Macquarie Surgical Innovation Identification Tool.

Project overview and scope

Across two phases of research, this project will test the usability of the MSIIT (phase I) and pilot its use in the surgical setting (phase II). This project will lay the foundation for a potential larger national and international trial of the MSIIT (figure 2).

Figure 2

MSIIT project overview. MSIIT, Macquarie Surgical Innovation Identification Tool.

Phase I of the project focuses on the usability of the MSIIT in two Australian hospitals, using focus groups, interviews and surveys with clinicians and hospital management. Phase I aims to ensure the MSIIT is clear and easy to complete, to determine the best implementation strategy for the MSIIT in the phase II trial, and to reveal current hospital practices surrounding identification of and support for innovation in surgery.

Phase II will be a pilot study trialling the MSIIT in two Australian hospitals. On the basis of the feedback from phase I, phase II will investigate operational aspects of using the MSIIT in a surgical setting. Phase II aims to assess the utility of the MSIIT in practice and determine the requirements for a large-scale trial of the tool.

Across both phases, this project aims to determine: (1) the current rate of surgical innovation and existing means of identifying it; (2) whether the MSIIT is appropriate for use in the surgical setting; (3) whether it is easy and quick to complete or requires modification based on feedback; (4) when the MSIIT should be completed and by whom; (5) where it best fits into established hospital processes and (6) its capacity to identify surgical innovation compared to existing means within participating hospitals.

Methods and analysis

Study design

The study will use multiple methods across two phases. Phase I will involve interviews, focus groups, a staff survey, observation and process mapping.9 Phase II will involve a trial of the MSIIT, follow-up interviews, focus groups and staff survey, as well as process mapping and collection of comparative data.

Participants

One private and one public hospital in New South Wales will participate in both phases of the project.

All participants will be employees of the participating hospitals, who will be recruited through recommendations from clinical directors and snowballing. Employees will be drawn from several groups of staff including leadership teams involved with hospital governance, champions within surgical departments and theatre staff, such as surgeons, junior surgeons, anaesthetists, anaesthetic nurses, scrub nurses and scout nurses. At the clinical level, each staff group is anticipated to consist of 5–10 individuals in each hospital. Recruitment of employees will continue, using the process outlined below, until at least 15 participants in each hospital representing all professional groups have volunteered.

Participant recruitment

The leadership group, who were initially contacted by the research team during hospital recruitment and will thus already be aware of the study, will be approached individually and provided with the participant information sheet for phase I. They will be invited to participate, and if they agree, will be given a participant consent form to sign and invited to nominate a convenient interview time. Potential participants in the leadership group will receive the research team's contact details to allow them the opportunity to ask questions and raise concerns about the study prior to participation.

After liaison with clinical directors, the unit clinicians will be given a short presentation about the study during a regular team meeting or approached individually if no such opportunity is available. Participant information sheets for phase I will be handed out to eligible staff, and volunteers will be invited to come forward during or after the meeting, by contacting the researchers directly or through the project officer. The presentation will explain the purpose and nature of the study, the timescales, and what it will involve in practical terms for participants. The two phases of the research will be explained, and information sheets and consent forms will be made available for all staff to keep. Potential clinical participants will receive the research team's contact details to give them the opportunity to ask questions and raise concerns about the study prior to participation. The leadership group will not be informed of the identity of clinicians participating in the study.

Data collection

Phase I: usability

Phase I will focus on usability of the MSIIT. It will involve interviews, focus groups and a survey. Hospital administrators (eg, quality managers) and surgical team members (eg, anaesthetists, surgeons, scrub nurses) will be interviewed on the usability of the MSIIT, and various surgical team members will participate in discipline-specific focus groups. Interviewees and focus group participants will complete a brief survey (survey 1).

Interviews

Interviews will be conducted with hospital administration and clinical staff not available for focus groups (see online supplementary appendix 1). The aim of these interviews is to understand governance arrangements relevant to surgical innovation. Approximately 10 interviews will be conducted with surgeons, quality managers and other relevant personnel.

After consent has been obtained, a mutually convenient time will be arranged with participants for face-to-face interviews in a private room at their hospital. At the start of the interview, participants will be reminded of the implications of their consent and will be told they can ask researchers to stop the recording at any time. Depending on participant responses, it is anticipated that each interview will take <1 hour.

During each interview, participants will view the MSIIT and be asked to provide feedback on its usability and their opinion on the best implementation strategy. Participants will be invited to complete survey 1. Interview recordings will be transcribed verbatim in preparation for analysis.

Focus groups

Focus groups with clinical staff organised by relevant subgroups will be conducted concurrently with interviews (see online supplementary appendix 1). These focus groups will take up to 1 hour with a maximum of five people per group.

After consent has been obtained, a mutually convenient time will be arranged with participants for focus groups, which will be held in a private room at the relevant hospital. At the start of the focus groups, participants will be reminded of the implications of their consent and will be told they can request that recording be stopped at any time. Depending on participant responses, it is anticipated that each focus group will take no longer than 1 hour.

In each focus group, participants will be shown the MSIIT and be asked to provide feedback on its usability and the contextual factors impacting the timing and applicability of its use in surgical practice. In addition, participants will be encouraged to practice using and to interact with the tool via scenarios and will be asked about current procedures around surgical innovation in their hospital. Participants will be invited to complete survey 1.

One member of the research team will facilitate the sessions, with a second member making handwritten notes on any questions of clarity asked. Both researchers will observe and record notes on the ease of simulated completion of the MSIIT (see the ‘Observation’ section). Focus group recordings will be transcribed verbatim in preparation for analysis.

Survey 1

Survey 1 (see online supplementary appendix 2) will be given to interviewees and focus group participants to collect individual, self-reported experiences of using the MSIIT as well as estimates of current levels and types of surgical innovation. Phase I surveys will be administered after initial interaction with the MSIIT during interview or focus group participation. The survey contains 12 questions relating to usability (eg, the questions on the MSIIT flow logically) and personal experience of using the MSIIT in practice scenarios and interactions with the tool (eg, the result of the MSIIT fits with my original perception of the level of innovation in our example case). These questions will be measured on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). Survey 1 will also include five extended response questions relating to context (eg, which members of staff should complete the MSIIT?).

Observation

Both researchers present at the focus groups will record observations of MSIIT completion. These observations will include how the tool was completed in simulations and if confusion or questions were raised with researchers, for instance, any requests for clarifications necessary to complete the tool.

Process mapping

The relevant existing surgical processes will be mapped based on the interviews, focus groups and hospital procedure documents. The purpose of this process mapping9 will be to understand when the MSIIT could or should be completed, for which surgeries, and by which staff.

At the end of the interviews and focus groups in phase I, researchers will ascertain participants' willingness to participate in phase II of the research and give them phase II participant information sheets and consent forms for their consideration. All participants will be advised that they can withdraw consent at any time without fear of consequence.

Phase II: pilot test

Phase II will involve the participation of personnel from multiple surgical theatres in the same two hospitals, who will independently complete the MSIIT for surgeries performed during the trial period. After the trial, follow-up interviews and focus groups will be conducted with trial participants. Interviewees and focus group participants will complete a brief survey (survey 2). Comparative data will be collected from any existing hospital processes that identify new procedures. Phase II is intended to form the pilot study for a larger project to implement a potentially refined MSIIT on a national and international scale.

MSIIT trial

The MSIIT will be provided to the two participating hospitals for a trial period of 6 weeks or 100 surgeries, whichever occurs first. Discussions with hospital-based colleagues have suggested that 100 surgeries per hospital will be adequate to capture a representative sample. During this time, researchers expect to have sufficient data for usability and piloting purposes, sufficient to confidently design a large-scale, international study with a greater variety of surgical settings. The timing of its use will depend on the process map developed during phase I. At least two surgical team members will be asked to individually complete the MSIIT for each surgery. This will provide data on the inter-rater reliability of the tool, with the understanding that having different staff roles will also influence perspectives on innovation. MSIITs will be coded so that they can be matched to specific surgeries and type of staff, but de-identified for individual staff and patients. A researcher will be on-site to provide participants with the MSIIT forms to ensure accurate coding.

Approximately 10 MSIIT trial participants, including hospital administration and clinical staff not available for focus groups, will be interviewed individually about their experience of the trial. Phase II interviews will follow the procedure used in phase I, without the interaction with the MSIIT. Participants will be asked for their feedback on the trial and invited to complete survey 2. Interview recordings will be transcribed verbatim in preparation for analysis.

Focus groups with clinical staff who participated in the MSIIT trial will be conducted organised by relevant subgroups. In these follow-up focus groups, staff who took part in the trial will be asked about their experience of using the MSIIT. Phase II focus groups will follow the procedure used in phase I, without the interaction with the MSIIT and the accompanying observations. Participants will be asked for their feedback on the MSIIT trial and invited to complete survey 2. Focus group recordings will be transcribed verbatim in preparation for analysis.

Survey 2

Survey 2 (see online supplementary appendix 3) will be given to interviewees and focus group participants to collect individual, self-reported experiences of the MSIIT trial. Participants will complete the survey at the end of phase II interviews and focus groups. In a similar format to survey 1, survey 2 contains twelve 5-point Likert scale questions relating to usability (eg, the questions on the MSIIT flow logically) and personal experience of the MSIIT trial (eg, the result of the MSIIT fit with my original perception of the level of innovation during the pilot). An additional question (did the MSIIT, in your opinion, appropriately identify surgeries as innovative?) will be included in survey 2, measured on 5-point Likert scale from 1 (not often enough) to 5 (too often). Survey 2 will also include six extended response questions relating to context (eg, over the pilot period how many surgeries do you think involved a surgical innovation?).

The actual use of the MSIIT during the trial will be mapped based on completed MSIITs, interviews, focus groups and surveys. This mapping will be compared to the model made based on phase I.

Comparative data

Rates of surgical innovation identified through established hospital mechanisms during the trial period will be collected and compared with those identified by the MSIIT during the trial.

Data analysis

MSIIT trial

For each completed MSIIT form, answers to four yes–no questions and open-ended follow-up questions will be recorded (table 1). Questions (1) and (2) concern the usability of the MSIIT, regarding whether surgical personnel can correctly complete the tool in practice. Questions (3) and (4) concern the MSIIT's capacity to reliably identify innovation. The MSIIT will be considered to have identified an innovation if at least one item on the checklist (figure 1) has a ‘no’ response. The total number of surgeries identified as innovative for each unit will also be recorded and compared with data collected from established hospital processes (see the ‘Comparative data’ section).

Table 1

Data recorded from MSIIT trial forms

Question (Y/N)	Follow-up (open-ended)
(1) Was the MSIIT completed fully?	If no, what was left blank?
(2) Were extra notes or comments required?	If yes, of what nature?
(3) Was the surgery identified as innovative?	If yes, for what reason?
(4) Does this match the completion by other theatre staff?	If no, how so?

MSIIT, Macquarie Surgical Innovation Identification Tool.

Interviews/focus groups

Data from phase I interviews and focus groups, including observations, will form part of the pool to develop a process map of existing procedures at the participating unit. Data from phase II interviews and focus groups will become part of the pool to develop a process map of MSIIT use during the trial (see the ‘Process mapping’ section). Data will also be coded and analysed via descriptive statistics.

Interview and focus group transcripts will be thematically analysed using a modified form of the ‘constant comparative method’.10 This approach systematically analyses textual data employing an inductive logic. It is inductive because the analyst does not impose categories on the data which she or he has developed a priori, but rather allows the participants' understandings to come to the fore and structure the analysis. However, it will be modified here because the data are unavoidably prestructured to a certain extent according to the interviewer's questions. Although all interviews and all focus groups will include the same set of questions, they will be semistructured, so that the interviewer or facilitator is not forced to maintain the same order of questions each time. This technique allows conversation to flow more naturally and means that participants can reveal how they structure their understanding of an issue without excessive prompting.11

Surveys

Surveys 1 and 2 will be collated, and non-identifiable data will be entered into a database. Scoring of Likert scales will be reversed where required to ensure questions are directly comparable, with scores of 5 being the most favourable and 1 being the least favourable. Means, median and mode of each question will be calculated, and these will be compared across different groups, including hospital, specialty and staff type. To directly compare results from the two surveys, a non-parametric test will be used as appropriate for Likert scale data. The answers to the extended response questions will be collated and analysed. This information will be synthesised with other data to complete the study aims.

Process mapping

Process mapping9 will be used to develop a model of the ideal procedure for MSIIT use based on survey, interview and focus group data from both phases. Process mapping will also be used to outline how the MSIIT could support or tie in with existing systems managing innovative surgery within each hospital.

Comparative data

Rates of surgical innovation identified through established hospital mechanisms will be compared with those identified by the MSIIT. Variability of identification over the trial period will be assessed and reported back to hospitals for verification.

Ethics and dissemination

This study is considered to be very low risk for participants. Low and negligible risk (LNR) ethical approval has been obtained from the Human Research Ethics Committee (HREC/16/WMEAD/91) for the public hospital and governance approval is currently being sought. Ethical and governance approval has been obtained from the HREC for the private hospital (Macquarie University HREC (Medical sciences) reference number 5201600426).

Patient and participant details and comparison data

Only the patient's gender and age will be collected on the MSIIT form. At no point will patients' names, addresses or other personal information be known to the researchers. Gender and age will be collected to justify questions 2b and 2c in the MSIIT (figure 1), which ask about the application of the surgery to that patient gender and age group. A study code will be generated for each procedure, and the date and time of surgery will be collected to ensure accurate counting and matching of forms completed by different surgical team members for the same procedure. This coding enables the researchers never to be in possession of identifying patient information.

Permission for access to the number and type of surgical innovations identified by existing hospital processes will be included in the governance approval. Clinical managers will supply relevant patient de-identified data to the research group.

The researchers are collecting de-identified data with the aim of analysing the tool, not the conduct of surgeons or other surgical staff. The data collected will not be used to audit surgeries and will not be matched to any measures of surgical outcome.

Limitations and considerations

As there is no gold standard for identifying surgical innovation, the validity and reliability of the MSIIT will be established using multiple participant completion for individual surgeries and user rating of accuracy. The researchers recognise that these will be influenced by roles, individual experience and perceptions.

The aim of this study is to test the MSIIT and does not extend to influencing existing hospital processes around surgical innovation. The amount of surgical innovation in the participating hospitals and its current level of oversight should remain unchanged during the course of this project. Should hospitals, after the completion of the project, find that the MSIIT identified more surgeries than their existing mechanisms during the course of the trial, they may independently wish to review hospital processes, however the collected MSIIT will be coded and individual staff data not supplied. Furthermore, it will be stressed that without further research, a larger trial of the MSIIT, such interpretations may be premature.

Dissemination

The results of this evaluation will be documented and presented to interested health service management and staff, made available to relevant stakeholders such as the IDEAL Collaboration, presented at conferences and published in a peer-reviewed MEDLINE-indexed journal.

Key milestones and timeframe

Phase I of the project is anticipated to take 6 months beginning August 2016, including recruitment of staff, interviews, focus groups and data analysis in preparation for phase II. Phase II of the project will start immediately after phase I. It is expected to require at least 6 months for implementation of the pilot, interviews, focus groups, data analysis and comparison and feedback to the participating hospitals. The project is expected to conclude by December 2017.

If this pilot project shows that the MSIIT can reliably identify surgical innovations and thereby support existing mechanisms for responsible innovation, a large-scale partnership project will be launched. The next project will involve trialling the MSIIT across a broad range of hospitals nationally and internationally.