Mixed-methods approach to understanding clinician macrocognition in the design of a clinical decision support tool: a study protocol.

INTRODUCTION: The anatomic variants of congenital heart disease (CHD) are multiple. The increased survival of these patients and disposition into communities has led to an increase in their acute presentation to non-CHD experts in primary care clinics and emergency departments. Given the vulnerability and fragility of these patients in the face of acute illness, new clinical decision support systems (CDSS) are urgently needed to better translate the best practice recommendations for the care of these patients. This study aims to understand the perceived confidence and macrocognitive processes of non-CHD experts (emergency medicine physicians) and CHD experts (paediatric cardiac intensivists) when treating children with CHD during acute illness and apply this to optimise the design of a CDSS (MyHeartPass™) for these patients. METHODS AND ANALYSIS: The first phase of the study involves a survey of non-CHD experts and CHD experts to understand their perceived confidence as it relates to treating acutely ill patients with CHD. The second phase is a qualitative cognitive task analysis using critical decision method to characterise and compare the macrocognitive processes used by non-CHD experts and CHD experts during the critical decision making. In phases 3 and 4, heuristic evaluation and usability testing of the CDSS will be completed. These results will be used to inform design changes to the chosen CDSS (MyHeartPass™). In the final phase, a within-participant simulation design will be used to study the effect of the CDSS on clinical decision making compared with baseline (without use of CDSS). ETHICS AND DISSEMINATION: Ethics approval from The Hospital for Sick Children in Toronto, Ontario, Canada has been obtained for all phases. Results will be published in peer-reviewed journals and presented at relevant conferences. On successful completion of these studies, it is anticipated that there will be a controlled implementation of the redesigned CDSS. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study applies a human factors research approach to understanding the differences in macrocognition between two groups of clinicians with different training in caring for patients with congenital heart disease who present with acute illness.

The mixed-method study design allows a thorough understanding of clinician macrocognition and their interaction with the user-interface which will inform improvements of the clinical decision support systems (CDSS).

Simulation testing allows for near-live testing of the CDSS in both qualitative and quantitative manner.

Due to an overall small sample size and specific experience of the clinicians studied, generalisability of clinician macrocognition may be limited.

Due to localised sampling of data and testing, the generalisability and application of the CDSS outside the institution in which it is being conducted cannot be determined.

Introduction

In response to the advances in medical care and need for domain expertise, as well as the myriad of diagnostic and therapeutic tools available, medical disciplines have evolved into sub-speciality fields. As an example, the complex and abnormal heart anatomy and physiology in patients with congenital heart disease (CHD), as well as the targeted management strategies used to treat CHD, has led to the development of sub-specialty expertise, experience and resources. CHD refers to the class of heart disease that is due to inborn abnormalities in structure and function of the heart and its great vessels. These abnormalities affect the function of the heart and increase the predisposition of these patients to acute deterioration in the setting of many common childhood illnesses, while also altering their response to conventional resuscitative therapies.1 Advances in diagnosis, intervention and management have improved the survival of these children, and most now live in diverse communities, often remote to subspecialised care.2–11 Therefore, when acutely ill, these patients present to community hospitals and emergency departments where the expertise to understand their complex heart disease may be limited. Many of these children present with heart failure symptoms that can be difficult to distinguish from more common childhood illnesses, and this, in turn, can lead to delays in diagnosis or inappropriate treatment.5 12–16 A survey of emergency department physicians in the state of Michigan revealed that nearly 72% were unsure of what the appropriate oxygen level in the blood should be in a patient with complex heart disease (one ventricle and normally cyanosed), and 58% worried and were uncomfortable caring for these patients.8

The use of clinical decision support systems (CDSS) can mitigate some of the challenges associated with the complexities of clinical decision making, such as those experienced by non-experts caring for acutely ill patients.17 CDSS encompass electronic systems that are designed to help clinical decision making, and include electronic health records (EHRs), protocols, practice guidelines and alerts. These systems, however, have been difficult to adopt due to usability and implementation issues with persistently high incidence of medical errors despite their adoption.18 19 For a CDSS to be effective, it needs to be designed with the cognitive processes of their users in mind. Careful consideration to the context and complexity with which data is presented can enhance clinician understanding and facilitate error prevention.18 20 Readily accessible CDSS can also allow for knowledge transfer in any setting. Finally, an organisational implementation of CDSS is required to avoid challenges with workflow integration and the need for workarounds which can further contribute to errors and poor implementation.18 20

Previous research has identified that novice and expert physicians use different sense-making styles (eg, novice physicians move to diagnosis quickly and dismiss cues that do not fit their initial diagnosis whereas experienced physicians are more prone to detecting subtle cues and considering alternative diagnoses) and that these differences help inform development of interventions.21 Although Schubert et al 21 studied the difference between expertise within the same clinical specialty, the difference in clinician macrocognition between experts of different disciplines caring for similar patients remains unexplored. Understanding such differences is particularly important in the design of CDSSs, which aim to reduce the knowledge gap between non-experts and experts.

In the absence of a CDSS specific for the acute management of paediatric CHD, we aim to optimise the design of our prototype CDSS (MyHeartPass™) based on clinicians’ cognitive processes. MyHeartPass™ is a web-based application for clinicians caring for patients with CHD who present with acute illness. It provides clinicians with specific historical information about the CHD, functional and physiological patient state, and relevant recommendations for management to facilitate clinical decision making. While the contents of this CDSS are based on the latest evidence and expertise in the field of CHD, in the absence of a firm understanding of the macrocognition of its potential users, its user-interface and overall design may not be ideally suited for non-expert and expert clinicians alike. The purpose of this study, therefore, is to describe the design and rational of a mixed method study to: (1) understand non-CHD expert and CHD expert clinicians’ perceived confidence in treating patients with CHD and the macrocognitive processes used, (2) inform the continued design of MyHeartPass™ in keeping with the mentioned principles of design and (3) evaluate its impact on the acute management of children with CHD.

Materials and methods

Participants

Non-CHD expert participants will include paediatric emergency medicine staff physicians, senior paediatric medicine residents and paediatricians who have not had specialised training in paediatric CHD management. These non-CHD experts are more likely to encounter a patient with CHD who presents with a common acute childhood illness. CHD experts will be recruited from paediatric cardiac intensive care physicians who practice in the cardiac critical care unit of a quaternary paediatric hospital with a large paediatric CHD programme in Canada. Expertise is established based on their specialised training in the field of CHD medicine, years of experience caring for these patients, academic publications and activities within this domain, and their primary clinical role in caring for critically ill paediatric patients with CHD. Nurses and respiratory therapists will be recruited for the simulation phase of the study and selected from those who work in the critical care unit or the emergency department of the same paediatric hospital. They are considered to have role specific expertise in paediatric resuscitation and, therefore, ideal for the simulation phase of this study.

Ethics and dissemination

Ethics approval has been obtained from the Research Ethics Board (REB 1000064565 and 1000064567) of The Hospital for Sick Children in Toronto, Ontario, Canada. Consent will be obtained from all participants for their audio recording during all phases of the study and video recording during the simulation phase. These will be anonymised and kept private and confidential behind institutional firewall with exclusive access rights with the study team and in keeping with the REB requirements. Participants will be recruited on a voluntary basis. The initial introduction to the study for possible enrolment will be done by the primary author and her emergency medicine collaborator through research presentations during scheduled research days in both departments of Paediatric Emergency Medicine and Critical Care Medicine at a quaternary paediatric hospital. To avoid undue influence, consent from CHD experts will be obtained by the collaborator from department of emergency medicine, while consent from non-CHD experts will be obtained by AA as they would not be in a position of authority with their corresponding participants to impose undue influence or coercion. The findings from this study will be disseminated to clinical teams, presented in various human factors and medical conferences, and submitted for publication.

Patient and public involvement

No patients or parents will be involved.

Phase 1: understanding the perceived confidence of clinicians caring for paediatric CHD patients

Study design

An electronic cross-sectional survey using the RedCap® survey system will be sent to non-CHD expert group of adult and paediatric emergency medicine physicians registered to practice in Canada and are members of Canadian Association of Emergency Physicians as well as all paediatric emergency medicine physicians registered with the Paediatric Emergency Research Canada network. CHD experts will be surveyed through the Paediatric Cardiac Intensive Care Society network. The survey will be anonymous and voluntary with implied consent. The survey is a modified version of that used in the study by Cashen et al to answer the question of confidence among emergency medicine physicians in the state of Michigan8. A copy of the survey questions is shown in online supplementary appendix 1a, b). This phase of the study will run from January to June 2020. To ensure an acceptable response rate and to comply with REB requirements, the initial electronic survey will be sent with a personal cover letter explaining the project along with subsequent two reminder emails on a monthly basis. The survey itself is designed to be short and respondent-friendly on an electronic platform to facilitate ease of response. As a financial incentive, all participants who complete the survey will have an opportunity to enter a draw for an Apple tablet. This is a modification to the Dillman strategy as an electronic survey platform is used in this study.22

Data analysis

The responses of adult and paediatric emergency medicine physicians, as well as that of CHD experts will be compared where possible using a two-sample t-test and analysis of variance (ANOVA) where more than two variables exist. To correct for multiple comparisons, the Bonferroni method will be applied. For qualitative responses, a thematic analysis will be undertaken and compared between non-experts and experts.

Phase 2: understanding clinical decision making

Critical decision method (CDM) is a specific cognitive task analysis (CTA) technique that focuses on understanding decision making under conditions of uncertainty. To understand the macrocognitive functions and processes of non-CHD experts and CHD experts, CDM will be used to guide semi-structured interviews of 4–6 clinicians from each of the non-CHD expert and CHD expert groups. Based on previous CTA studies of medical professionals, 4–6 interviews in each group would be sufficient to achieve saturations.21 23 Should new themes continue to emerge, however, additional participants will be recruited until saturation is achieved.

A researcher, trained in interviews, will ask questions similar to those posed by Crandall and Getchell-Reiter,24 Baxter et al 25 and Schubert et al 21 to understand the critical decision-making process of participants. In the interview, participants identify a challenging clinical scenario in which they were caring for an acutely ill patient with CHD and describe the events in detail with specific focus to the decisions made and the circumstances surrounding that decision. All the interviews will especially focus on understanding what information was being sought, in what order and when; what decisions they made and in what order; what treatments were selected, based on what data or patient finding, and how or why those treatments were deemed appropriate. The structured components of the interview are shown in table 1. This phase of the study will run parallel to the first phase between January and June 2020.

Table 1

Overview of the semi-structured CTA interview

Phase	Description
	Non-CHD expert	CHD expert
Introduction	Introduction to the study and description of how a semi-structured interview using CDM will operate. Consent will be obtained before starting the interview.
Scenario selection	Please think about a clinical scenario from ED involving a child with CHD that was particularly challenging, and your clinical expertise and experience made a difference in how things turned out. It could be a case where you think if someone else had managed the patient instead of you, things would have turned out differently.	Please think about a clinical scenario involving a child with CHD from your practice as a cardiac intensivist that was particularly challenging, and your clinical expertise and experience made a difference in how things turned out. It could be a case where you think if someone else had managed the patient instead of you, things would have turned out differently.
Event recall	Participants asked to describe the scenario in detail and in sequence where possible, to allow the creation of a timeline of events (particularly decision points)
Creation of a timeline and identification of decision points	Interviewer will use the timeline of events they have created based on the participant narrative to verify the story and timeline. In particular, the location of decisions and kinds of decisions will be clearly identified for further probing.
	Special attention will be paid to whether/when the participant sought additional help/consultation from CHD experts available to them as well as whether they considered transferring the patient to a cardiac ICU or cardiology ward.	Special attention will be paid to whether/when participants sought additional help from cardiovascular surgeons or other cardiologists, echocardiographers and interventionalists.
Probing questions	Interviewer will use a semi-structured interview format to ask probing questions to better understand the circumstances surrounding the decision points.Additional questions about what patient specific data was needed to understand the physiology or to make decisions and decide on interventions; what specialised/domain specific knowledge was used/needed to make decisions and suggests treatments; Why was that information/knowledge important; was it difficult to find this information; and if it effected their ability to make decisions and provide treatment.

CDM, critical decision method; CHD, congenital heart disease; CTA, cognitive task analysis; ED, Emergency Department; ICU, Intensive Care Unit.

Recordings from the interviews will be transcribed verbatim by the researcher (AA) and checked against the original recordings for accuracy. Transcribed interviews will be coded using NVivo software for macrocognitive processes. Two reviewers will code interview transcripts deductively based on a priori macrocognitive processes identified from Klein’s original macrocognition framework and Schubert et al’s subsequent modifications to that framework, and inductively based on the identified emerging processes.21 26 Raters will review their inductive codes for overlap (ie, both raters may identify the same emerging process principle, but will need to come to consensus on the wording moving forward). Coding discrepancies will be discussed among the coders until consensus is reached. The set of agreed on macrocognitive processes will comprise the ‘analytical framework’ that will be used by the coders to independently code subsequent interviews. Cohen’s kappa for inter-rater reliability will be calculated using NVivo’s coding comparison query function. Once an inter-rater reliability above 0.727 is reached on a subset of independently coded transcripts, the remaining transcripts will be coded by one coder. Common themes will be time-aligned to the various phases of the scenario described by participants to create a flow diagram or timeline of the macrocognitive processes engaged by each group. The overall differences in macrocognitive functions and processes used by the two groups will also be compared. These findings will be reviewed by domain experts (a CHD expert and non-CHD expert, respectively) to establish trustworthiness. A reflexive bracketing approach will be used to minimise impact of researchers’ pre-existing biases and experiences on the study.28 29

Phase 3: heuristic evaluation of MyHeartPass™

The purpose of this phase of the research is to identify the ways in which the design of the CDSS, MyHeartPass™, could be optimised to improve functionality. To achieve this goal, in January 2020, a human factors engineer and a clinician with expertise in paediatric CHD and familiarity with the MyHeartPass™ application will assess the interface of MyHeartPass™.

MyHeartPass™ (figure 1) is a prototype mobile/web-based application developed through a participatory design process with domain experts. It provides clinicians with five broad categories of information organised into five sections. The first category refers to the basic patient information, demographics and anatomical description of their CHD. The second category provides a more in-depth insight into the underlying cardiac physiology of these patients including a diagram of the CHD and functional features of its various parts, along with recommendations specific to the management of this physiology. These recommendations are based on the most recent resuscitation guidelines for children with CHD.1 The third category consists of additional pertinent, patient-specific medical information which, along with the demographic information from the first category will be auto populated through a connection with the hospital admission, discharge, transfer system. In the fourth category, recommendations specific to resuscitation considerations for these patients are provided. Finally, the fifth category contains a list of symptoms that could be considered as indicators of heart failure symptomatology in the patient. MyHeartPass™ is currently only a prototype and has yet to be tested or implemented in a clinical environment.

Figure 1

MyHeartPass™ prototype. ASA, Acetylsalicylic Acid; BT, Blalock-Taussig; BTS, Blalock-Taussig Shunt; IVC, Inferior Vena Cava; LA, Left Atrium; LPA, Left Pulmonary Artery; Neo-Ao, Neo-Aorta; NKA, No Known Allergies; PALS, Paediatric Advanced Life Support; PICC, Peripherally Inserted Central Catheter; RA, Right Atrium; RPA, Right Pulmonary Artery; RV, Right Ventricle; SVC, Superior Vena Cava.

Evaluators will independently assess the CDSS using the same version of the application on the same platform. Both evaluators will be asked to assess the user interface of MyHeartPass™ using Zhang et al’s30 criteria for heuristic evaluation of medical devices, which is captured in table 2. A rating system will be used to score the interface as outlined in table 3.

Table 2

Zhang et al’s heuristics criteria30

Criteria	Heuristic	Description
Consistency	Consistency and standards	Users should not have to wonder whether different words, situations, or actions mean the same thing. Standards and conventions in product design should be followed
Visibility	Visibility of system state	Users should be informed about what is going on with the system through appropriate feedback and display of information
Match	Match between system and world	The image of the system perceived by users should match the model the users have about the system
Minimalist	Minimalist	Any extraneous information is a distraction and a slow-down
Memory	Minimise memory load	Users should not be required to memorise a lot of information to carry out tasks. Memory load reduces users’ capacity to carry out the main tasks
Feedback	Informative feedback	Users should be given prompt and informative feedback about their actions
Flexibility	Flexibility and efficiency	Users always learn and users are always different. Give users the flexibility of creating customisation and shortcuts to accelerate their performance
Message	Good error message	The messages should be informative enough such that users can understand the nature of errors, learn from errors and recover from errors
Error	Prevent errors	It is always better to design interfaces that prevent errors from happening in the first place
Closure	Clear closure	Every task has a beginning and an end. Users should be clearly notified about the completion of a task
Undo	Reversible actions	Users should be allowed to recover from errors. Reversible actions also encourage exploratory learning
Language	Use users’ language	The language should always be presented in a form understandable by the intended users
Control	Users in control	Do not give users the impression that they are controlled by the systems
Document	Help and documentation	Always provide help when needed

Table 3

Zhang et al’s heuristics severity rating scale30

Score	Description of criteria
0	Not a usability problem at all.
1	Cosmetic problem only. Need not be fixed unless extra time is available.
2	Minor usability problem. Fixing this problem should be given a low priority.
3	Major usability problem. Important to fix this problem. Should be a high priority.
4	Usability catastrophe. Imperative to fix this problem before release.

Evaluators will then compare their scoring and discuss discrepancies until consensus is achieved. Findings will be analysed with respect to the total number of usability issues identified, the proportion of usability issues in each severity class, and the distribution of usability issues given the categories used for evaluation.

Phase 4: usability testing of MyHeartPass™

Usability testing of MyHeartPass™ with potential end users will allow the identification of safety and usability issues which can be addressed to optimise the user-technology interaction. Previous work in this area related to cost-benefit analysis has identified 5–7 users as the optimal number per test to identify approximately 85% or more of potential usability issues and achieve issue saturation.31 As such, 5–7 participants from each of the non-CHD expert and CHD expert groups will be recruited to participate in this phase of the study.

Between September 2020 and February 2021, participants will be asked to answer clinical questions related to CHD, the answers to which will be in the mock MyHeartPass™ records provided to the participants. Participants will be asked to think aloud as they proceed through answering the questions asked by the researchers. Usability scores will be assigned based on whether participants were able to answer a question independently (score of 2), with one hint (score of 1), with multiple hints or not at all (score of 0). If after two hints, the participants are unable to answer a question correctly, it will be considered an error. Two raters with expertise in usability testing will assign scores. Where there is disagreement on a score, agreement will be reached through discussion. Raters will also score the severity of errors such that major errors are assigned a score of 0, minor errors a score of 1 and inconsequential errors a score of 2. Major errors are those that could lead to significant patient morbidity and mortality (eg, unable to access correct information about the patient diagnosis and functional status leading to an incorrect or delayed decision about treatment) minor errors are those that can cause minor harm (eg, failing to find recent investigation results, and requesting new tests that may delay treatment), and inconsequential errors that would not lead to patient harm (eg, unable to find the gender of a patient where gender would not influence the decisions or management for CHD).

Once all tasks have been completed, participants will be asked to complete a short perceived usability survey (online supplementary file 2) to further evaluate the subjective usability of MyHeartPass™ and obtain basic participant demographic information (including role, years of experience in their role and years of experience caring for children with CHD).

Usability score distributions and averages will be calculated to evaluate the overall usability of MyHeartPass™. Descriptive data will be presented to identify if the types of usability issues and their associated severity vary between non-CHD experts and CHD experts.

Phase 5: MyHeartPass™ redesign

Findings from the previous phase regarding the user-interface of MyHeartPass™ will be used to improve the design of the application. This includes changes to the content of the application as well as the overall design and flow of the user-interface to address the issues identified by participants from the previous phases. The heuristic study will guide the development of the user interface (eg, use of colour, fonts, etc) based on standards of user interface design. Usability testing will shed light on the users’ response and interaction with the application and indicate the need to improve the existing system in terms of preventing errors, being consistent with what users expect, and displaying information such that user’s memory load is minimised. The type of information sought by clinicians during the CTA test will also inform the content design of the application such that additional relevant data would be added while other, less used data removed to better fit the thought process of clinicians. CTA results will provide an understanding of the cognitive processes used by CHD non-experts and experts when making critical decisions. Understanding differences in the use, and patterns of use, of macrocognitive processes across levels of expertise will provide insights to guide the interface redesign (eg, how data should be integrated and visually displayed) to support macrocognitive processes most prominent to each level of expertise. For example, if experts and non-experts differ in how they use macrocognitive processes during decision-making tasks, we will redesign the interface to support and enhance their respective cognitive processes for decision making. The original developers of MyHeartPass™ will be completing the redesign between March and June of 2021 which will then be used in the simulation phase described below.

Phase 6: simulation-based assessment of MyHeartPass™ compared to current state

The simulation-based assessment of MyHeartPass™ is to evaluate the impact of such a CDSS on the clinical decision-making practices of both non-CHD experts and CHD experts. The outcome of interest is the ease and time to navigate the CDSS to support decision making and actions. We hypothesise that the time to correct diagnosis, identifying appropriate recommendations, and seeking assistance where appropriate will decrease for all participants using MyHeartPass™, while the total number of appropriate recommendations will increase. In addition, the non-CHD experts are expected to improve their times more than CHD experts using MyHeartPass™.

Participants from both groups will be recruited to a counter balanced, within-subject study where they will complete two cardiac scenarios, one with MyHeartPass™ and another with the current standard of care using existing and routine sources of information. A total of 28 participants (14 in each group) will be recruited for this design to achieve a power of 0.8 for an alpha of 0.05 and a medium effect size based on power calculation by G*power calculator. The simulation scenarios will closely mimic near-live scenarios where a medium-high fidelity mannequin will be used to simulate a patient, and real-time monitors and equipment in the room will provide the clinical information and management that clinicians wish to use. MyHeartPass™ and the tablet version of the current EHR system will be made available to the participants on a tablet during the corresponding scenarios. The case exemplars themselves will be comparable in difficulty according to Risk Adjustment for Congenital Heart Surgery 1 (RACHS-1) category risk stratification32 but will represent different physiologies responsible for the clinical presentation at hand. The details of the cases are designed and reviewed by CHD domain experts (PL and AA) based on existing and tested simulation scenarios used for training. Participants will be asked to lead teams of nurses and respiratory therapists that will be involved in the simulated scenarios per their usual processes. While some participants may be those who had participated in the CTA phase of the study, a mix of new and previous participants will be recruited for this part of the study.

At the end of each simulation, all participants will be debriefed as a group, in keeping with the current practice of debriefing following simulations. Upon completion of the simulation sessions, a semi-structured interview will be held with the study participants (team leads) to understand their experience with the use of MyHeartPass™ during the simulation, its utility and usability and any recommendations to further improve the application. This phase of the study is projected to be completed between June 2021 and January 2022.

The video recordings will be used to measure: (1) study participant’s time to recognition of an issue, (2) study participant’s time to accurate diagnosis, (3) study participant’s time to correct action and (4) study participant’s time to activation of additional supports. These measures will be compared for significance between the two groups (with and without MyHeartPass™) for each participant. Where possible, data will be analysed using two-sample t-test and ANOVAs where more than two variables exist. To correct for multiple comparisons, the Bonferroni method will be applied.

Conclusion

Patients with CHD have complex disease and depend on subspecialists with expertise in CHD for their initial and longer-term care. As their survival and quality of life has substantially improved, they have become more dispersed and are more likely to attend an emergency room or clinic when acutely unwell, rather than directly to an expert in CHD. A well-designed CDSS will give non-CHD experts access to information and support for their clinical decision making. To this end, we have developed a prototype CDSS, called MyHeartPass™. Before considering implementation, however, it is essential to make sure the CDSS is well designed and functional for the end user. To do this, it is important to understand the macrocognitive differences in decision-making and information processing between non-experts and experts in CHD. Once these studies are completed, it will be possible to more accurately design the CDSS to fit the needs of both non-CHD experts and CHD experts. Heuristic and usability testing will further provide important information to improve the design of the CDSS, which in turn will be tested in a simulated setting. Ultimately, evaluating the difference in responses of non-CHD experts and CHD experts will determine the value of the CDSS with respect to clinical management, efficiencies and patient outcomes. An added benefit of understanding how experts think and make decision is the knowledge translation to non-experts and incorporation into novice and non-CHD expert clinician training to better equip them to care for the increasing population of CHD patients.