Development and implementation of a mobile device-based pediatric electronic decision support tool as part of a national practice standardization project.

Objective: Implementing evidence-based practices requires a multi-faceted approach. Electronic clinical decision support (ECDS) tools may encourage evidence-based practice adoption. However, data regarding the role of mobile ECDS tools in pediatrics is scant. Our objective is to describe the development, distribution, and usage patterns of a smartphone-based ECDS tool within a national practice standardization project. Materials and
Methods: We developed a smartphone-based ECDS tool for use in the American Academy of Pediatrics, Value in Inpatient Pediatrics Network project entitled "Reducing Excessive Variation in the Infant Sepsis Evaluation (REVISE)." The mobile application (app), PedsGuide, was developed using evidence-based recommendations created by an interdisciplinary panel. App workflow and content were aligned with clinical benchmarks; app interface was adjusted after usability heuristic review. Usage patterns were measured using Google Analytics.
Results: Overall, 3805 users across the United States downloaded PedsGuide from December 1, 2016, to July 31, 2017, leading to 14 256 use sessions (average 3.75 sessions per user). Users engaged in 60 442 screen views, including 37 424 (61.8%) screen views that displayed content related to the REVISE clinical practice benchmarks, including hospital admission appropriateness (26.8%), length of hospitalization (14.6%), and diagnostic testing recommendations (17.0%). Median user touch depth was 5 [IQR 5]. Discussion: We observed rapid dissemination and in-depth engagement with PedsGuide, demonstrating feasibility for using smartphone-based ECDS tools within national practice improvement projects. Conclusions: ECDS tools may prove valuable in future national practice standardization initiatives. Work should next focus on developing robust analytics to determine ECDS tools' impact on medical decision making, clinical practice, and health outcomes.

BACKGROUND AND SIGNIFICANCE

Electronic clinical decision support (ECDS) tools provide clinicians with information meant to aid in the diagnosis, treatment, and care of patients., These tools (which can be free-standing applications or integrated into existing health information technology platforms), can be passive or active transfers of information and are available in a wide variety of formats, including alerts, reminders, guidelines, and standardized order sets. A systematic review by Kawamoto et al. found four significant predictors to the success of ECDS tools: computer-generated support, integration into clinician workflow, support delivered at the time of decision making, and actionable recommendations. Cresswell and Sheikh additionally proposed that a tool’s success is influenced by the social, technical, and organizational framework supporting ECDS tools. Thus, a tool built with organizational support that incorporates evidence-based practices and that integrates end-user feedback may be more successful than one that does not. Despite these recommendations, there are limited reports of successful development, implementation, and adoption of ECDS tools, particularly across varied institutional settings and in pediatric care settings.,

In pediatrics, the evaluation of febrile infants for suspected severe infection/sepsis is a common clinical problem. Clinical practice varies widely in the evaluation and treatment of these patients despite widespread availability of validated risk-assessment tools., For example, in a study of 37 pediatric emergency departments, Aronson et al. (2015) found that use of antibiotics, invasive diagnostic tests, and hospitalization for febrile infants varied significantly. Implementation of institutional clinical practice guidelines (CPGs) is associated with substantially decreased variation in care, but the specific content of these CPGs remains unstandardized. Therefore, developing and disseminating standardized CPGs across a broad range of institutions could have a profound impact on the management of febrile infants nationally. ECDS tools have the potential to help standardize clinical practice, and could serve as an effective method of delivering evidence-based recommendations from CPGs to healthcare providers.

Experiences at our institution with mobile ECDS tool development demonstrated the feasibility and acceptability of using smartphone-based applications (apps) to disseminate CPG recommendations. From 2010 to 2013, our institution developed two apps for iOS devices (Figure 1). These apps, entitled CAP Guideline and iGuideline, provided clinicians with stepwise guidance adapted from our institution’s CPGs for managing children hospitalized with common acute illnesses, including: 1) community-acquired pneumonia (CAP); 2) asthma; 3) migraine; and 4) evaluation of the febrile infant <60 days of age. A survey of 242 healthcare providers at our institution found that 81% of smartphone users accessed at least one medically related mobile application, including pharmaceutical references, clinical calculators, and general medical references. Respondents most often reported application complexity and expense as reasons for not using medically related apps. These findings reflected previously published data on healthcare providers’ perceptions and use of mobile health applications.

Figure 1.

ECDS tool development timeline at Children’s Mercy Kansas City.

OBJECTIVE

In 2016, the American Academy of Pediatrics (AAP) Value in Inpatient Pediatrics (VIP) Network embarked upon a national project to standardize clinical practices in the evaluation of the febrile young infant entitled “Reducing Excessive Variation in the Infant Sepsis Evaluation (REVISE).” The focus of the VIP Network is to improve the value of care delivered to any pediatric patient in a hospital bed by helping providers implement CPGs and other best practices, with a special focus on eliminating harm and waste caused by over utilization. Previous projects sponsored by VIP have focused on a variety of common pediatric conditions. Given our prior experience developing a mobile ECDS tool for febrile infants, our group was selected to develop a similar tool for use by healthcare providers participating in REVISE. In this paper, we will describe the development, distribution, and initial usage patterns of a smartphone-based pediatric ECDS tool entitled PedsGuide in a national practice standardization project.

MATERIALS AND METHODS

Several members of our development team (SF, BK, KN) worked with the Project Expert Workgroup (which included RM and EB as members) to develop the “change package” for REVISE. The change package refers to all educational and clinical care materials furnished to project participants to assist with changing clinical practice at the institutional level. Prior VIP Network projects included print advertising, educational materials, and paper order sets, but electronic tools had not been previously included due to the wide variety of electronic health records (EHR) resources across institutions. After review by the AAP, our group was approved to develop an updated version of the iGuideline app for the REVISE change package.

A panel of national experts in pediatric infectious diseases, hospital medicine, emergency medicine, and general pediatrics convened to develop a set of core compliance metrics by which all participating sites would be assessed (Table 1).

Table 1.

Implementation metrics and balancing metric for the REVISE national collaborative

Metric	Target	Integrated into PedsGuide?
Appropriate hospitalizations	90% of hospitalized infants	Yes
Length of hospital stay (<30h for low risk; <42h for non-low risk)	80% of hospitalized infants	Yes
Urinalysis performed as part of initial evaluation	95% of infant evaluations	Yes
Chest X-ray obtainment	≤10% of infants without respiratory symptoms undergo chest X-ray	Yes
Recommended empiric antibiotics	90% of antibiotics prescribed are one of the recommended regimens	Yes
Incidence of missed serious bacterial infection (SBI)	<2% of evaluated infants discharged home return within 7 days with SBI	No

The five implementation and one safety/balancing metrics were selected based upon the strength of available evidence, clinical impact, and feasibility to monitor across all sites with only minimal resource use. These metrics were then incorporated into the clinical algorithms for the REVISE change package and the updated mobile ECDS tool, “PedsGuide.”

PedsGuide development

A local software company was identified to develop PedsGuide. The company’s selection was based on several factors, including: 1) its prior experience developing smartphone-based apps for healthcare and pediatrics; 2) its physical proximity for frequent and close communication with our institution and our team; and 3) the successful establishment of a master contract protecting our institution’s copyright and intellectual property rights and establishing cost schedules for development tasks. We deemed these factors to be critically important for a successful long-term relationship whereby our ECDS tool could be maintained and updated.

Once this relationship was formalized, we embarked upon a ground-up re-design of our ECDS tool. Although our institution owned the code for the original iGuideline, development of completely new software code was deemed the most appropriate course of action for several reasons, including: 1) the original app interface was outdated and inefficient on newer devices; 2) the original code was not adaptable for use on non-iOS devices; and 3) the integration of more comprehensive user analytics was not feasible. Thus, a new HTML-based hybrid software architecture was chosen to provide a more agile, cost-effective platform compatible with both iOS and Android operating systems, resulting in approximately 95% code reuse across platforms and reducing overall design and maintenance costs.

Next, our group took the management algorithms developed by the national expert panel and, in collaboration with the software developer, translated the algorithms into user workflow diagrams. These diagrams served as the basis for the app interface and allowed our team to supply the appropriate text and graphical and supplemental educational content for the app, including: 1) references; 2) risk checklists; 3) risk infographics; and 4) management information/recommendations.

Design elements

Previous studies have shown that available risk-prediction tools for evaluating febrile infants have a high negative predictive value (NPV >95%) yet clinicians still commonly perform invasive diagnostic tests on low-risk infants. Our group posited two reasons that could explain this variation: 1) difficulties with numeracy (the ability to work with or understand numbers) and risk estimation; and 2) base rate neglect cognitive bias. In base rate neglect, individuals tend to ignore general information on the rates of an event occurring in an at-risk population and instead use information relevant to only a specific case to predict the likelihood of disease. We therefore chose to incorporate two risk-assessment aids into the new app. First, we created checklists that allowed users to indicate the presence of a risk factor for bacterial infection. We also incorporated visual aids into the new tool’s interface that depicted the risk of bacterial infection pictographically (Figure 2A). Our choice of elements was based upon previous studies reporting that human figure pictographs are easily understood by most users, regardless of numeracy.,

Figure 2.

PedsGuide bacterial infection risk assessment sample workflow, including risk infographic screen (A, touch depth 2), diagnostic test recommendations (B, touch depth 3), and risk assessment checklist (C, touch depth 4). The check boxes on the risk assessment checklist represent potential user decisions that can be recorded.

Usability analysis

To optimize app design, a human factors expert at our institution (SF) consulted on the design. The field of human factors seeks to improve performance and decrease errors by understanding the cognitive strengths and limitations of people, and applying that knowledge to the design of tools and environments. For PedsGuide, the human factors researcher ran several high-level usability reviews of the prototype for adherence to usability guidelines, such as navigation architecture, readability, consistency, aesthetics, and layout. Ongoing iterative testing was performed throughout development to optimize the final product during development. The goal of this iterative process was to have a final product that would require less downstream redesign and updating and would facilitate effective use of the tool with a smooth workflow and high user satisfaction, potentially increasing adoption rates. Usability results were reconciled by the human factors expert along with clinician input, with formal recommendations sent to the developer for specific design changes.

User feedback, app updates, and analytics integration

User feedback was obtained during three phases: 1) pre-release beta testing; 2) pre-release content review and approval; and 3) post-release. Local pediatrician experts reviewed the app’s content and workflow and provided feedback throughout pre-release development. Application content also underwent review and approval locally by the Children’s Mercy Department of Evidence-Based Practice and nationally through the REVISE Expert Workgroup. Beta versions of PedsGuide were released and updated using TestFlight. Post-release, user feedback was obtained informally from individual participants in REVISE, from users who contacted the team directly via email, and formally through a survey tool sent out to all REVISE participants. User feedback from all sources, while overwhelmingly positive, also identified specific workflow and content issues that were addressed via multiple app updates (eg, placement of risk assessment checklists and inclusion of specific medication doses).

Participants in REVISE were invited to complete a survey to gain basic insight into participants’ comfort with ECDS tools and to explore individuals’ experience with PedsGuide. The survey’s questions were based off previous instruments (see Supplemental Materials). The survey was approved for release to REVISE participants after undergoing Institutional Review Board (IRB) approval by both the Children’s Mercy Kansas City IRB and the AAP’s IRB. Participants were contacted via the AAP’s Hospital Medicine email list-serve in January 2017 and again via the REVISE project list-serve in March.

Total number of app downloads was obtained from GooglePlay and the Apple App Store. PedsGuide usage data were aggregated and stored using Google Analytics. In Google Analytics, each user screen and action button within a mobile application can be labeled and tracked for future analysis. Each screen was labeled as a user action and each check box response (ie, a user-specific response) was labeled as a user decision (Figure 2C). Initial user action and user decision labeling within PedsGuide allowed for analysis of touch depth (the number of touches required to access specific screens), session duration, and geographic distribution. However, initial screen labeling prevented assessment of user interaction with guidelines-related app content. User action and decision labels were subsequently clarified and standardized in an update released in January 2017. For geographic analyses, session data were analyzed at the designated market area (DMA) level, based on prior evidence of its use in measuring user behavior across regions.

RESULTS

Participating clinicians and institutions

Clinical sites applied through the AAP to participate in REVISE and included a minimum of three institutional champions: a pediatric hospitalist, an emergency medicine physician, and a hospital administrator. A total of 132 sites across the United States, and one in Cameroon, applied to participate in REVISE, with 128 sites submitting data for review. Sites participated in a series of longitudinal web-based educational sessions throughout the project, including two sessions focused on the change package and PedsGuide app. The REVISE project was divided into pre-intervention and post-intervention cycles, with planned data collection by clinical sites starting 12 months prior to release of the change package through 12 months post-release. Data were entered into the Quality Improvement Data Aggregator (QIDA) database administered by the AAP, and periodic progress reports were distributed to sites for their review.

Initial downloads and usage patterns

PedsGuide was released as a free app to GooglePlay and the Apple App Store November 9, 2016. PedsGuide was downloaded by 3805 users (3119 iOS, 686 Android) across the United States leading to 14 256 use sessions between December 1, 2016 (the month the change package was released to REVISE participants), and July 31, 2017, resulting in an average of 3.75 use sessions/user during this time period. Users engaged in 60 442 total screen views, of which 37 424 (62%) involved accessing a portion of PedsGuide related to the REVISE clinical practice benchmarks. General app usage patterns are summarized in Table 2.

Table 2.

User downloads and initial usage patterns, December 1, 2016-July 31, 2017

Downloads	3805
Sessions	14 256
Avg. session duration	2 minutes 16 seconds
Screen views (avg. per session)	60 442 (6)
Unique screen views (avg. per session)	52 445 (5)
Metric related screen views (% of screen views)	37 424 (62%)
Median touch depth (IQR)	5 (5)
User decisions selected (avg. per session)	88 637(6)
DMAs with users (% of US DMAs)	135 (64%)

All averages are calculated by Google Analytics; we were unable to obtain raw data for calculation of medians and interquartile ranges.

Most (62%) user sessions lasted less than 60 seconds. Metric-related content accessed by users included hospital admission appropriateness (Metric 1 26.8%), length of hospitalization (Metric 2 14.6%), diagnostic testing recommendations (Metrics 3 and/or 4 17.0%), and recommended antibiotic regimen (Metric 5 3.4%). User decisions (defined as interaction with a checkbox on one of the checklists within the app; see Figure 2C for an example) occurred an average of six times per session. Geographic distribution of user sessions by DMA over time across the United States is depicted in Figure 3A–D.

Figure 3.

Map of cumulative PedsGuide sessions in US by designated market area (DMA) 12/1/2016-07/31/2017.

REVISE participant feedback

A total of 119 REVISE participants responded to the initial ECDS survey, an estimated 12% response rate. Of these, 79 (66%) had downloaded PedsGuide. Most respondents agreed that the app was easy to use (94%), that the functions were well integrated (78%), and that most people should learn to use the app very quickly (91%). Informal comments from users to date have been overwhelmingly positive. Sample responses included: “I recommend that my residents all download and use the app”; “I’ve used the app when an outside [healthcare] provider has called me before transferring a patient.” Site leaders for REVISE also requested information on their own local app usage patterns. Suggestions for improvements to PedsGuide from clinicians have focused on specific content concerns (eg, “I wish you provided more guidance for viral testing in the app”; “I disagreed with your app’s recommendation on this case”). Finally, no significant user interface or workflow criticisms have been identified in informal or formal user surveys.

DISCUSSION

In this report, we describe our longitudinal experience developing and deploying mobile ECDS tools for pediatric healthcare providers locally and nationally. Our findings provide insight for other healthcare providers, researchers, and health policy experts on the potential large-scale role that mobile ECDS tools can play in delivering evidence-based recommendations to healthcare providers in a broad variety of institutional and geographic settings. Furthermore, our experiences can be expected to inform future multi-site practice improvement efforts.

To our knowledge, there are no previous reports of ECDS use that describe the experience of deploying such a tool as part of a national practice standardization project on the scale of REVISE. We previously analyzed our regional deployment of the iGuideline ECDS tool using Apsalar and found that, from its release in June 2013, it was downloaded by 937 users in the first year, resulting in 3013 user sessions and a median session length of 31.8 seconds. Multiple previous reports have demonstrated the value of integrating ECDS into EHR systems to disseminate evidence-based practice recommendations. However, these reports are based primarily on individual health systems or integrated networks that utilize the same basic EHR infrastructure. Other practice improvement projects using ECDS across various clinical settings and practices have bypassed the EHR to provide effective decision support. Several previous studies report on the use of a smartphone-based ECDS tool used to disseminate clinical practice guidelines, and in pediatrics, previous mobile ECDS tools have focused on hyperbilirubinemia (web-based tool) and appendicitis.,

Our experience in developing and deploying PedsGuide provides several novel insights for successfully disseminating and encouraging the large-scale adoption of ECDS as a means of improving health and healthcare. Specifically, we found that a mobile ECDS tool can be intentionally designed to adhere to the Five Rights. First, we aligned app content with clinical benchmarks selected for the national standardization project (the right information). Next, by making PedsGuide freely available and usable on multiple smartphone platforms, we made clinical recommendations accessible to clinicians (right person) at the point of care (right time in clinical workflow) regardless of a clinician’s ability to access the EHR (right channel), thus increasing the value of recommendations at the time of ECDS use. Perhaps most importantly, throughout the development process, we ensured that our tool was maximally user friendly through user-centered design, human factors-based usability assessment, and extensive beta testing (right intervention format).

Our experience also highlights the current benefits of using of a smartphone-based ECDS tool to bypass organizational limitations imposed by integrating ECDS tools into EHRs. Given the resource and programmatic differences across health systems, the need for ECDS interventions that are interoperable across multiple settings is critically important if they are to be usable in large-scale practice improvement initiatives. Nearly half of the hospitals participating in REVISE are small- to medium-sized community hospitals with limited information technology (IT) resources. Unfortunately the wide diversity in EHR infrastructure and access across these sites, particularly at smaller community-based health centers with limited information technology resources, makes it prohibitively expensive at present to deploy an ECDS tool that efficiently integrates into every conceivable EHR build. Thus, we employed a strategy of delivering our ECDS tool through mobile devices outside of the EHR, with the assumption that such devices are generally available among healthcare providers. However, improving mobile ECDS tools’ integration with EHR infrastructures will likely become necessary for such tools to maintain their value to the end user, due to external pressures from healthcare funding agencies as well as internal cost and workflow concerns of healthcare institutions.

The resources required for developing PedsGuide were substantial, including hundreds of hours invested by physicians, researchers, software developers, and administration. However, this interdisciplinary, team-based approach helped ensure successful app development. Going forward, development of new ECDS tools should require progressively less time, as the basic software infrastructure has already been built, roles have been better defined, team members have acquired significant experience in app development, and sustainability efforts have been initiated.

Our experience has also highlighted future opportunities to assess the effectiveness of ECDS tool use in large-scale practice improvement projects. The integration of user analytics provides the opportunity to conduct in-depth analyses on the relationship between user behavior, medical decision making, clinical practice, and associated health outcomes, all critical factors when assessing the effectiveness of ECDS tools. Our experience provides insight into some of the analytical methods necessary to address these knowledge gaps, which we will apply to analyzing clinical practice and health outcomes data obtained at the conclusion of the REVISE project. Unfortunately, current coding of Google Analytics into PedsGuide allows for in-depth analysis of only session-level data (user-level data are primarily limited to downloads reported by the Apple App and Google Play stores); future updates to our analytics coding will allow for more in-depth analyses of user-level data.

Future studies planned as part of the REVISE project include assessing the relationship between app usage and reported clinical practices and health outcomes from abstracted medical records. Additionally, a simulation-based clinical trial is also being conducted to compare medical decision making and cognitive workload when using PedsGuide versus a standard medical reference tool (ie, the Harriet Lane Handbook) to manage febrile infants. Finally, in future VIP Network projects, we plan to use a phased rollout approach to better assess the impact of ECDS tools on clinical practice.

CONCLUSION

Smartphone-based ECDS tools that provide a mobile, easily accessible format and readily integrate into clinical workflow are a feasible and effective method of delivering evidence-based practice recommendations to clinicians as part of a national practice improvement project. However, developing an effective tool requires an interdisciplinary, team-based approach. Future work on developing robust analytics to determine the impact of ECDS tools on medical decision making, clinical practice, and health outcomes is needed. Our experience may help inform future efforts to deploy ECDS tools that are tailored to practice standardization and improvement projects.

FUNDING

This project was funded in part by a Blue Outcomes Research Award from the Kansas City Area Life Sciences Institute.

CONTRIBUTORS

Dr. McCulloh substantially contributed to the conception, design of the work, data acquisition, data analysis, and interpretation of data. He drafted the work, revised it critically for important intellectual content, and approved the final version submitted to JAMIA. He agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Dr. Fouquet substantially contributed to the conception, design of the work, data acquisition, data analysis, and interpretation of data. She helped draft the work, revised it critically for important intellectual content, and approved the final version submitted to JAMIA. She agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Dr. Herigon substantially contributed to the conception, design of the work, and interpretation of data. He revised the work critically for important intellectual content and approved the final version submitted to JAMIA. He agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Dr. Biondi substantially contributed to the conception of the work and interpretation of data. He revised the work critically for important intellectual content and approved the final version submitted to JAMIA. He agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Dr. Kennedy substantially contributed to the conception of the work and interpretation of data. He revised the work critically for important intellectual content and approved the final version submitted to JAMIA. He agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Ms. Kerns substantially contributed to the design of the work, data acquisition, data analysis, and interpretation of data. She helped draft the work, revised it critically for important intellectual content, and approved the final version submitted to JAMIA. She agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Dr. DePorre substantially contributed to the conception of the work and interpretation of data. She revised the work critically for important intellectual content and approved the final version submitted to JAMIA. She agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Dr. Markham substantially contributed to the conception of the work and interpretation of data. She helped draft the work, revised it critically for important intellectual content, and approved the final version submitted to JAMIA. She agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Dr. Chan substantially contributed to the conception of the work and interpretation of data. He revised the work critically for important intellectual content and approved the final version submitted to JAMIA. He agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Ms. Nelson substantially contributed to the conception of the work and interpretation of data. She helped revise the work critically for important intellectual content and approved the final version submitted to JAMIA. She agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Dr. Newland substantially contributed to the conception, design of the work, and interpretation of data. He revised the work critically for important intellectual content and approved the final version submitted to JAMIA. He agrees to be accountable for all aspects of the work and will help ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.