It's Just (Academic) Business: A Use Case in Improving Informatics Operations with Business Intelligence.

Academic biomedical informatics cores are beholden to funding agencies, institutional administration, collaborating researchers, and external agencies for ongoing funding and support. Services provided and translational research outcomes are increasingly important to monitor, report and analyze, to demonstrate value provided to the organization and the greater scientific community. Thus, informatics operations are also business operations. As such, adopting business intelligence practices offers an opportunity to improve the efficiency of evaluation efforts while fulfilling reporting requirements. Organizing informatics development documentation, service requests, and work performed with adaptable tools have greatly facilitated these and related business activities within our informatics center. Through the identification and measurement of key performance indicators, informatics objectives and results are now quickly and nimbly assessed using dashboards. Acceptance of the informatics operation as a business venture and the adoption of business intelligence strategies has allowed for data-driven decision making, faster corrective action, and greater transparency for interested stakeholders.

Introduction

Biomedical informatics exists as an academic entity within major medical institutions across the globe. Within the Unites States, every Clinical and Translational Science Awards (CTSA) and Cancer Center has a biomedical informatics core, functioning to expand informatics research and development as well as to offer informatics services.1 Public funding sources (e.g., NIH, NSF) and other financial stakeholders mandate cores report accomplished work in addition to publishing manuscripts and sharing developed work (e.g., software, algorithms). Furthermore, the informatics core in our institution has been required to develop sustainable cost models - billing investigators for services provided. Thus, while the strength of biomedical informatics lies in the applications, data, and knowledge developed, implemented, and shared within the field, from our experience, we have moved from small research shops to central cores for clinical research beholden to numerous parties for support and insight. Hence, informatics operations are also business operations.

A successful business is facilitated by both transparent actions and data-driven decisions.2 Transparency permits progress, allowing for greater accountability as well as performance evaluation.3 The earlier transparency is available, the greater the chance of overall success as corrections and modifications – if identified – can occur when there has been less investment and less is at stake.4 Data-driven decisions provide insight into the business operations. They are decisions based on both quantitative and qualitative data, whose usefulness relies on the quality of data and the effectiveness of interpretation. Harnessing business intelligence (BI) processes – the tools and techniques to move data to meaningful information and knowledge, such as data mining and complex analysis – can facilitate the reporting, monitoring and evolution of business operations and foster transparency and data-driven decisions,5 irrespective of whether the business is considered commercial, non-profit, or even academic.

To address these needs, one useful BI process is to create a dashboard displaying key performance indicators (KPIs). KPIs are “items of information collected at regular intervals to track the performance of a system”;6 when well-written, these allow for insight into business operations deemed relevant for success. A dashboard design typically has up to three purposes – strategic, analytical, and operational7 - to improve organizational understanding, transparency, and decision making. For more information, V. Smith offers a thorough overview of using dashboards for evaluation and dashboard creation,7 while Few8 has written a comprehensive and approachable book on the subject. The technical definition of a dashboard is a data display with a one-screen view that is updated daily, at the latest.8 Hence, what we are creating is really an Information Monitoring System. For ease of conversation within our institution, the term ‘dashboards’ has been accepted.

While KPIs and dashboards have existed since as early as the 1980s,9 and health business analytics has been explored,10 the use of these in biomedical informatics academic operations has not been described. As biomedical informatics has matured, coupled with the changing national funding reporting requirements, we posit that the implementation of these business practices is needed to more quickly assess the operational state of informatics cores. This paper describes solutions implemented within the Center for Biomedical Informatics (CBMI) at Washington University School of Medicine (WUSM) in incorporating BI within our workflow – specifically, determining pain points in our operations, managing operational information (e.g., billing and tasks), visualizing data through the use of dashboards, and incorporating data-driven decisions.

Methods

The CBMI was established in 2002 as a Core within the Washington University in St. Louis (WUSTL) Siteman Cancer Center (SCC) then expanded in 2007 through the WUSTL CTSA. The CBMI is now a multi-departmental initiative with faculty and staff supporting informatics in 73% (n=16) of the departments, and extends its academic partnership with other regional universities (e.g., Saint Louis University, University of Missouri) as well as our collaborating hospital, BJC Healthcare. The CBMI supports the development, adoption, and maintenance of many applications including: i) biospecimen data management system – OpenSpecimen11 (formerly caTissue12); ii) clinical data management system – ClinPortal13; and, iii) clinical research data warehouse – CIDER14. Moreover, the faculty and staff assist researchers with individual projects and informatics needs.

The growth of funding sources and funder reporting requirements along with the expansion of CBMI staff drove the need for improved monitoring and reporting. To improve the CBMI workflow, accountability, and transparency, we first accepted that we are now an academic business, and then moved forward assessing our operational needs. The requirements gathering phase stretched for over two years, when pain points were identified from collaborative input from the CBMI staff at project manager and lab meetings. CBMI staff and faculty collaboratively developed KPIs to illuminate core objectives, specifically pertaining to the inputs, outputs, and outcomes. This was an iterative process, prompting revisions of our data collection methods to more accurately measure the KPIs. Additionally, the authors reviewed previous reports submitted to grant funders and extracted the metrics reported in each of them. Next, we sought a method for monitoring and reporting the KPIs.

After the requirements were identified and supported by key stakeholders, the authors performed analysis of internal technical landscape, which included several software tools for project management and documentation, but lacked a comprehensive reporting aspect. External technical landscape review revealed that many of the pre-packaged BI tools available were either costly or limited in functionality, specifically they were restrictive from the point of view of design (charts and layout), data input formats and/or data processing on the back end. We needed an affordable, web-based product that could be hosted in-house and would allow for cleaning and consolidating data on the back end. We chose to build our dashboards as a web application built in R15 language and hosted on a Shiny Server16, which satisfied all the above requirements. This also allowed for sophisticated, interactive graphics while programming in a familiar language.

R is a programming language and environment providing a variety of data manipulation, statistical and graphical techniques, and is available as Free Software under GNU General Public License in source code form. Shiny17 is a framework for R, designed to turn code into interactive web applications, which can then be hosted online on a Shiny Server16. R software can be extended via packages; specifically rCharts18 package was used to design charts in the CBMI dashboards.

The development phase of a prototype was followed by thorough validation and user acceptance testing among CBMI employees, as well as a review by the CBMI Steering Committee and stakeholders within the WUSTL CTSA and SCC. Once the final dashboards were developed, CBMI gathered feedback from users on usability, usefulness and effectiveness of the dashboards as a reporting and operations monitoring tool.

Results

The identified operational pain-points hindering us from focusing on informatics included: documentation organization, project tracking, logging hours and billing, tracking services, and product monitoring and reporting (Table 1). In the previous two years, use of Atlassian19 JIRA and Confluence tools had been implemented to track software development and organize documentation. We also extended JIRA to include tracking of individual service projects; this had the benefit of linking product software bugs to the specific project tasks they blocked while improving project management. However, it was clear we lacked a facile method for viewing operations on a regular basis, thus spurring the development of dashboards.

Table 1.

Operational pain points in our biomedical informatics center.

Pain Points	Scenarios & Solutions
Documentation Organization	Documentation was stored on a mixture of a local Wiki, SharePoint, shared drive folders, and individual hard drives.Solution: Wiki - Atlassian Confluence19
Project Tracking	Project plans were tracked by project managers using a tool of their choice (e.g., Microsoft Project). Communications with researchers were typically recorded in e-mails.Solution: Ticketing System - Atlassian JIRA19
Hours Worked and Billing	Hours worked were tracked by each individual and reported to the administrator, who maintained a spreadsheet of billing and hours.Solution: Customized Screen and Report in Ticketing System - Atlassian JIRA19
Services Provided	Specific services and hours were logged in a custom developed tool (ServiceTracker). These were entered quarterly, thus only giving a snapshot of the past.Solution: Dashboards - Custom using R15, Shiny16,17, rCharts18
Product Monitoring & Reporting	Some queries were generated on a monthly basis for some products; however, a comprehensive picture of product use only came with unstandardized reports.Solution: Dashboards - Custom using R15, Shiny16,17, rCharts18

We quickly realized some project management solutions were not standardized across all of our products (i.e., tracking service project utilization of ClinPortal, OpenSpecimen, or CIDER) to allow greater flexibility for each project, but leading to discrepancies in how work was tracked. Thus, as we began identifying KPIs, we had to reevaluate and standardize data collection methods that would affect the validity of our KPI measurement data. This standardization also facilitated the implementation of a more efficient and automated process for billing, as a single report could be developed to handle this across our various software projects.

The KPI development process was rooted in the main goal of CBMI, which is to improve health through informatics. This goal is achieved when researchers get the help and the data they need to make biomedical discoveries and translate them into applications to successfully improve human health. Three key KPI domains surfaced as the essence of clinical informatics operations: service, products, and outcomes. ‘Service’ describes the operations of the people within the CBMI. For example, we need to track the researchers with whom we work and the hours CBMI members log. ‘Products’ elucidates the use of the primary CBMI software – in our case OpenSpecimen, ClinPortal, and CIDER. ‘Outcomes’ are the academic metrics deemed important - manuscripts, citations, and grants generated - as well as data and product downloads and adoption. Among the KPIs developed for CBMI were such performance metrics as billable hours logged by product and employee (service) and the number of research projects using our software (products) - all on a monthly basis. However, some KPIs that were identified presented difficulties in data collection, for instance the number of publications (or citations, or grants) supported by services of CBMI (outcomes).

The resulting dashboards (see Figure 1) allow for updated KPI monitoring of services, products, and outcomes through an interactive, web-based display giving an overview of CBMI in three ‘pages’ - main, services, and products. Some of the features include date range selection, data download, zooming, and category selection. The Main page presents a quick snapshot of count metrics related to services, products, and outcomes - such as the number of principal investigators (PIs) engaged with the CBMI during a user-provided time period. The Services page gives both analytic and operational views of the work completed by team members, including the hours logged grouped by PI and software application. The Products page illuminates the usage and historic trends of the software, for example: new CIDER projects created over time.

Figure 1.

A dashboard graphic: Clinportal usage by departments

The metrics are currently reviewed at the bi-weekly Project Management meetings, bi-monthly steering committee meetings, and as needed. There are tutorials for documentation and project tracking from Atlassian19, while sample dashboards and the code are freely available online (https://cbmiwustl.shinyapps.io/demo-dashboard/ and https://github.com/CBMIWU/demo-dashboard). For more detailed information on our processes or to request a full list of the KPIs, please contact one of the authors.

Discussion

Acceptance of the informatics operation as a business venture and the adoption of business intelligence strategies can allow for the analysis and improvement of informatics services. Previously, we used data solely for progress report preparation, rather than ongoing analysis of operations. While predictable and important, such reports are generally not prepared in advance, thus they increased the burden of deliverable work required within the busy informatics shop. Moreover, the reports only captured snapshots of the distant past, not ongoing trends and activities; and, identifying and acting upon events and trends was hampered by the lack of readily available and up-to-date information.

Implementing a project tracking system and improved wiki, elucidating KPIs, and developing dashboards has offered three primary improvements: transparency, data-driven decisions, and early correction opportunities. Access to these web-based tools are available to every CBMI member, and the dashboards are readily accessible to key stakeholders (i.e., CTSA and SCC directors and administrators), thus allowing for greater operational transparency among stakeholders. Decisions in supporting investigators and potential change of products are now more quickly made using the metrics available. In the limited time the dashboards have been available, we have been able to correct the way we enter and track the hours worked – thus saving us needing to correct these errors in three months when the next report is due. Although these improvements are difficult to quantify, we plan to monitor dashboard use by various parties and obtain ongoing feedback via periodic surveys to make further refinements as needed.

Benefits

An unanticipated benefit of our implementation process – programming our own dashboards in R – allowed us to focus on the KPIs that would truly be useful for improved decisions and operational insight. Implementing tools (particularly pre-packaged software) is quick and easy. However, many times it is difficult or impossible to differentiate between goals and means when adopting software because the software itself and the process for using the software are conflated. Because we used a software15 familiar to many of us and used sample working code, we knew any problems we encountered were due to either our thought process or lack of understanding of our KPIs, not the software. This focused our attention toward the information needed rather than distracting us with pre-defined features of a commercial product, in addition to the financial savings associated with open-source solutions.

Challenges

Challenges will exist for the implementation of any project tracking method or BI process. Navigating the Confluence wiki and identifying specific tasks in JIRA relies upon keywords entered, thus documentation may be difficult to find if the person who entered and the person searching for the information use different terms for the same concept. Additionally, adopting JIRA for tracking service projects and billing, while feasible and useful in certain aspects (e.g., association with software development activities), did not provide as much flexibility as found in readily available database software. Although the KPIs were developed within the CBMI based on approved metrics previously reported to external funders and WUSTL stakeholders, it is possible that we did not capture the information needed by all funding agencies or other unidentified stakeholders. Intentionally, the dashboards represent an overview perspective of CBMI operations, not a comprehensive picture; thus, there are metrics not presented. However, if important metrics are identified, they can be added at any time if quantifiable. For example, we would like to include metrics for the important outcome KPI of number of grants or publications resulting from projects utilizing our software, but have not yet identified a reliable source for these data. Finally, the data presented in the dashboards are only as accurate as the data collected. Our dashboard design requires weekly updates of the data rather than using an automated data-feed, and thus requires ongoing support and sufficient understanding of the data sources. Thus, we have carefully documented the R code and steps for deploying and updating the dashboards, as well as the data sources used for various metrics. The weekly update schedule also limits the use of the dashboards if the need for “real-time” metrics is required; to address this problem, we plan to implement automated data updates over time as feasible. While visualizing data can quickly identify and allow timely correction of data collection errors, it is possible not to see all errors and correcting some errors is challenging.

Conclusion

Despite the limitations, we have benefited in adopting these tools and practices, as we have been able to add needed business analytic processes to our group. We dedicated time to thinking about the processes before adopting or developing any tools to focus on adding operational value to the core. Now, with information readily available, we can make assessments and alter our course much earlier than before. Thus, this work in addressing the business needs of biomedical informatics has moved us from a core offering services to researchers, to an academic business, allowing us to organize, track, and visualize information, which provides operational transparency and the opportunity to make data-driven decisions.