Clinical information quality of digital health technologies: protocol for an international eDelphi study.

INTRODUCTION: Digital health technologies (DHTs) such as electronic health records, clinical decision support systems and electronic prescribing systems are widely used in healthcare. While adoption of DHTs can improve healthcare delivery, information quality (IQ) problems associated with DHTs can compromise quality and safety of care. The clinical information quality (CLIQ) framework for digital health is a novel approach to assessing the quality of clinical information from DHTs. This study aims to appraise the CLIQ framework by exploring clinicians' perspectives on the relevance, definition and assessment of IQ dimensions as defined in the framework. This study will adapt the CLIQ framework to the needs of clinical information users-the clinicians. The contextualised CLIQ framework will offer a pragmatic approach to assessing clinical information from DHTs and may help to forestall IQ problems that can compromise quality and safety of care. METHODS AND ANALYSIS: The electronic Delphi (eDelphi) approach will be used to engage a heterogeneous group of clinicians with patient-facing and/or information governance roles recruited through purposive and snowball sampling techniques. A semi-structured online questionnaire will be used to explore clinicians' perspectives on relevance, definition and assessment of IQ dimensions in the CLIQ framework. Survey responses on the relevance of dimensions will be summarised using descriptive statistics to inform decisions on retention of dimensions and termination of the study, based on pre-specified rules. Analysis of the free-text responses will be used to revise definition and assessment of dimensions. ETHICS AND DISSEMINATION: Ethics approval has been obtained from the Imperial College Research Governance and Integrity Team (Imperial College Research Ethics Committee (ICREC) Reference number: 20IC6396). The results of the study will be published in a peer-reviewed journal and presented at scientific conferences. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

A systematic, practical, affordable and transparent eDelphi approach will be used to engage clinicians on information quality (IQ) of digital health technologies (DHTs).

Heterogeneity of the expert panel, with panellists drawn from multiple clinical professions and countries, will enrich the findings and enhance the external validity of the clinical information quality (CLIQ) framework.

This study will simultaneously take advantage of the clinical experience and information governance expertise of clinicians.

Contextualising the CLIQ framework to the needs of the clinicians will result in a pragmatic approach to assessing IQ of DHTs in clinical practice.

Validation based on expert panel approach is limited to face and content validity with further assessment required for appraising the construct validity and applicability of the CLIQ framework in clinical practice.

Introduction

Digital health Technologies (DHTs) such as electronic health records, clinical decision support systems and electronic prescribing systems are widely used in healthcare.1 While widespread adoption of DHTs can improve healthcare delivery, information quality (IQ) problems associated with DHTs can compromise quality and safety of care.2 Patient safety incidents, relating to delayed, missing, partial or wrong information and resulting in patient harm or deaths, have been reported in the literature.3–6 For example, a patient had seizures due to incorrect mapping of different formulations of an epilepsy medication in the electronic prescription system.3

Although the negative impact of poor IQ of DHTs is well documented in the literature, not much is known about how to assess the quality of clinical information from DHTs. A systematic review published in 2021 identified 10 IQ frameworks that are relevant to assessment of clinical information from DHTs.7 Although these frameworks define fundamental dimensions that describe specific aspects of information, none offered a pragmatic approach to assessing information in clinical practice. Drawing on the findings of this systematic review, the clinical information quality (CLIQ) framework (table 1) was developed to provide a pragmatic approach to assessing the quality of clinical information from DHTs. This study aims to appraise the CLIQ framework by exploring clinicians’ perspectives on the relevance, definition and assessment of IQ dimensions as defined in the framework. This will help to contextualise the CLIQ framework to the needs of the information users as recommended in IQ literature.8 9 Clinicians are the end users of clinical information from DHTs.

Table 1

Clinical information quality framework for digital health

Informativeness directly concerns the usefulness of digital information for clinical purposes	Accuracy	The extent to which information is correct
	Completeness	The extent to which no required information is missing
	Interpretability	The extent to which information can be understood
	Plausibility	The extent to which information makes sense based on common knowledge
	Provenance	The extent to which the source of information is trustworthy
	Relevance	The extent to which information is useful for the intended task
Availability concerns the functionality of the system holding clinical information	Accessibility	The extent to which existing information is easily obtainable
	Portability	The extent to which information is accessible in different systems
	Security	The extent to which information is protected from unauthorised access and corruption
	Timeliness	The extent to which current information is available on time
Usability concerns the ease of use of clinical information	Conformance	The extent to which information is presented in the desired format
	Consistency	The extent to which information is presented in the same format
	Maintainability	The extent to which information can be maintained

Table 1 was originally published in an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. Kayode Philip Fadahunsi, Siobhan O'Connor, James Tosin Akinlua, Petra A Wark, Joseph Gallagher, Christopher Carroll, Josip Car, Azeem Majeed, John O'Donoghue. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.05.2021).

Methods and analysis

Study design

This study will use an electronic Delphi (eDelphi) approach, which is a systematic, practical, affordable and transparent method of engaging multiple stakeholders from different locations and integrating their opinions to achieve consensus.10 11 The eDelphi approach promotes equal participation and prevents dominance of the panel by outspoken participants, which often characterises physical committee meetings.12 In addition, the iterative process of the eDelphi method enables participants to reconsider their opinions based on collective responses.11

Steering Committee

This eDelphi study will be coordinated by a steering committee comprising of healthcare professionals and researchers with interest in digital health (KPF, NM, JG, PAW AM, JC). The steering committee developed the CLIQ framework,7 from which the initial items of the eDelphi study will be generated. The committee will be responsible for recruiting the panellists of the eDelphi study. In addition, the committee will make decisions regarding retention, removal or redefinition of IQ dimensions based on the inputs of the panellists according to prespecified decision and stoppage rules.

Generation of initial items

The initial survey for the eDelphi study (online supplemental appendix 1) has been generated from the infographic CLIQ framework7 and the accompanying assessment questionnaire developed based on evidence from literatures. The survey documentation and content cover the following:

Brief information about the study with a link to the participant information leaflet

Request for informed consent

Collection of demographic data of participants to confirm eligibility for the study and for descriptive purposes. This includes occupation.

Likert scale questions on relevance of IQ dimensions and categories.

Multiple choice questions on definition, assessment and categories of IQ dimensions.

Free-text questions on modification of definition, assessment and categories of IQ dimensions.

Collection of email addresses of participants for feedback purposes and as a contact method for the next round of survey.

Thus, the survey questions relating to the CLIQ framework are divided into two parts. The first part will explore the relevance of the dimensions in the CLIQ framework from the perspective of the panellists. The second part will obtain their suggestions on modification to the definitions, assessment and categories of the IQ dimensions in the CLIQ framework. The relevance of the IQ dimensions will be assessed based on the panellists’ perspective on the relevance of the dimensions to quality and safety of care using a five-point Likert scale. This captures different range of options and allows to distinguish between categories that people make naturally, without a strong cognitive load (strongly relevant, somewhat relevant, neither relevant nor irrelevant, somewhat irrelevant and strongly irrelevant).

Decision rules

Although there is no standard criteria for consensus in an eDelphi study, there is a need to predefine what constitutes a consensus to enhance objectivity and reduce analysis bias.11 Most previous Delphi studies use 60% agreement or higher as threshold for consensus.13 In this study, an IQ dimension will be considered relevant and retained in the final framework when at least 70% of the panellists, in any round of the survey, choose the options of strongly relevant or somewhat relevant when rating it. On the contrary, a dimension will be considered irrelevant and removed when at least 70% of the experts, in any round of the survey, choose the options of strongly irrelevant or somewhat irrelevant when rating it. The decision on whether to retain or remove any dimension for which no consensus is reached by the end of the study will be made by the steering committee based on the data from all the rounds.11

Stoppage rule

The eDelphi rounds will be stopped when consensus has been reached as described above on the relevance of at least 80% of all the IQ dimensions. The stoppage rule will be applied from the first round if no new dimensions are suggested by the respondents or from the second round after the respondents may have scored any suggested new dimension. The eDelphi study will be terminated at the end of the third round irrespective of the level of consensus achieved.13 This alternative stoppage rule is necessary to prevent the need to continue the eDelphi rounds if consensus is not achieved within a reasonable time frame,11 which will be regarded as 6–8 months in this study.

Participant recruitment

A heterogeneous group of clinicians will be selected including doctors, nurses, pharmacists and other healthcare professionals with patient-facing and/or information governance roles. Heterogeneity of panellists will allow a wide range of perspectives and enhance external validity of the framework.10 There are no clear guidelines about the sample size of an eDelphi study.13 However, the literature suggests 8–15 participants when the sample is homogeneous with a caveat to avoid extremely large sample sizes because the amount of data could be unmanageable.10 We therefore estimated that 40 participants will be required to accommodate different categories of clinicians (doctors, nurses, pharmacists and others), but increased the sample to 50 to account for 20% drop-out during the eDelphi rounds.13 Thus, we aim to recruit up to 50 participants to accommodate various clinician groups and compensate for drop-out during the eDelphi rounds as well as ensure geographical diversity.

The following eligibility criteria will be used to nominate clinicians that will be invited for the survey:

Prior or current experience with using DHTs in patients’ care.

Information governance role or personal interest in information governance.

Proficiency in English language to understand and complete the surveys.

Willingness to participate in a multiple-round eDelphi study (up to three rounds).

We are particularly interested in clinicians with information governance roles (chief clinical information officer, chief nursing information officer, Caldicott guardian, etc) as they typically have prior or current experience with using DHTs. Thus, the study will benefit simultaneously from their subject matter expertise and practical user experience. However, we did not limit participation to this group of clinicians with information expertise alone as we are aware that these roles do not exist in many countries especially in low-income and middle-income countries. In addition, recruiting clinicians with varying levels of expertise will encourage wide range of opinions.

Study procedures

The survey will be set up using Qualtrics software (Qualtrics, Provo, Utah, USA). The functionality of the survey will be tested by the members of the steering committee prior to its administration. The study will start with purposive nomination of the panellists by the members of the steering committee. Steering committee members will be asked to nominate panellists both within and beyond their professional networks. Nomination of the panellists by the steering committee members will be based on the pre-determined eligibility criteria discussed above, subject to confirmation by another committee member who will check the profile of the nominees against the eligibility criteria. Each of the panellists will be invited by an introductory email containing a brief overview of the study and the link to the survey. The snowball sampling technique will then be used to recruit additional panellists by asking the nominated panellists to share the eDelphi invitation to other eligible participants. Questions about participants’ occupation and prior digital health experience will be included in the survey to further confirm the eligibility of the panellists. Up to two reminders will be sent at least 2 weeks apart to encourage participation by those who did not respond to the initial email.

Only items on which consensus has not been reached and any newly suggested item(s) in the previous round will be included in the next round. The survey will be terminated based on the stoppage rule earlier listed. The first round of the survey started in June 2021. The study is expected to last between 6 and 8 months.

Data analysis plan

Survey responses on the relevance of dimensions will be summarised using descriptive statistics including frequencies, percentages, ranges and medians. The descriptive statistics will be used to provide concise feedback to the participants and to inform decisions on retention of IQ dimensions and termination of the study as already described. The feedback on the statistical summary of group response will be sent in the email inviting participants for the next round of the survey.

The free-text suggestions on the modification of the definition, assessment and categories of IQ dimensions will be analysed based on the reflexive thematic analysis approach.14 This will provide an opportunity to go beyond the texts to decode the intended meaning of the suggested modifications. It is however important to highlight that the purpose of thematic analysis in this study is to provide an in-depth understanding of the contributions of the panellists with the aim of revising the definition of IQ dimensions and the approach of assessment, as appropriate. We have therefore adapted the thematic analysis process to include the following steps:

Familiarisation with the data by reading the free-text contribution of the panellists repeatedly.

Coding of the data to highlight the issues raised with regard to the definition and assessment of CLIQ dimensions.

Development of themes by identifying patterns of the suggested modifications, reflecting on them in the context of the overall dataset and defining the essence of each theme.

The themes will then be considered by the steering committee and used to revise the definition and assessment of dimensions as appropriate. The feedback on the free text suggestions and the changes that have been made will be incorporated into the subsequent round of the survey.

Data management and storage

A data impact assessment and dataset registration were completed through the Imperial College Faculty of Medicine Data Privacy Impact Assessment Tool. This was done to address potential gaps and comply with relevant legal obligations. Data will be stored securely in an access-restricted Imperial College shared drive in accordance with General Data Protection Rules,15 the Data Protection Act (2018) and the Imperial College Data Protection Policy.16 Data will be stored for a minimum of 10 years after the study completion or longer if needed for further reference.

Ethics and dissemination

Ethics approval has been obtained from the Imperial College Research Governance and Integrity Team (Imperial College Research Ethics Committee (ICREC) Reference number: 20IC6396). Detailed information about the study will be presented in a participant information sheet containing information on the study objectives, expectation of the participants, duties of the researchers and relevant contacts (online supplemental appendix 2). Informed consent will be obtained electronically from each participant at the beginning of the online survey and before the eDelphi study questions. Participants may refuse to participate or withdraw from the study without giving any reasons at any point. However, any data collected and analysed prior to participant withdrawal will be retained.

Individual responses of the participants will be pseudonymised before being added to the secure drive. Feedback to each participant will only contain descriptive statistical summaries of the group responses. Each research participant will be assigned a research code, known only to the first author. Personal information, which could be used to directly identify participants such as their email addresses, will be kept confidential and known only to the first author. The results of the eDelphi study will be published in a peer-reviewed journal and presented at scientific conferences. Panellists will only be listed in the publication with their prior consent.

Patient and public involvement

Patients will not be involved directly in the design and conduct of the study as the study is aimed at DHTs used by healthcare professionals in a clinical setting. The members of the steering committee who designed and will oversee the study are mostly clinicians with research interest in digital health and the members of the expert panel will be clinicians with practical experience of using DHTs.

Discussion

This study seeks to appraise the CLIQ framework by exploring clinicians’ perspectives on the definition, relevance and assessment of IQ dimensions in the framework. The initial CLIQ framework defined IQ dimensions that are relevant to assessing DHTs, based on systematic review of literature, without obtaining inputs from information users or specifying how IQ could be assessed.7 However, this study will obtain direct inputs from clinicians, which will ensure that the dimensions in the contextualised CLIQ framework are those considered relevant by clinicians—the users of clinical information from DHTs. Inputs from the clinicians will also ensure that the questionnaire for assessing clinical information from DHTs is written in a clear and concise language that is well-understood by clinicians. The contextualised CLIQ framework from this study will comprise of two related instruments—an infographic framework and an assessment questionnaire. The infographic framework will define IQ dimensions that are relevant to assessing clinical information, thus providing a useful guide to understanding IQ requirements for DHTs. The questionnaire will offer a pragmatic approach to assessing clinical information from DHTs. The questionnaire could be used, for example, to obtain feedback about IQ of named DHTs from clinicians using them in clinical practice.

This study has several strengths and limitations. First, the eDelphi methods offers a systematic, practical, affordable and transparent approach to integrating opinions of clinicians on IQ of DHTs. Heterogeneity of the expert panel, with panellists drawn from multiple clinical professions and countries, will ensure variety of inputs and enhance the external validity of the CLIQ framework. In addition, this study will take advantage of the clinical experience and information governance expertise of participating clinicians thus combining practical user experience and subject matter expertise.

However, we acknowledge that validation based on expert panel approach is limited to face and content validity.17 We are therefore planning a pilot assessment to evaluate the construct validity of the contextualised CLIQ framework and assess its applicability in clinical practice. We acknowledge that the initial nomination of the panellists may lead to selection bias as steering committee members may tend to recruit colleagues they know personally, rather than via their wider professional networks. These colleagues may be more likely to participate than people invited through other sources. We have therefore put in place multiple measures to reduce the risk of selection bias. The snowball sampling technique will ensure that only a fraction of participants will likely be recruited directly by the steering committee members. The eDelphi approach will make it impossible for any of the panellists to dominate the decision-making process. Finally, we will compare the responses of the panellists who were recruited directly and those who were recruited by snowball techniques.

The contextualised CLIQ framework will offer a pragmatic approach to assessing clinical information from DHTs. The framework could be used in quality improvement initiatives relating to DHTs especially in health facilities. Such use may help to identify and forestall IQ problems that can compromise quality and safety of care.