Protocol for a systematic review and qualitative synthesis of information quality frameworks in eHealth.

INTRODUCTION: Electronic health (eHealth) applications have become a very large repository of health information which informs critical decisions relating to the diagnosis, treatment and prognosis of patients. Poor information quality (IQ) within eHealth may compromise patient safety. Evaluation of IQ in eHealth is therefore necessary to promote patient safety. An IQ framework specifies what aspects of information to assess and how to conduct the assessment. This systematic review aims to identify dimensions within existing IQ frameworks in eHealth and develop a new IQ framework for the assessment of eHealth. METHOD AND ANALYSIS: We will search Embase, Medline, PubMed, Cumulative Index to Nursing and Allied Health Literature, Maternity and Infant Care, PsycINFO (American Psychological Association), Global Health, Scopus, ProQuest Dissertations and Theses Global, Health Management Information Consortium and reference lists of relevant publications for articles published in English until November 2018. Studies will be selected by two independent reviewers based on prespecified eligibility criteria. Two reviewers will independently extract data in each eligible study using a prepiloted Microsoft Excel data extraction form. Thematic synthesis will be employed to define IQ dimensions and develop a new IQ framework for eHealth. ETHICS AND DISSEMINATION: Ethical approval is not required for this systematic review as primary data will not be collected. The result of the review will be disseminated through publication in an academic journal and scientific conferences. PROSPERO REGISTRATION NUMBER: CRD42018097142. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study will contribute an evidence-based framework for assessing information quality (IQ) in electronic health.

The protocol is based on Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols guidelines.

We used a theoretical framework to develop the search strategy.

The review will not provide specific information on the level of relevance of each IQ dimension included in the new IQ framework.

Introduction

Electronic health (eHealth), defined as the use of information and communication technology (ICT) in healthcare, is regarded as a modern driver of universal health coverage and quality healthcare delivery.1 A range of eHealth applications including telemedicine, electronic health records (EHRs), clinical decision support systems (CDSS), mobile health (mHealth) applications, computerised physician order entry (CPOE), electronic prescribing systems (EPS) and web-based health services (WHS), have all recorded varying levels of success in promoting access to quality health services.2 3 Over time, eHealth applications have become a very large repository of health information which informs critical decisions relating to the diagnosis, treatment and prognosis of patients.1 4 Against the backdrop of its increasing adoption, there are concerns that poor information quality (IQ) in eHealth may compromise patient safety.5 6 A number of patient safety problems associated with eHealth have been reported in the UK and the USA.7 8 These problems are classified as human factors, which are predominantly data entry errors; and technical factors, which are majorly IQ issues such as incorrect, partial and/or delayed information output.7 8 For example, incomplete information by CPOE led to medication overdose and subsequent acute renal failure in a patient.7 Evaluation of IQ in eHealth is therefore necessary to promote patient safety. Although human errors contribute to patient safety problems associated with eHealth, these factors could be addressed through clinical governance and other interventions which are beyond the scope of this review.

IQ describes the extent to which information is fit for purpose.9 Each dimension of IQ describes an aspect of information.10 11 For example, completeness is the extent to which data are sufficient for the task at hand, and timeliness is the extent to which up-to-date data are available when needed.12 An IQ framework is a systematic integration of IQ dimensions for the purpose of evaluating a specific information system.9 11 An IQ framework traditionally specifies what aspects of information to assess and how to conduct the assessment.10 11 An IQ framework also depicts the relationship existing among IQ dimensions by categorising them.12 However, some frameworks only conceptualise IQ without providing guidance on its assessment.9 11 For example, one IQ framework for EHR conceptualises IQ using 11 dimensions.12 The IQ framework depicts the relationship among ‘privacy’, ‘confidentiality’ and ‘secure access’ dimensions by grouping them in ‘security’ category.12 The dimensions and categories in the IQ framework are presented on table 1.

Table 1

Dimensions and categories of an existing information quality (IQ) framework for electronic health record

IQ category	IQ dimensions
Information	Accuracy
	Completeness
	Consistency
	Relevance
	Timeliness
	Usability
Communication	Provenance
	Interpretability
Security	Privacy
	Confidentiality
	Secure access

A number of IQ frameworks have been developed to evaluate different types of health information systems.13 However, IQ frameworks for newer types of eHealth, such as the mHealth apps, are virtually non-existent14 15 and there is no generic IQ framework for eHealth which is applicable across different eHealth applications. Also, there is no consensus on IQ dimensions that are relevant to eHealth and their definition. It is therefore necessary to synthesise the definition of the IQ dimensions within existing frameworks. Identification and definition of IQ dimensions are the first critical steps towards developing an IQ framework.16 Thus, this systematic review aims to identify and define dimensions within existing IQ frameworks in eHealth. In addition, the review will develop a new IQ framework for eHealth using the dimensions synthesised from the existing IQ frameworks for eHealth applications.

Methods and analysis

The protocol is based on the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) Protocols checklist17 presented as online supplementary file 1. The review team are healthcare and ICT professionals with research, teaching and clinical experience in eHealth. AM, JTA, KPF and JG are medical practitioners with hands-on experience in the use of eHealth applications in clinical practice. SO has a multidisciplinary background in nursing and information system. PAW is a nutritional and chronic disease epidemiologist with expertise in digital health technologies. JO has an expertise in ICT and IQ. CC is an expert in health technology assessment, systematic review and evidence synthesis. AM, JG, JO, PAW and SO also have a vast research and teaching experience in eHealth.

What IQ frameworks currently exist for evaluating eHealth applications?

How are dimensions within these existing IQ frameworks defined by the authors?

Which IQ dimensions indicate how well information in eHealth is fit for diagnostic, therapeutic or prognostic purposes?

How are these IQ dimensions in eHealth related to one another?

Eligibility criteria

The traditional systematic review approach based on Population Intervention Comparator and Outcome (PICO) is not fully applicable to this study as we aim to synthesise frameworks rather than interventions. The eligibility criteria are therefore based on the Behaviour/phenomenon of interest, Health context and Model/ Theory (BeHEMoTh) procedure, which is an approach recommended specifically for identifying frameworks, theories and models in systematic review.18 19 The inclusion and exclusion criteria are presented in table 2.

Table 2

Inclusion and exclusion criteria

Concept	Inclusion	Exclusion
Phenomenon of interest	Information or data quality	Information or data quality of administrative and non-clinical data
Health context	Use of eHealth for clinical purposes (ie, diagnostic, therapeutic or prognostic).	Online search for health-related information, e-learning, eHealth applications for self-management.
Model/theory	Multidimensional framework	Individual dimension assessment
Language	English	Non-English
Publication status	Published and grey literature	None
Date of publication	Any	None
Type of study	Any	None

eHealth, electronic health.

We will only include IQ frameworks for assessing eHealth applications used for clinical purposes. We will exclude IQ framework of eHealth applications that manage only non-clinical or administrative data, which are less likely to directly affect patient safety. We will also exclude IQ frameworks that assess online health-related information and e-learning because they are not directly used in clinical management of the patient at the point of care. We will exclude self-management applications, used by patients for health education and disease tracking purposes, as their IQ requirements are probably different compared with the applications used by healthcare professionals for clinical purposes. In addition, we will include multidimensional frameworks, but not individual dimension assessment. IQ is a multidimensional concept and individual dimension assessment cannot provide information about existing relationship between IQ dimensions. Both published and grey literatures will be included. There will be no restriction based on the date of publication. Thus, all relevant studies until November 2018 will be included. There will be no restriction based on study type as there is no evidence that one study type is superior to another when developing a framework. In addition, restriction based on study type may lead to exclusion of potentially relevant IQ frameworks.

Information sources

We will search Embase, Medline, PubMed, Cumulative Index to Nursing and Allied Health Literature, Maternity and Infant Care, PsycINFO and Global Health which are bibliographic databases for healthcare. We will search Scopus to identify eHealth publications in non-healthcare disciplines such as engineering and computer science. In addition, we will search Health Management Information Consortium and ProQuest Dissertations and Theses Global that are considered as good sources of grey literature.20 21 Finally, we will manually search the references of included studies and track their citations to identify other eligible studies using Scopus and Google Scholar.

Search strategy

The search terms will be based on three key concepts, information quality (behaviour or phenomenon of interest), eHealth (health context) and framework (models or theories). Search terms relating to each of these concepts will be developed based on the literature and thesauruses. A librarian will be consulted for inputs in the search strategy. Both Medical Subject Headings and free-text terms will be searched. Truncation and adjacency searching will be used to increase the sensitivity of the search as appropriate. The initial search strategy is available from https://www.crd.york.ac.uk/PROSPEROFILES/97142_STRATEGY_20180521.pdf

Data management

The search results will be imported into the Endnote reference management software (https://endnote.com) which will be used to delete duplicates. Duplicates not identified by the Endnotes will be manually removed. The study selection will be done with Covidence (https://www.covidence.org), a review-management software programme which is in partnership with Cochrane collaboration.

Study selection

Titles and abstracts of the studies will be screened for eligibility by two independent reviewers (KPF and JA) using the criteria outlined in table 2. Conflicts will be resolved by discussion between the two reviewers, and, if needed, by adjudication of a third independent reviewer (JO, SO or PAW). The full-text of all studies selected during screening will be reviewed independently by two reviewers (KPF and JA) with disagreement resolved as earlier described. A PRISMA flow chart will be used to show the details of the selection process.22

Data extraction

Two reviewers (KPF and SO) will independently extract data in each eligible study using a prepiloted Microsoft Excel data extraction form. Other reviewers (JO, CC, PAW, JG and AM) will review the extracted data to ensure accuracy and completeness of the data. Study details that will be extracted will include: author(s), year of publication, country, affiliation, study aim, study design and publication status. We will also extract data on the IQ framework and these will include: method of development; method of validation (if any); type of eHealth technology (eg, telemedicine, CDSS, WHS, EHR and EPS); IQ dimensions and their verbatim definition; categories of IQ dimensions (if any) and metrics of IQ dimension measurement (if any). The main data elements are further defined below:

IQ frameworks for eHealth applications: a systematic integration of IQ dimensions with the purpose of evaluating health information technologies used in the diagnosis, treatment and prognosis of patient.

IQ dimensions within the frameworks in eHealth: these are the evaluation criteria within the IQ frameworks that specify the extent to which health information technologies are fit for clinical use.

Definition of IQ dimensions in eHealth: a clear description of what aspect of information each dimension assesses.

Categories of dimensions within IQ frameworks in eHealth (if provided): IQ dimensions are often categorised to depict relationship among IQ dimensions in an IQ framework.

Metrics of measurement of IQ dimensions in eHealth (if provided): How each IQ dimension is measured, for example, questionnaire, mathematical formulae, and so on.

Quality assessment

We will assess the quality of the included studies using the appropriate Critical Appraisal Skills Programme checklist based on study design.23 Studies will not be excluded based on quality assessment outcome as this is unlikely to have any major impact on the ultimate definition of the dimensions and the construction of the IQ framework. However, the assessment is intended to provide a general idea about the quality of the existing IQ frameworks and the strength of evidence.24

Data synthesis

The IQ framework for eHealth will be developed using a thematic synthesis approach which comprises three stages.25

In the first stage, codes will be generated from the verbatim definition of IQ dimensions extracted from the existing frameworks. This will involve identification of unique concepts from each definition of IQ dimension.

In the second stage, the codes will be grouped into categories based on observed similarities and differences, and a descriptive theme will be created to capture the meaning of each category. These descriptive themes will be the IQ dimensions for the proposed framework. Each of the IQ dimensions will be defined based on the meaning of the original codes from which they were developed.

In the final stage, we will generate the analytical themes from the descriptive themes. Analytical themes are interpretation of the descriptive themes which usually go beyond the findings of the original studies. The analytical themes will be inferred from the descriptive themes (IQ dimensions) based on the interrelationship observed from the definition of the dimensions. This stage will involve organisation of the IQ dimensions (descriptive themes) into different categories conceptualised by the reviewers based on their understanding of the definition of the dimensions. Thus, the analytical themes will be the IQ categories in the new framework. All the reviewers will initially generate the analytical themes independently and then collectively as a group so as to minimise bias.25

Thus, the new IQ framework for eHealth will be derived from the thematic synthesis of the verbatim definition of IQ dimensions. The study details and other extracted framework-related information will provide an understanding of the context of the new IQ framework.

Patient and public involvement

Patients and the public will not be involved directly in the design and conduct of the review. However, the development of the review questions was informed by patient safety concerns and the experience of health professionals using eHealth applications in clinical practice.

Ethics and dissemination

Ethical approval is not required for this systematic review because primary data will not be collected. This systematic review protocol is registered in the International Prospective Register of Systematic Reviews (http://www.crd.york.ac.uk/PROSPERO).26 The result of the review will be disseminated through publication in an academic journal and scientific conferences.

Discussion

This systematic review aims to identify and define IQ dimensions as well as construct a new IQ framework for eHealth. This newly developed framework will specify aspects of eHealth information that should be assessed to determine if such information is fit for diagnostic, therapeutic or prognostic purposes.

This review is the first attempt to develop an evidence-based IQ framework using a systematic review approach, to the best of our knowledge. The use of a theoretical framework to develop the search strategy may also be considered as a strength of the review. However, the generation of analytical themes from descriptive themes in thematic synthesis has been described as controversial because it is influenced by the insight and judgement of the reviewers,25 but we believe that the multidisciplinary perspectives and vast experience of the reviewers will rather add values to data synthesis in this study. A limitation of this review is that the new IQ framework will be unable to provide specific information on the level of relevance of each IQ dimension. We are planning a subsequent international online Delphi study to address this limitation.

Finally, it is expected that the adoption of a transparent and rigorous systematic review approach methodology in this study will result in an evidence-based IQ framework for eHealth. Assessment of eHealth using the evidence-based IQ framework could identify poor IQ issues and potentially forestall associated patient safety problems.7 8