Database quality assessment in research in paramedicine: a scoping review protocol.

INTRODUCTION: The paramedic practice environment presents unique challenges to data documentation and access, as well as linkage to other parts of the healthcare system. Variable or unknown data quality can influence the validity of research in paramedicine. A number of database quality assessment (DQA) frameworks have been developed and used to evaluate data quality in other areas of healthcare. The extent these or other DQA practices have been applied to paramedic research is not known. Accordingly, this scoping review aims to describe the range, extent and nature of DQA practices within research in paramedicine. METHODS AND ANALYSIS: This scoping review will follow established methods for the conduct (Johanna Briggs Institute; Arksey and O'Malley) and reporting (Preferred Reporting Items in Systematic Reviews and Meta-Analyses extension for scoping reviews) of scoping reviews. In consultation with a professional librarian, a search strategy was developed representing the applicable population, concept and context. This strategy will be applied to MEDLINE (National Library of Medicine), Embase (Elsevier), Scopus (Elsevier) and CINAHL (EBSCO) to identify studies published from 2011 through 2021 that assess paramedic data quality as a stated goal. Studies will be included if they report quantitative results of DQA using data that relate primarily to the paramedic practice environment. Protocols, commentaries, case studies, interviews, simulations and experimental data-processing techniques will be excluded. No restrictions will be placed on language. Study selection will be performed by two reviewers, with a third available to resolve conflicts. Data will be extracted from included studies using a data-charting form piloted and iteratively revised based on studies known to be relevant. Results will be summarised in a chart of study characteristics, DQA-specific outcomes and key findings. ETHICS AND DISSEMINATION: Ethical approval is not required. Results will be submitted to relevant conferences and peer-reviewed journals. TRIAL REGISTRATION: 10.17605/OSF.IO/Z287T. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study will examine the range, extent and nature of database quality assessment practices in research in paramedicine.

The search strategy will capture a wide selection of potentially eligible studies, ensuring that the landscape of paramedic database quality assessment is comprehensively described, and unique considerations of the paramedic practice environment not overlooked.

A piloted data-charting form will structure extracted data according to identified parameters that allow comparisons among included studies and with database quality assessment frameworks from other areas.

Database quality assessment practices in use might not be fully represented in published literature, biasing the results.

Introduction

Paramedicine is a growing and evolving discipline that has been variously described as emergency medical services, prehospital care or emergency response. (Terms in common use to describe paramedicine inadequately characterise the range of care currently practiced internationally. We have chosen to use ‘paramedicine’ as a general term that includes traditional notions of ‘emergency’ and ‘prehospital care’, while accepting that emerging practice models frequently address non-urgent reports in the community, avoid transport to hospital and integrate with other allied health professions. We acknowledge international variety in the meaning of ‘paramedic’, and use it to include emergency medical technicians, responders and similar roles.) As paramedicine enters its fifth decade as a distinct area of practice, numerous studies have cited the need for more research capacity to support the unique subject matter.1–4 Although researchers are more frequently designing and conducting studies specifically about paramedicine in the paramedic practice environment,5 research in the field faces challenges in accessing high-quality administrative data, particularly those that can link to patient outcomes in related databases.1 2 6 This remains an ongoing issue at three basic levels.

First, data collection in paramedicine poses several unique challenges. These challenges begin with the nature of the work: often fast-paced and time-critical, paramedic care places simultaneous physical and cognitive demands on each provider’s attention and time.7–9 Additionally, the care environment can be unpredictable and disorganised, if not chaotic or unsafe, with frequent distractions and time pressures on scene.7–10 The main source of patient information—the patient—is also sometimes unconscious or uncommunicative for various reasons, all of which delay real-time documentation, with attendant potential loss of data or accuracy.9 11 Data input relies on individual care providers, not trained recorders in a dedicated role, which may result in questionable inter-rater and even intra-rater reliability.9 11

Second, not all data related to paramedic care are easily accessible. The vast majority of paramedic data are contained in the record of patient contact, known most often as the patient care report. Traditionally paper-based, the patient care report began transitioning to electronic platforms in urban areas with established systems in the mid-2000s and early 2010s.12 13 The process of adoption has been described as variable, non-linear and characterised by ongoing upgrades, revisions or changes instead of a single event.12 It is not uncommon for adjacent geographical areas to be served by providers with mixed reporting platforms. It has also faced challenges in terms of funding and maintaining technical expertise.13 Evaluations have noted the potential benefits of collecting large amounts of standardised data in electronic form, but that these have been inconsistently realised at the level of individual services.12 13

Third, where they do exist, electronic records have inherent limitations that apply to research in paramedicine. Healthcare in general has recognised the potential of electronic health records to support a wide variety of research, quality improvement, public health and administrative purposes.14–16 At the same time, there is also widespread acknowledgement of the limitations and pitfalls of conducting research with data collected for clinical use.14 15 17–19 These limitations fall into several categories, including: the gap between the reason for data collection and its research use; variations in clinical practice, documentation standards and data entry; and inconsistent use of electronic records within and among jurisdictions.14 15 18 19 As an additional challenge, records of paramedic care are typically based on the event that occurred, not individual patients. Connecting paramedic care to patient outcomes therefore requires linkage based on data collected during the clinical encounter, and linkage success has been shown to vary widely and be subject to potential bias.20 Acknowledging both the potential benefits and limitations of research based on electronic data, studies argue for clear and consistent ways of describing, evaluating and sharing information about the data quality of electronic health records.15

Data quality practices are no less important in paramedic research than in general electronic health records, particularly considering the unique difficulties of data collection and the relatively recent integration of electronic record keeping. The continuing growth of paramedic research will depend on measures to improve, standardise and communicate confidence in the source material. In some areas of paramedic practice, this process has begun with standards for and position statements on data capture and reporting.21 22 Related healthcare fields, however, have developed numerous database quality assessment (DQA) frameworks that provide a conceptual structure as well as a technical map to assessing the quality of databases as a whole and the suitability of particular data for any specific use. No comparable DQA frameworks have been developed to address the unique circumstances of paramedic care, and the applicability of existing ones to the paramedic practice environment remains to be determined.

An overview of DQA frameworks

In the most general terms, data quality is defined as ‘the extent that the data fulfil users’ expectations and suit its intended purposes,’ or more simply, fitness for use (Mashoufi et al, p20).11 DQA frameworks commonly use a series of thematic domains to subdivide various components of data quality. Several reviews have noted that different frameworks use similar terminology, but frequently with slightly different meanings that reflect a particular setting or purpose. As an extreme example of the variety of terminology within the field, a review of DQA practices in public-health information databases counted 49 different terms used to describe various DQA attributes (analogous to domains) among the studies.23 Within these, completeness, accuracy and timeliness were evaluated most often, with the number of attributes assessed ranging from one to eight in any individual paper.

Individual studies showcase the variety of domains and assessment methods specifically related to a wide range of healthcare settings. These settings include a provincial-level administrative repository,24 emergency nursing,25 a framework synthesis of national-level clinical research networks26 and a model proposed for the Canadian primary-care environment.16

Potentially relevant to paramedic research, a recent review summarised DQA practices in emergency medicine.11 These authors proposed five domains applicable to the field (accuracy, completeness, timeliness, accessibility and consistency), but did not address how the small number of included prehospital studies differed from or resembled their in-hospital counterparts.11 Rather than targeting a specific healthcare setting, the Canadian Institute of Health Information (CIHI), a national-level data repository, uses a DQA framework that can be applied to a broad range of healthcare systems. It includes the following domains: relevance, accuracy and reliability, comparability and coherence, timeliness and punctuality and accessibility and clarity.27 Both comprehensive and general, this framework appears most adaptable to a range of settings and purposes.

Scoping review rationale

With paramedic research emerging as a distinct field with its own unique characteristics of data collection, future research will require common standards of methodological rigour. In the absence of a paramedic-specific DQA framework, DQA practices in paramedic research remain sporadically reported. Without area-specific guidance and in the context of a literature base that has not been described, a scoping review is an appropriate method to begin to define the boundaries of this topic. Metaphors of mapping are commonly applied to scoping review purpose, and multiple authors employ specific terminology to describe the dimensions of a research landscape, including range, extent and nature.28 29 Accordingly, this scoping review aims to assess the range, extent and nature of DQA practices in paramedic research. Findings from this review will be used to assess whether a unique paramedic DQA framework might be needed or possible, and whether it could be developed using a ‘best fit’ approach to combining a systematic review and qualitative evidence synthesis as described elsewhere.30 31

Methods and analysis

This protocol has been informed by guidance from the Johanna Briggs Institute (JBI) and is presented according to the stages proposed by Arksey and O’Malley.32 33 It has been registered with the Open Science Framework. The review will follow the guidelines of the scoping extension to the Preferred Reporting Items in Systematic Reviews and Meta-Analyses extension for scoping reviews (PRISMA-ScR).34

Stage 1: identifying the research question

Using the framework proposed by the JBI guidance on scoping reviews, parameters of this review are defined by the Population, Context and Concept of related research.32 Here, population identifies paramedic studies related to DQA (including quality improvement). This corresponds with elements of range, which will characterise the paramedic studies based on the location, date of publication and clinical area of paramedic data being assessed. The context situates paramedic DQA studies within their setting and defines the extent of their assessment. Specifically, extent describes the level and breadth of data, where level distinguishes between, for example, data collected at the level of an individual service as compared with a country and breadth reports the number of institutions included or connected at each level. The concept is defined as data quality, and includes DQA, information quality or data accuracy, as distinct from clinical performance or measures of quality of care. The concept is further defined by specific characteristics that describe the nature of the assessment, such as the data fields assessed, methods of assessment, DQA framework (if specified) and applicable assessment domain.

Stage 2: identifying relevant studies

A search will be undertaken to identify research studies that explicitly assess paramedic data quality as a stated goal. Studies will be limited to those that report quantitative results of DQA using data that relate primarily to the paramedic practice environment. These criteria exclude protocols, commentaries, case studies, interviews, simulations and experimental data-processing techniques. They also exclude studies that are not primarily focused on paramedic data or ones that evaluate databases that only incidentally include paramedic information. The paramedic practice environment will be interpreted broadly (encompassing urban, rural, remote and military contexts), but will exclude special circumstances outside of regular practice, such as disaster and mass-casualty situations. No restrictions will be placed on language. If abstracts or articles in languages other than English are identified as potential candidates, arrangements for translation will be attempted on a case-by-case basis.

In consultation with a professional librarian, a provisional search strategy was developed using keywords and subject headings identified in available articles that represent the population, concept and context. It has been iteratively revised with input from pilot assessments of draft versions. Aiming to include a wide selection of possibly relevant research, we initially applied no date filters. Next, we compared searches limited to the most recent 5, 10 and 15 years to balance numbers of citations with comprehensiveness. We choose approximately the last 10 years (rounded to include all of 2011) as a reasonable compromise between including all possibly relevant articles and those that are most recent and likely most applicable, while ensuring a sample of at least 10 000 citations (not counting duplicates). Therefore, studies will be limited to those from 2011 through 2021. This search will be applied to the following databases: MEDLINE (National Library of Medicine), Embase (Elsevier), Scopus (Elsevier) and CINAHL (EBSCO). The search strategy, as applied to these databases, is included as online supplemental file 1. Search results will be imported into a data-management software platform, Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia; available at www.covidence.org).

Stage 3: study selection

After removal of duplicates, title and abstract screening will be performed on a small sample of records (approximately 2.5%) to ensure a consistent application of the inclusion criteria. All remaining titles and abstracts will be screened by two reviewers independently. Any record selected by either reviewer will be included for full-text screening. Next, full-text screening will be performed by two reviewers, with any differences resolved by discussion, with a third reviewer available if necessary.

Stage 4: extracting data

Two reviewers will assess each paper selected for inclusion independently using a custom-designed data-charting form piloted on key articles. This form was developed by the reviewers using consensus on a sample of key articles known among the team and believed to be relevant to the study prior to the search (included as online supplemental file 2). This form includes 12 fields grouped by the three parameters (Range/Population, Extent/Context, Nature/Concept). Fields under the heading of Range include geographical location, year of publication, study purpose and clinical area (if applicable). The level, breadth and duration of data being assessed will be documented under the heading of Extent. Fields that make up the Nature parameter include the specific paramedic data assessed, the methods of assessment, summarised results of assessment and domain of data quality being assessed, both as identified by the study and under the framework proposed by the CIHI (if possible).27 Categorisation under a framework has been included to provide information in cases where a domain was not identified, and to provide a consistent reference point for comparing all included studies. In the absence of any framework directly applicable to paramedic research, it is possible that no existing domains will apply to some identified DQA practices. To minimise this potential bias, the CIHI framework was chosen as being accessible, comprehensive and broadly applicable to a range of topics.

Data-charting meetings will be held at regular intervals to compare results and assess the adequacy of the extraction form. If necessary, modification will be made, and additional data included (or removed). This process will occur iteratively until all records have been assessed by all reviewers with the ability to capture all relevant data.32 Any modifications to the form will be recorded as changes to protocol and reported in the final results. Results from each reviewer using the final form will be compared and reconciled through discussion of all included studies. In accordance with methodological guidance for scoping reviews, critical appraisal will not be conducted on included studies.34

Stage 5: Collating, summarising and reporting the results

Results of the search and screening process will be presented in text and using the PRISMA-ScR flow diagram.34 Included studies will be summarised in a chart of characteristics for which data were charted (PRISMA-ScR item 18), and results will be synthesised in table or narrative format, depending on findings.34

Patient and public involvement

None.

Ethics and dissemination

As a review of publicly available studies, this study does not require ethical approval. The results will be submitted for publication to a peer-reviewed journal and presented to conferences and research gatherings.