Development and validation of hospital information system-generated indicators of the appropriateness of oral anticoagulant prescriptions in hospitalised adults: the PACHA study protocol.

INTRODUCTION: The appropriateness of oral anticoagulant prescriptions is a major challenge to improve quality and safety of care. As indicators of the appropriateness of oral anticoagulant prescriptions are lacking, the aim of the study is to develop and validate a panel of such indicators, in hospitalised adults, from the hospital information system of two university hospitals in France. METHODS AND ANALYSIS: The study will be carried out in four steps: (1) a literature review to identify indicators of the appropriateness of oral anticoagulant prescriptions and their conditions of appropriateness; (2) a Delphi consensus method to assess the potential utility and operational implementation of the selected indicators; (3) techniques of medical data search to implement indicators from the hospital information system and; (4) a cross-sectional study to assess the ability of indicators to detect inappropriate oral anticoagulant prescriptions, performance of medical data search techniques for tracking or retrieving information and the ability of tools to be transferred into other institutions. The fourth step will include up to 80 patient hospital stays for each indicator, depending on the prevalence of inappropriate prescriptions estimated in interim analyses. ETHICS AND DISSEMINATION: This work addresses the current lack of quality indicators of the appropriateness of oral anticoagulant prescriptions. We aim to develop and validate such indicators for integrating them into hospital clinical practice, as part of a structured approach to improve quality and safety of care. As each hospital information system is different, we will propose tools transferable to other healthcare institutions to allow an automated construction of these indicators. The PACHA study protocol was approved by institutional review boards and ethics committees (CPP Sud-Ouest et Outre Mer III-DC 2016/119; CPP Ile-de-France II-CDW_2016_0014). REGISTRATION DETAILS: Clinical Trial.gov registration: NCT02898090. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

The study will address the current lack of indicators of the appropriateness of oral anticoagulant prescriptions, based on a comprehensive literature review and consensus of European experts.

The study will present a comprehensive approach to develop and validate indicators of the appropriateness of oral anticoagulants which can be used as a model in the future for the development and validation of quality and safety indicators automated from hospital information systems.

The study will propose transferable tools for the automatisation of quality indicators from the hospital information system of any healthcare institution.

The study will perform a semi-automated retrospective retrieval of eligible hospital stays for sampling design due to operational difficulty in implementing a large manually prospective retrieval within care units.

The study will be performed in an exploratory approach and will be limited to the assessment of about 15 indicators due to the time-consuming manually extraction of data within patient records for the reference test.

Introduction

Appropriateness of care, defined as the adequacy of any care to patient needs in accordance with practice guidelines, is a key dimension of quality of care and a major challenge for patient safety and efficiency of healthcare systems.1 Inappropriate care includes misuse, underuse and overuse.2–4 Chassin et al  reported that underuse frequency varies between 40% and 60% of all provided care,2 while Morgan et al reported that overuse frequency varies between 10% and 30%.3 Improving appropriateness of care is a priority in North America5 and Europe, including France,6 especially for its potential impact on quality and cost of care.

Drug use holds a special place within appropriateness issues. The impact of drug adverse events is important; in France, they could be implicated in 20% of emergency room visits and in 3%–5% of hospitalisations,7–9 while 25% of patients admitted to intensive care units would suffer from at least one organ failure related to drug adverse event.10 Assessment of the appropriateness of drug prescriptions must ensure that patients receive the appropriate drug, for the appropriate indication, at an appropriate dosage, with an appropriate treatment adaptation depending on clinical and biological monitoring. Up to 30%–50% of serious adverse events related to drugs could be avoided11 12 and prevalence of drug adverse events concerns 10%–20% of hospitalised patients; more than 1 in 10 patients would present such an event during hospitalisation.13–15 The French national study of adverse events related to drugs (ENEIS 2) estimated the incidence of severe drug adverse events to 0.7 per thousand days of hospitalisation in 2009.9 Furthermore, international studies reported that 20%–60% of drug prescriptions could be inappropriate.16–18 Thus, the frequency, severity and impact of adverse events could be reduced by more appropriate prescriptions both in outpatient and hospitalised patients.

Oral anticoagulants: a priority drug class

Oral anticoagulants (vitamin K antagonists (VKA) and direct oral anticoagulants (DOA)) are among the drugs most frequently involved in adverse events.19 Epidemiological studies report that bleeding events related to VKA are among the first drug adverse events.7 11 12 In France, VKA could be involved in 17 000 hospitalisations per year, of which more than half could be avoided.20 VKA are especially difficult to use due to a narrow therapeutic index and the need for regular laboratory monitoring based on the International Normalised Ratio.20

The development of indicators of oral anticoagulation prescriptions is especially justified by the recent arrival of DOAs, which share the same bleeding risk than VKA. Furthermore, the appropriateness of oral anticoagulant use and its economic impact is a major public health issue due to a large targeted population and frequent use.21

Identified inappropriate practices of oral anticoagulant prescriptions at hospital may explain some bleeding or thrombotic complications9 22–24; such adverse events could be reduced by improving prescriptions and monitoring practices.25 Thus, the large targeted population, the high frequency of prescriptions and the strong risk of adverse events justify the importance of developing tools for improving the appropriateness of oral anticoagulant prescriptions in hospitalised patients.

Importance of validated indicators of the appropriateness of oral anticoagulants

Despite clear guidelines for clinical practice of oral anticoagulant prescriptions,20 21 26–32 validated indicators measuring the appropriateness of oral anticoagulant prescriptions are lacking. Such indicators could provide a way to regularly monitor oral anticoagulant prescriptions for validated clinical indications (atrial fibrillation, valvular heart disease and prosthetic heart valve, venous thromboembolic disease) or clinical situations in patients under oral anticoagulants (trauma or planned surgery). Published studies mainly focused on the development of indicators on atrial fibrillation,33 especially on the indication of warfarin therapy, but did not assess their accuracy.34–39 The availability of indicators able to detect inappropriate oral anticoagulant prescriptions would improve the effectiveness and safety of prescriptions.

Feedback of indicators of the appropriateness of oral anticoagulant prescriptions to health professionals could strengthen tools for self-assessment and quality improvement at hospital. As there is a need to provide regular feedback of indicators to health professionals to improve practices,40 it implies that such indicators may be automated from the hospital information system.

Challenges

The challenge is to be able to provide useful, implementable and valid final tools in any healthcare institution in France or elsewhere; this supposes the ability of tools to be transferred for developing indicators from other hospital information systems.

Definition, selection and validation of indicators

Because many guidelines refer to oral anticoagulant prescriptions for a wide range of clinical situations,41–45 there might be a large panel of indicators potentially measuring the appropriateness of oral anticoagulants. The construction of a dashboard of indicators requires defining these indicators, selecting and prioritising them according to their utility and interest for operational implementation, implementing them from the hospital information system and assessing their ability to detect inappropriate oral anticoagulant prescriptions. This assessment is justified by potential simplifications of the definition of indicators that might occur during their implementation, especially due to availability of the required data.

Construction of indicators from a data warehouse and ability of tools to be transferred

Information needed to establish a dashboard of indicators measuring the appropriateness of oral anticoagulant prescriptions is generally dispersed in many commercial applications (eg, biology results, imagery results, drug prescriptions or medical diagnoses coded with the International Classification of Diseases) within the hospital information system. These applications are not always interfaced, and the information system structure might differ between healthcare institutions. Moreover, data collected are often heterogeneous (structured or not, coded with international, national or only local terminologies) and redundant. Thus, there is a need to calculate the indicators from a data warehouse, which is a database used to collect, arrange and store information previously collected within the hospital information system through different commercial applications.

We hypothesise that the integration of data from the hospital information systems of two French university hospitals into an i2b2 (Informatics for Integrating Biology and the Bedside) data warehouse will allow homogenising and structuring information contained in patient records. This should guarantee better performance of indicators to detect inappropriate oral anticoagulant prescriptions. The i2b2 is an open source data warehouse including a set of modules developed by Harvard Medical School to provide independent investigators with tools necessary to collect and manage project-related medical research data.46 As this interoperable software framework is used worldwide by a large scientific community and many hospitals, the implementation of this data warehouse will allow secondary use of data for all indicators, including those of other studies, without having to carry again the integration work of each of data sources.

Aims of the PACHA study

The main objective of the PACHA study (indicateurs de ‘Pertinence des prescriptions d’AntiCoagulants oraux à l’Hôpital Automatisés’) is to develop and validate indicators of the appropriateness of oral anticoagulant prescriptions in hospitalised adults from the hospital information systems of two French university hospitals (Bordeaux University and Georges Pompidou European hospitals). As such, we aim to define a panel of useful, implementable, valid, reliable and robust indicators to alert on the existence of inappropriate prescriptions. These indicators would cover the main clinical situations of oral anticoagulant prescriptions and could be used by any health professional interested in strengthening tools for self-assessment and quality improvement at hospital.

Methods and analysis

The PACHA study protocol was approved by institutional review boards and ethics committees (CPP Sud-Ouest et Outre Mer III—DC 2016/119; CPP Ile-de-France II—CDW_2016_0014) and registered in Clinical Trial (registration number: NCT02898090). The steering and scientific committees of PACHA include epidemiologists, hospital information system specialists, pharmacologists, pharmacists, cardiologists, neurologists, geriatricians, emergency physicians, anaesthesiologists, specialists in biology or haematology, and statisticians.

Structure of the study

The study will be carried out in four steps (Figure 1): (1) identification of possible indicators measuring the appropriateness of oral anticoagulant prescriptions and their conditions of appropriateness; (2) selection by experts of a panel of indicators judged both useful and implementable; (3) construction of indicators from the hospital information system; (4) assessment of the ability of selected indicators to detect inappropriate oral anticoagulant prescriptions, of the performance of medical data search techniques for tracking or retrieving information needed for the construction of indicators, and of the ability of tools to be transferred to other institutions. The four steps will be carried out, in parallel, at Bordeaux University Hospital (Bordeaux, France) and Georges Pompidou European Hospital (Paris, France), while Rennes University Hospital (Rennes, France) will only be involved in the fourth step of the study (table 1). These three investigating centres are university hospitals combining multidisciplinary hospital, academic and research activities; in 2015, tens of thousands of patients were hospitalised in each hospital.

Figure 1

Flow chart of the PACHA study.

Table 1

Description of the main methodological characteristics of the PACHA study in four steps

Step	Study design and objective	Study samples	Measures, analysis and outcomes	Expected results
1	Systematic literature review to identify indicators of the appropriateness of oral anticoagulant prescriptions	Guidelines on: (1) clinical situations for which a prescription of oral anticoagulants is indicated or contraindicated; (2) the appropriate use of oral anticoagulants for validated clinical indications	Critical appraisal of guidelines; definition of a panel of indicators identified from guidelines whose quality was judged sufficiently good	Identification of a panel of indicators covering misuse, underuse and overuse of oral anticoagulants
2	Delphi consensus method to assess the potential utility and operational implementation of the indicators	Experts with clinical expertise about the targeted clinical situations and prescriptions of oral anticoagulants	Rating of the potential utility and operational implementation (in terms of frequency and severity); selection of indicators for which at least 8 out of 10 experts judged them both useful and implementable	Validation of the final list of indicators and their order of implementation from the hospital information system
3	Prospective phase during which techniques of medical data search will be used to implement the selected indicators from the hospital information system	Samples of hospital stays of patients cared at Bordeaux University or Georges Pompidou European hospitals in 2015 for targeted clinical situations or having had a prescription of oral anticoagulants (defined as type 1 samples; one sample per indicator)	Extraction and integration of all the concepts of interest from the hospital information system into the i2b2 data warehouse and construction of indicators from this i2b2 data warehouse	Values of indicators
4	Cross-sectional study to assess the ability of indicators to detect inappropriate prescriptions of oral anticoagulants	Samples of patient hospital stays (type 1 samples as described in step 3) for primary and secondary analyses; three additional samples of patient hospital stays (defined as samples of types 2, 3 and 4; see the 'Methods and analysis' section for definitions) for secondary robustness analyses	Primary statistical parameter for each indicator: prevalence of inappropriate prescriptions of oral anticoagulants estimated from the data warehouse in comparison to the reference (information manually extracted by clinical research technicians in the patient hospital stay); secondary statistical parameters for each indicator: accuracy, reliability and robustness	Validation of indicators taking into account their ability to detect inappropriate prescriptions in relation to criteria of validation that will have been defined a priori
	Cross-sectional study to assess performance of medical data search techniques for tracking or retrieving information needed for the construction of indicators	Samples of patient hospital stays (type 1 samples as described in step 3) for the analysis of performance of tools tracking information for the construction of indicator; supplementary sample of patient hospital stays (defined as type 5 sample; see the ‘Methods and analysis’ section for definition) for assessing the ability of the data warehouse to identify targeted clinical situations or oral anticoagulant prescriptions covered by indicators	Primary analysis for assessing the performance of tools tracking information for the construction of indicators: recall, precision and F-measure; secondary analysis for assessing the performance of tools tracking information for the construction of indicators: qualitative analysis; secondary analysis for assessing ability of the data warehouse to identify targeted clinical situations or oral anticoagulant prescriptions: qualitative analysis	Provision with useful information on performance of medical data search techniques using or not a data warehouse or Natural Language Processing tools
	Cross-sectional study to assess the ability of other healthcare institutions to implement the same indicators with another data warehouse than i2b2	Sample of hospital stays of patients cared at Rennes University Hospital in 2015 for targeted clinical situations or having had a prescription of oral anticoagulants, whose data will have been included in the eHOP data warehouse	Extraction of data required for the construction of indicators from another data warehouse than i2b2: from the eHOP data warehouse at the Rennes University Hospital; primary statistical parameter: proportion of indicators that will be successfully automated from the eHOP data warehouse	Validation of the ability of tools to be transferred to other institutions in relation to an expected proportion of indicators sucessfully automated that will have been defined a priori

Step 1: Identification of indicators of the appropriateness of oral anticoagulant prescriptions

Study design

A systematic literature review will be performed to identify indicators of the appropriateness of oral anticoagulant prescriptions in hospitalised adults and their conditions of appropriateness. These indicators will thus be designed to identify inappropriate prescriptions of oral anticoagulants in hospitalised patients.

Literature search

The literature review will identify guidelines, from European high authorities for health and European learned societies, focusing on clinical situations for which a prescription of oral anticoagulants (VKA or DOAs) is indicated or contraindicated, and focusing on the appropriate use of oral anticoagulants for validated clinical indications in terms of dosage, duration of treatment, pretherapeutic assessment, compliance with contraindications and monitoring during treatment. The targeted clinical situations will be: (i) frequent cardiovascular situations with thromboembolic risk that could justify a prescription of oral anticoagulants: atrial fibrillation (estimated prevalence between 1.5% and 2.0% of the general population),41 valvular heart disease and prosthetic heart valve (prevalence of valvular heart disease estimated at 2.5% of the general population, of whom a quarter of them benefit from a prosthesis)45 47 48 and venous thromboembolic disease (estimated annual incidence between 100 and 200 per 100 000 persons in the general population),43 for which misuse, underuse or overuse of oral anticoagulant as well as inappropriate implementation of guidelines might occur22–24 49–52; (ii) trauma or planned surgery in patients under oral anticoagulants42 53 for which an increased haemorrhagic risk due to inappropriate adaptations of oral anticoagulants might occur.

Measures and analysis

We will perform a critical appraisal of all selected guidelines. We then will define a panel of indicators of the appropriateness of oral anticoagulants. For each indicator, we will specify its name, objective, construction (numerator and denominator for proportions), domain of application (inclusion and exclusion criteria) and methods for data collection (data sources and sampling). Appropriateness of care will be defined for each indicator in terms of drug dosage, duration of treatment, pretherapeutic assessment, compliance with contraindications and monitoring during treatment.

Expected results

The identified indicators should cover misuse, underuse and overuse of oral anticoagulants. Indicators of misuse will refer to patients presenting a clinical situation for which an oral anticoagulant is indicated and has been prescribed, but for which prescribed modalities or choice of the anticoagulant do not respect: (1) strict contraindications or contraindicated associations; (2) dosage, rhythm or duration of oral anticoagulant treatment; (3) conditions for pretherapeutic assessment and monitoring of oral anticoagulant; (4) adaptations required during treatment, especially in case of trauma or planned surgery requiring to stop temporarily or permanently the oral anticoagulant, a change in dosage, rhythm or duration of prescription, or a relay towards injectable anticoagulants. Indicators of underuse will refer to patients with a clinical situation for which no oral anticoagulant has been prescribed, despite an indication for such a prescription and the absence of contraindication for oral anticoagulants. Indicators of overuse will refer to patients who have been prescribed an oral anticoagulant despite the absence of a clinical situation justifying such a prescription, or when the benefits of the treatment do not exceed the risks.

Step 2: Selection of useful and implementable indicators

In accordance with guidelines,54 a Delphi consensus method54–56 in two rounds will be conducted among experts to assess the potential utility and operational implementation of the indicators among those previously identified during the first step of the study.

Expert selection

We will select experts with clinical expertise about the targeted clinical situations and prescriptions of oral anticoagulants, while not being members of the study steering nor scientific committees. These experts will be specialists in cardiology, vascular medicine, neurology, geriatrics, emergency medicine, anaesthesia and intensive care, pharmacology or haematology, practising in France, Belgium or Switzerland. Experts meeting these criteria will be selected by members of the study scientific committee from their professional networks and lists of authors who have participated in the development of European guidelines focusing on prevention or treatment of the targeted clinical situations or focusing on appropriate use of oral anticoagulants for validated clinical indications.

Experts will be asked to judge the potential utility (defined as the indicator ability to detect inappropriate oral anticoagulant prescriptions and to implement improvement actions) and operational implementation in terms of frequency and severity (defined as the indicator ability to detect inappropriate oral anticoagulant prescriptions that are frequent or severe enough to justify changing practices). Experts will be asked to rate utility and operational implementation of the proposed indicators on a qualitative binary scale (yes or no). Experts will also be asked to give their opinion on the way the proposed indicators are defined and formulated, especially for some definitions or criteria that are not consensual in the literature. Each expert will also have the opportunity to propose up to three new clearly defined indicators, and to specify from which guidelines they have been identified.

At each round, an indicator will be selected if at least 8 out of 10 experts considered the indicator both useful and implementable (strong consensus for selection). An indicator will be deleted if at least 8 out of 10 experts considered the indicator both not useful and not implementable (strong consensus for deletion).  All expert ratings will be reported in an anonymised database.

At the end of the two rounds of the Delphi process, a final synthesis meeting will be organised by the steering committee to validate the final list of selected indicators, especially by considering the indicators for which no consensus will have been obtained. We will also establish the order in which the final panel of indicators will be implemented from the hospital information system and assessed for their ability to detect inappropriate oral anticoagulant prescriptions.

Step 3: Construction of the selected indicators from the hospital information system

During this prospective phase, techniques and tools of medical data search will be applied to implement the selected indicators from the hospital information system.

Study population

The study population will include hospital stays of patients aged 18 years and over, cared at Bordeaux University or Georges Pompidou European hospitals for targeted clinical situations (atrial fibrillation, valvular heart disease and prosthetic heart valve, venous thromboembolic disease, trauma or planned surgery in patients under oral anticoagulants) or having had a prescription of oral anticoagulants during their hospital stay in medicine, surgery, emergency or postemergency unit, from 1 January 2015 to 31 December 2015, with medico-administrative data available in the hospital information system (electronic patient record). Hospital stays of patients who would have denied the computer processing of their electronic record will not be included in the study.

Study samples

For each indicator, the medical information units of Bordeaux University and Georges Pompidou European hospitals will carry out a semi-automated retrospective retrieval of eligible patient hospital stays. This retrospective retrieval will be carried out from the hospital information system: (i) from queries listing patient hospital stays concerned by the targeted clinical situations within electronic patient records (codes defined according to the International Classification of Diseases 10th version); (ii) from queries listing patient hospital stays during which a VKA or DOA has been prescribed within the drug prescription commercial application (codes defined according to the Anatomical, Therapeutic and Chemical Classification of Drugs, ATC) or (iii) from these two types of queries.

A first series of samples of patient hospital stays (type 1 samples) will be constituted for constructing each of the selected indicators (one type 1 sample per indicator). These type 1 samples might be stratified, if deemed necessary, on up to three out of five possible stratification variables (type of hospital stay; type of prescribed oral anticoagulant; age and sex of patients; type of trauma) depending on the indicator. Indicators will be considered independently of each other.

Three additional samples of patient hospital stays (samples of types 2, 3 and 4) will be constituted to investigate hospital stays which could have been missed by queries within the hospital information system during the retrospective retrieval (one sample of types 2, 3 and 4 per indicator). These samples might also be similarly stratified, if deemed necessary, up to three stratification variables.

For indicators whose lists of patient hospital stays will be identified directly from queries, we will proceed in sampling of the required number of patient hospital stays in each investigating centre. For indicators whose sampling will require investigation of the randomly selected patient hospital stays, clinical research technicians will investigate patient files to ensure eligibility; eligible stays will be included in the corresponding sample according to the above-mentioned procedure. The procedure will be performed following the order of prioritisation for the construction of indicators (as defined at the end of the study step 2) and will be continued until the required number of patient hospital stays is obtained for each sample of each indicator; a real-time monitoring of inclusions will be performed.

Measures and analysis: construction and calculation of indicators from the data warehouse

Information needed for the construction of the selected indicators from the hospital information system will be integrated into the specific i2b2 data warehouse46 in both the Bordeaux University and Georges Pompidou European hospitals. The indicators of the appropriateness of oral anticoagulants prescriptions will be constructed from this data warehouse, which will allow answering quickly to queries that may involve millions of patient records, while constructing indicators in other healthcare institutions using the same data warehouse. The data warehouse will both contain structured data (eg, medical diagnoses, acts, drug prescriptions, biological prescriptions and their results, prescriptions of radiology) and unstructured data (data only available in a textual format).

Concepts of interest that will need to be available in the data warehouse for the construction and calculation of each indicator will be defined during the first and second steps of the study. We will need to identify, at least, clinical data, symptoms or diagnoses, drugs, laboratory tests, imaging tests or procedures for indirectly identifying a diagnosis or an adverse effect (Figure 2). Concepts of interest related to structured data will be automatically extracted from the data warehouse. Other techniques of medical data search will be used to identify concepts of interest related to unstructured data from the data warehouse, especially Natural Language Processing (NLP)57 58; NLP techniques will be needed for identifying concepts of interest in text areas (eg, targeted clinical situations, drug treatments, treatment monitoring, comorbidities or adverse events). To improve the ability of the NLP techniques to retrieve the concepts of interest, we will use a morpho-syntax analyser, which will allow identifying, in free-text boxes, the syntactic role of each word (namely verb, adjective or name). We will develop a version of the module of NLP that uses the morpho-syntax analyser and another that will directly process raw text. Analysis of each concept will be reinforced by a judgement of its contextual relevance, taking into account negation (whether it is a concept really present or described as absent) and temporality (whether the concept is present for a patient at the time t of investigation). Both Bordeaux University and Georges Pompidou European hospitals will extract data needed for the construction of indicators from an i2b2 data warehouse, especially by developing and using an NLP module that will be integrated into the tools associated with this data warehouse.

Figure 2

Concepts of interest needed to be available in the data warehouse for the construction of indicators.

All indicators will be calculated from patient hospital stays of type 1 samples. Patient hospital stays which would have been missed by queries during the retrospective retrieval within sample types 2, 3 and 4 will be only taken into account for calculating indicators in the context of a robustness analysis presented in the fourth step of the study. A limited number of priority indicators will be commonly constructed at both hospitals; half of the other indicators will be constructed in Bordeaux University Hospital and the other half will be constructed in Georges Pompidou European Hospital.

The expected results will be values of indicators.

A) Assessment of the ability of indicators to detect inappropriate prescriptions

The study design will be a cross-sectional study.

Study population and samples

The study population will be the same as those previously described for the construction of the selected indicators (step 3 of the study). All samples previously described will be used (samples of types 1, 2, 3 and 4).

Measures and analysis: data collection for analysis of accuracy, reliability and robustness of indicators

Data for analysis of accuracy, reliability and robustness of indicators will be collected at both hospitals for common indicators. The measure of accuracy, reliability and robustness will be conducted according to methodological standards.56 59–62

Data collection for the accuracy analysis

Data collection and analysis of accuracy will be carried out independently for each indicator. For each patient of a given type 1 sample, the value of the indicator will be calculated: (i) from data related to patient hospital stays that will be manually extracted by the clinical research technicians (one for each investigating centre) within electronic patient records; (ii) from data related to patient hospital stays that will be automatically extracted from the data warehouse. Information manually extracted by the clinical research technicians will be collected on a specific electronic case report form. The prevalence of inappropriate oral anticoagulant prescriptions could thus be estimated in real-time from data collected by the clinical research technicians as well as from data automated from the data warehouse.

The reference defining the actual appropriateness of prescriptions will be defined from information manually extracted by the clinical research technicians in patient records, and then, secondly, by the judgement of clinical experts based on information extracted by the clinical research technicians, patient records and clinical expertise. In each investigating centre, a same group of three experts will be responsible for judging the appropriateness of prescriptions for each patient hospital stay of the sample constituted for a given indicator. In case of disagreement between the three experts, the opinion of the majority of them will be considered.

Data collection for the reliability analysis

For each indicator, patient records will be randomly drawn from subpopulations concerned by sources of coding variations. The results of the appropriateness measure will then be compared between two different time periods of coding or between two units with different coding practices.

Data collection for the robustness analysis

For each indicator, patient records will be randomly selected and results of the appropriateness measure will be compared by varying their conditions of appropriateness (as defined at the end of the first step) or by using or not techniques of NLP (without using NLP; with tools of NLP integrating the morpho-syntax analyser and; with tools of NLP directly processing raw text).

Similarly, for each indicator, results of the appropriateness measure will be compared taking into account or not patient hospital stays that would have been missed by the hospital information system among the samples of types 2, 3 and 4 investigated by the clinical research technicians. For each patient hospital stay of these samples, the indicator value will be produced by the automated system. In parallel, the clinical research technicians will carry out a manual extraction of information needed to measure the appropriateness, by a complete review of the patient hospital stays; this information will be collected on a specific electronic case report form.

Statistical parameters

The primary statistical parameter will focus on the prevalence of inappropriate prescriptions of oral anticoagulants estimated, for each indicator, from data warehouse in comparison to the prevalence estimated from the reference (information manually extracted by the clinical research technicians in the patient hospital stay), and the estimation of its 95% CI.

The secondary statistical parameters will focus on:

Accuracy of each indicator (sensitivity, specificity and predictive values) to detect inappropriate oral anticoagulant prescriptions from data related to patient hospital stays that will be automatically extracted from the data warehouse compared with the reference.

Reliability of each indicator (Kappa coefficients for unordered qualitative indicators, weighted Kappa coefficients for ordered qualitative indicators and intraclass coefficients for quantitative indicators) by analysing their results’ variations depending on variations of information coding in the hospital information system.

Robustness of each indicator (Kappa coefficients for unordered qualitative indicators, weighted Kappa coefficients for ordered qualitative indicators and intraclass coefficients for quantitative indicators) by analysing their results’ variations depending on variations of their conditions of appropriateness, or the use or not of techniques of NLP, or by taking into account or not patient hospital stays that would have been missed within the hospital information system among the samples of types 2, 3 and 4 investigated by the clinical research technicians.

Sample size

For the accuracy analysis, the required sample size of 80 patient hospital stays has been calculated to estimate a prevalence of 5%, with a 95% CI of 1.4% to 12.3% according to the exact binomial distribution. In the perspective of warning indicators, we considered a prevalence of 5% to guarantee a minimum precision for low prevalence of inappropriate prescriptions that are severe enough to justify changing practices. Furthermore, by considering an exploratory approach, an interim analysis will be carried out at the end of investigation of the first 40 patient hospital stays to optimise the balance between the expected precision and required resources for data collection. If the observed number of patient hospital stays with inappropriate prescription is ≥10 (corresponding to prevalence ≥25%), data collection will be stopped for this indicator since the increase in the number of patient hospital stays would not significantly increase precision of estimates in comparison to required resources. On the contrary, if the observed number of patient hospital stays with inappropriate prescription is <10 (corresponding to prevalence strictly <25%), data collection will be continued until 80 patient hospital stays, as initially planned. This sample size of 40 patient hospital stays has been fixed according to the 95% CI of the exact binomial distribution, which is 12.7% to 41.2% with a prevalence of 25%.

For the robustness analysis, the estimation of the number of patient hospital stays required for samples of types 2, 3 and 4 of each indicator is based on statistical hypotheses derived from the lot quality assurance sampling technique63 64 (null hypothesis (H0): the group of stays is not acceptable if the proportion of patient hospital stays that have been missed by the hospital information system in the sample is greater than or equal to the defined value P0; alternative hypothesis (H1): the group of stays is acceptable if the proportion of patient hospital stays that have been missed by the hospital information system is strictly lower than the defined value P0; N: size of the lot, which corresponds to the population resulting from the medical diagnoses queries for additional patient hospital stays needing to be investigated to detect any missed patient hospital stays; d+1= number of missed patient hospital stays whose observation on the sample would imply to reject the group of stays). Using tables for lot quality assurance sampling technique at 1 df,63 the number of patient hospital stays required is 25 with the following hypotheses: P0=10%, α=5%, d=0 and size of the lot between 100 and 1 00 000 patient hospital stays.

The scientific committee of the study will proceed to a collective validation of indicators, by taking into account their ability to detect inappropriate prescriptions in relation to criteria of validation that will have been defined a priori.

B) Assessment of performance of medical data search techniques for tracking or retrieving information

The study design will be a cross-sectional study.

The study population will be the same as those previously described for the construction of the selected indicators (step 3 of the study).

The study samples will focus on the type 1 samples previously described, as well as on a supplementary sample of 50 randomly selected patient hospital stays (type 5 sample). The latter will be used, for each indicator, to assess the ability of the data warehouse to identify targeted clinical situations or oral anticoagulant prescriptions covered by indicators in comparison to retrieval conducted without the data warehouse from electronic patient records and the drug prescription commercial application.

For the constitution of the type 5 sample, we will identify all patient records identified in the data warehouse for which the concerned clinical situation or prescription of oral anticoagulant is present, on the same period than those during which the type 1 samples will have been selected. Within this group of patient records, we will remove those that will have already been included in type 1 samples; we will then randomly select 50 patient records from these subgroups.

For the assessment of performance of tools tracking information for the construction of indicators from patient hospital stays included in the type 1 samples, we will estimate three statistical parameters during primary analysis: recall (defined as the number of concepts that are correctly returned by the evaluated tool in comparison to the number of concepts returned by the reference), precision (defined as the number of concepts that are correctly returned by the evaluated tool in comparison to the total number of concepts returned by the search engine for a given query) and F-measure (which combines recall and precision).57 This assessment will be performed by comparing information extracted from the data warehouse with information manually extracted by the clinical research technicians from patient hospital stays. It will especially be carried out for the concepts that will have been identified by the NLP module. An analysis of recall, precision and F-measure will be performed according to the level of NLP used during search of concepts: (i) no use of NLP module; (ii) use of NLP module integrating the morpho-syntax analyser; (iii) use of NLP module directly processing the raw text. A secondary qualitative analysis will be performed by another independent clinical research technician who will not have participated to manual extraction of data. It will aim at assessing discordances between information extracted from the data warehouse and information manually extracted by the clinical research technicians in patient records for each indicator.

To assess the ability of the data warehouse to identify targeted clinical situations or oral anticoagulant prescriptions, covered by indicators, from patient hospital stays included in the type 5 samples, we will perform another secondary qualitative analysis. This assessment will aim at analysing discordances between the retrieval of targeted clinical situations and/or oral anticoagulant prescriptions by two compared methods: (1) without using the i2b2 data warehouse, only on the basis of coded medical diagnoses and oral anticoagulant prescriptions from commercial applications and (2) by using the i2b2 data warehouse.

The scientific committee of the study will be provided with useful information on performance of medical data search techniques using or not a data warehouse and NLP module.

C) Assessment of the ability of tools to be transferred to other institutions

The study design will be a cross-sectional study.

Study sample

The study sample will focus on the hospital stays of patients aged 18 years and over, cared at Rennes University Hospital, between 1 January 2015 and 31 December 2015, for targeted clinical situations (atrial fibrillation, valvular heart disease and prosthetic heart valve, venous thromboembolic disease, trauma or planned surgery in patients under oral anticoagulants) or having had a prescription of oral anticoagulants during their hospital stay in medicine, surgery, emergency or postemergency unit, whose medico-administrative data will have been included in the eHOP data warehouse. Hospital stays of patients who would have denied the computer processing of their electronic record will not be included in this analysis.

Rennes University Hospital will extract the structured and unstructured data needed for the construction of these indicators from their eHOP data warehouse.65 Unstructured data will be extracted by using the NLP module that would have been especially implemented in a local web-service for improving the ability of tools to be transferred, with or without a morpho-syntax analyser. For this assessment, the primary statistical parameter will focus on the proportion of indicators that will be successfully automated.

The validation of the ability of tools to be transferred will focus on an expected proportion of indicators successfully automated that will have been defined a priori.

Ethics and dissemination

This study will improve the quality and safety of care by addressing the current lack of indicators measuring the appropriateness of oral anticoagulant prescriptions, in France and Europe, based on guidelines for clinical practices in validated clinical indications. Such improvements will exceed the scope of healthcare institutions, since a high number of oral anticoagulant prescriptions initiated in such institutions are pursued in outpatients.

We will propose a panel of useful, implementable, valid, reliable and robust indicators of the appropriateness of oral anticoagulant prescriptions that will be automated from the hospital information system and generalisable to other healthcare institutions in France and Europe. These indicators will be gathered in dashboards whose impact will then be assessed based on clinical and medico-economic criteria, which will improve the ability of healthcare institutions answering to current institutional requirements in quality of care.

Validated indicators will be regularly conveyed to prescribing health professionals by individual, electronic, visual graphic feedbacks,66 as part of a continuous process of professional practices improvement. Such feedbacks will help the familiarisation of health professionals with quality indicators by integrating them into the heart of a dialogue on practices. Thus, they will be important tools for updating their knowledge about the appropriateness of oral anticoagulant prescriptions.

In the future, the validated indicators would potentially be assessed for their sensitivity to change56 through the implementation of interventions to improve the appropriateness of oral anticoagulant prescriptions. We can also consider collaborative work to generalise these indicators and develop continuity of care between healthcare institutions and ambulatory medicine.

At last, this study will propose transferable tools to other French or European healthcare institutions to allow an automatisation of indicators of the appropriateness of oral anticoagulants as well as future other quality and safety indicators.

Conclusion

This study addresses the current lack of indicators of the appropriateness of oral anticoagulant prescriptions, based on guidelines and validated clinical indications. We aim at integrating them into hospital clinical practice, as part of a structured approach to improve quality and safety of care.

By selecting useful and implementable indicators from judgement of European experts and by providing transferable tools for their automatisation from the hospital information system of other healthcare institutions, we will strengthen their potential of generalisation in French and European healthcare institutions to improve quality and safety of care. Such ability of tools to be transferred between different contexts and countries is important to improve the effectiveness of quality of care strategies based on the use of indicators.67 Furthermore, this study will provide a comprehensive model for the development and validation of other indicators of the appropriateness of care which may be automated from hospital information systems.