Use of an electronic medication management support system in patients with polypharmacy in general practice: study protocol of a quantitative process evaluation of the AdAM trial.

BACKGROUND: Interventional studies on polypharmacy often fail to significantly improve patient-relevant outcomes, or confine themselves to measuring surrogate parameters. Interventions and settings are complex, with many factors affecting results. The AdAM study's aim is to reduce hospitalization and death by requiring general practitioners (GPs) to use a computerized decision-support system (CDSS). The study will undergo a process evaluation to identify factors for successful implementation and to assess whether the intervention was implemented as intended.
OBJECTIVE: To evaluate our complex intervention, based on the Medical Research Council's guideline dimensions. RESEARCH QUESTIONS: We will assess implementation (reach, fidelity, dose, tailoring) by asking: (1) Who took part in the intervention (proportion of GPs using the CDSS, proportion of patients enrolled in them)? Information on GPs' and patients' characteristics will also be collected. (2) How many and which medication alerts were dealt with? (3) Was the intervention implemented as intended? (4) On what days did GPs use the intervention tool?
METHODS: The process evaluation is part of a stepped-wedge cluster-randomized controlled trial. Characteristics of practices, GPs and patients using the CDSS will be compared with the non-participating population. CDSS log data will be analyzed to evaluate how the number of medication alerts changed between baseline and 2 months later, and to identify the kind of alerts that were dealt with. Comparison of enrolled patients on weekdays versus weekends will shed light on GPs' use of the CDSS in the absence or presence of patients. Outcomes will be presented using descriptive statistics, and significance tests will be used to identify associations between them. We will conduct subgroup analyses, including time effects to account for software improvements. DISCUSSION: This study protocol is the basis for conducting analyses of the quantitative process evaluation. By providing insight into how GPs conduct medication reviews, the evaluation will provide context to the trial results and support their interpretation. The evaluation relies on the proper documentation by GPs, potentially limiting its explanatory power.

Introduction

As a result of improvement in medical treatments for formerly fatal chronic diseases, life expectancy, and the number of people with multiple chronic conditions (multimorbidity), has increased throughout the world. Multimorbidity is associated with polypharmacy, the concurrent use of multiple drugs by a patient. There is no consensus on the definition of polypharmacy, but it is commonly defined as the daily intake of at least five different drugs. Polypharmacy has become more prevalent over the years, with estimates of the number of affected patients ranging from one in five adults to more than half the older population (65+ ). As large numbers of drugs may be required to treat multiple diseases, polypharmacy may often be appropriate. Nonetheless, inappropriate therapy regimens, with unrecognized duplicate prescriptions or drug–drug and drug-disease interactions may lead to therapy failure, deterioration in conditions or effect reinforcement.

In order to manage patients’ medication and avoid inappropriate polypharmacy, many clinical trials, with or without computerized support, have been conducted to examine the effectiveness of medication reviews in recent years. The findings of these trials have been inconsistent and have often failed to generate significant results. In particular, beneficial effects of medication reviews on patient-relevant outcomes could not be proven.[6,7]

Furthermore, as practices are complex systems in themselves, it is not only the interventions that are often complex. This implies that many factors influence the measurable outcomes. By assessing factors that may influence intervention outcomes and determining whether the intervention is implemented in the target population as intended, process evaluations can show how the intervention works. Process evaluations can facilitate the interpretation of results and provide insight into reasons for success or failure. The underlying theoretical framework, which is based on a proposal by Wierenga et al., can be found in Figure 1. This theoretical framework was originally formed by synthesizing relevant frameworks that had previously been used in implementation science.[10-12] We chose it because it includes a wide range of factors that influence both implementation and the implementation process itself. The framework also addresses several implementation levels: a macro level (the socio-political system), a meso level (implementing organizations) and a micro level (physicians/staff and patients). Furthermore, the framework takes into account the influence of the employed implementation strategies. It thus provides a comprehensive basis for the discussion of the results of this process evaluation.

Figure 1.

Theoretical framework of the process evaluation based on Wierenga et al. Maintenance of the results is part of another publication of the AdAM study.

The most important domains of a process evaluation are Reach (who received the intervention and who did not?), Dose (how intense was the intervention?), Fidelity (was the intervention delivered as intended?), and Tailoring (how was the intervention altered to imbed it in a daily routine?).

In the AdAM study ( ‘Anwendung für ein digital gestütztes Arzneimitteltherapie- und Versorgungsmanagement’, or ‘application of digitally supported drug-therapy and care management’), the electronic medication management system ‘eMMa’ is used by general practitioners (GPs). In this paper, we describe the process evaluation of the AdAM study, based on log data from eMMa.

The AdAM study

The primary aim of the AdAM study is to determine whether a yearly medication review supported by eMMa effectively reduces the combined endpoint of all-cause hospital admissions and all-cause mortality in adult patients with five or more chronic prescriptions in primary care. The evaluation of eMMa assesses cost-effectiveness, physicians’ and patients’ perspectives on the intervention, a sustainability assessment, a qualitative comparative analysis of contextual and implementation process factors, and the process evaluation described in this paper.

Design, setting and population

The full details of the trial design and its methodology will be published elsewhere. Briefly, the AdAM study is a stepped-wedge cluster-randomized controlled trial (cRCT) with open cohorts conducted in general practices in the German region of Westphalia-Lippe from 2017 to 2021. General practices are the units of randomization and all practice patients are treated in accordance with the practice’s group assignment. General practices are invited to participate when one or more physicians in the practice is a GP, an internist, or a physician without specialization that provides primary care in accordance with the KVWL (Kassenärztliche Vereinigung Westfalen-Lippe, a regional Association of Statutory Health Insurance Physicians) and that claims data shows to have at least 11 potentially eligible patients. Every month, newly recruited general practices are randomized into either the intervention group or the waiting control group using block randomization of variable block length based on practice IDs. In total, the target number of included practices was 1080. The statutory health insurance company BARMER provides randomized practices in the intervention group with lists of potentially eligible patients based on claims data. As the lists are updated quarterly, the cohort is open. Eligible patients are 18 years or older, have five or more chronic prescriptions (at least five different codes according to the Anatomical Therapeutic Chemical Classification System in two consecutive quarters) and are insured by BARMER. They are invited to participate by general practices in the intervention group and are enrolled in AdAM after giving their written informed consent. They are then provided with the experimental treatment foreseen for the intervention group. BARMER also generates lists of potentially eligible patients at general practices in the waiting control group based on claims data, but do not disclose this information to the practices, which continue to provide usual care. After 15 months in the waiting control group, these practices also receive a list of potentially eligible patients and updated quarterly lists of potentially eligible patients are disclosed to them from then on. Patients on the lists are invited to participate and enrolled in AdAM after giving written informed consent. They then also receive the experimental intervention.

Experimental intervention

Supported by eMMa, GPs conduct medication reviews for patients in the intervention group at least once a year. eMMa is unlocked for general practices in the intervention group, while general practices in the waiting control group have no access to it during the 15-month control period. Practice patients that are potentially eligible for AdAM are identified in eMMa. After a patient is enrolled, the practice receives access to their entire claims data, including diagnoses, prescriptions, and other data on health services utilization such as hospital stays and contacts with other physicians. eMMa provides the opportunity to update information (e.g. on new diagnoses and prescriptions for which claims have not yet been made) and to add specific details that are not included in claims data (e.g. height, weight, laboratory test results on kidney function, over-the-counter drugs, and medication doses). GPs then examine patients’ medication regimens, supported by alerts from eMMa in case of inappropriate prescriptions (e.g. drug-drug and drug-disease interactions, inappropriate dosages, or potentially inappropriate medication because of age).

Alerts are provided by eMMa according to a four-level system denoting severity ( ‘red’ for severe, ‘yellow’ for less critical medication alerts, ‘grey’ for alerts that are probably not clinically relevant, ‘info’ for informative content). The severity rating in eMMa is based on continuous screening of medical publications and the notifications of German and international regulatory authorities provided by physicians and pharmacists. The sources are systematically analyzed in accordance with the structured WHO UMC algorithm for the categorization of the causality of adverse drug reactions. Quality of evidence is based on the GRADE system.

GPs can optimize treatment accordingly or document reasons for not changing a potentially inappropriate medication (PIM), save the changes in eMMa, print a medication plan, and discuss changes with the patient. This optimization process can be carried out with or without the patient present. A detailed description of the process can be found in Figure 2. Although the process evaluation compares outcomes at baseline with those 2 months later, GPs can access eMMa at any time and check the alerts both before T1, and afterwards, when data collection for the process evaluation has been completed. Once per year, physicians receive an annual reimbursement of €85 for each patient treated using eMMa.

Figure 2.

Schematic working process of GPs with eMMa.

Intervention training and support

General practice teams are invited to a 2-hour continuing medical education (CME) session on polypharmacy and the main functions of eMMa on a voluntary basis. CME was provided by consulting pharmacists and IT specialists familiar with eMMa at two central locations (Münster and Dortmund), as well as in decentralized KVWL district offices. In addition, FAQ and training videos are provided on the KVWL website. Support hotlines for questions relating to administration, IT and polypharmacy are accessible for GPs upon request. KVWL contacts participating general practices in case of low numbers of visits to the eMMa site, or low rates of enrolled patients, via fax or by phone.

Outcomes

Apart from the combined primary outcome hospitalization or death from any cause, a number of secondary outcomes will be assessed: (1) indicators of high-risk prescribing and (2) specific-cause hospital admissions preceded by high-risk prescribing (e.g. increased risk of gastrointestinal bleeding due to prescription of oral anticoagulants with either non-steroidal anti-inflammatory drugs (NSAIDs) or platelet aggregation inhibitors without a gastroprotective drug, increased cardiovascular risk due to prescription of oral NSAIDs for either heart failure patients or those receiving angiotensin-converting enzyme inhibitors[18,19]), and (3) process measures, such as number of potentially inappropriate medications and underuse of medication (START criteria).

Methods

Aims and objectives of the process evaluation of the AdAM study

The process evaluation of the AdAM study accompanies an evaluation of the effectiveness of the experimental intervention eMMa, a computerized decision-support system (CDSS). In accordance with the United Kingdom’s Medical Research Council guidelines, our process evaluation will determine whether the intervention was implemented as intended by assessing the following dimensions and questions:

Reach: The question behind this dimension is: Was the recruitment of patients, GPs, and general practices successful? For this purpose, the study population that was enrolled in eMMa is compared with the eligible but non-enrolled population. Figure 3 provides an overview of the definition of the populations under consideration.

Figure 3.

Different populations compared for the reach dimension.

Dose: The overarching question in this dimension deals with how comprehensively the intervention is applied. For this purpose, we examine the number, as well as the type (prioritization), of alerts that were responded to, and those that were not.

Fidelity: Was the intervention implemented in such a way that success was possible? To achieve this, we defined several parameters assessed for the ‘Dose’ domain that we regarded as crucial to the intervention’s success and mandatory if the intervention was to be considered implemented as intended.

Tailoring: How did physicians ensure the intervention fitted in well with their daily routine? We will assess the temporal dimensions (each day of the week as well as a comparison between working days and weekends) of eMMa usage.

The evaluation of these dimensions will provide greater insight into how GPs deal with the intervention and help explain findings in the main study relating to the primary outcome of hospitalization and death. It will also identify potential pitfalls that should be circumvented in future interventions. More precisely, we gathered information on (1) characteristics of participating GPs and patients, that is, information on bias in the GP population, and on the health of included patients, as this may help explain success and failure (reach); (2) how intensely the intervention was applied, with the aim of determining whether the intensity of the application might explain success or failure (dose); (3) whether tasks the research team considered to be crucial to the intervention’s success were actually carried out (fidelity); and (4) how GPs adapted the intervention to facilitate its use on an everyday basis (tailoring).

Data and methods of the process evaluation

Time

Like the trial evaluation, data is collected at two time points, at baseline (T0), and 2 months later (T1). T0 is defined as the moment when the anamnesis has been completed and confirmed by the GP, after which he or she will be able to see the alerts provided by eMMa. If this is not confirmed, surrogate triggers will function as T0, depending on any information entered into eMMa (e.g. adding missing dosage, input of physical parameters). Sensitivity analyses will be conducted for patients with a surrogate T0. The 2-month time interval between T0 and T1 was a compromise: On the one hand, GPs should have enough time to agree with the patient on medication changes, while bearing in mind that such adjustments should be gradual.[21,22] On the other hand, the interval should not be so long that the health status of patients could deteriorate for reasons unconnected with the intervention.

Population

Different study populations are defined for the four dimensions as follows:

• For the Reach dimension, the enrolled study population consisting of patients, GPs, and general practices (=intervention group) are compared with their non-participating counterparts at T0.

• Patients: The population of enrolled patients is compared with the eligible (non-enrolled) BARMER population, as well as with those patients that gave their informed consent for AdAM but have no activity documented in the eMMa software (=inactive population).

• GPs: The enrolled GPs are compared with the eligible GP population in the study region, as well as with those GPs that gave their informed consent for AdAM but have no patient with documented activity in the eMMa software (=inactive population).

• General practices: The enrolled general practices are compared with the eligible general practice population in the study region, as well as with those practices that gave their informed consent for AdAM but without any patient with documented activity in the eMMa software (=inactive population).

• Dose & Fidelity: The enrolled patient population (=intervention group) is defined as the study population and comparisons are made between T0 and T1.

• Tailoring: The enrolled GP population (=intervention group) is defined as the study population at T0.

The main focus here is on group comparisons (e.g. proportions, mean values):

• Patients (Is there a prioritization in patient recruitment?):

• Proportion of the enrolled patient population vs eligible (non-enrolled) BARMER population vs inactive patient population.

• Group differences (enrolled vs non-enrolled vs inactive patient population) in terms of patient characteristics (e.g., age, gender, disease score, economic status).

• We will evaluate whether there is an association between the enrollment rate of patients and disrupting factors. These factors will be seasonal influenza, the COVID-19 pandemic, and technical difficulties that make it temporarily impossible to access the software. A time protocol including software updates and technical problems will be provided by KVWL and BARMER.

• GPs (Was there a selection bias in GP recruitment?):

• Proportion of GPs using eMMa compared to the overall GP population in the study region and the inactive GP population.

• Group differences (enrolled vs overall vs inactive) in terms of GP characteristics (e.g., age, sex, specialization).

• General practice (Did selection bias influence recruitment at the general practice level?):

• Proportion of general practices using eMMa compared to the entirety of practices in the study region and the inactive GP population.

• Group differences (enrolled vs overall vs inactive) in terms of practice characteristics (e.g., employed GPs, number of patient visits)

A detailed overview of the outcome variables is provided in the data collection section.

To assess the intervention dose, the summed differences in alerts per GP (representing the number of patients treated in eMMa) at T0 compared to T1 will be calculated. We will also assess how the number of alerts per patient changes from T0 to T1. Seven different alert categories (see basic alerts in Figure 4) are registered by eMMa, with up to four severity levels for each. This process evaluation only assesses changes in the clinically relevant severity levels 1 (‘red’) and 2 (‘yellow’). Different alert categories will be analyzed separately and aggregated over all alerts. The alerts will be grouped in clinically fitting categories to facilitate interpretation of the results. See Figure 4.

Figure 4.

Flowchart of analyzed alerts.

All analyses will take account of alert severity levels and the way the alert is dealt with by the GPs:

(i) Joint analyses of severity levels 1 and 2 and (ii) only severity level 1 to determine whether severity level 1 alerts were prioritized.

If the GP provides a note to confirm that he or she is aware of the risk but has nonetheless decided not to change the medication, the respective alerts will include the note. These are referred to as ”justified alerts” in Table 1. Alerts that were left unchanged but without a note will be referred to as “unjustified alerts”. A sensitivity analysis will be conducted only considering those alerts at T1 that remain unjustified.

Table 1.

Overview of the analyses.

	Main analysis	Sensitivity analysis
(i) Severity levels 1 and 2	Differenceinallalerts(T_1−T_0)=→unjustifiedalerts(T1)+justifiedalerts(T1)−[unjustifiedalerts(T0)+justifiedalerts(T0)]	Differenceinallalerts(T_1−T_0)=unjustifiedalerts(T1)−[unjustifiedalerts(T0)+justifiedalerts(T0)]
(ii) Severity level 1	Differenceinseverealerts(T_1−T_0)=unjustifiedalerts(T1)+justifiedalerts(T1)−[unjustifiedalerts(T0)+justifiedalerts(T0)]	Differenceinseverealerts(T_1−T_0)=unjustifiedalerts(T1)−[unjustifiedalerts(T0)+justifiedalerts(T0)]

The analysis will be performed for the overall patient population and include exploratory age-sex stratifications.

We will also analyze the percentage of enrolled patients for whom the bodily parameters kidney function, height and weight were entered. Finally, we will record the share of patients that had a printed medication plan and the share for whom a medication change had been documented.

We define ‘Implementation as intended’ as the case when the user reacts to the triggers of all identified severity 1 alerts from the time of enrollment (T0) to 2 months later (T1), or provides notes explaining why he or she did not respond to an alert by changing the medication. A software change during the course of the study made it necessary for GPs to confirm that the patient’s anamnesis had been completed before they saw the alerts. For this reason, we will conduct specific sensitivity analyses of interventions before and after that date, as well as of the GP population that confirmed completion of the anamnesis.

To assess whether eMMa was used in the physicians’ spare time or during patient consultations, we will look at the days of the week that patients were enrolled. The ratio of enrollment rates on Fridays, weekends, and public holidays to the rest of the week will be calculated. The number of patients receiving the intervention per day will provide insight into whether GPs worked with eMMa en bloc or sporadically.

Data collection

We will collect data from three different sources:

eMMa software saves changes made to a patient’s medication by the GP and counts alerts at baseline and 2 months later.

KVWL delivers data on physicians. Physicians must be members of the KV to participate in the study, and work in the involved practices. The data will be delivered in aggregated form to ensure anonymity.

The BARMER statutory health insurer generates a data warehouse containing information on whether a patient was enrolled in eMMa by their GP or not. Aggregated information on the two patient groups will be provided in order that populations can be compared.

Table 2 provides an overview of the variables that contributed to outcomes. Pharmacovigilance in Germany includes the use of ‘Dear-doctor letters’ (Rote-Hand-Brief in German). In accordance with federal health authority regulations, pharmaceutical manufacturers provide informational material on drugs to physicians and pharmacists when alarming new data on specific drugs are available. Pharmacotherapy for specific patients might then be adjusted accordingly.

Table 2.

Dimensions with all outcomes and the responding variables with their source.

Outcome	Characteristics	Time	Source
Reach
Group differences between enrolled, non-enrolled and inactive patients	• Age• Sex• Economic status• Number of chronic prescriptions at intervention start• Disease score (disease count, 23 medCDS, 21 Charlson comorbidity index, 22 HRQoL comorbidity index) 24	T0	BARMER
Group differences between enrolled, non-enrolled and inactive GPs	• Age• Sex• Specialization type• Percentage of enrolled patients	T0	KVWL
Group differences between enrolled, non-enrolled and inactive practices	• Number of employed physicians• Number of patient visits per quarter• Percentage of eligible patients that were enrolled• Type of practice	T0	KVWL
Change of enrollment rate per month (overall)	• Percentage of eligible patients that were enrolled per month	Continuously	eMMa
Dose
Differences in medication interaction alerts per patient and per GP	• Number of medication interaction alerts• Number of severity level 1 medication interaction alerts	T0 to T1	eMMa
Differences in duplicate prescription alerts per patient and per GP	• Number of duplicate prescription alerts• Number of severity level 1 duplicate prescription alerts	T0 to T1	eMMa
Differences in kidney function alerts per patient and per GP	• Number of kidney function alerts	T0 to T1	eMMa
Differences in dosage alerts per patient and per GP	• Number of dosage alerts• Number of severity level 1 dosage alerts	T0 to T1	eMMa
Differences in age-related PIM alerts per patient and per GP	• Number of age-related PIM alerts	T0 to T1	eMMa
Differences in dear-doctor-letter alerts per patient and per GP	• Number of dear-doctor-letter alerts	T0 to T1	eMMa
Differences in allergy alerts per patient and per GP	• Number of allergy alerts	T0 to T1	eMMa
Differences in medication regimen-related (= combined interaction and duplicate prescription) alerts per patient and per GP	• Number of medication regimen-related alerts• Number of severity level 1 medication regimen-related alerts	T0 to T1	eMMa
Differences in dose-related (= combined kidney function and dosage) alerts per patient and per GP	• Number of dose-related alerts• Number of severity level 1 dose-related alerts	T0 to T1	eMMa
Differences in substance-related (= combined age-related PIM, allergy and dear-doctor-letter) alerts per patient and per GP	• Number of substance-related alerts	T0 to T1	eMMa
Differences in total (= combined medication regimen-, dose-, and substance-related) alerts per patient and per GP	• Number of total alerts• Total number of severity level 1 alerts (= interaction, duplicate prescription, kidney function, dosage, and allergy)	T0 to T1	eMMa
Percentage of patients whose physician parameters were entered into eMMa	• Number of patients with vs without documented kidney function• Number of patients with vs without documented height• Number of patients with vs without documented weight function	T1	eMMa
Percentage of patients with a printed medication plan	• Number of patients with vs without a printed medication plan in German• Number of patients with a printed medication plan in a foreign language vs a printed medication plan in German	T1	eMMa
Percentage of patients whose medication was changed	• Any changes occurred in patient’s medication (binary variable)	T1	eMMa
Fidelity
Number of interventions per GP with reductions of unexplained severity level 1 alerts to zero	• Number of medication-related alerts of severity level 1 reduced to zero (binary variable)• Number of dose-related alerts of severity level 1 reduced to zero (binary variable)• Number of allergy alerts reduced to zero (binary variable)	T0 to T1	eMMa
Tailoring
Distribution of day of enrollment in eMMa	• Number of patients involved in the intervention, stratified by day of the week (Monday through Sunday) for all GPs	T0	eMMa

BARMER, a statutory healthcare service company operating in Germany; eMMa, electronic medication management; GPs, general practitioners; HRQoL, health-related quality of life; KVWL, Kassenärztliche Vereinigung Westfalen-Lippe; PIM, potentially inappropriate medication.

Statistical analysis

All outcomes will be presented using descriptive statistics. Mean, standard deviation, median, interquartile range and 95% confidence intervals will be provided for continuous, normally distributed variables. Frequency and percentages will be provided for binary and categorical variables. Poisson confidence intervals will be calculated for count variables that do not follow a normal distribution. For statistical testing, a significance level of alpha = 0.05 (5%) and two-sided hypothesis testing (if not specified otherwise) will be applied. If variables are not normally distributed, non-parametric tests will be conducted. All analyses will be stratified according to the time of the practice’s first enrolled patient according to the eMMa software version that was in use at that time. Impact of the COVID-19 pandemic will also be analyzed regarding regional shutdowns and incidences.

Univariate analysis

To ensure data privacy, group comparisons in the Reach dimension will only be carried out on a descriptive level. To determine any association between enrollment rates and disrupting factors, we will conduct a time-series analysis and report the number of days on which such disruption occurred.

Differences in the fall in alerts between T0 (independent variable) and T1 (dependent variable) will be analyzed using the paired T-test and the Wilcoxon rank sum test for all types of alerts (Figure 2, lowest level). The Chi²-Test will be used at T1 to compare documented vs non-documented and printed vs non-printed items in eMMa.

The Chi²-Test will be used to test the decline in overall alerts to zero between T0 and T1 for the alert types displayed in Figure 2 (medium level).

Differences in the number of patients on individual days of the week, between weekday and weekend as well as between quarters, months and years will be tested using the Chi² test at T0.

Further multivariable (explanatory) analyses/subgroup analyses

:

Age- and sex-stratified analyses of the different alert categories (GP level) and stratification of the number of enrolled patients per GP

Linear correlations between the different alert types will be represented using the Pearson correlation coefficient plus p-values and corresponding scatterplots.

Linear or logistic regression will be used to model the relationship between the reduction of all alerts (response variable) and both the different alert categories and patient characteristics (explanatory variables)

:

Age- and sex-stratified analyses of medication regimen*dose*allergy alerts

:

Analyses of weekdays stratified according to number of enrolled patients per GP

Software: R and R Studio will be used to perform data quality checks, data transformation, statistical analysis, graphical visualization, and for reporting. The MySQL database will be used for data storage. To ensure data is imported and exported automatically, a link between the MySQL database and R will be created.

Discussion

This process evaluation will provide insight into the way GPs implement the AdAM intervention. By evaluating the kind of alerts that decrease and those that stay mostly unchanged, it can elucidate which patient groups are prioritized in terms of enrollment and how their medication is assessed. This will help explain findings in the main study. This is one of the first studies to evaluate GPs’ decisions in the medication management process and – to the best of our knowledge – the first to support findings with log data from a CDSS. A study protocol describing a plan to conduct a process evaluation on another polypharmacy trial has also recently been published elsewhere. Our process evaluation, however, has the strength to use claims data and a digital decision support by a software program. This quantitative research is part of a more comprehensive process evaluation that includes qualitative data on the perspectives of participating physicians and their patients. This data has already been published.[14,15]

However, there are some limitations to consider. The process evaluation relies on proper documentation by GPs since log data only show changes made in the system and may not display real life consequences. Still, these changes may be reflected in the potential improvement in primary and secondary outcomes without full visualization in the results of this process evaluation.

Since digital software like eMMa is continuously updated, this intervention will be affected by changes in the usability of the CDSS. Some of these may make it more difficult to compare the actions of GPs at different time points. For example, in the final version of the eMMa software, confirmation that the patient’s anamnesis has been completed is required before alerts can be seen by GPs. If changes introduced during the study turn out to be relevant to the process evaluation, sensitivity analysis will be used to examine them. As log data can be used to compare time protocols for updates and technical difficulties that might have hampered the use of eMMa, we will be able to identify links between technical disruptions and the conduct of GPs.