Validation of ACCESS: an automated tool to support self-management of COPD exacerbations.

BACKGROUND: To support patients with COPD in their self-management of symptom worsening, we developed Adaptive Computerized COPD Exacerbation Self-management Support (ACCESS), an innovative software application that provides automated treatment advice without the interference of a health care professional. Exacerbation detection is based on 12 symptom-related yes-or-no questions and the measurement of peripheral capillary oxygen saturation (SpO2), forced expiratory volume in one second (FEV1), and body temperature. Automated treatment advice is based on a decision model built by clinical expert panel opinion and Bayesian network modeling. The current paper describes the validity of ACCESS.
METHODS: We performed secondary analyses on data from a 3-month prospective observational study in which patients with COPD registered respiratory symptoms daily on diary cards and measured SpO2, FEV1, and body temperature. We examined the validity of the most important treatment advice of ACCESS, ie, to contact the health care professional, against symptom- and event-based exacerbations.
RESULTS: Fifty-four patients completed 2,928 diary cards. One or more of the different pieces of ACCESS advice were provided in 71.7% of all cases. We identified 115 symptom-based exacerbations. Cross-tabulation showed a sensitivity of 97.4% (95% CI 92.0-99.3), specificity of 65.6% (95% CI 63.5-67.6), and positive and negative predictive value of 13.4% (95% CI 11.2-15.9) and 99.8% (95% CI 99.3-99.9), respectively, for ACCESS' advice to contact a health care professional in case of an exacerbation.
CONCLUSION: In many cases (71.7%), ACCESS gave at least one self-management advice to lower symptom burden, showing that ACCES provides self-management support for both day-to-day symptom variations and exacerbations. High sensitivity shows that if there is an exacerbation, ACCESS will advise patients to contact a health care professional. The high negative predictive value leads us to conclude that when ACCES does not provide the advice to contact a health care professional, the risk of an exacerbation is very low. Thus, ACCESS can safely be used in patients with COPD to support self-management in case of an exacerbation.

Introduction

COPD is a heterogeneous disease with fluctuations in daily respiratory symptoms. Many patients with COPD suffer from acute exacerbations. These episodes are defined as “sustained worsening of the patient’s condition from the stable state, and beyond normal day-to-day variations, that are acute in onset and necessitate a change in regular medication”.1 Exacerbations pose a substantial burden on patients and society, since they are the main cause of hospitalization,2 mortality,3 and health care utilization4 in COPD, and lead to a more rapid decline in lung function5,6 and health status.6–8

When exacerbations are treated promptly, recovery time9,10 and hospital admissions10,11 can be reduced. However, many patients poorly recognize the difference between regular day-to-day variations and exacerbations and do not respond to an imminent exacerbation adequately.8,9,11 Self-management programs that include patient education and the use of written action plans have been developed to improve patients’ self-management behavior at times of an exacerbation. A written action plan contains instructions about the medication to use and the actions to take when a patient notices that respira-tory symptoms worsen. Research has shown that the use of written action plans may enhance prompt treatment of exacerbations and, as a result, improve health-related quality of life,12 reduce recovery time9,13–15 and hospital admissions.12,16,17 Therefore, current clinical COPD guidelines recommend a written exacerbation action plan for every COPD patient with frequent exacerbations to keep at home.18,19 However, it has been reported that in ~50% of exacerbations, patients do not adhere to the instructions in the action plan.9 Exacerbation self-management appears to be negatively influenced by habituation to symptoms, heterogeneity of exacerbations, low self-empowerment, and patients’ belief that they should not bother others with their problems.20,21

To optimize patients’ self-management during symptom worsening that may indicate an exacerbation in development, there is a need for easy-to-use tools to better support patients in the recognition and interpretation of symptoms than a paper action plan does, and to further improve self-empowerment and stimulate a sense of urgency to seek medical treatment, when needed. Therefore, we have previously developed a software application called “Adaptive Computerized COPD Exacerbation Self-management Support” (ACCESS).22 This software application aims to 1) identify exacerbations in an early phase and 2) directly provide self-management advice without the intervention of a health care professional. The advice enables patients to take prompt and adequate action in case of an imminent exacerbation. In this study, we deter-mined whether patients with COPD can safely use ACCESS as a self-management support tool in case of symptom worsening. More specifically, our research question was “What is the validity, expressed as sensitivity, specificity, positive and negative predictive value, of ACCESS’ clinically most important advice, ie, to contact a health care professional today, which is given when a COPD exacerbation is imminent?”

Methods

Study design, population, and sample size

We performed secondary analyses on data that were obtained in a 3-month prospective observational study that was designed to cross-validate the Bayesian network model (Supplementary materials). Patients with COPD were recruited from the outpatient clinic and the pulmonary rehabilitation program of the Radboud University Medical Center, Dekkerswald (Nijmegen, the Netherlands). Patients were eligible if they had a spirometry-confirmed diagnosis of COPD (ie, post-bronchodilator FEV1/FVC ,0.7), were known by their health care professional to have a high exacerbation risk, were able to communicate in the Dutch language, and were not familiar with physical or mental problems that would prevent adherence to study protocol. Additionally, in the outpatient population, patients had to be able to visit Dekkerswald if their symptoms worsened. At the pulmonary rehabilitation center, eligible patients received oral and written information about the study from a research team member. In the outpatient clinic, eligible patients received a written invitation to participate from their pulmonary nurse. After a week, patients who agreed to participate received individual oral and written instructions from a member of the research team. Within the inclusion period, patients from the outpatient clinic were invited to participate in sets, simultaneous to the consecutive inclusion of patients from the rehabilitation program. No formal sample size calculation was performed, but 50 exacerbations were considered sufficient for the cross-validation of the Bayesian network model (Supplementary material). In previous research in this patient population, an average of 4.2 exacerbations per year per patient was observed,23 so, theoretically, 50 patients would need to participate for 3 months to reach the number of 50 exacerbations.

Written informed consent was obtained from all the par-ticipating patients. The Medical Ethics Committees of the regions Arnhem and Nijmegen approved the study (approval number 2013/385).

ACCESS software application

ACCESS is an innovative automated exacerbation self-management support system. The system is innovative as it combines exacerbation detection with automated treatment advice without the interference of a health care professional. Exacerbation detection is based on the results of 12 yes-or-no questions about worsening of symptoms, emotions, and bronchodilator use and the outcomes of measurement of peripheral capillary oxygen saturation (SpO2 in %, Contec Pulse Oximeter, CMS50D), FEV1 (in liters, Vitalograph asma-1), and body temperature (in degrees Celsius, Medisana FTN infrared thermometer) (Figure 1). After data recording and depending on the outcomes, ACCESS directly generates 1–6 automated treatment options to support self-management to alleviate symptom burden. In case the responses to the questions and/or the measurements indicate an imminent/ possible exacerbation, ACCESS always advises to use the system again the next day. Which advice ACCESS will give is based on a decision tree that was designed by an expert panel of clinicians (Figure 2) and the risk of an exacerbation as predicted from a Bayesian network model that is incorporated in ACCESS.22 The system is adaptive in the sense that it is personalized by the patient’s health care professional, who sets personal reference values for FEV1 and SpO2, inserts patient-specific medication instructions, and decides on which pieces of advice are relevant for the patient. We refer to the supplementary file for more details about the development, content, and evaluation of the system.

Figure 1

Content of ACCESS.

Abbreviations: ACCESS, Adaptive Computerized COPD Exacerbation Self-management Support; SpO2, peripheral capillary oxygen saturation; FEV1, forced expiratory volume in one second.

Figure 2

Expert decision model of ACCESS advice.

Abbreviations: ACCESS, Adaptive Computerized COPD Exacerbation Self-management Support; SpO2, peripheral capillary oxygen saturation; FEV1, forced expiratory volume in one second.

Data collection

During the 3-month follow-up, patients completed paper diary cards with the 12 ACCESS questions and three physiological measurements (SpO2, FEV1, and body temperature; Supplementary material) every day. All data were gathered on paper to prevent missing data due to difficulty in the use of the software. The diary cards were collected weekly. The automated treatment advice was calculated by the ACCESS computer model after the 3-month observation period for each entry and added to the study database (see Figure 2 for the decision model).

Patients were instructed to contact their chest physician or respiratory nurse every time they had a worsening of any respiratory symptom. The chest physician diagnosed whether an exacerbation was present or not and recorded diagnosis and treatment plan on a standardized form. Additionally, at the end of follow-up, the patients’ medical records were examined in detail by the investigators (LB, EB) for exacerbation occurrences.

Furthermore, each patient completed a separate “best value” form on a stable day together with a health care professional or an investigator (LB) to establish the reference values for the patient’s personal best measurements of FEV1 and SpO2.

Exacerbations

We used a symptom-based and an event-based definition of exacerbations to compare ACCESS’ advice to contact a health care professional with.

Symptom-based definition: the diary card data were used to identify symptom-based exacerbations using the following definition: a change for at least 2 consecutive days in two or more major symptoms (dyspnea, sputum purulence, sputum amount), or a change in any one major symptom plus one or more minor symptoms (cold, wheeze, sore throat, cough).5,24,25

We counted a new exacerbation episode when it was preceded by at least 2 days without exacerbation.

Event-based definition: all contacts documented by health care professionals that led to a new prescription of prednisolone and/or antibiotics or hospitalization were considered event-based exacerbations.

Statistical analyses

Missing diary card data were excluded from analyses, since on a day with missing data, ACCESS cannot determine whether or not one or more pieces of advice should be given. We used descriptive statistics to examine the occurrence of the 6 automated pieces of advice. When an exacerbation was imminent, ACCESS should have advised patients to contact a health care professional. So, because of the clinical importance, we examined the relationship between ACCESS’ advice to contact a health care professional and symptom-based exacerbations with cross-tabulations. Sensitivity, specificity, and positive and negative predictive values were calculated. To consider ACCESS as being safe, sensitivity should be high (ie, if an exacerbation is indeed present, ACCESS should advise to contact a health care professional in virtually all cases), and negative predictive value should be high (ie, if ACCESS does not advise to contact a health care professional, there should be no exacerbation in virtually all cases). We looked at the second day of each exacerbation episode, because we wanted to examine if ACCESS provides timely advice, and this was the first day that an exacerbation could actually be established according to our definition.5,24,25 We performed post hoc descriptive analyses to further explore the precise reasons behind ACCESS’ advice to contact a health care professional in the absence of a symptom-based exacerbation.

To compare ACCESS’ advice to contact a health care professional to the actual diagnosis of the health care professionals, the ACCESS advice was also cross-tabulated with event-based exacerbations. Again, sensitivity, specificity, and positive and negative predictive values were calculated.

IBM Statistical Package for Social Sciences (SPSS) 22 was used for all analyses.

Results

Patients and exacerbations

Twenty-six patients from the rehabilitation department and 28 patients from the outpatient clinic were included between November 2013 and April 2014 (Figure 3). Together, these 54 patients completed 2,928 of the 2,992 diary cards (97.9% completeness) during a mean participation period of 55.4 (±SD 25.1) days. One hundred and fifteen symptom-based exacerbations were reported by 42 patients, with a median duration of 6.0 (IQR 7.4) days. Twenty-nine event-based episodes were documented from 22 patients. There were four hospital admissions in three patients, and no deaths during the follow-up period. Table 1 shows the baseline characteristics and median number of exacerbations of the study population.

Figure 3

Flowchart of patient participation in the study.

Table 1

Baseline characteristics of the study population (n=54) represented in mean (SD), median (IQR), or frequency (%)

Age, years (SD)	64.7 (8.6)
Male (%)	32 (59.3)
FEV1, litersa (SD)	1.26 (0.6)
FEV1 % predicteda (SD)	44.4 (18.2)
BMI, kg/m2,a (SD)	26.2 (4.8)
Follow-up time, days (SD)	55.4 (25.1)
Symptom-based exacerbation episodes per month (IQR)	1.1 (3.5)
Event-based exacerbation episodes per month (IQR)	0.0 (0.7)

Note:

n=44.

Abbreviations: IQR, interquartile range; BMI, body mass index; FEV1, forced expiratory volume in one second.

Automated advice of ACCESS

On 2,099 of the 2,928 days (71.7%), ACCESS provided one or more pieces of treatment advice. In 285 cases (9.7%), there was only the advice to measure again the next day, and in 544 cases (18.6%) there was no advice given, meaning that no symptom changes were reported that day. Figure 4 shows that the advice to contact a health care professional was given on 1,493 days (51.0% of 2,928 days). The advice to be thoughtful about how to distribute one’s energy over the day was given most often, in 1,829 cases (62.5%). The advice to use breathing techniques was provided in 1,285 cases (43.9%), the advice to maximize bronchodilator use was provided in 1,249 cases (42.7%), and the advice to use coughing techniques in 730 cases (24.9%).

Figure 4

Representation of percentages of times each advice was provided by ACCESS, based on 2,928 diary cards. On each day, more than one advice could be provided.

Abbreviation: ACCESS, Adaptive Computerized COPD Exacerbation Self-management Support.

Validity of ACCESS advice compared to symptom-based exacerbations

Table 2 shows the cross-tabulation between the advice to contact a health care professional and the actual presence (the second day) of a symptom-based exacerbation. In 112 of the 115 exacerbations, ACCESS advised to contact the health care professional, so sensitivity was 97.4% (95% CI 92.0–99.3). Specificity was 65.6% (95% CI 63.5–67.6), positive predictive value was 13.4% (95% CI 11.2–15.9), and negative predictive value was 99.8% (95% CI 99.3–99.9).

Table 2

Cross-tabulation of symptom-based exacerbationsa vs the advice to contact a health care professional given by ACCESS

		Symptom-based exacerbation
		Yes	No	Total
ACCESS advice to contact a health care professional	Yes	112	725	837
		13.4%	86.6%	100%
		97.4%	34.4%	37.7%
	No	3	1,382	1,385
		0.2%	99.8%	100%
		2.6%	65.6%	62.3%
	Total	115	2,107	2,222
		5.2%	94.8%	100%
		100%	100%	100%

Note:

Based on the second consecutive day of worsening of at least one major symptom (dyspnea, sputum purulence, sputum amount) and one minor symptom (cold, wheeze, sore throat, cough).5,24,25

Abbreviation: ACCESS, Adaptive Computerized COPD Exacerbation Self-management Support.

In-depth analyses were performed to examine the cases where the advice to contact a health care professional was given by ACCESS, but no symptom-based exacerbation turned out to have occurred (n=725). Figure 5 shows that the main reasons for ACCESS to advise patients to contact their health care provider were: 1) a drop in FEV1 of at least 10% compared to the patient’s personal best value; 2) increased fatigue in combination with a worsening of at least one major symptom for at least two consecutive days; and 3) increased dyspnea that day in combination with increased dyspnea and/or wheeze the previous day.

Figure 5

Representation of reasons for ACCESS’ advice to contact a health care professional, in the absence of a symptom-based exacerbation (n=725). In each event, more than one reason may apply.

Notes: 1) A worsening of at least 2 consecutive days in two or more major symptoms (dyspnea, sputum purulence, sputum amount) or a change in any one major symptom plus a cold and/or wheeze and/or sore throat and/or cough and/or fatigue; 2) dyspnea and/or wheeze and dyspnea and/or wheeze the previous day; 3) decrease in FEV1 is 10% or more of personal best; 4) increased dyspnea and increased bronchodilator use; 5) decrease in SpO2 is 4% or more of personal best; 6) risk calculation of Bayesian network is higher than 85%; 7) a worsening of two or more major symptoms (dyspnea, sputum purulence, sputum amount); 8) fifth consecutive day that advice “measure again tomorrow” (indication of worsening of any symptom) is given; 9) increased dyspnea and fear of increased dyspnea; 10) third day of increased wheeze and increased bronchodilator use; 11) any one major symptom (dyspnea, sputum purulence, sputum amount) and fever (see also Figure 2).

Abbreviation: ACCESS, Adaptive Computerized COPD Exacerbation Self-management Support; SpO2, peripheral capillary oxygen saturation; FEV1, forced expiratory volume in one second.

Validity of ACCESS advice compared to event-based exacerbations

A total of 29 event-based exacerbations were documented (Table 3). For the event-based exacerbations, sensitivity of the ACCESS advice to contact a health care professional was 96.6% (95% CI 80.4–99.8), specificity was 52.2% (95% CI 37.1–66.9), positive predictive value was 56.0% (95% CI 41.4–69.7), and negative predictive value was 96.0% (95% CI 77.7–99.8).

Table 3

Cross-tabulation of event-based exacerbationsa vs the advice to contact a health care professional given by ACCESS

		Event-based exacerbation
		Yes	No	Total
ACCESS advice to contact a health care professional	Yes	28	22	50
		56.0%	44.0%	100%
		96.6%	47.8%	66.7%
	No	1b	24	25
		4.0%	96.0%	100%
		3.4%	52.2%	33.3%
	Total	29	46	75
		38.7%	61.3%	100%
		100%	100%	100%

Notes:

Exacerbations treated with a course of antibiotics and/or prednisolone or resulted in hospitalization.

Patient started a course of prednisolone on his first day of study participation, reporting only minor symptoms that day.

Abbreviation: ACCESS, Adaptive Computerized COPD Exacerbation Self-management Support.

In one occasion, a course of prednisolone had been prescribed but ACCESS had not advised to contact a health care professional (Table 3). It turned out that this patient had started his study participation during an exacerbation, and on his first day, he reported minor symptoms and prednisolone was prescribed. The advice to contact a health care professional was given by ACCESS on the next day.

Discussion

In this study, we reported on the automatically provided treatment advice of the ACCESS software application, which aims to support COPD patients to take timely and adequate action in their self-management in case of symptom changes and exacerbations. We found that on the majority of days (71.7%), ACCESS provided one or more pieces of treatment advice to reduce symptom burden. In 51.0% of the cases, the advice was given to contact a health care professional that day. Hence, in 20.7% of all cases, ACCESS did not detect an (developing) exacerbation, but the pieces of advice – other than to contact a health care professional – were provided to lower the burden of day-to-day symptom fluctuations.

Next, we investigated the safety of ACCESS by examining the validity of its advice to contact a health care professional. The high sensitivity of this advice for both symptom- and event-based exacerbations (97.4% and 96.6%, respectively) showed that when an exacerbation is imminent, ACCESS is very likely to advise the patient to contact a health care professional. The high negative predictive value of the advice for both symptom- and event-based exacerbations (99.8% and 96.0%, respectively) showed that when ACCESS does not advise to contact a health care professional, there is a very low risk of an imminent exacerbation. Based on these results, we conclude that ACCESS’ advice to contact a health care professional is safe for patients to follow, because the chance of under-treatment of exacerbations seems negligible. However, specificity was moderate to low (65.6% and 52.2% for symptom- and event-based exacerbations, respectively) and positive predictive values were low (13.4% and 56.0% for symptom- and event-based exacerbations, respectively). This suggests that contact with a health care professional may not be needed in all the cases that ACCESS provides the advice. It should be noted that ACCESS has not been developed to replace the health care professional but to decrease patient delay in case of a developing exacerbation. Thus, part of the moderate to low specificity and low positive predictive value can be explained by the discrepancy between our reference standard (an exacerbation) and the aim of ACCESS’ advice: contact a health care professional to consult about alarming symptoms which may (or which may not) be caused by an exacerbation. Furthermore, some of these advice-triggering signs or symptoms were measures that were not included in the definition of a symptom-based exacerbation that we used as reference standard, such as a drop in FEV1 and peripheral capillary oxygen saturation values, or fatigue. This definitely has influenced specificity and positive predictive value, but the advice still may have been clinically relevant, since these parameters are mentioned in the literature as possible indicators of an exacerbation.26–31 Still, the expert panel may also have been overly cautious in assigning the pieces of advice in the decision model, which is understandable because of heterogeneity of symptoms and onset of exacerbations.

Comparison to other literature

In most other telehealth systems that have been developed for COPD in recent years, a health care professional monitors the input of patients actively, and patients are contacted in case of an imminent exacerbation.32–38 These health care professionals may be cautious in their monitoring too. More-over, these systems require continuous availability of health care professionals, and patients may come to rely on the health care professional to contact them in case of alarming symptoms instead of the other way around. Thus, instead of improving patient’s self-management and reducing patient delay, the latter may even increase. To our knowledge, only one other electronic support system provides automated advice to patients with COPD.39 This Web-based application advises on the start of prednisolone or antibiotics in case of an exacerbation, whereas our system provides various pieces of self-management advice right from the onset of symptom changes, even before an exacerbation can be established. Besides, so far as we know, the validity of the Web-based application has not been tested yet. We found a high rate of symptom-based exacerbations (ie, a median of 1.1 exacerbations per month) compared to other studies.5,14,23 This may be caused both by the inclusion of patients who were likely to be frequent exacerbators and our methodological approach to count a new episode when there were 2 days without exacerbation or missing days preceding the entry. In our view, this approach resembles a home situation best, where we aim to improve early recognition and patients should be advised at any time of a symptom worsening.

Strengths and limitations

Patients recorded the exacerbation-related data daily on paper diary cards that were compiled weekly. This resulted in a very high completeness of data (97.9%) and reduced the potential for recall bias. Also, the use of paper diary cards instead of using ACCESS as software application prevented potential bias caused by differences in digital skills among our study population. The mean follow-up time in our study was shorter than planned. Our initial plan was to include the pulmonary rehabilitation patients in the first week after starting rehabilitation and follow them for the complete duration of the program, which is 12 weeks. However, many patients were not enrolled in our study until week 4 of the rehabilitation program, resulting in a shorter follow-up period than the expected 12 weeks. Still, we found a high rate of symptom-based exacerbations, because of the inclusion of patients who were known by their health care professional to have a high risk of exacerbations. We did not perform a sample size calculation, although this is recommended in the STARD 2015 guideline,40 as we conducted secondary analyses on data that were obtained for the cross-validation of the Bayesian network model. Furthermore, at this moment, there is no other tool with a well-studied diagnostic accuracy with which we can compare ACCESS. The validity of ACCESS was tested in a population that was recruited in an outpatient clinic and in our pulmonary rehabilitation department and were prone to have exacerbations. This could have caused selection bias and it may be questioned whether our results can be generalized to a less severe COPD population with a lower risk of having exacerbations, such as in primary care. However, we previously demonstrated that in primary care also, the frequency of symptom-based exacerbations among patients with COPD is high.41

Future research

Further research to assess the clinical effectiveness of this support is ongoing (ClinicalTrials.gov Identifier: NCT02553096). If we find that moderate specificity results in unnecessary health care utilization, extended analyses are needed to investigate if specificity could be increased, without losing the high sensitivity.

Conclusion

In nearly three-quarters of the cases, ACCESS provided one or more of the following pieces of self-management advice – increase bronchodilator use, use breathing techniques, use coughing techniques, be thoughtful on how to distribute one’s energy over the day, and to contact a health care professional. This shows that ACCESS supports patients in their self-management of day-to-day symptom variations. Additionally, this study showed that ACCESS provided the advice to contact a health care professional on time, ie, on the second day of an exacerbation, with high sensitivity and negative predictive value, and thus appears to be a safe tool to use at home for patients with COPD. ACCESS has not been developed to replace the health care professional but to assist patients to take timely and adequate action in case of symptom worsening. In this way, ACCESS may improve self-management, decrease patient delay in seeking medical treatment when an exacerbation is imminent, and reduce disease burden by COPD.

Data sharing statement

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Supplementary materials

Previous steps in the development of ACCESS

The aim of the ACCESS system is to support COPD patients in exacerbation self-management by detecting exacerbations and providing tailored and timely treatment advice.

Building and optimizing the Bayesian network model

van der Heijden et al1 have described the first develop-mental phase of ACCESS (Figures S1 and S2, Step 1). In summary, a Bayesian network model was constructed based on expert opinion of two pulmonary physicians, who pro-vided relevant parameters based on their clinical experience and contemporary literature. A Bayesian network model is a flexible probabilistic model that can compute the probability of the presence of a disease based on available symptoms, also in case of missing data, and predictions can be personalized by entering patient-specific data. Based on the expert-driven model and available data, a data-driven model was created and compared to the occurrence of symptom-based exacerbations.1 This resulted in a risk prediction of exacerbations based on 12 yes or no questions which are complemented with measurements of peripheral capillary oxygen saturation (SpO2), forced expiratory volume in one second (FEV1), and body temperature. Furthermore, a pilot study among patients with COPD was performed to examine usability of the smart phone and sensors.1

Primary aim of the study described in the current paper was to gather data for a cross-validation procedure to optimize and test the Bayesian network model to automatically detect exacerbations. The data were divided in six groups of nine randomly selected patients. In each of the six cross-validity checks, the model learned to identify exacerbations based on the entries of the five training sets and was then tested for diagnostic accuracy on the sixth set. Gold standard were symptom-based exacerbations. The final model, resulting from the cross-validation procedure, showed an area under curve of the receiver operating curve of 91.4% (95% CI 90.3–92.5; Figure S3).

After the cross-validation procedure, a cutoff point of 85% was chosen to indicate when an exacerbation was at hand. Any risk calculation beyond this point should lead to an automatic advice of ACCESS to contact a health care professional or start a course of prednisolone.

Development and integration of automatic treatment advices

The next goal was to formulate treatment advice based on the risk calculation of the Bayesian network model to be given automatically by ACCESS. However, when discussing the matter in detail with clinicians, it became clear that self-management advice could not solely be based on the risk calculation of ACCESS. To decide in which specific situation what advice should be given by ACCESS, clinical expertise was used (Figure S2, Step 2). Expert involvement was realized in a three-step procedure: 1) an expert panel consisting of a pulmonary physician, a general practitioner, and a clinical psychologist discussed with which symptoms they wanted a patient to take what action, based on their clinical experience and contemporary guidelines for COPD exacerbation management; 2) various health care professionals – pulmonary physicians, pulmonary nurses, general practitioners, practice nurses – provided the advices they would give in four standardized cases on paper; and 3) these advices were then discussed by the expert panel and minor adjustments were made to the decision model.

In contrast to what we had previously anticipated, the expert panel focused on its clinical judgment, rather than using the risk calculation of the Bayesian network model as a base for the advices. This was perhaps inevitable in many cases, since several advices are initiated based on specific symptoms. Eventually, a cutoff point of the risk calculation of the Bayesian network model was chosen to underlie the expert panel’s decision model for automatic advice. This cutoff point was set at 85%, with a sensitivity of 65.8% (95% CI 62.4–69.0), specificity of 91.5% (95% CI 90.2–92.6), positive predictive value of 75.0% (95% CI 71.6–78.1), and negative predictive value of 87.3% (95% CI 85.8–88.6). When this cutoff point is reached, ACCESS will always provide the advice to contact a health care professional.

The advices are personalized by the patient’s health care professional, who enters the patient’s FEV1 and SpO2 values in the system as reference values, and adds specific medication instructions for maximal bronchodilator use. Specific advices can be blocked for a patient, if his health care professional deems this best. For instance, if the patient has not been taught coughing techniques, the advice to use coughing techniques would be irrelevant, maybe even confusing.

As a last step in the developmental phase, we have examined in the current study if patients are advised safely and correctly, by comparing the clinically most important ACCESS advice – “Contact a health care professional today” – with a symptom-based and an event-based definition of exacerbations (Step 4). For a clinical evaluation of ACCESS, a randomized controlled trial is in progress, comparing the effect of ACCESS with the effect of a written action plan on exacerbation time (Step 5).

Schematic presentation of the ACCESS system.

Schematic overview of the development of the Adaptive Computerized COPD Exacerbation Self-management Support (ACCESS) System.

ROC curve of cutoff values of the Bayesian network model’s risk calculation to identify a symptom-based exacerbation.

Abbreviations: ROC, receiver operating curve; AUC, area under the curve.