A computer decision aid for medical prevention: a pilot qualitative study of the Personalized Estimate of Risks (EsPeR) system.

BACKGROUND: Many preventable diseases such as ischemic heart diseases and breast cancer prevail at a large scale in the general population. Computerized decision support systems are one of the solutions for improving the quality of prevention strategies.
METHODS: The system called EsPeR (Personalized Estimate of Risks) combines calculation of several risks with computerisation of guidelines (cardiovascular prevention, screening for breast cancer, colorectal cancer, uterine cervix cancer, and prostate cancer, diagnosis of depression and suicide risk). We present a qualitative evaluation of its ergonomics, as well as it's understanding and acceptance by a group of general practitioners. We organised four focus groups each including 6-11 general practitioners. Physicians worked on several structured clinical scenarios with the help of EsPeR, and three senior investigators leaded structured discussion sessions.
RESULTS: The initial sessions identified several ergonomic flaws of the system that were easily corrected. Both clinical scenarios and discussion sessions identified several problems related to the insufficient comprehension (expression of risks, definition of familial history of disease), and difficulty for the physicians to accept some of the recommendations.
CONCLUSION: Educational, socio-professional and organisational components (i.e. time constraints for training and use of the EsPeR system during consultation) as well as acceptance of evidence-based decision-making should be taken into account before launching computerised decision support systems, or their application in randomised trials.

Background

In France, a considerable number of deaths are due to preventable diseases. Ischemic heart diseases and cerebrovascular diseases still account for more than 80,000 premature deaths and breast cancer, more than 10,000 deaths. The control of the main cardiovascular risk factors, such as hypertension, remains poor, both because of lack of awareness and insufficient blood pressure control [1]. The objectives of breast cancer screening (mammography every 2 years in women aged between 50 and 69 years recommended in French guidelines) are not completely attained either [2].

In France, the promotion of prevention strategies in the ambulatory setting remains a national challenge. Several initiatives have been undertaken by Public Health authorities and by Health Insurance National Office, to prompt general practitioners to modify their prevention practices and improve their adherence to national guidelines. The development of the EsPeR system (Personalised Estimate of Risks) is one of these actions. It relies on three hypotheses:

1. for a given individual, an accurate estimation of his (her) health risks allows a more pertinent choice of prevention interventions

2. presentation of a patient-customized clinical guideline, will promote its implementation.

3. the combination of risk estimations and patient-specific guidelines embedded in a computerised decision aid can promote physician application of guideline recommendations during consultations and stimulate the communication with the patient.

The objective of this paper is to describe EsPeR, a computerised decision support system designed to help prevention practices by use of risk estimates and guidelines and to present a pilot qualitative evaluation of its ergonomics, understanding and acceptance by general practitioners.

Methods

Design of EsPeR

The EsPeR system is designed for health professionals and it is freely accessible on the Internet, after a simple identification process. An interactive decision support system allows an estimation of risks and a presentation of clinical guidelines. A non-interactive part describes the scientific rationale and background of the system (the documentation part) and provides its sources of medical and epidemiological knowledge (the library part). The following functional description focuses on the decision support system.

Functional description

The decision support system on the EsPeR website includes four primary functions:

1. Presentation of the 10 highest risks of avoidable causes of death according to demographic characteristics (age, sex and region of residence).

2. Estimation of specific risks, using published risk prediction models.

3. Individualized presentation of French guidelines on cardiovascular risk management and cancer screening

4. Printable summary of prevention messages for physicians and patients.

Highest risks of death

For a specific patient, the user is asked to enter the system via a screen that presents the 10 highest risks of avoidable causes of death according to demographic characteristics (age, sex and region of residence) (see figure 1). The total 10-year mortality risk is displayed along with the absolute risk and the relative part of risk for each cause. These probabilities have been computed from 1) the French national statistics on the causes of deaths (along with available explicative variables (age, sex and region of residence) and 2) the French population data given by 1990 national census and intermediate estimates by INSEE (the French national Institute of Statistics and Economics). The probability of death from one cause, at a given age is estimated thanks to the method used by INSEE, based on a the use of fictitious cohort, which allows to estimate prospective probabilities from the transformation of transversal observational data [3]. The aim of this part is to suggest a hierarchy of preventable risks that should be adjusted according to the patient specific risk factors (e.g. consumption of tobacco and alcohol, blood pressure level, etc.).

Figure 1

Presentation of the ten first causes of mortality in a women aged 55 years, in the EsPeR system

Specific risks and individualized guidelines

After this first screen the physician can choose the specific risks he(she) wants to further estimate and consider for prevention in his(her) patient (cardiovascular risk, alcohol abuse screening, breast cancer, colorectal cancer, uterine cervix and prostate cancer screening, depression and suicide risk). A data input form is dynamically created with relevant clinical and biological items, once the physician has selected the risks (Figure 2). For breast cancer and cardiovascular disease, the form is integrated with a module which allows to create a family tree, enter disease history for each person in the pedigree and estimate familial risk of the patient [4] (Figure 3). The validation of the form provides a risk estimate for each selected risk, and a presentation of guidelines.

Figure 2

Data form after selection of breast cancer risk

Figure 3

Presentation of the pedigree in the familial history module

The method of implementation of guidelines has been previously described [5]. It relies on a generic guideline representation model derived from the GLIF 2 (Guideline Interchange Format) model [6], and the use of XML as a language of representation for any specific guideline, according to a specific DTD (document type definition) of this model. An inference engine implements the guideline model and executes any specific guideline represented in XML.

All original guidelines have been analysed to specify a decision algorithm, represented in XML language, and a list of elementary messages (e.i. extracts of guideline for a given patient profile). Two physicians experienced in medical informatics performed these analysis and specifications. For a given patient, the presentation of guideline consists in 1) the list of variables (i.e. the patient's characteristics) used in the decision algorithm, and 2) the successive elementary recommendations related to his(her) characteristics.

Knowledge base

All knowledge used in the system has been selected on the basis of extensive validation (according to evidence-based medicine methodology). Cardiovascular risks and breast cancer risk are computed thanks to published risk prediction models [7,8]. We considered the Framingham models as sufficiently validated for cardiovascular risk stratification in various cohorts, including French ones [9-11]. Since the Gail model for breast cancer has never been tested on any French data, we chose to only present an estimate of relative risk. Provided guidelines have been selected from those published by the French National Agency for Health Evaluation (ANAES) for:

• Diagnosis and management of hypertension, hyperlipidemia, and type 2 diabetes

• Screening for breast cancer, colorectal cancer, cervical cancer, and prostate cancer

• Diagnosis of depression and suicide risk

• Diagnosis and management of alcohol abuse.

Printable summary

Finally, a printable summary of all risk estimates and guidelines is made accessible for both physician and patient, with words and format adapted to each.

Technical architecture

The EsPeR system is based on a 3-tier client-server architecture. The concepts used by the decision support modules (patient data, risks estimates) are implemented in JAVA classes; these classes encapsulate relational databases with SQL queries and generate with the results HTML pages by using Active Server Pages technology.

Qualitative evaluation

Following published recommendations [12], we designed a pilot qualitative study to evaluate the ergonomics, understanding and acceptance of the interactive decision support system of EsPeR by general practitioners. The main objective of this preliminary evaluation in laboratory was to appreciate how general practitioners could use and react to EsPeR and to go in depth with them into their reactions and beliefs. A secondary objective of this study was to prepare a larger qualitative study to be organised in ambulatory practice, with use of EsPeR during consultation.

Study design

The study was performed in two sessions, each consisting in two focus groups. The first session was organised in January 2002 with the version 1 of the system and the second one in October 2002, with the version 2, improved according to several results of the first session. The differences between the first version and the second one concern the number of provided guidelines and the global ergonomics and navigation within the site. All focus groups were animated by three investigators (GC, IC and TD for the first and second ones, GC, IC and LL for the third and fourth ones)

Each focus group was divided into four phases: 1) a structured discussion to evaluate skills and culture on prevention (at the beginning of the session) 2) a presentation of the project and short training to EsPeR (1 hour), 3) an individual work on 6 to 12 structured clinical scenarios with or without EsPeR (2 to 4 hours) and 4) a structured discussion to draw a consensus synthesis on the evaluation of EsPeR (at the end of the session) (1 to 2 hours).

The same investigator (GC) conducted discussion during phases 1 and 4 in all groups, according to outlines presented in Table 1, while the others (IC and TD for the 1st session in January and IC and LL for the 2nd session in October) took notes of the discussion. The later were asked to transcribe original comments of physicians as faithfully as possible, rather than their own interpretations. Both topics of discussion (prevention and synthesis of evaluation) were also addressed thanks to a structured individual questionnaire filled out by physicians before the corresponding discussion.

Table 1

Outlines for structured discussion in focus groups

Discussion on prevention	Consensus synthesis
• Physician point of view	• Ergonomics and adoption of the computer decision support system (by main functionalities)
What is prevention?
Knowledge on risk (vocabulary, interpretation)	• Patient medical record (PMR)
Knowledge on guidelines	Unique/multiple forms
Current use of guidelines	Archives
Part of prevention in their practice	Interoperability with personal practice PMR
• Patient point of view (as perceived by physicians)	• Presentation of risks
Expectations for prevention	Were they puzzled by risk estimates?
Attitudes and adherence to the process of risk control	Perceived utility of risk estimates for the care of individual patients
• Expectations of physicians for computer decision support systems:	• Presentation of guidelines
	Perceived appropriateness of advices in GPs field practice
For medical decision making
To be usable while facing the patient	• Printable synthesis for patient
For communication with patients	Usability for communication with patients
For continuous medical education

During the second phase, the three investigators observed the physicians' behaviour in using the system. They collected physicians' remarks, difficulties and suggestions, along with their personal comments or opinion on these reactions. Structured clinical scenarios were formulated according to a priori hypotheses on physician's practices or beliefs, which could be in contradiction with the scientific content of EsPeR. For example, as current French guideline recommend systematic mammography screening of breast cancers in women aged 50 to 70 years, we depicted a scenario of a 43 aged women with average personal risk of breast cancer according to the Gail model (particularly, with one history of breast cancer in her mother, diagnosed at the age of 70). Physicians were asked to answer to each scenario on a structured form, about what would be their estimates of health risks and prevention strategies, whether concordant or not with EsPeR, used as decision aid.

Recruitment of general practitioners

Physicians were recruited differently for both sessions. For the January session, we requested volunteers in two different professional groups by the intermediate of the executive manager of each group (the National French Federation of Mutuality, which manages non for profit health centres with full-time of part time salaried GPs, and the National College of Teaching GPs). For the October session, we recruited volunteers by a mail shot to physicians who practice in two administrative departments of France (n = 437 in Indre et Loire, n = 135 in Seine et Marne) and who already participated in a prevention program organized by the CANAM (Health Insurance National Office for non salaried workers). Volunteers were asked to have minimal computer and Internet skills. In each focus group, the participants were not supposed to know each other and had no working contacts with the investigators before the sessions. All GPs were paid for their participation in the study.

Data collection and analysis

Demographic and professional characteristics of participants were collected by a questionnaire mailed to each participant before the session.

Evaluation data were collected:

1. During the discussion phases: from discussion notes collected by two investigators

2. During individual work on clinical scenarios: from standardized structured answer form filled out by each physician for all scenarios, with open commentaries and observation notes from the three investigators

3. Using two structured questionnaires evaluating the degree of knowledge (beginning of the session) and the satisfaction of physicians with EsPeR (end of the session).

Observations and discussion notes collected by the different investigators were all pooled together, with indication of the origin of each item (original comments by physicians or interpretation by investigators). They were summarized by one investigator (IC) and validated by two others (by GC and TD in January then by GC and LL in October). This qualitative verbatim of the focus group was further classified into:

• Ergonomics of the system (navigation, speed, intuitiveness)

• Understanding of contents

• Acceptance of advices (barriers and facilitators to put guidelines into practice).

We also checked the consistency of this verbatim with answers to questionnaires and we tried to infer from this verbatim some pragmatic interpretation potentially leading to concrete improvement of the EsPeR system.

Finally, we analysed the answers to each individual clinical scenario as concordant or discordant with the answers recommended in EsPeR. We expressed this concordance in terms of "acceptance" ("not acceptable": no concordance with the recommendation of EsPeR and negative comments from physicians; "intermediate": no concordance but no negative comments or concordance but negative comments; "acceptable": concordance with the recommendation of EsPeR and positive comments). We tried to explain this acceptance in the light of the verbatim of the focus group.

Results

Participants characteristics

The four focus groups were performed respectively with a total of 36 participants (11 and 11 in January session, 8 and 6 in October session). Mean age was 48 years, 89% were male and 80% used electronic medical record software. The four groups were different in terms of professional setting and use of informatics (Table 2).

Table 2

Characteristics of participants (figures are numbers)

	January session		October session
	Group A (n = 11)	Group B (n = 11)	Group A (n = 8)	Group B (n = 6)
Age, median (IQR)	46 (11)	51 (10)	42.5 (11.5)	51 (10)
Sex (number of males)	8	11	7	6
Practice setting
Health centre	5	1	0	0
Teaching activity	3	8	0	0
Ambulatory (urban)	-	-	6	4
Ambulatory (semi-rural)	-	-	2	2
Use of computer during consultation	6	10	8	6
Use of an electronic medical record (EMR)	6	10	8	5
Use of EMR during consultation	4	10	7	5
Use of decision support systems for diagnosis	2	4	1	1
Use of decision support systems for prescription	2	9	8	5

Problems of ergonomics

Several problems of ergonomics were clearly identified with the two first groups in January (both through observation by investigators and reports by physicians). These problems concerned:

• Navigation in the family tree module

• Navigation between data forms and results (risk and guidelines)

• Absence of archive of medical record and absence of an export function to avoid duplicate data entry (in EsPeR and in personal EMR software)

Some solutions to these problems were developed for the version 2 and GPs of the October sessions found the ergonomics and navigation correct, easy to use and fast enough (see Table 3). Physicians needed 15 to 20 minutes to work on one clinical scenario with EsPeR.

Table 3

Extracts of discussion notes and written open comments by participating physicians

Discussion notes on prevention
January 2002:	October 2002:
Prevention is:"individual prevention or population prevention?""to inform the patient""to educate the patient for health: nutrition, etc.""cancer prevention consists in regular and systematic exam, more specifically according to particular risks of patients""a lot of time !" "the activity of prevention is guided more by a the reasons of consultations or the circumstances than by any structured prevention plan."Demand of patients:"there is an harmful role of the medias: sometime, patients are informed before we are""patients would accept messages of prevention if we had more time with them"Guidelines:"there are too many of them" "we lack time to read them""we can agree with them but not use them""general practitioners are not involved enough in their development" "guidelines avoid to do bibliography and the ANAES is independent from Health Insurance institutions"	Prevention is:"Prevention is included in the time of regular consultation, according to its context""to use appropriate means to avoid accidents and diseases""a specific consultation for prevention would be twice longer than regular consultation""Guidelines:"ANAES guidelines are more acceptable for us when we have the opportunity to work on them in our continuous medical education group""we trust the ANAES guidelines more than experts opinions""there are a strong pressure by the media on patients concerning PSA screening"
Written open comments on clinical scenario answer forms
January 2002:	October 2002:
"the keyboarding is too long in the family history module""EsPeR does not take into account the treatment of risk factors""Two myocardial infarctions and one stroke in the family (uncles and aunts) do not constitute a familial risk""there is no estimated familial risk despite her mother had breast cancer""I will prescribe a mammography, even though EsPeR tells me it is not recommended.""Don't forget that we care of a patient who does not care about his(her) probability or about the cost of his(her) screening test""EsPeR helps to be aware of intensity of risks","EsPeR helps to balance risk of cardiovascular diseases and risk of cancer","I thought I knew the guidelines... finally: I don't",	"I did not find any answer with EsPeR""We cannot refuse a mammography screening to a 43 years aged women, even though screening is recommended at 50 years of age.""There is a strong incentive from laboratories to use PSA for systematic prostate cancer screening" * (it is not recommended by guideline in EsPeR)"I am questioned in my idea on the pertinence of screening"
Discussion notes on evaluation synthesis
January 2002:	October 2002:
"EsPeR is not ready for use in consultation""This tool is not appropriate for daily practice""EsPeR is easy to use, but difficult to integrate in one consultation""EsPeR allow to learn""Criteria used to define risk are too strict. In practice, we use fuzzy criteria""We trust our clinical experience to estimate risks" "the statistical truth is not the clinical truth""I need to adapt guideline to my personal practice""cognitive interest of the individualisation of guidelines brought at the moment of decision""I was able to integrate the concepts of risk only when I started working with EsPeR.	"I learned something today...I have practiced prevention for 23 years, only based on my common sense""The presentation of mortality data provides an interesting tool to communicate with the patient""We need a framework to interpret risks or some labels; we do not know how to relate cardiovascular risk with mortality risk""more warnings and more active alerts are needed from the estimations of risks""highlight individual messages better in guideline messages""we often overestimate familial risks"

Problems of understanding

The physicians' understanding of risks was highly variable. Several of them were not familiar with a quantitative estimate of risk. They did not know how to apply the absolute risk provided in EsPeR and asked for explanations. Few participants asserted being more confident in their own risk estimate. Few GPs did not know clinical practice guidelines. Most had heard about them but did not refer to them, and few both read and used them routinely (Table 3).

Acceptance of EsPeR content

The acceptance of the guideline messages was evaluated from GPs answers to clinical scenarios (adherence or no adherence to the messages), from their reactions observed during individual work and from the comments reported in discussion notes during open group discussion (Table 3). The cardiovascular guidelines were in general better accepted than guidelines on cancer screening. Several recommendations given by EsPeR concerning cancers screening were perceived as inappropriate because of patients' demands and social or organizational pressures.

Overall, physicians perceived the system as useful. Few participants would have liked to be guided more directly to select the risks that should be considered in priority for prevention for a given patient.

Table 4 presents examples of the acceptance of some recommendations of EsPeR, for different clinical situations addressed in scenarios, along with plausible explanations and solutions inferred from discussion notes and from physicians' comments.

Table 4

Degree of acceptance of guideline messages, possible interpretation and solutions

Examples of clinical scenarios (and corresponding guideline message)	Acceptance of EsPeR advice#	Characterization of the problem of acceptance	Possible solution
Tobacco smoking (only the last year consumption is taken into account in EsPeR)	Intermediate	Lack of knowledge (cardiovascular risk returns to baseline after smoking cessation)	More explicit reasoning and detailed explanations
Familial cardiovascular risk of a 60-year old man with 2 brothers who had myocardial infarction at more than 70 years of age... (no familial risk according to the definition used in the guideline)	Not acceptable	Over estimation of familial risk. Knowledge and evidence disagree with common sense and inherited cultural belief	More explanation and information on evidence
Absence of familial risk of breast cancer in a woman whose mother had a breast cancer at 60 and had no other family member having had a cancer	Intermediate	Knowledge and evidence disagree with common sense and inherited cultural belief	More explanation and information on evidence
Breast cancer screening in a 43-year old women at average risk (mammography not recommended as systematic screening, genetic screening only if high familial risk)	Not acceptable	Over-estimation of familial risk. High pressure (anxiety) of patients for screening without knowledge on benefit and risk of screening	More accurate information targeted on both the physician and the patient
Colorectal cancer screening by colonoscopy in a 60-year old man whose father is dead from colorectal cancer at 80 (not recommended in average risk patients except in research programs)	Not acceptable	Over-estimation of familial risk. Over estimation of the benefit/risk of colonoscopy	More explanation and information on evidence
Cervical cancer screening in a 55-year old women (pap smear recommended until 70 years of age)	Acceptable	-	-
Prostate cancer mass screening in men aged more than 50 (screening not recommended)	Intermediate	Poor knowledge of evidence	More information on evidence

# Acceptance: "not acceptable"= no concordance with the recommendation of EsPeR and negative comments from physicians; "intermediate" = no concordance but no negative comments or concordance but negative comments; "acceptable" = concordance with the recommendation of EsPeR and positive comments

Discussion

The present paper describes a computerised system devoted to the prevention of avoidable morbidity, based on risk assessment and valid recommendations. The main objective of EsPeR is to put in perspective the different health risks for one individual and to present guideline messages appropriate for this individual. The hierarchy of risks currently relies on the presentation of French mortality data and is therefore poorly "individualized", taking into account age, sex and region of residence. A more accurate estimate of several risks is possible thanks to specific models such as the Framingham model. However, these models have been demonstrated as being adequately discriminent but poorly calibrated in French population [10,13]. Other models have been recently published thanks to the SCORE project and are recommended by the Third Joint Task Force of European Societies on Cardiovascular prevention [14,15]. We intend to integrate them in EsPeR. We also experienced difficulties in implementing the French guidelines on cardiovascular prevention because of the heterogeneity of recommended tools for risk stratification.

We ran a pilot qualitative study in laboratory, in order to observe physicians' reaction and behaviour while using EsPeR and to find with them some explanations. We interpreted the collected verbatim and observations as potential barriers and facilitators for the diffusion of this tool. No focus group session was recorded or videotaped and no social scientist was involved in the study: the groups were animated by three different investigators among whom one was not involved directly in the design of EsPeR and supposed to be "neutral". Recruitment was different for both sessions (in January and October) and led to very different groups which interestingly suggested that different kinds of reactions may pertain to different backgrounds (socio-professional environment, use of informatics, etc.).

Despite these limitations, ergonomics problems were easily identified thanks to the first two focus groups. Corrective measures were taken in the version 2 and led to solve these problems, as verified with the second session of focus groups. The main problems pertained to the content of EsPeR and presentation of knowledge. Analysis of these results led to several actions. The first one consisted in giving a feedback to the ANAES (the French national agency in charge of the development of guidelines) on difficulties encountered while computerizing guidelines. This resulted in a partnership between our laboratory and the ANAES to set up a guideline development framework designed for the guideline authors, in order for them to write directly and unequivocally "computerizable" guidelines. Our results also suggest that improving the quality of preventive strategies in the primary care setting requires considering the multiple competing demands faced by patients and physicians during each consultation. The promotion of the EsPeR system, designed for assisting physicians in setting rational priorities in prevention strategies, should therefore include both an educational intervention (i.e. teaching evidence and the concept of risk for increasing physician's confidence in recommendations) and an organisational intervention (i.e. increase of the consultation duration to allow the complete use of the computerised decision support system, or promote the delegation of certain tasks to nurse practitioners).

The key assumption made by people developing decision aids is that providing decision support (recommendations, alerts, reminders, calculations) as well as useful clinical information at the point of care can reduce unwarranted variations in care and improve patients' health. It is generally acknowledged that computerised reminders and alerts improve the practice of simple preventive measures (screening, vaccinations...) [16]. However, more complex decision support systems such as computerised guidelines [17], or calculation of cardiovascular risk [18] did not improved quality of care in several recent well designed randomised trials. These findings can have several explanations: either the system does not work, or it is not used because it does not fit the physicians' needs or else it is in conflict with physicians beliefs and constraints. In keeping with this idea, some authors encourage evaluating computerised decision support systems not only with randomised trials [19]. In some examples, a qualitative study helps to explain the negative of poor results of a randomised controlled trial [17,20]. We therefore plan a qualitative on-field evaluation of the system to assess its usability and its acceptance by general practitioners on larger scale, in two different regions of France, before considering evaluating its impact in a randomised trial.

Conclusion

We presented an original computerized decision support system for medical prevention based on concepts like risks and complex knowledge found in clinical practice guidelines. The pilot qualitative evaluation of this system reinforces the need of training on its content, prior to its on-field evaluation. The main perspective of this work is the design of a larger qualitative study to evaluate the utilization and barriers to acceptance of the EsPeR system in the day to day practice. Potential important implication of this project could be the use of EsPeR as a structuring tool for prevention strategies in ambulatory medicine.

Competing interests

None declared.

Pre-publication history

The pre-publication history for this paper can be accessed here: