Post-marketing health technology monitoring. The analysis of an experience from a clinical perspective.

INTRODUCTION: A system for monitoring the use of aphaeresis in the treatment of ulcerative colitis (UC), named system for monitoring aphaeresis in ulcerative colitis (SiMAC), was designed in 2006 in the Basque Country. In the present study, the opinion of the clinicians who participated in SiMAC was evaluated, in order to identify the barriers and gather suggestions that could improve implementation of this kind of system.
METHODS: A mixed questionnaire was designed, in order to gather clinicians' assessments of the SiMAC monitoring system.
RESULTS: The response rate was 73.9% (17/23). The data from 40.96% (159/388) of patients with UC treated with aphaeresis was recorded. The main reasons for not including the data from all treated patients were a lack of required data or not meeting the study inclusion criteria. Positive aspects of the SiMAC were identified, as the simplicity of data collection and its systematic, multi-center approach. The negative aspects mentioned were the use of a local computer application and the lack of time for health professionals to enter data. DISCUSSION: The use of monitoring systems helps to formalize the introduction of technologies of little-known effectiveness; involve clinicians and medical societies in coming to agreement and obtaining information about the safety, effectiveness or efficiency of new technologies; and provide relevant information to healthcare administrations for making decisions about the introduction of new technologies into healthcare practice. In order for a monitoring system to work, the process must be straightforward. A minimum set of key variables that are easy to collect must be selected, and an effort made to involve a range of stakeholders, especially institutions and scientific societies, to support the work group.

Introduction

Health technology assessment (HTA) is designed to provide information, based on scientific knowledge and a real-life context, to inform decision-making at various levels in the health system. However, when making decisions about the introduction of new technologies, there may be doubts due to a lack of information about effectiveness or because, when using the technology under normal conditions, problems, or side effects arise that were not predicted in the preliminary studies (Varela Lema et al., 2007).

In the case of medical devices, the CE mark guarantees that the product meets the requirements of the European Directives designed to ensure their safety and performance. However, the CE mark does not guarantee the clinical effectiveness or cost effectiveness of these products. For this reason, various initiatives have been implemented in the past decade with the aim of observing when and how technology is used in real conditions, or in order to carry out studies in which the safety and effectiveness of the technology is assessed within a specific population or scenario (Frønsdal et al., 2010). In Spain, a procedure called “Coverage under review,” has been designed for assessing technologies with high potential for producing substantial improvements in health or in quality of life, but for which there is not sufficient information to make pronouncements about their safety, effectiveness, and efficiency. This procedure is used before the technologies can be funded in general terms by the public health system.

In the Basque Country and Galicia, specific regulations determine that, depending on the available scientific evidence, some new technologies may be considered as subject to special limitations and controls or to special monitoring. This means that, following authorization for use, they remain subject to the specific indication, implementation, and assessment criteria established by the relevant bodies.

Under this law, the Basque Office for Health Technology Assessment (Osteba) produced an assessment report on the use of aphaeresis systems in treating inflammatory bowel disease (IBD; Gutiérrez-Ibarluzea and Arcelay, 2004), technology that has borne the CE mark since 1999. This report suggested that the technology be monitored for treating ulcerative colitis (UC), given the lack of scientific evidence found about its effectiveness and the short monitoring periods previously implemented, which are inappropriate for detecting safety problems in the medium- to long-term. A research project was consequently designed, using a computer application known as the system for monitoring aphaeresis in ulcerative colitis (SiMAC, abbreviated from the Spanish; Ibargoyen-Roteta et al., 2006). This project was based on the prior related recommendations of the Spanish Work Group on Crohn’s Disease and Ulcerative Colitis (GETECCU); Cabriada et al., 2006).

Twenty-three hospitals from all over Spain participated in this project and data were collected from 195 patients with UC treated with this type of system.

Because this was the first system designed to monitor the use of a new technology, the present work aims to reflect the opinions of the clinicians who participated in data collection via SiMAC, with the aim of identifying the difficulties that were encountered and gathering suggestions about aspects that could improve the implementation and participation of clinicians in data collection of other monitoring systems.

Materials and Methods

Study population

Clinicians responsible for collecting data from the 23 hospitals participating in SiMAC.

Structure of the questionnaire

A mixed questionnaire was designed, with two sections (see Figure 1). The first section, labeled “Data of interest,” contained questions related to the hospital, the situation in terms of IBD and UC, and the use of aphaeresis systems in each of the hospitals.

Figure 1

System for monitoring aphaeresis in ulcerative colitis assessment questionnaire. The outcomes of the questionnaire had not been validated.

The second part of the questionnaire was designed in order to evaluate the SiMAC by the participating clinicians. This section dealt with topics such as the suitability of the variables defined for treatment monitoring, the most appreciated aspects of the system, the difficulties encountered, and the measures that could improve participation of health professionals in this type of monitoring system.

Study design

Transversal, descriptive study, carried out at the end of the SiMAC data collection by sending an assessment questionnaire via email to the representative health practitioner at each hospital. The email was used because of difficulties to contact health professionals by telephone. The questionnaire was first sent on 15.12.2009. Two weeks later, a reminder was sent to those hospitals that had not answered. A certificate accrediting participation in the assessment study was sent to all practitioners who had responded to the questionnaire.

Results analysis

The numerical data related to the hospital and the number of registered patients with IBD and UC, as well as the number of patients treated with aphaeresis, have been summarized in the form of absolute and relative frequencies. The answers to the open questions have been described in the text.

Results

Participation in the study

The response rate to the questionnaire was 73.9% (17/23).

Figure 2 shows the location of the hospitals that participated in the SiMAC, with those that responded to the assessment questionnaire shown in black.

Figure 2

Distribution of participating hospitals according to whether or not they responded to the questionnaire.

Data of interest about participating hospitals

These are tertiary hospitals, for which the population coverage, the number of registered patients with IBD and specifically, with UC, are presented in Table 1.

Table 1

Characteristics of the hospitals that responded to the questionnaire.

	Population covered	Patients with IBD	Patients with UC	Percentage of IBD patients with UC
Hospital 1	300,000	750	375	50
Hospital 2	700,000	1000	500	50
Hospital 3	300,000	700	300	42.86
Hospital 4	500,000	368	190	51.63
Hospital 5	250,000	800	350	43.75
Hospital 6	500,000	–	–	–
Hospital 7	450,000	986	450	45.64
Hospital 8	350,000	800	450	56.25
Hospital 9	400,000	1525	741	48.59
Hospital 10	250,000	700	320	45.71
Hospital 11	210,000	550	350	63.64
Hospital 12	215,000	750	319	42.53
Hospital 13	–	15	13	86.67
Hospital 14	600,000	300	200	66.67
Hospital 15	300,010	348	156	44.83
Hospital 16	250,000	500	250	50
Hospital 17	800,000	700	–	–

Table 2 shows the year in which aphaeresis treatment was first used in each hospital, the number of patients with treated IBD and UC and the number of treated UC patients included in SiMAC. It can be observed that there are six hospitals that began using aphaeresis systems before 2004.

Table 2

Aphaeresis in the participating hospitals that responded to the questionnaire.

Hospital	Aphaeresis start year	IBD patients with aphaeresis	UC patients with aphaeresis	UC patients with aphaeresis in SiMAC	Percentage UC patients with aphaeresis in SiMAC
Hospital 1	2004	27	24	22	91.67
Hospital 2	2006	20	15	11	73.33
Hospital 3	2003	45	40	3	7.5
Hospital 4	2005	19	19	11	57.89
Hospital 5	2003	16	13	7	53.85
Hospital 6	2004	50	45	8	17.78
Hospital 7	2006	10	8	3	37.5
Hospital 8	2005	14	13	4	30.77
Hospital 9	2003	27	20	2	10
Hospital 10	2004	12	8	8	100
Hospital 11	2003	20	17	8	47.06
Hospital 12	2007	9	9	6	66.67
Hospital 13	2007	3	3	3	100
Hospital 14	2002	80	60	10	16.67
Hospital 15	2002	35	25	20	80
Hospital 16	2004	40	36	32	88.89
Hospital 17	2006	33	33	1	3.03
Total		460	388	159	40.98

SiMAC assessment by clinicians

Table 2 reveals that 8 of the 17 hospitals included less than 50% of the UC patients that they had treated with aphaeresis. When asked why they had not included all the patients in SiMAC, of the 14 centers that answered this question, eight blamed a lack of data about these patients; three explained that they were patients with Crohn’s disease or indeterminate colitis; another four indicated that it was due to the inclusion criteria not being met (fundamentally, because no endoscopy had been performed, which was necessary for calculating the level of UC activity within the stated time periods); and one cited a work overload and not knowing how to involve the other department which was responsible for the treatment, in the data collection.

Six clinicians answered the question about the existence of specific indications for the use of aphaeresis, with each citing different indications: steroid-dependent or -resistant IBD; UC in general; steroid-dependent UC following the failure of immunosuppressive agents; steroid-dependent or resistant UC; as a treatment prior to surgery; or in mild to moderate and steroid-dependent UC.

Only two participants considered that the variables that had been defined for treatment monitoring were not the right ones. One justified this by saying that there are too many and that some do not provide information that is of interest to the study, which hinders data collection.

Of the 15 participants who considered the defined variables to have been well-chosen, three suggested making some changes related to defining collection times, with a clearer definition of the possible indications of aphaeresis and the use of an activity index that does not require endoscopy. With regard to those aspects of the SiMAC that received the best feedback, five participants appreciated the design of a systematic collection method for patient treatment, monitoring, and control; three referred to the simplicity of data collection and use of the computer application; four highlighted the multi-center nature of the study; and finally, two participants referred to the persistence of the sponsor group and the ease of communication with the same.

The difficulties encountered by participants were the following: seven had some kind of problem installing the application or with data-sending; three stated that they did not have time to enter the data; another three participants pointed to their lack of information about patients for whom the data was collected before the start of the project; two indicated that they had too many variables to collect; and one claimed to have had sparse information about the results of their patients in the medium- and long-term.

Incentives or measures pointed out to improve participation in a monitoring system were the following: provide a data manager or person to collect the required data (four participants); have a permanent database to facilitate publications (three participants); facilitate the inclusion of more patients in the database (three participants); prioritize one or more meetings between researchers and participants (three participants); reduce the work load of healthcare professionals (one participant); financial and curricular incentives (one participant); and the creation of an online (rather than local) database for data entry (one participant).

Of the 14 participants who answered the question on the participation of industry, one believed that industry players should not participate or, if they did, should do as little as possible; five participants considered that they could provide the material (the aphaeresis columns, in this case) or finance treatment in centers that are without coverage; four suggested that they could provide data managers; another participant suggested that they could take charge of maintaining the project database and webpage; and another that they could send reminders to participants to collect the pending data. One participant pointed out that the legal framework would pose barriers to industry players providing this type of help.

With regard to how to improve data collection for technology monitoring, six participants suggested that an online database could be designed, easier to use and with less variables; three proposed that the care load of the healthcare professional could be reduced or that the research could be included as part of their professional responsibilities; one indicated that having an electronic patient record that already includes these data would facilitate data collection; another that it would be improved by creating databases into which all treated patients could be entered (and not only some cases); and finally, one considered that the process would improve if it was done prospectively, using a data collection program with an easier interface. All these results are presented in Table 3.

Table 3

System for monitoring aphaeresis in ulcerative colitis assessment by clinicians.

Questions	n
Which are the reasons for not including all treated patients? (n = 14)
Lack of data about those patients	8
Crohn’s disease or Indeterminate colitis	3
Not meeting inclusion criteria	4
Work overload and not knowing how to involve the other department which was responsible for the treatment, in the data collection	1
Defined monitoring variables are the correct ones? (n = 17)
Yes/No	15/2
Would you modify any of them? (n = 17) Yes/No	4/13
What aspects did you like about SiMAC (n = 16)
Systematic collection method for patient treatment, monitoring and control	5
Simplicity of data collection and the use of computer application	3
Multi-center nature of the study	4
Persistence of the sponsor group and the ease of communication with the same	2
They liked the system	2
Which difficulties encountered when participating in this project? (n = 17)
Problems installing the application or with data-sending	7
No time to enter data	3
Lack of information about patients for whom data was collected before the start of the project	3
Too many variables to collect	2
Sparse information about patient results in the medium-and long-term	1
None	1
Which kind of incentives or measures would improve participation of health professionals in data collection? (n = 13)
Provide a data manager or person to collect the required data	4
Permanent database to facilitate publications	3
Facilitate the inclusion of more patients in the database	3
Prioritize one or more meetings between researchers and participants	3
Reduce the workload of health care professionals	1
Financial and curricular incentives	1
Creation of an online (rather than local) database for data entry	1
Its sufficiently incentivized	1
In which way could industry participate in such a system? (n = 14)
Industry players should not participate, or if they did, should do as little as possible	1
They could provide the material (the aphaeresis columns, in this case) or finance treatment in centers that have no coverage	5
Provide data managers	4
Take charge of maintaining project database and webpage	1
Send reminders to participants to collect the pending data	1
There are legal framework that pose barriers to industry players providing this type of help	1
Industry is necessary to carry out these type of projects	1
How do you think we could improve data collection in a technology monitoring system? (n = 13)
Designing an online database	6
Reducing the care load of health professionals or including research as part of their professional responsibilities	3
Having an electronic patient record that already includes these data	1
Creating databases into which all treated patients could be entered	1
Improving the process if done prospectively, using a data collection program with an easier interface	1

Discussion

Some options for obtaining the necessary information for monitoring health technology include the use of administrative databases, surveillance activities, or databases obtained from more specific activities, such as records of particular technologies or diseases (Frønsdal et al., 2010). In the present case, in order to monitor the effectiveness and safety of aphaeresis systems in treating UC, the patients treated with these systems were specifically registered. An installable application was designed for this purpose, with variables defined in advance by a group of IBD experts from the GETECCU group (Ibargoyen-Roteta et al., 2006). The reason for developing an installable application was to provide every participating center with the option of maintaining and exploiting their own local database. It was considered that this might be an incentive for increasing the level of participation in the SiMAC. However, once the evaluation study had been completed, it became clear that participants had had some problems, specifically with installing the application and sending data to the central database. This is a very important point, as it may have influenced the data collection and the ease of the whole process. In fact, as indicated by some participants, the use of an online application, as well as the existence of the electronic patient record (Varela Lema et al., 2007) are options that could have simplified data collection. The ease of data collection has a major impact on whether or not participants meet the deadline for inclusion of data from any study of this type (Ministerio de Sanidad y Consumo, 2007).

Even though initially SiMAC was designed for collecting data in a prospective manner, it was decided that the inclusion of data from those patients who had started aphaeresis treatment before the start of the study should be allowed, since many hospitals had already started to use these systems prior to 2004.

When working with technologies used in pathologies or conditions with low incidence, or with applications in a small number of patients, national and international collaborations could facilitate the data collection process. In the present case, an attempt was made to involve the greatest possible number of Spanish hospitals in order to collect data from the greatest possible number of patients, and thus build a sufficiently large sample as to be able to draw conclusions that are relevant to clinical practice.

This type of collaboration facilitates mechanisms for sharing protocols, and even data on results from individual patients, which makes it possible to draw more reliable conclusions within the shortest possible time. In this respect, it is worth mentioning the effort carried out at European level by the EUnetHTA network in order to describe the situation in Europe in this area, and search for parameters for standardizing the design and operation of a monitoring system.

With regard to the low percentages of UC patients treated with aphaeresis systems that were included by some centers, clinicians state that this was mainly due to the impossibility of filling in the data for some of the variables required in SiMAC. According to opinions presented in other studies, focusing the monitoring on a minimum set of key data is essential in order to be able to meet the proposed target (Frønsdal et al., 2010). In fact, the clinicians who participated in SiMAC proposed reducing the number of variables to be collected for each patient, and replacing the index used for measuring UC activity with another that would not require endoscopy.

Another of the issues flagged by participants as a factor hindering data collection is their lack of time for dedicating themselves to this type of task. Reducing the care load for professionals participating in this type of initiatives, or including the research as an activity that is part of standard hospital tasks, are some of the suggestions made to remedy this. In this regard, the potential role of the electronic patient record should not be overlooked. This is starting to be used, and should be sufficiently flexible so as to be able to include the data necessary for monitoring studies, and from which it should be easy to extract data for analysis, without overloading medical practitioners even more. Study participants also suggested contracting data managers as another possible solution, although this would pose a funding problem for healthcare organizations, since the funds dedicated to these ends are limited (Frønsdal et al., 2010).

The funding obtained for carrying out the present work was restricted, since the design and setting up of the SiMAC monitoring system was funded through the Spanish national health system’s Quality Plan 2006, but not the monitoring of the data collection by the hospitals participating in the study. The project was able to be completed thanks to the involvement of GETECCU, which enabled closer contact with clinicians, provided information about the project in its annual meeting and promoted data collection. Increased funding would have made it possible to hold regular meetings between the principal researchers and the participating professionals, which the latter raised as an aspect that could improve the functioning of this type of monitoring system.

Industry players also participated in completing the study to a certain extent, by personally reminding the participants, during their visits, to complete the monitoring of the patients that they had included in the records. As a possible solution to the funding problems, the European Commission signaled that technology monitoring could be funded by the industry in general (European Commission, 2004, 2009), despite the fact that it has already invested large amounts of money in the process of technology acceptance by the competent authorities and in safety monitoring systems. In the present case, only one participant held the opinion that industry should not participate, or that it should do so as little as possible. However, most participants offered differing visions of how industry could participate, such as facilitating the contracting of data managers or funding the columns or the treatment. Whatever the case, it would be fundamental to recognize whether industry participation influenced data collection in any way, disclosing the existence or absence of conflicts of interest. There are currently a variety of initiatives for bridging the “know-do” gap and for implementing health technologies in which agreements are reached between service providers and technology producers. In this vein, NICE (the UK’s National Institute for health and Clinical Excellence) has proposed mechanisms for improving access to new treatments (Raftery, 2001); and joint risk agreements with industry have now been developed in order for health technologies to be introduced in a prudent manner, while simultaneously reducing timescales (Hutton et al., 2007; Klemp et al., 2011; US Department of Health and Human Services, 2011).

Conclusion

Finally, while keeping in mind the small sample size (despite having included the opinions of almost 74% of the healthcare professionals), we can conclude that the use of monitoring systems can help us to: (a) create protocols for the introduction of technologies of little-known effectiveness, so that it is better controlled and has a better guarantee in terms of patient benefits; (b) involve clinicians and medical societies in agreements about and obtainment of information on new technologies; (c) obtain corroborated, objective information about the safety, effectiveness, and efficiency of new technologies; and (d) provide relevant information to the health administrations for making decisions about the introduction of new technologies into medical practice.

However, in order for a monitoring system to work, the following points need to be taken into account: (a) the necessity of facilitating the process; (b) the selection of a minimum set of relevant variables that are easy to collect [methods for identifying outcomes of interest could be used, such as the one developed by GRADE (Ibargoyen-Roteta et al., 2010)]; and (c) that the involvement in these systems of different stakeholders is fundamental, particularly institutions (recognition of healthcare professionals’ work and the necessity of funding the procedure) and scientific societies (support for the work group), an example of which is the role played by GETECCU in this project.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.