Comparison of two methods to estimate adverse events in the IBEAS Study (Ibero-American study of adverse events): cross-sectional versus retrospective cohort design.

BACKGROUND: Adverse events (AEs) epidemiology is the first step to improve practice in the healthcare system. Usually, the preferred method used to estimate the magnitude of the problem is the retrospective cohort study design, with retrospective reviews of the medical records. However this data collection involves a sophisticated sampling plan, and a process of intensive review of sometimes very heavy and complex medical records. Cross-sectional survey is also a valid and feasible methodology to study AEs.
OBJECTIVES: The aim of this study is to compare AEs detection using two different methodologies: cross-sectional versus retrospective cohort design.
SETTING: Secondary and tertiary hospitals in five countries: Argentina, Colombia, Costa Rica, Mexico and Peru. PARTICIPANTS: The IBEAS Study is a cross-sectional survey with a sample size of 11 379 patients. The retrospective cohort study was obtained from a 10% random sample proportional to hospital size from the entire IBEAS Study population.
METHODS: This study compares the 1-day prevalence of the AEs obtained in the IBEAS Study with the incidence obtained through the retrospective cohort study.
RESULTS: The prevalence of patients with AEs was 10.47% (95% CI 9.90 to 11.03) (1191/11 379), while the cumulative incidence of the retrospective cohort study was 19.76% (95% CI 17.35% to 22.17%) (215/1088). In both studies the highest risk of suffering AEs was seen in Intensive Care Unit (ICU) patients. Comorbid patients and patients with medical devices showed higher risk.
CONCLUSION: The retrospective cohort design, although requires more resources, allows to detect more AEs than the cross-sectional design. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

The identification of adverse events (AEs) is the first step to improve patient safety. We know cross-sectional studies are easier and less expensive to measure AEs.

This article adds the comparison between different study designs, and finds the most efficient to find AEs in the clinical practice. The study was made in five Latin American countries, so the data are strong for analysis.

We learn with this study that retrospective cohort design allows to detect more AEs compared with cross-sectional. ICU patients and patients with comorbidities have more AEs.

The sample used to evaluate the retrospective cohort was 10% of medical records used in the cross-sectional study, and proportional to hospital size, which could not be representative of all the population attended to in the healthcare system. Tertiary hospitals are more complex and the number of AEs could be overestimated, in comparison with the total number of patients attended to in the country.

Another limitation is the quality of medical records. If the variabilities in the accomplishments between the different countries and healthcare systems were high the comparison between them could be weaker.

Introduction

Valid and timely information about the frequency and impact of healthcare related adverse events (AEs) and about the system’s ability to detect, prevent and manage these AEs is extremely important to understand the failures of healthcare, and to design and evaluate the effectiveness of risk reduction strategies. Increasingly, a large number of research studies have estimated such type of information in various health systems and organisational contexts,1 leading to a growing body of evidence about the burden and nature of AEs caused by healthcare. One of the most important sources of information for such type of data are the patients’ medical records, most frequently through the practice of retrospective reviews following agreed protocols and standard abstract forms. This methodology has consolidated itself as one of the most valid references in the field of patient safety research.2 Nevertheless, despite the advantages of retrospective records reviews in identifying important and observable AEs, there are also some concerns about the capacity to conduct such methodology in facilities with weaker data and research infrastructure, and moreover when certain periodicity is desirable for monitoring the effectiveness of risk reduction strategies.

Every research methodology and data collection system has its strengths and drawbacks.3 Routine information systems have limitations related to compliance and coding bias. Events reporting systems also show preference for the type of events that reporters consider more relevant and have difficulties tracing duplicates, in addition to still facing unresolved legal issues in many contexts, which penalise reporting and limit their effective use. Prospective studies tend to focus on the analysis of higher-risk patients in detriment of other patients. Medical records, electronic or not, are threatened as well by lack of completeness and recording bias, since clinicians tend to record the data that are more meaningful to them from a clinical point of view. In addition, medical retrospective records review involves a sophisticated sampling plan, and a resource intensive process of record retrieval, reviewing and abstracting of sometimes very heavy and complex medical records.

A data collection process that has been less frequently used in the field of patient safety research, despite its potential, consists of running periodic cross-sectional surveys aiming to assess the point prevalence of AEs.4 5, This design has been commonly used to monitor the frequency of healthcare associated infections in many hospitals across Europe and elsewhere,6 where it has proven to be a feasible and valid methodology, capable of being run with no excessive resources on a large scale and across many institutions and organisational cultures. One of the advantages of this design is that instead of requiring a statistically savvy sampling plan, all patients admitted at a given time to the hospital can be surveyed at once, simplifying the sampling process as well as the search and retrieval of records from the archives, since these are usually located near the patients in the wards.7 This design also gives researchers the opportunity to ask the attending clinicians for some clarifications in the records, including some missing data. The unit of observation in this design is typically 1 day of admission, which makes it much shorter and simpler for the reviewers, and gives an estimate of a 1 day prevalence, as opposed to the cumulative incidence of a retrospective record review.8 Because of its greater simplicity, the management of large and multicentred research studies is also simplified.

The IBEAS Study was a multicountry effort aiming to estimate for the first time the frequency of hospital related AEs in a selection of hospitals from Argentina, Colombia, Costa Rica, Mexico and Peru.9 The study was conducted in 2007, in 58 hospitals of the five countries, with the collaboration of Spain, and pan-American organisation and WHO. It used a 1-day cross-sectional design due to the perceived simplicity, lesser demands and greater opportunities for strengthening local capacity and eventual replication of this approach. The researchers involved in designing IBEAS, aware of its innovative approach in the field of AE measurement, were mindful of determining the relationship between the estimates of the 1-day point prevalence design and the more traditional retrospective cohort record review approach. In this same context Michel et al10 evaluated the rates comparing three different methods: cross-sectional, prospective and retrospective. Therefore, we selected a randomised sample of all IBEAS patients to fully examine retrospectively their medical records.

Methods

The AE definitions used in both designs were those published by WHO in the International Classification for Patient Safety.11

A patient safety incident is an event or circumstance that could have resulted, or did result, in unnecessary harm to a patient.

An AE or harmful incident is an incident that results in harm to a patient.

Harm implies impairment of structure or function of the body and/or any deleterious effect arising therefrom, including disease, injury, suffering, disability and death, and may be physical, social or psychological.

In the IBEAS project the AE was defined as:9 ’Any event causing harm to the patient that is perceived to be more related to the healthcare management rather than to the patient’s underlying condition’.

The IBEAS Study had two parts,12 a cross-sectional study and a retrospective cohort study.

The cross-sectional study involved determining how many patients admitted to the participating hospitals experienced harmful incidents attributable to healthcare on a given day (day 0). A prevalent AE is defined as one that originates during hospitalisation and is clinically present on the day of the study, either as an after-effect or undertreatment. This also includes those AEs that were occasioned prior to hospitalisation at any care level and which led to subsequent admission. AEs that had occurred prior to the survey and whose effects had disappeared without prolonging the hospitalisation on that particular day were not included.

The retrospective cohort study was conducted using a sample of patients with the aim of confirming whether the cross-sectional study could replace the conventional retrospective cohort study used to date. Specifically, the study involved reviewing the case notes of a random sample of 10% of patients (1101 patients) hospitalised on day 0, proportional to hospital size, from the entire IBEAS Study population. Case notes were scanned to ascertain whether, at some point during their hospitalisation (or in a previous admission to the hospital), inpatients had experienced a harmful incident, regardless of whether the consequences of the incident were still present on day 0. Patients continued to be monitored until discharge. The sampling strategy and forms are available on request. An incident AE is defined as one that occurs during any patient care process, as it may be detected at another level of care or in another hospital. In practical terms, as we carried out a retrospective study based on clinical hospital records, primary care AEs were not included. Those that led to readmission in the same or another hospital were compensated by the AEs which were detected during this hospitalisation and which had been originated in a previous hospitalisation.

The IBEAS Study was carried out in five countries: Argentina, Colombia, Costa Rica, Mexico and Peru. The number of hospitals included was 58, all of them secondary-level and tertiary-level hospitals. We used a purposive sample of hospitals and the inclusion was voluntary.

The sample size was of 11 379 patients, with a minimum of 2000 patients per country.

In both studies (cross-sectional and retrospective cohort), researchers used two tools to detect harmful incidents, namely a Screening Guide and a Modular Questionnaire13 14 using the medical record review.15 16

First, the screening guide was applied to the patients in the study. This served as an alert and tracking system for possible incidents. All the patients admitted at hospital (except emergency room) were studied. The screening was made by well trained nurses.

If a patient screened positive for one or more of the 19 alert criteria in the screening guide, the case was studied using the case history. An in-depth study of case histories enabled researchers to conclude whether a patient did in fact present with the consequences of a harmful incident (true positive) and if so, to classify the type of event, its severity, any associated factors, and whether or not the incident could have been avoided, and so on. This second confirmatory review was made (in both cross-sectional and retrospective) by medical doctors with at least 5 years of clinical experience. A patient could have more than one AE in the same hospitalisation, and in this case the study collects all of them.

The reviewers' training took place in two stages. First, the trainer workshop addressed the national coordinating teams in Buenos Aires in 2007. Second, the national coordinators, in turn, trained the national investigators. A concordance study was carried out in Bogotá in 2008 using clinical records from each country. The most complex cases were assessed and an agreement was reached.

The preventable AE and the gravity were assessed according to the recommendations in the Modular Questionnaire, and the reviewers were also trained in these criteria.

The cross-sectional and the retrospective cohort studies were made by the same reviewers in each country.

The completed review forms of the retrospective study were entered in electronic files and submitted to a central repository managed exclusively by the principal investigators. Descriptive and multivariate analyses were conducted using SPSS V.14. Logistic regressions were used to estimate the prevalence and incidence of AEs, once taking into account the effect of some covariates, such as patient’s age and comorbidity (intrinsic factors), presence of catheter lines and medical devices (extrinsic factors), type of admission and type of hospital.

Results

In the cross-sectional study 11 379 patients were included (figure 1); 3853 of them (33.9%) fulfilled at least one of the screening criteria. In the second phase of the cross-sectional study 1191 patients had an AE.

Figure 1

Study patients in the cross-sectional study.

For the retrospective cohort study (figure 2), 1101 patients (10%) were randomly selected from all the 11 379 patients included in the cross-sectional study. The medical records of 13 of these patients (1.2%) could not be retrieved and were excluded from the study.

Figure 2

Study patients in the retrospective cohort study.

The screening phase of the retrospective review found about 44.5% of the medical records, corresponding to 484 patients, positive for at least one of the 19 triggers included in the forms. At the confirmatory phase, it was determined that 40 of those patients had experienced one or more patient safety incidents without harm or prolonged stay, and 288 patients had experienced at least one AE (harmful patient safety incident). Of these, in 215 patients the AE was considered to be mostly related to the healthcare received rather than to the patient intrinsic vulnerability.

The characteristics of patients in the two types of study are presented in table 1. Patients in the retrospective cohort study were of similar sex composition than the cross-sectional study sample. Though they were slightly older, they did not show significant differences in their intrinsic factors (comorbidity). It seemed there were more patients in surgical wards and with slightly more procedures in the retrospective review sample than the patients in the 1-day cross-sectional study. The composition of participating hospitals and type of admission was comparable in the two types of designs.

Table 1

Characteristics of the study population

		Cross-sectional			Retrospective cohort			p Value
		n	%	95% CI	n	%	95% CI
Sex	Women	5975	52.5	51.6 to 53.4	547	50.3	47.3 to 53.2	n.s.
Age	Mean (SD)	40.2	26.9		42.1	26	31.3 to 36.9	0.02
Department	Medical wards	4045	35.5	34.7 to 36.4	371	34.1	31.3 to 36.9	.001
	Surgery/gynaecology	3898	34.3	33.4 to 35.1	435	40.0	37.1 to 42.9
	Obstetrics	1241	10.9	10.3 to 11.5	109	10.0	8.2 to 11.8
	Paediatrics	1701	14.9	14.3 to 15.6	128	11.8	9.9 to 13.7
	Intensive care	494	4.3	4 to 4.7	45	4.1	3 to 5.3
Hospital complexity	Tertiary	10 520	92.5	92 to 92.9	1011	92.9	91.4 to 94.4	n.s.
	Secondary (with surgery and ICU wards)	859	7.5	7.1 to 8	77	7.1	5.6 to 8.6
Admission type	Unplanned admission	8031	70.6	69.7 to 71.4	726	66.7	63.9 to 69.5	n.s.
	Planned admission	2099	18.4	17.7 to 19.2	190	17.4	15.2 to 19.7
Intrinsic risk factors	Yes	6128	53.9	52 to 54.8	615	56.5	53.6 to 5.5	n.s.
	No	5251	46.1	45.2 to 47.1	473	43.5	40.5 to 46.4
Extrinsic risk factors	Yes	8484	74.6	73.8 to 75.4	844	77.6	75.1 to 80.1	0.03
	No	2895	25.4	24.6 to 26.2	244	22.4	19.9 to 24.9
Patients studied		11 379			1088

n.s., not significant (p>0.05).

Table 2 shows the prevalence of patients with AEs was 10.47% (95% CI 9.90% to 11.03%). As a patient can have more than one AE, the total number of AEs detected was 1349, so the global prevalence of AEs was 11.85% (1349/11379) (95% CI 11.26% to 12.46%).

Table 2

Differences in result measures in both study designs

	Cross–sectional (prevalence)	Retrospective cohort (cumulative incidence)
Patients with adverse events (AEs)	1191/11379=10.47% (95% CI 9.90% to 11.03%)	215/1088=19.76% (95% CI 17.35% to 22.17%)
Total number of AEs	1349/11379=11.85% (95% CI 11.26% to 12.46%)	314/1088=28.86% (95% CI 26.12% to 31.61%)

Table 2 also shows the cumulative incidence of patients suffering at least one AE related to the care received before or during their hospitalisation was 19.76% (95% CI 17.35% to 22.17%) (215/1088). In total, there were 314 AEs (because a patient could have more than 1 AE) related to healthcare corresponding to a cumulative incidence of total AEs of 28.86% (95% CI 26.12% to 31.60%) (317/1088).

Table 3 shows the results of the cross-sectional study and the retrospective cohort record review per country, showing the rate of positive screening and its positive predictive value (PPV) and the corresponding final estimate in terms of 1-day prevalence and the proportion of patients with at least one AE during their hospitalisation. In the 1-day cross-sectional study, the rate of positive screening review form (SRF) seemed to range more homogeneouslybetween 30% to 39% of all records, with PPVs between 25% and 37%. In the retrospective cohort review, however, the range of positive screening was wider going from about 17% to almost 64% of all records, and also reaching higher PPVs from 24% to over 60%. In all countries, the percentage of patients suffering at least one AE during their hospitalisation was significantly higher than the rate observed in the 1-day study, with values going from 11% of patients to more than 36%.

Table 3

Adverse event frequency measures and screening form performance

	Cross-sectional study			Retrospective study
	Positive screening review form	Positive predictive values	Prevalence of adverse events	Positive screening review form	Positive predictive values	Cumulative incidence of adverse events
Country 1	39.0% 926/2373 (95% CI 37.0% to 41.0%)	33.7% 312/926 (95% CI 30.6% to 36.8%)	13.1% 312/2373 (95% CI 11.8% to 14.5%)	61.7% 145/235 (95% CI 6.7% to 8.1%)	51.7% 75/145 (95% CI 43.2% to 60.2%)	31.9% 75/235 (95% CI 25.7% to 38.1%)
Country 2	30.6% 887/2897 (95% CI 28.9% to 32.3%)	25.3% 224/887 (95% CI 22.3% to 28.2%)	7.7% 224/2897 (95% CI 6.7% to 8.7%)	38.9% 112/288 (95% CI 33.1% to 44.7%)	32.1% 36/112 (95% CI 23.0% to 41.2%)	12.5% 36/288 (95% CI 8.5% to 16.5%)
Country 3	35.4% 578/1632 95% CI 33.1% to 37.8%)	34.3% 198/578 (95% CI 30.3% to 38.2%)	12.1% 198/1632 (95% CI 6.7% to 8.7%)	63.7% 107/168 (95% CI 56.1% to 71.3%)	57.0% 61/107 (95% CI 47.2% to 66.9%)	36.3% 61/168 (95% CI 28.7% to 43.9%)
Country 4	34.5% 692/2003 (95% CI 32.4% to 36.7%)	24.7% 171/692 (95% CI 21.4% to 27.9%)	8.5% 171/2003 (95% CI 7.3% to 9.8%)	46.9% 82/175 (95% CI 39.2% to 54.5%)	24.4% 20/82 (95% CI 14.5% to 34.3%)	11.4% 20/175 (95% CI 6.4% to 16.4%)
Country 5	31.1% 770/2474 (95% CI 29.3% to 32.9%)	37.1% 286/770 (95% CI 33.7% to 40.6%)	11.6% 286/2474 (95% CI 10.3% to 12.8%)	17.1% 38/222 (95% CI 11.9% to 22.3%)	60.5% 23/38 (95% CI 43.7% to 77.4%)	10.4% 23/222 (95% CI 6.1% to 14.6%)
Total	33.9% 3853/11379 (95% CI 32.9% to 34.7%)	30.9% 1191/3853 (95% CI 29.4% to 32.4%)	10.5% 1191/11379 (95% CI 9.9 to 11.0)	44.5% 484/1088 (95% CI 41.5% to 47.5%)	44.4% 215/484 (95% CI 39.3% to 48.3%)	19.8% 215/1088 (95% CI 17.3% to 22.2%)

In both studies (table 4), the highest risk of suffering AEs was seen in ICU patients. Surgical patients were associated with more risk than patients admitted in the medical wards. Whereas, in the cross-sectional study, obstetrics and paediatric patients also showed higher risk than medical patients. Comorbid patients showed higher risk of suffering AEs in both studies, as well as patients with catheter lines, and other procedures. Similarly, the length of stay before the day of study in the cross-sectional study and the total length of stay in the retrospective cohort were associated witha higher risk of suffering AE. In the retrospective cohort review, emergency hospitalisations did not seem to be associated to the risk of suffering AEs as seemed to be the case in the cross-sectional study. Patient age was not retained as an independent variable or as a confounding factor in the final model in both studies.

Table 4

Correlates of adverse events in multiple logistic regression analyses

	Cross-sectional				Retrospective cohort
Variables	p Value	OR	95% CI for OR		p Value	OR	95% CI for OR
Department (1)
Surgery and gynaecology	0.06	1.17	0.99	1.38	0.01	1.75	1.17	2.61
Obstetrics	0.02	1.37	1.06	1.78	0.05	0.38	0.15	0.99
Paediatrics	0.00	1.50	1.21	1.85	0.15	0.40	0.12	1.40
Intensive care	0.00	2.52	1.96	3.26	0.01	2.77	1.25	6.17
Complexity of the hospital (2) tertiary	0.02	1.45	1.07	1.97
Type of admission (3) urgent	0.00	1.34	1.12	1.61	0.59	1.14	0.71	1.83
Length of stay until the day of study	0.03	1.00	1.00	1.00	0.01	1.00	1.00	1.01
Patient comorbidity (4) any	0.00	1.42	1.22	1.64	0.00	2.02	1.28	3.19
Use of medical devices (5) any	0.00	2.59	2.14	3.14	0.00	3.24	1.79	5.85
Country 1
Country 2	0.00	0.46	0.38	0.56	0.00	0.34	0.20	0.59
Country 3	0.37	0.91	0.73	1.12	0.16	1.44	0.86	2.41
Country 4	0.00	0.65	0.52	0.81	0.00	0.22	0.12	0.42
Country 5	0.04	0.82	0.69	0.99	0.01	0.38	0.19	0.76

Reference categories: (1) medical specialities; (2) secondary hospitals of intermediate complexity with at least surgical theatres, and postsurgical resuscitation wards; (3) planned admission; (4) no comorbidities (5) no devices.

The types of AEs identified in both the cross-sectional and the retrospective cohort studies showed similar distribution. The most frequent types of AEs identified in any study were related to the occurrence of healthcare associated infections (more than 35% of all AEs), followed by AEs related to procedures (more than 26%). Medication related AEs represented less than 10% of all AEs in each of the studies (table 5).

Table 5

Adverse event (AE) types and proportion of total AE

Type of AE	Prevalence	95% CI	Cumulative incidence	95% CI
Care provided	13.27%	11.46 to 15.08	12.16%	16.24 to 20.32
Medication	8.23%	6.76 to 9.69	6.57%	9.87 to 13.17
Healthcare associated infections	37.14%	34.56 to 39.72	30.68%	35.99 to 41.30
Related to procedures	28.69%	26.27 to 31.10	21.86%	26.75 to 31.65
Diagnostic issues	6.15%	6.15 to 7.44	2.66%	5.10 to 7.53
Nosocomial urinary tract infection	4.08%	2.98 to 5.17	5.09%	2.50 to 7.69
Nosocomial pneumonia	9.41%	7.82 to 11.01	6.37%	3.51 to 9.23
Postsurgical haematoma	2.89%	1.96 to 3.82	3.50%	1.31 to 5.69
Phlebitis	3.41%	2.40 to 4.41	5.73%	3.00 to 8.46
Neonatal complications	1.12%	0.51 to 1.71	0.32%	0.01 to 1.76

There were some differences in the impact caused by the AEs identified in the cross-sectional study, versus the retrospective cohort study. The AEs identified through the cross-sectional study seemed to be associated more frequently with hospital readmissions and slightly more with prolonged stay, whereas the frequency of AEs which did not cause prolonged stay or readmission was higher in the retrospective cohort study (table 6).

Table 6

Impact of adverse events in hospitalisation, n (%)

	Cross-sectional		Retrospective cohort
Did not have prolonged hospital stay	228	18.91% (95% CI 16.65% to 21.16%)	87	29.9% (95% CI 24.46% to 35.33%)
Prolonged hospital stay	759	62.9% (95% CI 60.17% to 65.70%)	178	61.2% (95% CI 51.88% to 63.58%)
Extra days same hospitalisation	Mean: 16.1 days SD (29.6)		Mean: 14.9 days SD (19.9)
Causing admission	219	18.16% (95% CI 15.94% to 20.38%)	26	8.9% (95% CI 5.48% to 12.38%)
Extra days new hospitalisation	Mean: 21.4 days SD (69.7)		Mean:19.0 days SE (22.3)

The preventability of AE (See online supplementary appendix) was very similar, with about 65% in the retrospective cohort review and 60% for the cross-sectional study.

Discussion

The choice of the most appropriate epidemiological design in the study of magnitude of AE is not a trivial issue. The question has been analysed in different studies and the generalised consensus is that the choice of method should be based on the aims of the study and the need to combine the minimisation of bias and the validity of AE identification with the reproducibility of value judgements on their iatrogenic nature and/or preventability.17

The retrospective design for the study of AE has been the method used in all national studies.1 Nevertheless it is a method which produces results which may be heavily influenced by the quality of clinical records.

A prospective study offers pedagogical and communicative advantages and facilitates a concomitant analysis of the root causes which provide the conditions for the occurrence of AEs. However not only might it prove too costly, but it would also involve a high workload and excessive complexity.

On the other hand, the cross-sectional design is more time-efficient and resource-efficient and easier to perform. Although it does not allow for a study of the total hospitalisation episode, it has proved capable of sustaining over time a more stable system of observation. We also need to bear in mind that, as a result of a possible survival bias, those AEs which lead to hospital admission will be over-represented, as will those related to nosocomial infection or those which are difficult to identify if the patient is not examined (such as bruising), due to the systemic approach itself of a prevalence study. As in the prospective approach, communication with the ward staff (the patient is hospitalised at this time) makes it easier to judge the causality of the AE and its preventability.

The relationship between prevalence and incidence generally depends on the duration of the event under review and the period of observation.8 In our case we calculated prevalence on a given day and not during the whole period. Consequently this relationship will not be well reflected. In figure 3 we see the possible AEs which may occur and those that are detected on the basis of this approach.

Figure 3

Study of prevalent AE on a given day. AE, adverse event; HCC, healthcare centre; PC, primary care.

When we compare the results of the cross-sectional study with those of the retrospective cohort study within the context of the IBEAS project, the differences are due exclusively to the design, as the methodology and sample are the same (assuming the representativeness of the incidence subcohort). In (online supplementary appendix), figure 3 and figure 4) which represent the scheme followed in the methodology of this study, we see that the difference between the prevalence and incidence values on a given day are due to those AEs which, having occurred during hospitalisation, are not prevalent on the day of the study (represented by a yellow arrow in figure 3). This also explains why the patients of the retrospective cohort study present more extrinsic risk factors (devices) than in the cross-sectional study.

Figure 4

Scheme of incident AE study. AE, adverse event; HCC, healthcare centre; PC, primary care.

The SRF has been used in American18 19   and Australian20 21 cohort studies and in different European22 23 countries. It is highly sensitive (84%) in the detection of AEs and we therefore assume that the number of false negatives should be small. We can also detect with the revision of the modular questionnaire.

Appropriateness of the review forms to a point prevalence study was discussed during the training workshop. Modifications to adapt them to the context of Latin America were done bearing in mind vocabulary, and adding common risk factors like malaria or prematurity.

The percentage of patients flagged in the SRF and the predictive value of this phase in the detection of AE are totally compatible with those found in those other AE studies of which we are aware. We can therefore state that the materials are sensitive enough and appropriate for the identification of both prevalence and cumulative incidence of AE. However, in the retrospective cohort study, the PPV of the SRF is higher. This may be due to the fact either that the guide was originally designed for an incidence study and proves more efficient in this type of study or that as the retrospective cohort study was performed after the cross-sectional study, it is possible that the experience of the reviewers raised the performance level of the first questionnaire.

The Spanish version of the modular review form was adapted in Spain for the Identification of Adverse Events (IDEA) Project and modified after the ENEAS (National Study of Adverse Events) Study.4 The researcher must make value judgements through implicit criteria on most occasions. Characterisation of AEs caused by the care rather than the pathological process itself, is done by the reviewer scoring from 1 to 6 the probability that the AE is due to the care. A value of ≥4 is required to confirm this. The same criterion is used to evaluate the AE as preventable. Cross-sectional design allows researchers to consult the medical staff while they are collecting data in order to clarify any uncertainty or doubts associated with the AE. The reliability of the questionnaire in other studies has been assessed as moderate.23

A limitation of this study could be the sample used to evaluate the retrospective cohort. It is 10% of medical records used in the cross-sectional study and proportional to hospital size, which could be not representative of all the population attended to in the healthcare system. Tertiary hospitals have more complexity and the number of AEs could be overestimated, in comparison with the total number of patients attended to in the country. Another limitation is the quality of medical records. If the variability in the accomplishment between the different countries were high the comparison between them could be weaker.

The frequency of AEs in both the cross-sectional and retrospective cohort studies was greater than that found in previous studies, which may be due to the different characteristics of the patients, who had higher average age and more risk factors. The nature of the sample selection and the peculiarities of the different search systems prevent statistical inferences and comparisons either within each country or between the countries which are part of the study.

In any case, higher prevalence means higher cumulative incidence. In some way the interdependence of these frequency measures remains when we use prevalence on a given day. The fact that prevalence is sensitive to the differences in the characteristics of the patients and that it reflects the differences found between countries, would make it a useful tool in the study and follow-up of the frequency of AEs and in comparative studies. Furthermore, as the explanatory model for the occurrence of AE is the same, studying the factors which influence prevalence may provide the same clues when designing strategies for AE control and therefore provide a more efficient tool.

Moreover, the fact that the prevalence design detects proportionally more serious AEs is not a drawback. On the contrary, these are precisely the AEs which need to be prioritised when designing control strategies, and as we commented above, the detected AEs were equally preventable in both designs. This reinforces the idea that preventability and seriousness of the AEs are independent factors.

Due to the fact that the point prevalence design is more efficient in terms of time and resources, its validity is less dependent on the quality of the clinical records and allows simultaneous study through other observation and audit systems, regular prevalence on a given day studies might provide an efficient AE monitoring and control strategy.