The modified Ottawa method to establish the update need of a systematic review: glass-ionomer versus resin sealants for caries prevention.

OBJECTIVE: To demonstrate the application of the modified Ottawa method by establishing the update need of a systematic review with focus on the caries preventive effect of GIC versus resin pit and fissure sealants; to answer the question as to whether the existing conclusions of this systematic review are still current; to establish whether a new update of this systematic review was needed.
METHODS: Application of the Modified Ottawa method. Application date: April/May 2012.
RESULTS: Four signals aligned with the criteria of the modified Ottawa method were identified. The content of these signals suggest that higher precision of the current systematic review results might be achieved if an update of the current review were conducted at this point in time. However, these signals further indicate that such systematic review update, despite its higher precision, would only confirm the existing review conclusion that no statistically significant difference exists in the caries-preventive effect of GIC and resin-based fissure sealants.
CONCLUSION: In conclusion, this study demonstrated the modified Ottawa method as an effective tool in establishing the update need of the systematic review. In addition, it was established that the conclusions of the systematic review in relation to the caries preventive effect of GIC versus resin based fissure sealants are still current, and that no update of this systematic review was warranted at date of application.

INTRODUCTION

In 2009, the authors of this study published a systematic review (search cut-off date: 15 January 2008), in order to appraise the then current evidence regarding the caries-preventing effect of glass-ionomer cement (GIC) in comparison to that of resin-based fissure sealants[16]. This systematic review with meta-analysis found no evidence that either material was superior to the other in the prevention of dental caries. Therefore, both appeared to be equally suitable for clinical application as fissure sealant materials.

In 2011, this systematic review was updated and the overall outcome was judged to be in agreement with the conclusions of the original systematic review[9]. The cut-off date for the systematic literature search of this update was the 26th of August 2010. At the beginning of 2012, the authors considered the question as to whether a further update would be needed. The Cochrane collaboration advocates a preset time-based approach, with a recommended frequency of two years, for initiating the update of an original systematic review[5].

In contrast, the use of a priority approach towards the updating of systematic reviews has been recently suggested[14]. Such an approach relies on the identification of signals indicating update needs. The Agency for Healthcare Research and Quality's (AHRQ) Evidence-based Practice Center (EPC) program conducted studies aimed at developing reliable methods for assessing the need for updating of systematic reviews[4]. Consequently, the EPC at the University of Ottawa developed the "Ottawa method". The modified Ottawa method relies on identification of qualitative, quantitative and "other" signals from new systematic review-, meta-analysis- or trial reports published (or otherwise made available) after the search date of an original systematic review. In order to identify new reports, a systematic literature search is conducted. Criteria for signals that indicate the need to update a systematic review report are shown in Figure 1[4]. Quantitative signals (B1 and B2) are obtained by pooling data extracted from identified new trials with the data from the original systematic review, using a fixed effects meta-analysis. The pooled results are then compared with the original systematic review results for any possible changes in statistical significance (B1) or effect size (B2). As the purpose of the meta-analysis is to identify signals indicating a potential need to update an original systematic review, and not to establish a revised effect estimate, this simplified use of meta-analysis (without considering pooled data heterogeneity) has been reported as sufficient[4]. The underlying criteria of this method were found to be effective during a study of 100 systematic reviews published from 1995 to 2005[7].

Figure 1

Modified Ottawa method: signal criteria

Signal type	Signal code	Operational definitions
(A) Qualitative signals of potentially invalidating changes in evidence	A1	Opposing findings: a pivotal trial1 or systematic review/meta-analysis/guideline including at least one new trial that characterized the treatment in terms opposite to those earlier
	A2	Substantial harm: a pivotal trial1 or systematic review/meta-analysis/guideline whose results called into question the use of the treatment based on evidence of harm or that did not proscribe use entirely but did potentially affect clinical decision making
	A3	A superior new treatment: a pivotal trial1 or systematic review/meta-analysis/guideline whose results identified another treatment as significantly superior to the one evaluated in the original systematic review, based on efficacy or harm
(A) Qualitative signals of major changes in evidence	A4	Important changes in effectiveness short of "opposing findings" (A1)
	A5	Clinically important expansion of treatment
	A6	Clinically important caveat
	A7	Opposing findings from discordant pivotal trial1 or systematic review/meta-analysis
(B) Quantitative signals	B1	A change in statistical significance (e.g. from non-significant to significant)
	B2	A change in relative effect size of at least 50%
(C) Other signals		Other signals were sought for key questions for which there were no prior systematic review or RCT, e.g. questions for which only large cohort studies or CCTs were identified. The criteria included a major increase in the number of new studies or a new study with at least 3x the number of participants as in previous.

RCT=Randomised control trial; CCT=Case control trial

A trial with at least 3x the number of participants as the largest trial included in the original systematic review

As the modified Ottawa method is still fairly new, this study aims to demonstrate its application by establishing the update need of a systematic review with focus on the caries preventive effect of GIC versus resin pit and fissure sealants[9]. As health care providers and policymakers depend on up-to-date evidence from systematic reviews, this study aims further to answer the question as to whether the existing conclusions of this systematic review[9] are still current and whether a new update of it is needed.

MATERIAL AND METHODS

The modified Ottawa method was applied[4]. In accordance with this method; a basic literature search was conducted on the 26th of April 2012, using the search strategy presented in Figure 2. The search strategy included the same string of English search terms and databases as those utilized in the last systematic review update, concerning the caries-preventive effect of GIC-based fissure sealants versus that of resin[9]. The search period for the PubMed search chosen was from 26 August 2009, being one year before the date of the last systematic review update (26 August 2010), to 26 April 2012. All other databases were searched from 2009 to the present. In addition, an ad hoc hand search was conducted through issues of pivotal journals (Dental Materials, Caries Research, Journal of Dentistry and Journal of Dental Research) published between 2009 and 2012.

Figure 2

Details of applied search strategy

	Citations included	Articles accepted
Expanded Academic ASAP PLUS search strategy: 26.04.12
(GIC sealant* OR glass ionomer cement sealant) AND (caries OR tooth decay) BMC search strategy: 2009 - 2012 (Boolean search) Online: http://www.biomedcentral.com/search/
(GIC sealant* OR glass ionomer cement sealant) AND (caries OR tooth decay) Cochrane Oral Health Reviews search strategy: 2009 - 2012 (Advanced search) Online: http://www.thecochranelibrary.com/view/0/index.html	4	0
(GIC sealant* OR glass ionomer cement sealant) AND (caries OR tooth decay) DOAJ search strategy: Date of search 26.04.12 Online: http://www.doaj.org	5	0
(GIC sealant* OR glass ionomer cement sealant) AND (caries OR tooth decay) Meta Register Of Controlled Trials search strategy: Date of search 26.04.12 Online: http://www.controlled-trials.com/isrctn/search.html	0	0
(GIC sealant* OR glass ionomer cement sealant) AND (caries OR tooth decay) Science-Direct search strategy: 2009 - present/26.04.12 (Expert search) Online: http://www.sciencedirect.com	0	0
(GIC sealant* OR glass ionomer cement sealant) AND (caries OR tooth decay) PubMed search strategy: 26.08.09 - 26.04.12 Online: http://www.pubmed.org	8	0
(GIC sealant* OR glass ionomer cement sealant) AND (caries OR tooth decay)	29	1
* fissure sealant AND glass ionomer AND resin	28	2
Reference check: 26.04.12		0
Ad-hoc hand search: 26.04.12		1
Total articles included:		4

Additional PubMed search

In order to verify the validity of the search strategy applied in the last systematic review update, a further string of search terms: "fissure sealant AND glass ionomer AND resin" for search in PubMed was also included as the search in this database had generated most of the reviewed articles, so far[9].

Both authors independently scanned titles and abstracts of the citations that were identified through the literature search. The selection criterion for systematic reviews and/or prospective clinical control trials was relevance to the original systematic review question[9]: whether teeth sealed with either GIC or resin developed equivalent caries occurrence on pits and fissures. Articles with titles of potential relevance but without abstracts were traced in full for scanning. Disagreements between reviewers were solved by discussion and consensus. Accepted articles were traced in full and reviewed for signals.

Quantitative signals (B1 and B2) were obtained by pooling data extracted from identified new trials with the data from trials included in the original systematic review, using a fixed effects meta-analysis. These pooled results were then compared with the pooled results of the datasets from the last systematic review update alone[9], for any possible changes in statistical significance (B1) or effect size (B2)[4].

Fixed effects meta-analysis for the identification of quantitative signals was conducted, using RevMan Version 4.2 statistical software of the Nordic Cochrane Centre, The Cochrane Collaboration (Copenhagen; 2003). The Risk ratio (RR) with 95% confidence intervals (CI) and p-values was computed. Alpha level was set at 5%.

The following information was extracted from the accepted trial reports for each intervention group: type of dentition (permanent or primary), length of follow-up period (in months or years) and dichotomous datasets (DS) concerning the outcome measure. The outcome measures included the number of sealed teeth that developed caries (n), from the total number of evaluated teeth (N), for each intervention group. On this basis, datasets (DS) were defined as any extracted set of n/N for the glass-ionomer cement (GIC) and resin groups. The result of one or more datasets concerning one particular type of dentition, tooth cavity, evaluation criteria and follow-up period was considered as a key finding.

Both authors extracted data from accepted trials independently, without being blinded to authors, institutions, journal names and trial results. Disagreements between authors were solved through discussion and consensus.

RESULTS

Literature search and data collection

Figure 3 provides information on the number of articles identified. The database search using the same search strategy as that of the last systematic review update[9] generated 29 citations, the PubMed search with the additional string of search terms generated 28 citations and ad hoc hand search identified one citation (Figure 2). From the total of 58 citations, 53 were excluded through scanning of titles and abstracts, and 5 articles[1-3,6,15] were included. Only clinical trial reports, and no systematic review, meta-analysis or guideline, were found. Of the clinical trial reports, one[2] was excluded, as its investigation covered comparison of resin with resin-modified GIC (Vitremer), instead of conventional GIC. All accepted trial reports were published in 2012.

Figure 3

Flow diagram of article selection

Of the four accepted trial reports, only one[15] was identified through use of the search strategy of the previous systematic review update[9]; two were found through use of the additional string of search terms: "fissure sealant AND glass ionomer AND resin" in PubMed[1,6] and one was identified during ad hoc hand search[2]. The discrepancy between search results from the original[8] and additional search strategies was considered to be one signal according to criteria C (Figure 1).

Data analysis and signal identification

From the four accepted trial reports[1,3,6,15], ten separate datasets (DS) could be extracted:

Three datasets: "4-DS 1 new"[6], "5-DS 2 new"[3] and "6-DS 3 new"[15], from three trials related to comparison of the effect size of resin with that of GIC-based fissure sealants after 6 months (Figure 4);

Figure 4

Meta-analysis to key finding: Caries on sealed teeth/6 months follow-up period n=Number of sealed teeth with caries; N=Number of teeth evaluated; RR=Risk ratio; CI=Confidence interval; DS=Dataset number; Original=Data from original systematic review; New=Data from new identified trials

Three datasets: "3-DS 1 new"[6], "4-DS 2 new"[15] and "5-DS 3 new"[3], from three trials related to the effect size of GIC-based fissure sealants in comparison to resin after 12 months (Figure 5);

Figure 5

Meta-analysis to key finding: Caries on sealed teeth/12 months follow-up period n=Number of sealed teeth with caries; N=Number of teeth evaluated; RR=Risk ratio; CI=Confidence interval; DS=Dataset number; Original=Data from original systematic review; New=Data from new identified trials

Four datasets: "5-DS 1 new"[1], "6-DS 2 new"[6], "7-DS 3 new"[3] and "8-DS 4 new"[15], from four trials, related to the effect size of GIC based fissure sealants in comparison to resin after 24 months (Figure 6).

Figure 6

Meta-analysis to key finding: Caries on sealed teeth/24 months follow-up period n=Number of sealed teeth with caries; N=Number of teeth evaluated; RR=Risk ratio; CI=Confidence interval; DS=Dataset number; Original=Data from original systematic review; New=Data from new identified trials

These new datasets were pooled together with the corresponding datasets of the last systematic review update[9] in three fixed-effect meta-analyses for sealed permanent teeth after 6, 12 and 24 months (Figures 4, 5 and 6, respectively).

The 6- and 12-month results do not suggest significantly different outcomes for the pooled results of the original datasets on their own: RR 1.81; 95% CI 0.39-8.37; p=0.45 and RR 2.61; 95% CI 0.61-11.14; p=0.19, respectively, and thus do not represent a signal in terms of criteria B1. Nonetheless, the point estimate of the 12-month results changed from RR 2.61 to 1.66, constituting a signal in line with criteria B2 (Figure 1).

The 24-month result (Figure 6) of RR 1.51; 95% CI 1.00-2.28; p=0.05 is distinct from the original result RR 2.10; 95% CI 1.23-3.58; p=0.0006 that suggested a statistically significant difference between the two types of interventions, in favour of resin sealants. Even though the p-value of 0.05 still suggests borderline significance, the lower confidence level of 1.00 for the pooled result indicates no difference between the two interventions and may thus meet signal criterion B1. In addition, the risk ratio (RR) change from the original 2.10 to 1.51 represents a change in relative effect size of over 50% and thus meets signal criterion B2 (Figure 1).

Besides the identified signal types B1, B2 and C, no further signals were identified, and the result was summarized in Figure 7.

Figure 7

Signal search results

Literature search	Articles found	Signal criteria	Signal found	Details of key-finding	Remarks
Systematic review/meta-analysis /guidelines	No	A - C	No
Individual trials	Yes	A1	No
		A2	No
		A3	No
		A4	No
		A5	No
		A6	No
		B1	Yes	Lower confidence limit of the pooled Risk ratio after 24 months = 1.00 (Figure 5)	This result contrasts with the Lower confidence limit of the pooled Risk ratio from the original results = 1.23
		B2	Yes	Pooled Risk ratio after 24 months = RR 1.51 (Figure 5)	This result contrasts with pooled Risk ratio from the original results = RR 2.10
				Pooled Risk ratio after 12 months = RR 1.66 (Figure 4)	This result contrasts with pooled Risk ratio from the original results = RR 2.61
		C	Yes	Identification of only one out of four accepted trial reports relevant to the systematic review question on the basis of the search strategy from the last systematic review update

RR=Risk ratio

DISCUSSION

This study aimed to demonstrate the application of the modified Ottawa method by establishing the update need of a systematic review.

A potential limitation of this method is that it is still fairly new. The effectiveness of this method has been compared to that of a second method, the RAND method[13]. In contrast to the former, the RAND method relies on the combination of external expert opinion and an abbreviated literature search[4,13]. This investigation involved identification of update needs in four selected systematic reviews[4,13]. The result showed substantial agreement (kappa 0.64, 95% CI 0.45-0.83) between the two methods[4,13] and suggests that the modified Ottawa method is an effective tool for identifying update needs of original systematic reviews. However, the comparison of the modified Ottawa method with the RAND method did not include actual systematic review updates and the need for further research was emphasized by its authors[4,13].

In this study, the modified Ottawa method assisted, within a relatively short study-period (between April and May 2012), in the identification of new clinical evidence relevant to the systematic review question after the previous systematic review update[9]. The existence of such evidence would have remained obscure within the context of this systematic review if a general 24-month period, recommended for systematic review updates by the Cochrane collaboration[5], had been followed. This result is in contrast to that reported in a recently published systematic update on the failure rate of atraumatic restorative treatment (ART) versus that of amalgam fillings[10]. In this analysis, despite the update's extended search strategy[10], no trial report was identified three years after the cut-off date of the original systematic review[11]. These >2-year-results, as well as the <2-year-results of this study call into question the feasibility of using a two-year approach as a guide for systematic review updates and suggest that application of a priority approach, i.e., by use of the modified Ottawa method, would be more suitable.

This study further aimed to answer the question as to whether the existing systematic review conclusions regarding the caries preventive effect of GIC versus resin pit and fissure sealants are still current and whether a new update is needed. In this context, it has to be emphasized that the aim of the modified Ottawa method is not to provide clinically relevant review updates in an abridged form. Instead, it only assists in the identification of signals that may indicate whether or not an - often cost-intensive and time-demanding/consuming - update of an existing systematic review is indeed needed[4].

This study identified three quantitative and one qualitative signal. The first quantitative (B1) signal shows a change in the 95% confidence interval of pooled datasets relevant to the occurrence of caries on sealed surfaces on permanent teeth after 24 months (Figure 6). The lower confidence level of datasets included in the previous systematic review update[9] extended far into positive values (1.23), thus suggesting a statistically significant higher caries occurrence on tooth surfaces sealed with conventional GIC than on those sealed with resin 24 months after placement. However, the new result (1.00) indicates no caries occurrence difference between the two types of sealant. This observation is in line with the observed trend, previously established through cumulative meta-analysis, towards a Risk ratio (RR) of 1.00[9]. Such trend gave higher plausibility to the hypothesis of "no difference" between GIC and resin sealants than to the alternative hypothesis of 'resin is superior to GIC'[9]. While the overall conclusion of the previous systematic review update was in favour of the former hypothesis, the original 24-month data remained in contradiction to such conclusion. The identified B1 signal indicates an alignment of the 24-month data results with the overall systematic review conclusion. Therefore, although a clear signal was identified, this signal does not indicate a change in the overall systematic review conclusion and thus may not be indicative of the need for a systematic review update. It does, however, suggest that the new published evidence basically confirms the current view that no difference in clinical performance exists between GIC and resin-based fissure sealants.

The first identified B2 signal (Figure 7), presenting a >50% change of the point estimate for the 24-month data, further confirms the B1 signal and, thus, also may not be regarded as contributive to the indication of an update need. The second B2 signal presents a >50% point estimate reduction, from RR 2.61 to RR 1.66, after 12 months. However, the 95% confidence intervals, as well as the p-values of both point estimates do not indicate a change in statistical significance. Therefore, both results remain within limits of the overall systematic review conclusions. The change of the point estimate itself may be regarded as a shift towards the RR 1.00 value of no effect difference, thus reconfirming the previously established trend[9] of higher plausibility for the hypothesis of "no difference" between GIC and resin sealants.

The established C-Signal (Figure 7) raises concerns regarding the feasibility of the search strategy applied in the systematic review, so far. In addition, the scope of the previous systematic review update included a search of the English literature only, and thus excluded all relevant non-English evidence that may exist regarding this topic. However, the strength of this signal in indicating the need for a systematic review update may have been reduced because an extended search strategy in the English literature only generated results confirmatory to the overall review conclusion. Furthermore, it has been suggested that exclusion of non-English trial reports has little effect on summary treatment effect estimates[8,12] and this appeared to be confirmed in the recent update, without language restrictions, of another systematic review by the authors[10].

CONCLUSION

In conclusion, this study demonstrated the modified Ottawa method as an effective tool in establishing the update need of the chosen systematic review. In addition, it was established that the conclusions of the systematic review, in relation to the caries preventive effect of GIC versus resin based fissure sealants, are still current and that no update of this systematic review was warranted at date of application of the modified Ottawa method.