Quality Assessment of Published Articles in Iranian Journals Related to Economic Evaluation in Health Care Programs Based on Drummond's Checklist: A Narrative Review.

Health economic evaluation research plays an important role in selecting cost-effective interventions. The purpose of this study was to assess the quality of published articles in Iranian journals related to economic evaluation in health care programs based on Drummond's checklist in terms of numbers, features, and quality. In the present review study, published articles (Persian and English) in Iranian journals related to economic evaluation in health care programs were searched using electronic databases. In addition, the methodological quality of articles' structure was analyzed by Drummond's standard checklist. Based on the inclusion criteria, the search of databases resulted in 27 articles that fully covered economic evaluation in health care programs. A review of articles in accordance with Drummond's criteria showed that the majority of studies had flaws. The most common methodological weakness in the articles was in terms of cost calculation and valuation. Considering such methodological faults in these studies, it is anticipated that these studies would not provide an appropriate feedback to policy makers to allocate health care resources correctly and select suitable cost-effective interventions. Therefore, researchers are required to comply with the standard guidelines in order to better execute and report on economic evaluation studies.

What’s Known

Recent years have seen a rise in the publication of studies on the economic evaluation of health care.

In Iran, decisions regarding medical technology, pricing medicines, and reimbursement rules are made without considering economic evaluation and it seems that many decisions are based on experts’ opinion and previous experiences.

What’s New

Economic evaluation studies in Iranian journals fail to inform policymakers in choosing cost-effective interventions.

Significance should be attached to health economic evaluation research in Iran, and the quality of this research should be increased by implementing standard guidelines.

Introduction

Economic evaluation compares the costs and consequences of two or more health interventions to achieve the best choice. Its overall objective is to maximize the benefits due to resource constraints.[1] The cost-effectiveness analysis can be applied as a guide to rank priority setting and make a rational decision when introducing drugs or new health technologies.[2]

In recent years, there has been an upsurge of using economic evaluation in health care resource allocation and decision-making across the developed and Asian countries. Countries such as Australia, England, Wales, and Sweden have formally adopted the use of economic evaluation in pricing drugs, development of clinical practice guidelines and communication with health professionals. In Asian countries, although countries such as South Korea, China, Thailand and Taiwan use the methods of economic evaluation, among them, only South Korea has adopted economic evaluation as a formal tool for medical decision making.[3]

Iran is a country with a middle-income and the gross national income (GN) per capita in 2013 was US $15,600. Approximately 6.7% of the gross domestic production (GDP) is spent on health care.[4] Over the past few decades, health care expenditure in Iran has risen for many reasons, including aging population, increased prevalence of noncontiguous and chronic diseases, increased prescription and taking drugs, and improved usage of new medical technologies.[5,6] In recent years, studies published in Iran on economic evaluation of health care have increased. Nevertheless, decisions regarding medical technology, pricing of medicines, and reimbursement rules are taken without economic evaluation considerations. It seems that many decisions are based on experts’ opinion and experience.[7]

The present study examines the quality of the structure of all articles published in Iranian journals (Persian and English) dealing with the economic evaluation of health care during 1990 and 2014. We aimed at assessing the quality of published articles related to economic evaluation in health care programs based on Drummond’s checklist. The results of this research would assist researchers and decision-makers of the health system to properly design, implement, and present the results of economic evaluation studies.

Articles that had carried out full economic evaluation, including cost-effectiveness analysis, cost-utility analysis, and cost-benefit analysis were included. In this review, the methodological quality of the structure of articles was analyzed by Drummond’s standard checklist (table 1). Drummond’s checklist is used to assess the methodological quality of full economic evaluations with limited use of decision-analytic modeling (such as Markov modeling). To conduct the critical assessment of the methodological quality of economic modeling, Phillips checklist can also be used.[8] Given that only one article conducted Markov model in the present study, we therefore, used Drummond’s checklist.[1] All items in the checklist were scored according to positive, negative, or unclear.

Table 1

Drummond’s criteria for the assessment of economic evaluation studies

Row	Criteria
1	Was a well-defined question posed?
2	Was a comprehensive description of the competing alternatives offered?
3	Was the evidence of the effectiveness of the program offered?
4	Were all important and relevant costs and consequences identified?
5	Were all important and relevant costs and consequences measured accurately?
6	Were all important and relevant costs and consequences have been properly valued?
7	Were the costs and consequences adjusted for different times?
8	Was an incremental analysis of costs and consequences of competing alternatives done?
9	Was the effect of uncertainty (sensitivity analysis) investigated in estimating the costs and consequences?
10	Were the presentation and analysis of all issues related to users of the results included?

Based on the inclusion criteria, we reviewed 27 articles that fully covered economic evaluation.[9-35] The results showed that, of all the 27 studies, 12 (45%) were published in Persian. The number of studies in English related to health economic evaluation research has increased remarkably from 2011 to 2013 and the number of Persian articles reached a peak in 2013 (figure 1). In tables 2 and 3, the quality assessment of studies written in both English and Persian are reported. The results showed that 23 out of 27 studies (85%) in the fifth Drummond’s criteria, 22 studies (81%) in the sixth criteria, 21 studies (77%) in the tenth criteria, 19 studies (70%) in the fourth, and 18 studies (66%) in the seventh criteria had flaws. In table 4, the number of flaws in articles based on Drummond’s criteria is listed. Overall, the results indicated that in 6 criteria (second, third, fifth and eighth to tenth) the number of flaws in Persian articles were more than the English articles. In addition, the most common defects in Persian and English articles were in the fifth and sixth criteria, respectively. The most common methodological weakness of the articles was in the measurement and valuation of costs. Considering their perspective, only two articles performed full identification, measurement and valuation of costs.[15,28] The economic characteristics of the studies are presented in table 5. The results showed that among the 27 reviewed articles, 1 article performed the cost-benefit analysis[32] and the number of articles in connection with the cost-effectiveness and cost-utility analysis were 17 and 9, respectively. Among the studies that used cost-utility analysis (i.e. combined consequences such as quality-adjusted life years (QALYs) or disability-adjusted life years (DALYs) to measure consequences), five studies used QALYs indicator,[18,19,28,31,35] and four studies used DALYs indicator,[20-22,33] in order to measure effectiveness. Most studies were carried out on health care services such as medical and pharmaceutical interventions, 8 studies evaluated screening programs,[12,14,15,18,19,21,22,27] and only one study evaluated thalassemia prevention and care.[9] Figure 2 shows the number of full economic evaluations publications in the Iranian journals by disease categories. Based on the figure, more economic evaluation studies were conducted on endocrine, nutritional, and metabolic diseases. In terms of study design, most studies were observational and two were randomized controlled trial.[24,34] Regarding the model type, 9 studies used the decision tree model[9,15,17-19,24,26,31,33] and only one article used Markov model.[28] In the study conducted by Sari, the decision tree was not drawn.[31] Among the reviewed articles, 7 studies were conducted from the perspective of the provider,[10,13,21,23,27,33,34] 6 from the perspective of the society,[15,18,19,24,28,30] one from the perspective of the patient,[17] one study used two perspectives (patient and provider),[26] and the remaining studies lacked a perspective. The findings of this study also show that 66% of the studies neither use the discount rate nor paid attention to this issue. Moreover, in the studies that used the discounted rate, no sensitivity analysis was performed on the discounted rate. Furthermore, most of the studies clearly stated the time horizon of the study. The time horizon of most studies was less than one year and two studies used a time horizon of 20 years,[11,30] and one study had used lifetime.[28]

Figure 1

The number of full economic evaluations published in English and Persian per year in the Iranian journals. From 2011 to 2013, the number of English studies has risen considerably and in 2013 the number of Persian articles reached an apex.

Table 2

Assessment of methodological quality of English articles based on Drummond’s checklist

Row	Study	1-Was a welldefined question posed?	2-Was a comprehensive description of the competing alternatives offered?	3-Was the evidence of the effectiveness of the program offered?	4-Were all important and relevant costs and consequences identified?	5-Were all important and relevant costs and consequences measured accurately?	6-Were all important and relevant costs and consequences have been properly valued?	7-Were the costs and consequences adjusted for different times?	8-Was an incremental analysis of costs and consequences of competing alternatives done?	9-Was the effect of uncertainty (sensitivity analysis) investigated in estimating the costs and consequences?	10-Were the presentation and analysis of all issues related to users of the results included?
1	Sari31	N/A	+	+	N/A	N/A	N/A	+	+	+	N/A
2	Nikfar28	+	+	+	+	+	+	+	+	+	+
3	Mehrazmy24	+	+	+	-	+	+	-	+	+	+
4	Hatam19	+	+	+	-	N/A	N/A	+	+	+	N/A
5	Keshtkaran22	N/A	+	+	N/A	N/A	N/A	+	+	+	N/A
6	Soleymani34	+	+	+	+	N/A	N/A	-	+	N/A	-
7	Gharibnaseri17	+	+	+	-	-	-	-	+	+	N/A
8	Rahbar29	N/A	+	+	N/A	N/A	N/A	-	-	-	-
9	Aghili10	+	+	+	-	-	-	-	-	+	N/A
10	Shamshiri33	+	+	+	+	-	N/A	+	+	+	+
11	Naghipour25	N/A	+	+	N/A	N/A	N/A	-	-	-	N/A
12	Allameh11	N/A	+	+	N/A	N/A	N/A	-	-	-	-
13	Divsalar13	+	+	+	+	+	N/A	-	+	N/A	N/A
14	Yaghoubi35	N/A	+	+	N/A	N/A	N/A	-	+	-	+
15	Shajar32	N/A	-	+	N/A	N/A	N/A	-	-	-	-

Yes (+), No (-) and unclear (N/A)

Table 3

Assessment of methodological quality of Persian articles based on Drummond’s checklist

Row	Study	1-Was a welldefined question posed?	2-Was a comprehensive description of the competing alternatives offered?	3-Was the evidence of the effectiveness of the program offered?	4-Were all important and relevant costs and consequences identified?	5-Were all important and relevant costs and consequences measured accurately?	6-Were all important and relevant costs and consequences have been properly valued?	7-Were the costs and consequences adjusted for different times?	8-Was an incremental analysis of costs and consequences of competing alternatives done?	9-Was the effect of uncertainty (sensitivity analysis) investigated in estimating the costs and consequences?	10-Were the presentation and analysis of all issues related to users of the results included?
1	Ghaderi16	N/A	+	+	N/A	N/A	N/A	-	+	+	N/A
2	Keshtkaran21	+	+	+	+	N/A	N/A	+	+	+	N/A
3	Hatam18	+	+	+	-	N/A	N/A	+	+	+	N/A
4	Keshtkaran23	+	+	+	+	N/A	+	-	+	+	+
5	Asefzade12	N/A	-	-	N/A	N/A	N/A	-	-	-	-
6	Rasuli30	+	+	+	+	N/A	+	+	-	+	N/A
7	Foruzanfar15	+	+	+	+	+	+	-	-	+	+
8	Farajzadegan14	N/A	+	+	N/A	N/A	N/A	-	-	N/A	N/A
9	Nakhaee27	+	+	+	-	-	-	-	-	N/A	N/A
10	Abolghasemi9	N/A	+	+	N/A	N/A	N/A	-	+	-	N/A
11	Karimi20	N/A	+	+	N/A	N/A	N/A	+	-	-	N/A
12	Nakhaee27	+	+	+	-	N/A	N/A	-	-	N/A	N/A

Yes (+), No (-) and unclear (N/A)

Table 4

The number of flaws in Persian and English articles based on Drummond’s criteria

Row	Drummond’s criteria	English articles		Persian articles
		N	%	N	%
1	Was a well-defined question posed?	7	46	5	41
2	Was a comprehensive description of the competing alternatives offered?	2	13	2	16
3	Was the evidence of the effectiveness of the program offered?	0	0	1	8
4	Were all important and relevant costs and consequences identified?	11	73	8	66
5	Were all important and relevant costs and consequences measured accurately?	12	80	11	91
6	Were all important and relevant costs and consequences have been properly valued?	13	86	9	75
7	Were the costs and consequences adjusted for different times?	10	66	8	66
8	Was an incremental analysis of costs and consequences of competing alternatives done?	5	33	7	58
9	Was the effect of uncertainty (sensitivity analysis) investigated in estimating the costs and consequences?	7	46	6	50
10	Were the presentation and analysis of all issues related to users of the results included?	11	73	10	83

Table 5

Economic features of included studies

Feature	English journal		Persian journal		All
	N	%	N	%	N	%
Type of economic evaluation
CEA*	10	67	9	75	19	70
CUA*	4	27	3	25	7	26
CBA*	1	7	0	0	1	4
Study design
RCT *	2	13	0	0	2	7
Observational	7	47	8	67	15	56
Decision tree	5	33	4	33	9	33
Markov model	1	7	0	0	1	4
Perspective evaluated
Social	3	20	3	25	6	22
Provider	4	27	3	25	7	26
Patient	1	7	0	0	1	4
Patient and provider (mixed)	0	0	1	8	1	4
Not stated	7	47	5	42	12	44
Type of sensitivity analysis
One-way	6	40	5	42	11	41
Multi-way	2	13	1	8	3	11
Probabilistic	0	0	0	0	0	0
Not performed	7	47	6	50	13	48
Time horizon
≤1 year	7	47	9	75	16	59
1-10 years	2	13	1	8	3	11
Over 10 years	2	13	1	8	3	11
Not specified	4	27	1	8	5	19
Type of outcome
QALY/DALY*	6	40	3	25	9	33
Intermediate (physiological, functional, etc.)	10	67	10	83	20	74
Money units	0	0	0	0	0	0
Discount rate
3%	3	20	3	25	6	22
5%	0	0	1	8	1	4
>5%	2	13	0	0	2	7
Not stated	10	67	8	67	18	67

CEA: Cost-effectiveness analysis; CUA: Cost-utility analysis; CBA: Cost-benefit analysis; RCT: Randomized controlled trial; QALY: Quality-adjusted life years; DALY: Disability-adjusted life years

Figure 2

The number of full economic evaluations published in the Iranian journals based on disease categories. “Endocrine, nutritional and metabolic diseases” were the most common diseases category covered by the economic evaluation studies.

Discussion

Economic evaluation evidence can play a prominent role in making decisions about resource allocation. Based on Drummond’s checklist, the present review examined the quality of the structure of all articles published in Iranian journals dealing with the economic evaluation of health care during 1990 and 2014. The results show that many published articles did not comply with the international standards for economic evaluation guidelines and had major methodological flaws. Such poor quality of economic evaluation studies is not unique to Iran. Studies from other developing and developed countries suffer from similar methodological faults.[36-55] This is possibly due to a limited number of health economics specialists as well as the lack of knowledge by clinicians, policy makers, and managers in the field of economic evaluation.

Hosseinpour et al. showed that financial and administrative managers of hospitals in Iran do not have the appropriate level of knowledge in the field of health economics.[56] Lack of specialized journals on health economics could also have exacerbated the situation. Neumann et al. reported that medical journals had a higher risk of reporting economic studies of poor quality.[57] Determining the viewpoint of economic evaluation studies is important and has an effect on both costs and consequences.[58] However, only 55% of the studies (15 out of 27) had a viewpoint. It shows that many authors are not aware of the importance of determining a viewpoint and its effect on costs and consequences. Among the studies that determined a viewpoint, many did not measure the costs associated with the viewpoint of the study. For example, in Mehrazmy and Hatam’s study, although the viewpoint of the study was the society, but indirect costs were not measured.[18,19,24] Moreover, in studies with a viewpoint on service providers, the capital expenditures, equipment, and overhead costs had not been measured. Therefore, it can be argued that one of the major flaws in these studies is the lack of transparency in the recognition, measurement, and valuation of costs. Our results show that 44% of the studies did not calculate the incremental cost-effectiveness ratio. A number of studies reported the mean of cost-effectiveness ratio (total cost divided by total effectiveness). However, in turn, this could affect the main conclusions of a study and would lead to biased results. Another major weakness in economic evaluation studies in Iranian journals is the limited use of sensitivity analyses, to the extent that 48% of the studies (13 out of 27) had not conducted sensitivity analyses. Sensitivity analyses in economic evaluation studies are important. It measures the change rate of the incremental cost-effectiveness ratio in comparison with changes in parameters. It is used when dealing with the effect of uncertainty in results and in the generalizability of results.[1] Another weakness of the studies is the authors’ lack of attention to the generalizability of results to the national level or to other settings, to the extent that only 25% of the studies had paid attention to this topic. Moreover, the results showed that cost-effectiveness analysis is the most common method for economic evaluation in published articles. A possible reason is that obtaining data on intermediate consequences (physiological consequences) is easier and less costly in comparison with the final consequences (QALYs or DALYs). Similarly, the results of Teerawattananon’s study showed that cost-effectiveness analysis was the most common type of economic evaluation in Thailand.[53] Considering limited resources, it is expected that economic evaluation studies must be conducted on interventions, which have a significant effect on the health of the population. However, the results of this study showed that most economic evaluation studies were carried out on diseases that are of no concern to the Iranian health system. For example, more than 40% of the burden of the diseases in our country is due to injuries and mental illnesses[59] However, none of the studies covered the economic evaluation analysis of injuries and mental illnesses. Moreover, the findings of this study suggest that researchers paid more attention to cure rather than preventive health care. These findings are consistent with studies by Teerawattananon, Neumann, and Catalá-López.[53,57,60]

There are a couple of limitations in the present study. We only reviewed studies published in the national journals of Iran. Articles published in international journals regarding the economic evaluation of health care were excluded from the study since they had a better quality compared with their Iranian equivalent. Another limitation of this review was that unpublished studies were not identified by our literature search. These were typically reports from the department of health technology assessment and health economics department of the ministry of health and medical education, medical schools, reports from health insurance organizations as well as pharmaceutical companies and academic thesis.

Conclusion

The results showed that the economic evaluation literature in Iran is still in its infancy and many of these studies suffer from common methodological faults. Furthermore, the majority of these studies did not cover the main health concern in Iran. Therefore, researchers are required to comply with the standard guidelines in order to better execute and report on economic evaluation studies. Recommended guidelines are Drummond’s checklist, consolidated health economic evaluation reporting standards (CHEERS checklist) and the grading system for the quality of cost-effectiveness studies (Chiou’s system).