Quality of published Iranian medical education research studies: a systematic review.

BACKGROUND: Research in medical education has been paid more attention than before; however the quality of research reporting has not been comprehensively appraised. To evaluate the methodological and reporting quality of Iranian published medical education articles.
METHODS: Articles describing medical students, residents, fellows or program evaluation were included. Articles related to continuing medical education or faculty development, review articles and reports, and studies considering both medical and nonmedical students were excluded. We searched MEDLINE through PubMed in addition to major Iranian medical education search engines and databases including Scientific Information Database (SID) from March 2003 to March 2008. The Medical Education Research Quality Index (MERSQI) scale and the Consolidated Standards of Reporting Trials (CONSORT 2001) were used for experimental studies and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) was utilized for observational studies.
RESULTS: Ninety five articles were found to be related to the medical education research in Iran including 16 (16.8%) experimental studies. Total MERSQI scores ranged between 3.82 and 13.09 with the mean of 8.39 points. Mean domain scores were highest for data analysis (1.85) and lowest for validity (0.61). The most frequently reported item was background (96%) and the least reported was the study limitations (16%).
CONCLUSION: The quality of published medical education research in Iran seems to be suboptimal.

Introduction

Medical education research has been improved extensively during recent years (1-2) as evidenced by several national and international journals and the increasing number of education reports published in scientific journals (3-4). Medical education research is frequently criticized for lack of generalizability and rigor by stakeholders such as professional organizations, journal editors, universities, teaching institutions and education researchers, maintaining that the quality of medical education research is inadequate and greater methodological rigor is needed (5-9).

Evaluating and improving methodological quality of the medical education research should be a central consideration to ensure that current educational efforts for training future physicians can improve patient care (10). A few methods and scales have been proposed for quality assessment in medical education. Among these, Medical Education Research Study Quality Instrument (MERSQI) (11) has been used with strong content, criterion, and predictive validity as well as inter-rater and intra-rater reliability. This metric has been shown to be a useful tool for educators, reviewers, and journal editors to assess the quality of medical education research (11-13). As for evaluating the quality of reporting, Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) and Consolidated Standards of Reporting Trials (CONSORT) statements are used as the most comprehensive guides and standards for reporting observational and interventional studies, respectively (14-15).

In Iran, medical students, residents, and fellows are now being trained in more than 45 medical universities and affiliated teaching hospitals (16-17). Following establishment of medical educational development centers in these universities, medical education research has been improved quantitatively during recent years (18). However, the quality of these published articles and reports have not been appraised and evaluated from methodological point of view.

Therefore, we conducted this systematic review with two aims: a) to explore the quality of reporting of experimental and not experimental research studies in medical education; b) to evaluate the methodological quality of medical education research studies.

Methods

The study protocol was approved by the research council of the Medical Education and Development Center of Iran University of Medical Sciences.

Design

This study is a systematic review of the methodological and reporting quality of the published articles in the field of medical education in Iran.

Search strategy

Published articles related to medical education research conducted in Iran were included in this review. Studies published between March 2003 and March 2008 were searched by two of authors (PG and FS) in MEDLINE through PubMed (accessed 15 October 2009). A search on the Scientific Information Database (SID) in which all Iranian journals are indexed was also conducted (accessed 15 October 2009). Combinations of the words related to medical education (medical education, teaching, and learning) and learners’ level (student, intern, resident and fellow) were used. Three major Iranian medical education journals (Strides in Development of Medical Education, Journal of Medical Education, and Iranian Journal of Medical Education) were hand-searched to identify additional studies published during the time interval. For the studies published in 2 different journals or in different languages, only the first publication was included regardless of the language or the journal.

Study selection

Two reviewers (PG and FS) independently reviewed the titles and abstracts of the 762 retrieved articles and selected the eligible studies for inclusion. Medical education research was defined as any research study pertaining to medical students, residents, fellows, faculty members, or program evaluation. Studies with cross-sectional, case-control, cohort, and post-test only designs, as well as uncontrolled trials, non-randomized trials, and randomized-controlled trials which had been conducted in Iran were included. Articles related to continuing medical education or faculty development, as well as review articles and reports were excluded. Studies considering both medical and nonmedical students (nursing, allied medicine, dentistry, etc.) were also excluded. When titles and abstracts were not sufficient for determining the eligibility, the full articles were identified and reviewed. Any disagreements between the reviewers were resolved by consensus.

Study review and data extraction

Full-texts of relevant studies were obtained and reviewed by reviewers (PG and FS). The viewers were not blind to the study location, authors affiliation or citation information. A standardized form was designed to extract the data from included studies. The following information was extracted from each article: location of the study, the highest academic degree of the authors, source of funding, study design, study population, sample size, outcomes, and the axis of the study. We categorized the studies into 3 axes of teaching, learning, and evaluation.

Methodological quality assessment

Medical Education Research Study Quality Instrument (MERSQI) (11) was used for assessing the methodological quality of included studies. This instrument can assess the methodological quality of medical education studies in 6 domains including study design, sampling, type of data, validation of evaluation instrument, data analysis, and outcomes measured. By scoring in these domains, each medical education research can be scored between 6 and 18; 6 showing the lowest quality and 18 for the highest quality. Strong content, criterion, and predictive validity as well as inter-rater and intra-rater reliability have been reported previously for MERSQI scores (11-13).

Reporting quality assessment

Quality of reporting was assessed using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist (14) and the 2001 revision of the Consolidated Standards of Reporting Trials (CONSORT) statement (15) for observational and experimental studies, respectively.

Data analysis

Quantitative synthesis of data and meta-analysis was not possible to perform due to the heterogeneity of aims, scopes, designs, and population studied. All statistical analyses were performed using the Statistical Package for Social Sciences, version 15 for Windows™ (SPSS® Inc., Chicago, IL). Data were summarized as mean ±SD and count (percent) for continuous and categorical variables, respectively. The normality of data was assessed using One-Sample Kolmogorov-Smirnov test. For non-normal data, median and range was used. MERSQI score calculation was used based on the reported method by Reed et al (11). Student's t-test and one-way analysis of variances (ANOVA) was used when comparing continuous variables (i.e. MERSQI scores) between two and more than two subgroups, respectively. All tests of significance were two-tailed and considered to be significant at P value less than 0.05.

Results

Study search and selection results

Overall, a total of 856 articles were retrieved from MEDLINE, SID and hand search which is illustrated in figure 1. Screening of the titles and abstracts reduced this number to 174 articles. For these articles, full-texts were obtained and reviewed. Finally, 95 articles were found to be eligible for inclusion. Using time limits were not possible in SID search engine, thus we manually excluded the articles published out of the selected time period.

Fig.1

Literature search and study selection process for identifying Iranian medical education articles published between 2003 and 2008.

Characteristics of included studies

Table 1 summarizes the characteristics of the included studies. The articles were published in 6 international journals indexed in MEDLINE, 3 exclusive Iranian medical education journals, and 19 Iranian non-medical education journals. Sample size of included studies ranged from 11 to 1370 with a median of 75 participants.

Table 1

Characteristics of included studies

Characteristics		Number (%)
Source of article	MEDLINE	7 (7.3)
	Iranian medical education journals	60 (63.1)
	Iranian general journals	28 (29.5)
Study design	Observational	79 (83.2)
	Experimental	16 (16.8)
Centers	1	87 (91.6)
	2	3 (3.2)
	>2	5 (5.2)
Axis	Teaching	44 (46.3)
	Learning	30 (42.1)
	Assessment/evaluation	11 (11.6)
Highest academic rank of authors	Professor	17 (17.9)
	Associate professor	24 (25.3)
	Assistant professor	36 (37.9)
	Instructor	2 (2.1)
	Student	1 (1)
	Not mentioned	15 (15.8)
Funding source	Ministry of health and medical education	3 (3.2)
	Academic	17 (17.9)
	Research center	2 (2.1)
	Not mentioned	73 (76.8)
Number of study participants	<50	29 (30.5)
	50-100	27 (28.4)
	>100	39 (41.1)
Population studied *	Basic science students	18 (18.9)
	Physiopathology students	8 (8.4)
	Clinical clerkship students	36 (37.9)
	Interns	42 (44.2)
	residents	16 (16.8)
	Fellowship	0
	Faculty members	13 (13.7)
	Other **	3 (3.3)

* Percents do not add up 100 due to multiple target populations in some studies

** Including morning reports, medical faculties, exams or multiple choice questions each with one study.

Teaching with 44 (46.3%) studies was the most common axis of the medical education articles and assessment/evaluation was the least with 11 (11.6%). The majority of studies evaluated interns (42 studies) and clerkship students (36 studies) as the main target population. None of the studies assessed fellowship training. The highest academic rank of the authors was assistant professor in 36 (37.9%) articles and associate professor and professor were next with 24 (25.3%) and 10 (10.5%), respectively. In 23 articles, the academic rank for authors was not mentioned. Financial support was reported only in 22 articles in which 17 (17.9%) had university support, 3 (3.2%) were sponsored by the Iranian Ministry of Health and Medical Education and for the other 2 articles a research center support was provided.

Methodological quality

The majority of studies (79, 83.2%) had a single-group cross-sectional or single group post-test only design and only 5 studies (5.3%) were designed as a randomized-controlled trial. Almost all studies had been conducted in one center (87, 91.5%) and only 5 studies were multi-institutional. Among 56 studies with applicable response rate, 18 (32.2%) failed to report or had less than 50% response rate. In 62 studies (65.3%), outcome assessments were based on the study participants’ assessment while objective measurements were performed only in 33 (37.4%) studies. Only 15, (15.8%) studies reported the content validity evidence for the evaluation instrument they used for outcome assessment. Almost one third of the studies (32, 33.7%) had an inappropriate data analysis and among 63 studies with appropriate analysis, 13 studies (21%) used descriptive analysis only.

More than two thirds of studies (67, 70.5%) assessed satisfaction, attitude, opinion, perception, or general facts as an outcome. Twenty five (26.3%) assessed the knowledge or skills of the participants and in the remaining 3, participants behavior was assessed. None of the studies assessed any patients or healthcare system outcomes (Table 2).

Table 2

Methodological quality of Iranian medical education published articles between 2003 and 2008 based on the MERSQI Scale (N=95)

Domain		MERSQI item	Studies Number (%)	Item Mean ±SD	Domain Mean ±SD
Study design	Design	Single group cross-sectional or single group post-test only	79 (83.2)	1.18 ±0.49	1.18 ±0.49
		Single group pre-test & post-test	6 (6.3)
		Nonrandomized, 2group	5 (5.3)
		Randomized controlled trial	5 (5.3)
Sampling	No. of institutions studied	1	87 (91.5)	0.56 ±0.23	1.20 ±0.72
		2	3 (3.2)
		>2	5 (5.3)
	Response rate, %	Not applicable	41 (43.2)	1.12 ±0.46
		<50 or not reported	18 (18.9)
		50-74	5 (5.3)
		≥75	31 (32.6)
Type of data	Type of data	Assessment by study participant	62 (65.3)	1.69 ± 0.95	1.69 ± 0.95
		Objective measurement	33 (37.4)
Validity of evaluation instrument	Internal structure	Not applicable	20 (31.1)	0.52 ±0.50	0.61 ±0.70
		Reported	39 (41.1)
		Not reported	36 (37.8)
	Content	Not applicable	20 (21.1)	5 (5.3)	0.20 ±0.4
		Reported	15 (15.8)
		Not reported	60 (63.2)
	Relationships to other variables	Not applicable	20 (21.1)	0.05 ±0.22
		Reported	4 (4.2)
		Not reported	71 (74.7)
Data Analysis	Appropriateness of analysis	Data analysis inappropriate for study design or type of data	32 (33.7)	0.66 ± 0.47	1.85 ±1.36
		Data analysis appropriate for study design or type of data	63 (66.3)
	Complexity of analysis	Inappropriate	32 (33.7)	1.18 ±0.91
		Descriptive analysis only	13 (13.7)
		Beyond descriptive analysis	50 (52.6)
Outcomes	Outcomes	Satisfaction, attitudes, perceptions,opinions, general facts	67 (70.5)	1.16 ±0.26	1.16 ±0.26
		Knowledge, skills	25 (26.3)
		Behaviours	3 (3.2)
		Patient/health care outcome	0

Total MERSQI scores of the 95 included studies ranged between 3.82 and 13.09 with the mean ±SD of 8.39 ±2.28 points. The highest mean domain score was in data analysis domain with 1.85 ±1.36 while the validity of evaluation instrument domain showed the lowest MERSQI score (0.61 ±0.70). Although not statistically significant, the total MERSQI score was higher for experimental studies comparing to the observational studies (9.07 ±2.79 versus 8.26 ±2.16; p=0.019). The total MERSQI score was not significantly different between the teaching (8.19 ±2.42), learning (8.66 ±2.11), and assessment/evaluation (8.22 ±2.41) axes (p=0.63).

Reporting quality

Tables 3 and 4 summarize the frequency of each item in observational and experimental studies. Among 79 observational studies, 23 (29.1%) studies did not report their study design in the abstract and in 10 studies (12.7%), the reported design was incorrect according to the method section. In method section, participant eligibility was the most reported item (71 articles, 82.9%) and the sample size calculation method was the least (2 articles, 2.5%). In results section, slightly more than half of the studies (46, 58.2%) reported the descriptive data. Key results, limitations of the study and generalizability of the results were discussed in the discussion section of 72 (91.1%), 11 (13.9%) and 12 (15.2%) studies, respectively.

Table 3

Reporting quality of 79 observational Iranian medical education published articles between 2003 and 2008 based on the modified STROBE checklist

Paper Section	Item	Descriptor	Frequency (%)
Title and abstract	Study design	Yes	46 (58.2)
		No	23 (29.1)
		Incorrect*	10 (12.7)
Introduction	Background/rationale	Yes	75 (94.9)
		No	4 (5.1)
	Objectives	Yes	74 (93.7)
		No	5 (6.3)
Methods	Study design	Yes	50 (63.3)
		No	23 (29.1)
		Incorrect*	6 (7.6)
	Setting	Yes	66 (83.5)
		No	13 (16.5)
	Participants eligibility	Yes	71 (89.9)
		No	8 (10.1)
	Data sources/ measurement	Yes	68 (86.1)
		No	11 (13.9)
	Sampling method	Yes	51 (64.6)
		No	28 (35.4)
	Study size calculation	Yes	2 (2.5)
		No	40 (50.6)
		Not applicable	37 (46.8)
	Statistical methods	Appropriate	53 (67.1)
		Inappropriate**	12 (15.2)
		No	14 (17.7)
Results	Participants flow	Yes	1 (1.3)
		No	0
		Not applicable	78 (98.7)
	Descriptive data	Yes	46 (58.2)
		No	33 (41.8)
	Other analyses	Yes	2 (2.5)
		No	77 (97.5)
Discussion	Key results	Yes	72 (91.1)
		No	7 (8.9)
	Limitations	Yes	11 (13.9)
		No	68 (86.1)
	Interpretation	Yes	68 (86.1)
		No	11 (13.9)
	Generalizability	Yes	12 (15.2)
		No	67 (84.8)
Other Information	Funding	Yes	21 (22.6)
		Not mentioned	63 (79.7)

* Inappropriateness of the reported study design with method section or ambiguous designs reported (e.g. descriptive-analytic study)

** Inappropriateness of the statistical tests with variables, not using the statistical tests reported in method section, or not reporting the p values

Table 4

Reporting quality of 16 experimental Iranian medical education published articles between 2003 and 2008 based on the modified CONSORT 2001 checklist

Paper Section	Item	Descriptor	Frequency (%)
Title and abstract	Random allocation	Yes	4 (25)
		No	1 (6.3)
		Not applicable	11 (68.7)
Introduction	Background	Yes	16 (100)
		No	0
Methods	Participants (Inclusion/Exclusion)	Yes	16 (100)
		No	0
	Setting	Yes	10 (62.5)
		No	6 (37.5)
	Interventions	Yes	13 (81.2)
		No	3 (18.8)
	Objectives	Yes	15 (93.7)
		No	1 (6.3)
	Sampling method	Yes	8 (50)
		No	8 (50)
	Sample size calculation	Yes	2 (12.5)
		No	11 (68.8)
		Not applicable	3 (18.7)
	Randomization-Sequence generation	Yes	5 (31.3)
		No	5 (31.3)
		Not applicable	6 (37.5)
	Randomization-Allocation concealment	Yes	0
		No	5 (31.3)
		Not applicable	11 (68.8)
	Randomization-Implementation	Yes	0
		No	5 (31.3)
		Not applicable	11 (68.8)
	Blinding (masking)	Participant	1 (6.3)
		Outcome assessor	2 (12.5)
		Outcome assessor & analyst	1 (6.3)
		No	1 (6.3)
		Not applicable	11 (68.8)
	Statistical methods	Appropriate	10 (62.5)
		Inappropriate*	4 (25)
		No	2 (12.5)
Results	Participants flow	Yes	2 (12.5)
		No	4 (25)
		Not applicable	10 (62.5)
	Recruitment/Follow-up	Yes	8 (50)
		No	6 (37.5)
		Not applicable	2 (12.5)
	Baseline data	Yes	9 (56.3)
		No	7 (43.7)
	Numbers analyzed	Yes	8 (50)
		No	8 (50)
	Ancillary analyses	Yes	0
		No	16 (100)
Discussion	Interpretation	Yes	14 (87.5)
		No	2 (12.5)
	Generalizability	Yes	2 (12.5)
		No	14 (87.5)
	Overall evidence	Yes	11 (68.8)
		No	5 (31.3)
	limitations	Yes	5 (31.3)
		No	11 (68.8)
Other Information	Funding	Yes	6 (37.5)
		No	10 (62.5)

* Inappropriateness of the statistical tests with variables, not using the statistical tests reported in method section, or not reporting the p values

As for experimental studies, all 16 studies mentioned the eligibility criteria. Intervention description was not provided in 3 (18.8%) studies and sampling method was reported in only half of the studies. Randomization-sequence generation was reported only in half of the applicable articles and none of the studies reported the allocation concealment method. In results, baseline data was not provided in 7 (43.7%) of the experiments and half of the studies did not report the number analyzed in statistical analyses. Generalizability was the least reported item in the discussion section with only 2 (12.5%) studies reporting it.

Discussion

Although medical education research had been known as an important field of research in medical universities in Iran, to our best of knowledge, this systematic review is the first study aimed to assess the methodological quality of these reports. Our results revealed that the overall methodological quality is suboptimal and some important elements are not routinely reported in published papers. These findings are in line with other internationally reports.

Other studies in medical education have described suboptimal reporting methodological quality (11-13). For example Cook et al (19-20) in two systematic reviews showed that many essential elements of scientific reporting were frequently missing from articles describing medical education experiments, including a critical literature review, study design statement, definition of the comparison or control group.

The lowest MERSQI score was observed in the validity of evaluation instruments domain in our systematic review which is consistent with other studies (11-13) and confirms previous reports that studies rarely report validity assessments for their evaluation instruments (21-22). The most reported measure of validity observed in our review was internal structure while in other studies content validity was more frequently reported (19-20). One explanation for this difference is that authors and journal reviewers in Iran might pay more attention to the measures of internal validity (e.g. Cronbach’s alpha) than the description of how adequately items represent the content of the items.

Our systematic review demonstrated that only 17% of studies in this review used an interventional design, that only 8.5% have been conducted in more than one center, and that only 3.2% assessed the learner behaviors and none of them assessed the patient/health care outcomes which highlights the need to increase methodological rigor in medical education research. More interventional studies with control groups are needed to make comparisons meaningful and applicable in the field of medical education. Such interventional studies can produce high level of evidence for educators and help them select the best methods and approaches for training medical students. Multi-institutional research is essential in order to test the effects of educational interventions across educational sites and environments. What works in one university or faculty may not work in another depending on the culture and processes conducted in each site. In medical education research, a significant amount of studies are measuring trainee satisfaction and performance and relatively little attention is paid to the effect of medical education on patient or healthcare outcomes which are fundamental goals of training physicians (23).

The majority of studies evaluated interns (44.2%) and clerkship students (37.9%) as the main target population. Few studies assessed the residency training (16.8%) and none of them assessed the fellowship training, highlighting the gap in this field.

Financial support was reported only in 23.2% of the studies, showing the need for greater funding for medical education research. Reed et al (11) showed a significant association between funding and study quality (as measured by the MERSQI), providing evidence to support the call to increase funding for medical education research in the USA. Policy reform that increases funding support may promote high-quality medical education research (11, 24).

Evaluation of the reporting quality of the reviewed articles revealed that although peer-review process is performed in medical education journals, several essential elements of scientific reporting were still missing. This is consistent with previous studies (19) and highlights another weakness in medical education articles and their reviewing process for publication in journals. We noticed that in 13% of the studies, the implied study design was different from what the authors stated in their articles; this finding is confirmed by another study in the USA stating that statement of the study design is missing in almost 80% of abstracts of published medical education articles (20).

Limitations

This study faced some limitations. Firstly, some medical education articles published in Iranian non-medical education journals may be missing from this review due to the lack of a reliable search engine in the SID, the main electronic database of scientific journals in Iran. Secondly, MERSQI is developed to assess the methodological quality of the studies rather than their reporting quality. However, low reporting quality has influence on the MERSQI score. It is probable that authors conducted better studies but simply did not report in the manuscript all the items that MERSQI needs.

Implications for future research

This suboptimal quality status may reflect the need for using statistical consultants in research teams for improving the quality of inferences and also using more methodological and analytical oriented reviewers which could correct analytic flaws before publication or reject studies with fatal flaws.

Authors and journal reviewers in Iran should pay more attention to the measures of internal and external validities for medical education research design or report before publishing. Comparing to medicine, research in education may be more complex, confounding factors may be more apparent, content may be more implicit and controlled trials may be difficult. Thus, higher quality of research is mandated and correct designing research studies in this field is of paramount importance.

Conclusion

It could be concluded that a) majority of published medical education research have been observational studies rather than experimental; b) the reliability and validity of assessment tools used in the Iranian educational studies to assess the success of their educational efforts have not been properly reported and indicated; c) overall, the quality of the reporting of research studies in medical education in Iran seems not strong; and d) more robust and rigorous research studies are needed in the future.

Acknowledgements

The authors would like to thank Dr. Kamran Soltani-Arabshahi and Dr. Mohammad E Khamseh from Iran University of Medical Sciences and also Dr. David Cook from Mayo Clinic for their expert advice. This project has been granted funding (No. M/396) by deputy of research of Iran University of Medical Sciences. Ethical approval is not applicable.

Conflict of Interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.