The Effect of Problem-Based Learning on Improvement of the Medical Educational Environment: A Systematic Review and Meta-Analysis.

OBJECTIVES: The aim of this study was to evaluate the effect of problem-based learning on improving the medical educational environment.
MATERIALS AND METHODS: All relevant studies on problem-based learning and the medical educational environment were searched for in PubMed, the Education Resources Information Center (ERIC) catalogue, Google Scholar, China National Knowledge Infrastructure (CNKI) and WanFang Data (WF) databases for material dating from 1969 to May 2015 without any language limitation. Six randomized controlled trials of problem-based learning compared to traditional lecture-based learning were included. The Cochrane risk of bias tool was used to assess the quality of the included studies. Review Manager (Revman) version 5.3 software was used for data analysis. The effect size of the improvement on the medical educational environment was calculated as the mean difference and 95% confidence interval. Heterogeneity was evaluated with Cochrane's χ2 test and I2. Publication bias was assessed by funnel plot, Begg's rank correlation test, and Egg's linear regression test.
RESULTS: The six included studies were at low risk of bias in all domains except for three that were at high risk of bias in the domain of allocation concealment. The pooled effect size showed that problem-based learning was better than lecture-based learning in improving the medical educational environment, as measured by the Dundee Ready Education Environment Measure (DREEM), with statistically significant differences. No significant publication bias was observed. The sensitivity analysis showed that the result was reliable.
CONCLUSIONS: This study showed that problem-based learning was able to improve the medical educational environment as measured by DREEM. However, further studies with larger sample sizes and high-quality data are needed.

Introduction

The educational environment is complex and contains all elements of teaching and learning in educational institutions. A growing number of educators are paying more attention to the importance of the educational environment in medical schools. Their studies show that the educational environment within which students study has a significant impact on their satisfaction with the course of study, sense of well-being, aspirations and knowledge acquisition [1,2,3,4,5,6,7,8]. Several instructional strategies have been implemented to improve the medical educational environment in recent decades, and problem-based learning (PBL) has become more prominent among them [9].

The PBL approach to medical education was instigated at McMaster University in Canada in 1969 [10] to solve the problem in which traditional lectures failed to prepare medical students for problem-solving in clinical settings [11]. The PBL is a nontraditional, active, inductive, student-centered approach to learning, and it enables students to gain competence in self-learning, collaboration, problem solving and critical thinking [12,13]. The distinguishing feature of PBL is the use of small tutorial groups. Each group contains 6–8 students and a tutor who plays the role of group facilitator or guide, rather than expert or purveyor of knowledge [14]. There are five basic steps in the PBL process: problem analysis, establishment of learning objective, collection of information, summarizing, and reflection [15].

By contrast, traditional lecture-based learning (LBL) is characterized by large class sizes and the instructor-driven, lecture-based delivery of a curriculum. In LBL classes, teacher-directed information is presented with no need for free inquiry [16].

Some studies on the relationship between PBL and the medical educational environment have shown that PBL has a positive effect compared to LBL educational approaches [17,18,19,20,21], while others have not shown the same outcomes [22]. In addition, most of these studies were confined to one course, one major or one school. To the best of our knowledge, there has been no systematic review and quantitative evaluation of the effect of PBL on improving the medical educational environment on a worldwide scale.

The Dundee Ready Educational Environment Measure (DREEM) is considered to be the most suitable such instrument for evaluating the medical educational environment [23]. Therefore, we used DREEM as the outcome measure in this review.

The objective of this meta-analysis was to compare the effects of the PBL and LBL teaching methods on improving the medical educational environment, thereby providing a scientific basis for determining whether PBL remains a valid and effective environment for medical education at a rapidly changing and challenging time for curriculum development.

Materials and Methods

Search Strategy

A wide variety of electronic databases were screened, including PubMed, the Education Resources Information Center (ERIC) catalogue, Google Scholar, China National Knowledge Infrastructure (CNKI) and WanFang Data (WF). The following terms or keywords were used: (‘problem-based learning’ OR ‘PBL’) AND (‘education environment’ OR ‘education climate’ OR ‘learning environment’) AND (‘medical’ OR ‘medicine’). Studies were selected on the basis of the ‘abstracts’ or ‘all fields’. The search was restricted from 1969 (when medical PBL appeared) to May 2015, and no language restrictions were imposed.

The general inclusion criteria were a comparison between a PBL (intervention) condition and a control (LBL) condition, with a quantitative outcome focused on the medical educational environment, without any language limitation. The specific inclusion criteria were: (a) using PBL as an educational approach in the intervention group, (b) using traditional lectures as the only teaching method in the control group, (c), describing randomized controlled trials (RCTs), (d) reporting the sample size, the mean difference and standard deviation of medical educational environment scores for the intervention group and control group, and (e) using the Dundee Ready Educational Environment Measure (DREEM) to evaluate the medical educational environment as an outcome.

DREEM was developed at Dundee University Medical School in the UK in 1997 by an international Delphi panel [24]. It is a standardized, valid, and reliable tool with 50 items, including the following five subscales: students' perceptions of learning (12 items), students' perceptions of teachers (11 items), students' academic self-perceptions (8 items), students' perceptions of atmosphere (12 items), and students' social self-perceptions (7 items). Each of the 50 items are scored 0–4 on a 5-point scale, where 4 means ‘strongly agree’, 3 means ‘agree’, 2 means ‘unsure’, 1 means ‘disagree,’ and 0 means ‘strongly disagree’. Negative statements in items 4, 8, 9, 17, 25, 35, 39, 48 and 50 are scored in reverse. Therefore, higher scores indicate a positive evaluation. The maximum score of DREEM is 200, representing an ideal medical educational environment. DREEM has been translated into Spanish, Portuguese, Arabic, Chinese, Dutch, Swedish, Norwegian, Malay and Thai, and is widely used internationally to assess the medical educational environment [25].

The exclusion criteria were: (a) studies that lacked a control group, (b) studies that did not use traditional lectures as the only teaching method in the control group, (c) studies that were non-RCTs, (d) studies that did not evaluate the medical educational environment or used other tools to evaluate the outcome, (e) studies with incomplete data, such as not reporting the mean difference and standard deviation of the medical educational environment scores, and (f) studies that were duplications of other studies.

The literature search identified 1,001 articles, including 989 articles from database searches and 12 potential articles from citations in a retrieved paper. After reading the titles and abstracts, 184 duplicates, 52 reviews and 642 obviously not relevant articles were excluded. We then identified 123 articles for which full texts were retrieved for further evaluation. However, following a detailed review, 117 studies were excluded based on the inclusion and exclusion criteria, as follows: no control group (n = 43), not measuring the educational environment outcome (n = 57), not using DREEM to evaluate the educational environment outcome (n = 1), not using the traditional lecture as the only teaching method in the control group (n = 1), duplicating data reports of another study (n = 2), being non-RCT (n = 5), and not providing outcome data (n = 8). Finally, 6 studies [26,27,28,29,30,31] met the stipulated criteria and were included in this systematic review and meta-analysis (fig. 1).

Fig. 1

The study selection process.

Data Extraction

Two reviewers (Yongjie Qin and Lei Yu) extracted data independently from the included studies and met to discuss their findings until a consensus was reached. The following information was recorded for each included study: (a) the first author, (b) the year of publication, (c) the country of origin, (d) the sample size (intervention group and control group), (e) characteristics of the participants, (f) the intervention method and teaching methods in the control group, (g) outcomes (the mean difference and standard deviation of the medical educational environment scores), (h) the time of measuring outcomes, (i) the tool for measuring outcomes, and (j) the length of intervention.

Quality Assessment

The quality of the included studies in this review was independently assessed by two reviewers (Yongjie Qin and Lei Yu) using the Cochrane Collaboration's tool [32], which provided seven criteria to assess risk of bias in these studies: (a) sequence generation, (b) allocation concealment, (c) blinding of participants and personnel, (d) blinding of outcome assessment, (e) incomplete outcome data, (f) selective outcome reporting, and (g) other sources of bias. For each domain, the assessment was denoted as ‘low risk’, ‘high risk’ or ‘unclear risk’ according to the descriptions of each study. Any disagreement was resolved by discussion until a consensus was achieved.

Statistical Analysis

Data analysis was performed with Review Manager (Revman) version 5.3 software. Effect sizes were presented by weight mean difference (WMD) and 95% confidence intervals (CIs). There were two models of meta-analysis. We used the fixed-effects model if there was no heterogeneity, otherwise the random-effects model was used. Heterogeneity was evaluated using Cochran's χ2 test (p value) and I2. Significant heterogeneity was considered with p < 0.10 and I2 > 50% [33]; I2 of 25, 50 and 75% indicated low, moderate and high heterogeneity, respectively. The sensitivity analysis was used to determine whether there was heterogeneity. The results from each study were recalculated using the pooled estimates to see if these recalculations would alter the results. Publication bias was assessed by funnel plot, Begg's rank correlation test, and Egg's linear regression test, which were conducted using STATA 11.0 (StataCorp LP, College Station, Tex., USA). p values <0.05 were considered statistically significant.

Results

Search Results

All of the studies were published between 2006 and 2013, and all of them were in Chinese. The sample sizes of the 6 studies ranged from 53 to 300 participants, and the pooled sample size was 673 (PBL group: 335, control group: 338). All of the studies were performed in China, including 4 studies at the Chinese Medical University [26,27,29,31], 1 studyat Liaoning Medical College [30] and 1 study in Wuxi Hospital [28]. The participants were medical students in 5 studies [27,28,29,30,31] and nursing students in 1 study [26]. The length of intervention varied between several class hours and several months. All studies measured the medical educational environment after using PBL (table 1).

Table 1

Characteristics of the included studies

First author [Ref.]	Year	Country (exact place)	Sample (IG/CG)	Participant characteristics	Interventions	Comparator	Outcome measurements		Duration of intervention
							timing	scores (IG/CG)
Dai [26]	2006	China(China Medical University)	60(30/30)	Undergraduate nursing students studying the clinical practice of emergency treatment	With PBL as the educational approach, there was a 2-hour class discussion each week	With LBL as the teaching method, there was a 2-hour course lecture each week	After	137.33 ± 6.47/131.07 ± 4.19	4 weeks
Ou [27]	2008	China(China Medical University)	53(25/28)	Students of clinical medicine in a Japanese program undertaking a pediatrics course	PBL as the educational approach	LBL as the teaching method	After	133.36 ± 13.86/124.39 ± 11.78	A whole course
Liu [28]	2011	China (Wuxi Hospital of Jiangsu Province)	80(40/40)	Students taking part in clinical clerkship of general surgery	With PBL as the educational approach, there were 4–6 students and 1 mentor in each group; students followed a PBL process with one scenario every 2 weeks	LBL as the teaching method	After	142.2 ± 8.3/126.9 ± 11.0	3 months
Liu [29]	2012	China(China Medical University)	120(60/60)	Students of clinical medicine studying topographic anatomy	Using PBL as the educational approach there were 4 groups, each consisting of 15 students; students followed a PBL process with one scenario during 4 class hours; there were 4 scenarios in total	With LBL as the teaching method, each scenario lasted 4 class hours	After	150.25 ± 21.54/141.80 ± 10.74	16 class hours
Lu [30]	2012	China(Liaoning Medical College)	300(150/150)	Medical postgraduates (2010 entrance) taking clinical epidemiology	Using PBL as the educational approach there were 5 PBL classes and each class consisted of 30 students (5 groups of 6 students)	LBL as the teaching method	After	135.39 ± 17.57/128.11 ± 14.38	A whole course

After = Data were collected after the intervention; IG = intervention group; CG = control group.

Risk of Bias in the Studies

All of the included studies were RCTs. In 1 study [31], the allocation sequence was adequately generated and concealed by use of a random number table. The allocation sequence in two studies [26,28] was adequately generated by stratified randomization. The allocation sequence of the remaining 3 studies [27,29,30] was generated by the preference of the researchers, who assigned the students to the experimental group or control group; these studies were therefore judged to be high risk in this domain. The blinding of participants and personnel or outcome assessment was not described in any of the studies. In all of the studies, data collection was clearly described and reported, so we judged them as complete outcomes with no selective reporting of results. According to the definition of the Cochrane Collaboration, all studies seemed to be free from ‘other sources of bias’. Overall, most of the studies were found to be at low risk of bias and of high quality (fig. 2).

Fig. 2

Summary of the risk of bias assessment.

Effects of Interventions

Overall DREEM Scores

Moderate heterogeneity (I2 = 61%, p < 0.10) was revealed in the included studies, so the random-effects model was used to perform the meta-analysis. Compared with the control LBL group, the pooled effect size showed a significant difference in medical educational environment scores (WMD 9.10, 95% CI 5.98–12.21, p < 0.00001) in favor of the PBL group (fig. 3).

Fig. 3

Mata-analysis and forest plot of medical education environment scores after using PBL compared with LBL.

Subscale Scores

Subscale scores of the DREEM were reported in the 6 included studies. The pooled effect sizes for students' perceptions of learning (WMD 2.84, 95% CI 2.23–3.45, p < 0.0001), students' perceptions of teachers (WMD 1.77, 95% CI 0.92–2.61, p < 0.0001), students' academic self-perceptions (WMD 1.06, 95% CI 0.61–1.50, p < 0.00001), students' perceptions of atmosphere (WMD 1.76, 95% CI 1.16–2.35, p < 0.00001), and students' social self-perceptions (WMD 1.19, 95% CI = 0.77–1.62, p < 0.00001) favored the PBL group compared with the control LBL group (table 2).

Table 2

Pooled effect sizes of the subscales

Subscale item	Sample size (PBL/LBL), n	Analysis model	WMD (95% CI)	p value of effect	Heterogeneity
					χ2	d.f.	p	I2
Students’ perceptions of learning	335/338	fixed	2.84 (2.23, 3.45)	<0.0001	2.8	5	0.81	0%
Students’ perceptions of teachers	335/338	random	1.77 (0.92, 2.61)	<0.0001	11.28	5	0.05	56%
Students’ academic self-perceptions	335/338	fixed	1.06 (o.61, 1.50)	<0.00001	5.91	5	0.32	15%
Students’ perceptions of atmosphere	335/338	fixed	1.76 (1.16, 2.35)	<0.00001	8.68	5	0.12	42%
Students’ social self-perceptions	335/338	fixed	1.19 (0.77, 1.62)	<0.00001	8.15	5	0.15	39%

Sensitivity Analysis

Given the moderate heterogeneity, it was necessary to carry out a sensitivity analysis to verify the reliability of the results. After excluding the study in which the weight was largest [26], the pooled effect size was in favor of the PBL group (WMD 10.01, 95% CI 6.44–13.58, p < 0.00001). Furthermore, when the study with the largest sample was excluded [30], the pooled effect size was superior for the PBL group (WMD 9.62, 95% CI 5.57–13.67, p < 0.00001). Also, when the study with the smallest sample was excluded [27] the pooled effect size still showed that the PBL group was better able to improve the medical educational environment (WMD 9.14, 95% CI 5.56–12.73, p < 0.00001). Thus, the reanalyses performed in light of heterogeneity did not yield results different from those in the the primary analysis.

Publication Bias

In assessing publication bias, the shape of the funnel plot was symmetrical (fig 4). No significant bias was observed using Begg's test [Z = 1.32, p = 0.260 (>0.05)] or Egger's test [t = 1.74, p = 0.157 (>0.05)].

Fig. 4

Funnel plot analysis for overall medical education environment scores. SMD = Standard mean difference; se = standard error.

Discussion

Our meta-analysis showed that PBL is associated with higher overall DREEM scores than those in LBL groups, with statistically significant differences. This suggests that PBL is more efficient than LBL in improving the medical educational environment. Indeed, our result was consistent with most previous studies pertaining to DREEM scores, carried out in a variety of national contexts. A study in Saudi Arabia compared two medical schools, one of which used a conventional LBL curriculum and the other a hybrid PBL curriculum. The study found that students experiencing the hybrid PBL curriculum had significantly higher DREEM scores (p < 0.001) [21]. Some studies in China also indicated that the students' DREEM scores in PBL classes was significantly (p < 0.001) higher than those in LBL classes [34,35]. Similar findings existed in American studies using the Medical School Learning Environment Survey (MSLES), which found that PBL students were significantly more satisfied with their learning environment than students in the LBL groups (p < 0.05 and p < 0.001) [17,18]. Conversely, a Brazilian study showed that DREEM scores did not differ between PBL courses and hybrid LBL courses that also used some PBL approaches [36].

Prior individual studies showed positive effects for some subscales of DREEM, but not for others. In particular, studies were consistent in finding that PBL could improve students' perceptions of learning [21,26,27,28,29,30,31], but inconsistent in whether or not they found significant positive differences for the other 4 aspects. In contrast, our meta-analysis found that the PBL method was superior to the LBL method in all subscales of DREEM.

Considering the inconsistences of the findings across individual studies, we speculate that the following factors may be the reasons for the differences. One factor possibly explaining the variation is the difference in the duration of intervention from several class hours to several years [21,22,26,27,28,29,30,31]. A second factor might be the different characteristics of the participants across the studies. Some of them were nursing students [26] and others were medical students [21,22,27,28,29,30,31]. Meanwhile, some participants were postgraduate students [30] and the others were undergraduate students [21,22,26,27,28,29,31]. They were also in different phases: some were in the clinical knowledge study phase [26,27,28,31] and some in the basic knowledge study phase [22,29,30]. The differences in major, grade, study phase, and so on might also have affected the results. A third factor might be differences in the characteristics of the instructors. The instructors of some studies [31] were the same in both the experimental and control groups, while in other studies the instructors varied across the groups [26]. A fourth factor might be variation in the implementation of PBL as an educational method.

Though inconsistences existed in the individual studies, our meta-analysis, combining evidence across all studies, provides evidence that PBL could improve the medical educational environment. Looking at results from particular studies, we see that, compared to traditional LBL, PBL had the following advantages: (a) it was more effective on the development of students' knowledge and skills [37], (b) it generated a better performance in clinical examinations and faculty evaluation [38], (c) it was superior in developing students' critical thinking [14], (d) it is more nurturing and enjoyable, and (e) it is significantly superior with respect to students' attitudes and opinions about their programs [39]. Obviously, the 5 advantages were related to the particular components assessed with DREEM, which are the students' perceptions of learning, teachers, academic self-perceptions and atmosphere, and their social self-perceptions. Therefore, PBL possibly improved the medical educational environment by its influence on these aspects.

The results of the sensitivity analysis showed that there were no significant differences with the primary analysis. Thus, we consider the result of this meta-analysis to be reliable.

Publication bias is always a concern when making decisions based on the results of a meta-analysis. This is also known as the ‘file drawer problem’, which refers to the researcher's preference to report mostly positive results. The results of the funnel plot, Begg's rank correlation test and Egg's linear regression test indicated that there was no significant publication bias in the included studies.

The strengths of this systematic review were: the literature was searched with terms and keywords in ‘abstracts’ or ‘all fields’ in multiple databases, avoiding leaving any possible articles out; the sensitivity analysis showed that the result of the meta-analysis was reliable, and the statistical analyses showed no evidence of publication bias, which is a major threat to the validity of reviews. The limitations of this study were: the small number of high-quality studies that compared the PBL with the LBL medical educational environment; the 6 studies were done in Chinese institutions, with 4 of them done at different departments in the same hospital, and the quality of methodology of the included studies could be higher − 3 studies lacked definitive randomization, and none of the studies described were blinded, hence the included studies had a high risk in the domain of allocation concealment and there was heterogeneity in the results among these studies.

PBL has swept the world of medical education since its introduction in 1969, and this revolution has had a huge impact on the development of medical school curricula [40]. Although PBL is not perfect, our findings provide evidence that it is a teaching method likely to improve the medical educational environment. Undoubtedly, the educational environment is dynamic and related to many influencing factors. Thus, we would recommend that medical schools evaluate their students' perceptions of the medical educational environment frequently, and develop educational methods with flexibility and the effectiveness necessary to accommodate the challenges they face.

Conclusion

In this study, we found that existing evidence supports the claim that PBL is more effective than LBL. However, further high-quality studies with larger sample sizes are needed to confirm this finding.

Disclosure Statement

The authors have no conflicts of interest.