Assessing the effectiveness of problem-based learning of preventive medicine education in China.

Problem-based learning (PBL) is defined as a student-centered pedagogy which can provide learners more opportunities for application of knowledge acquired from basic science to the working situations than traditional lecture-based learning (LBL) method. In China, PBL is increasingly popular among preventive medicine educators, and multiple studies have investigated the effectiveness of PBL pedagogy in preventive medicine education. A pooled analysis based on 15 studies was performed to obtain an overall estimate of the effectiveness of PBL on learning outcomes of preventive medicine. Overall, PBL was associated with a significant increase in students' theoretical examination scores (SMD = 0.62, 95% CI = 0.41-0.83) than LBL. For the attitude- and skill-based outcomes, the pooled PBL effects were also significant among learning attitude (OR = 3.62, 95% CI = 2.40-5.16), problem solved skill (OR = 4.80, 95% CI = 2.01-11.46), self-directed learning skill (OR = 5.81, 95% CI = 3.11-10.85), and collaborative skill (OR = 4.21, 95% CI = 0.96-18.45). Sensitivity analysis showed that the exclusion of a single study did not influence the estimation. Our results suggest that PBL of preventive medicine education in China appears to be more effective than LBL in improving knowledge, attitude and skills.

Problem-based learning (PBL) was first implemented at McMaster University Medical School in the 1960s where it has revolutionized the field of medical education since then1. PBL is defined as a student-centered pedagogy in which participants are allocated in groups of up to 8 persons under non-directive tutors and given tasks or challenges that reflect situations relevant to the working environments they will experience2. In this way, the learners are empowered to integrate theory and practice, and apply knowledge and skills to develop a viable solution to the problem34. PBL represents a paradigm shift from traditional pedagogy, which is more often lecture-based learning (LBL). LBL pedagogy focuses on factual knowledge and memorization, providing little chance for application of knowledge acquired from basic science to the working situation. There are many other advantages of PBL over LBL, including flexible knowledge, improved communication, collaborative skills and self-directed learning skills, and a more enjoyable and motivational format5.

Originally, PBL was devised in an attempt to develop a teaching method for use for clinical medical education. A variety of medical schools in Europe (the Netherlands, Sweden, Norway, and the United Kingdom), Asia (Japan, Korea, and China), Canada, the United States, South Africa, Australia and New Zealand already have or are introducing PBL as a learning modality3678. Now it has also been used in multiple disciplines including public health and preventive medicine9. For example, Dyke et al. had used PBL approach for teaching epidemiology, and found that the PBL students got a far richer learning experience than their counterparts in the traditional stream10. As the public health system in China has been developing rapidly for the past decade, PBL is increasingly popular among preventive medicine educators striving to produce graduates who can solve complex public health problems efficiently. In recent years, a growing number of Chinese studies have investigated the effectiveness of PBL pedagogy in preventive medicine education. As one of the most well-known public health schools in China, the School of Public Health from Nanjing Medical University (NMU) has adopted a PBL in preventive medicine education from September 2012. NMU has established multidisciplinary cases throughout the course schedule to improve the positive outcomes, such as motivating learning, problem solved skill, collaborative skill and critical thinking. To the best of our knowledge, however, there are no reports or reviews summarizing the trials done to evaluate the effectiveness of PBL for preventive medicine education.

The uncertainty about the effectiveness of PBL and the heterogeneity in the published literatures provided the impetus for this study. Thus, the aim of the study was to perform a meta-analysis on all published studies to obtain an overall estimate of the effectiveness of PBL on learning outcomes of preventive medicine education in China.

Results

Study characteristics, and quality assessment

The study selection process is depicted in Fig. 1. The search strategy yielded 110 entries, of which 64 in Chinese were considered to have potential value and were subjected to further examination. Based on the inclusion and exclusion criteria, we excluded 52 articles (23 were reviews or editorials; 16 no relevant quantitative outcomes; 9 no comparison with LBL; 3 no full text and 1 duplicate publication). Three additional articles were included from the reference review. Ultimately, a total of 15 articles involving 2,061 subjects published from 2006 through 2013 were included in this meta-analysis111213141516171819202122232425. Characteristics of the selected studies are presented in Table 1.

Figure 1

Flow chart summarizing identification of included studies.

Table 1

Characteristics of included studies

			No. of participants
Study	Year	Disciplines or curricula	PBL	LBL	Source of participants	Outcome assessment
Zhang Jie et al.	2013	Environmental health	54	50	Rural doctors in Han Chinese	Examination scores and questionnaire surveys
			55	61	Rural doctors in minority nationalities	Examination scores and questionnaire surveys
Li Guangyou et al.	2013	Medical statistics	45	48	Undergraduate students	Examination scores and questionnaire surveys
Gao Lanyue et al.	2013	Occupational health and medicine	61	61	Undergraduate students	Examination scores
Zhao Yingzheng et al.	2013	Toxicology	40	38	Undergraduate students	Examination scores and questionnaire surveys
Zhang Zhihong et al.	2012	Environmental health	77	57	Undergraduate students	Examination scores and questionnaire surveys
Zhang Qin et al.	2012	Child and adolescent health	50	51	Undergraduate students	Examination scores and questionnaire surveys
Wang Shuran et al.	2011	Nutrition science and food hygiene	10	116	Undergraduate students	Examination scores and questionnaire surveys
Lu Zhiquan et al.	2011	Epidemiology	150	150	Postgraduate students	Examination scores and questionnaire surveys
Hu Dongmei et al.	2011	Medical statistics	90	90	Postgraduate students	Examination scores and questionnaire surveys
Sun Rong et al.	2010	Medical statistics	86	85	Undergraduate students	Examination scores and questionnaire surveys
Zhang Yanqi et al.	2010	Medical statistics	19	39	Eight-year program medical students	Examination scores and questionnaire surveys
Zhang Yanqi et al.	2010	Medical statistics	39	39	Eight-year program medical students	Examination scores and questionnaire surveys
Wu Songwen et al.	2010	Medical statistics	60	60	Undergraduate students	Examination scores and questionnaire surveys
Zhang Yixi et al.	2009	Medical statistics	93	78	Higher vocational school students	Examination scores and questionnaire surveys
Deng Shusong et al.	2006	Preventive medicine	53	56	Undergraduate students	Examination scores and questionnaire surveys

PBL, problem-based learning; LBL, lecture-based learning.

The methodological quality of each included study is summarized in Table 2. The quality scores ranged from 4 to 6. A majority of the studies are cluster randomized trials, while only one is randomized controlled trial (RCT). Blinding is a critical measure to protect against bias, however, none of the studies mentioned the application of this method. Some other bias protection measures such as control for important factors and incomplete data bias were often used in the publications. Whether the outcome assessors and data collectors were blinded to subjects' assignments or not was not mentioned in the studies.

Table 2

Methodologic quality of studies included in the meta-analysis

Study	Year	Randomization	Allocation concealment	Blind	Control for important factorsa	Control for incomplete data bias	Assessment of outcomeb	Total quality scores
Zhang Jie et al.	2013	-	Δ	-	ΔΔ	Δ	ΔΔ	6
Li Guangyou et al.	2013	-	Δ	-	Δ	Δ	ΔΔ	5
Gao Lanyue et al.	2013	-	Δ	-	ΔΔ	Δ	Δ	5
Zhao Yingzheng et al.	2013	-	Δ	-	ΔΔ	Δ	ΔΔ	6
Zhang Zhihong et al.	2012	-	Δ	-	ΔΔ	Δ	ΔΔ	6
Zhang Qin et al.	2012	Δ	Δ	-	Δ	Δ	ΔΔ	6
Wang Shuran et al.	2011	-	-	-	ΔΔ	Δ	ΔΔ	5
Lu Zhiquan et al.	2011	-	Δ	-	ΔΔ	Δ	ΔΔ	6
Hu Dongmei et al.	2011	-	Δ	-	ΔΔ	-	ΔΔ	5
Sun Rong et al.	2010	-	Δ	-	ΔΔ	-	ΔΔ	5
Zhang Yanqi et al.	2010	-	Δ	-	-	Δ	ΔΔ	4
Zhang Yanqi et al.	2010	-	Δ	-	-	Δ	ΔΔ	4
Wu Songwen et al.	2010	-	Δ	-	ΔΔ	Δ	ΔΔ	6
Zhang Yixi et al.	2009	-	Δ	-	ΔΔ	Δ	ΔΔ	6
Deng Shusong et al.	2006	-	Δ	-	ΔΔ	Δ	ΔΔ	6

aA maximum of 2 triangles could be awarded for this item. Studies that controlled for age received 1 triangle, controlled for previous academic performance received an additional triangle.

bA maximum of 2 triangles could be awarded for this item. Studies that measured by examination scores or questionnaire surveys received 1 triangle, measured by both examination scores and questionnaire surveys received two triangles.

Quantitative synthesis

The effects of PBL methods were evaluated by synthesizing theoretical examination scores, pass rate, the improvement on learning attitude, problem solved skill, self-directed learning skill and collaborative skill in this meta-analysis. The effects of PBL on theoretical examination scores were reported by all the included studies. When all the eligible data were pooled into the meta-analysis, PBL was associated with a significant increase in students' examination scores (SMD = 0.62, 95% CI = 0.41–0.83, Pheterogeneity < 0.001) (Fig. 2). Four studies reported pass rate of the theoretical test, when they were pooled together, there was a lack of heterogeneity (P = 0.83) and the PBL effect was significant (OR = 2.88, 95% CI = 1.44–5.75). For the attitude- and skills-based outcomes, the pooled PBL effects were also significant (OR = 3.62, 95% CI = 2.40–5.16, Pheterogeneity = 0.444 for learning attitude; OR = 4.80, 95% CI = 2.01–11.46, Pheterogeneity < 0.001 for problem solved skill; OR = 5.81, 95% CI = 3.11–10.85, Pheterogeneity = 0.048 for self-directed learning skill; and OR = 4.21, 95% CI = 0.96–18.45, Pheterogeneity < 0.001 for collaborative skill) (Table 3).

Figure 2

Forest plot for the effects of PBL on examination scores compared with LBL.

Table 3

Summary of effect sizes for PBL and LBL

Outcome	No. of studies	No. of subjects	SMD/OR (95%CI)a	I2	P for heterogeneity
Examination scores	15	2061	0.62 (0.41–0.83)	80.1%	<0.001
Pass rate of examination	4	495	2.88 (1.44–5.75)	0.0%	0.833
Questionnaire surveys
Learning attitude	4	546	3.62 (2.40–5.16)	0.0%	0.444
Problem solved skill	6	851	4.80 (2.01–11.46)	82.6%	<0.001
Self-directed learning skill	5	690	5.81 (3.11–10.85)	58.3%	0.048
Collaborative skill	3	426	4.21 (0.96–18.45)	87.5%	<0.001

aRandom-effects model was used when P value for heterogeneity test <0.10; otherwise, fix-effects model was used.

PBL, problem-based learning; LBL, lecture-based learning; SMD, standardized mean difference; OR, odds ratio; CI, confidence interval.

We then evaluated the effects of PBL in students' examination scores according to the quality scores. For quality scores = 6, SMD = 0.70, 95% CI = 0.37–1.04, Pheterogeneity < 0.001; scores = 5, SMD = 0.53, 95% CI = 0.37–0.70, Pheterogeneity = 0.153; scores = 4, SMD = 0.38, 95% CI = 0.03–0.73, Pheterogeneity = 0.924. The result indicated that studies with high quality scores would yield significantly larger effect sizes than those with low quality scores.

Test of heterogeneity

There was significant heterogeneity (P < 0.001) among included studies. We assessed the source of heterogeneity by quality scores, source of participants and sample size (participants more than 50 in both PBL and LBL groups). As a result, quality scores (P = 0.035) and source of participants (P = 0.048), but not sample size (P = 0.146), were found to contribute to the substantial heterogeneity.

Sensitivity analyses

Sensitivity analyses indicated that Lu's study was the main origin of heterogeneity. However, when this article was excluded the overall result (SMD = 0.57, 95% CI = 0.38–0.77, P < 0.001) and the heterogeneity (P < 0.001, I = 72.6%) was not materially influenced. Further exclusion of any single study got similar results with a range from 0.56 (95% CI = 0.37–0.75) to 0.67 (95% CI = 0.46–0.87), suggesting that the results of this meta-analysis are statistically reliable.

Publication bias

Begg's funnel plot and Egger's test were performed to evaluate the publication bias of the literatures. As a result, the funnel plots did not show any obvious asymmetry. Then, the Egger's test was adopted to provide statistical evidence of funnel plot symmetry. The results also indicated no evidence of publication bias (P = 0.432).

Discussion

Many medical schools internationally are changing their curricula and moving to PBL programs26. While in China, the application of PBL pedagogy is still in the initial stage, especially among preventive medicine education. In this meta-analysis, we found that compared with LBL, PBL was more effective in all domains including theoretical knowledge, attitude and skills. Nevertheless, the results of attitudes and skills were clearly less precise than that of the theoretical knowledge. It may be due to the less eligible studies and smaller sample sizes. Moreover, the assessment of theoretical knowledge by a test or exam seems more objective than that of attitudes and skills by the participants' sensation.

PBL is one of the best described interactive learning methods27. According to the questionnaire results and the tutors, students showed more enthusiasm for PBL rather than LBL. It is reported that learning is most effective when students are actively involved in PBL28. Therefore, our results were consistent with previous conclusions, which may improve the application of lesion-based theory to practical knowledge as well as achieving specific goals. Although the heterogeneity amongst studies was large (I = 80.1%), we found that the sources of heterogeneity were from quality scores and source of participants. Furthermore, the results of the subgroup analysis only partially explained the heterogeneity. Variations in other aspects such as study design, involved discipline, and the duration may also play a part.

In the analysis stratified by quality scores, elevated effects were more pronounced among studies with higher quality score, suggesting the importance of methodological quality during PBL intervention. Overall, the methodological quality of the included studies in this meta-analysis was not high. First, most of the studies are not randomized controlled trials (RCTs). In China, students are divided into classes when they entered the university, so the researchers prefer to allocate according to the classes rather than the individuals. Second, as an investigation in the field of education, it is impossible for the researchers to use the blinding method during the whole curriculum. Third, there is no standard criterion for evaluating the effectiveness of the PBL pedagogy. Whether the outcome assessors and data collectors were blinded to subjects' assignments or not was also poorly reported. In a word, high risks for selection bias, performance bias, as well as measurement bias do exist in the literatures we included.

In conclusion, our meta-analysis shows that PBL in preventive medicine education appears to be more effective than LBL in improving knowledge, attitude and skills. For future studies assessing the implementation of PBL, high quality of problems, uniform and objective outcome measurement, and well-designed RCTs are required. Only through studies which are strategic planned and conducted with carefully quality controlled, can we ever expect to achieve the practical goals.

Methods

Search strategy

Relevant biomedical sciences and educational databases [PubMed, EMBASE, China National Knowledge Infrastructure (CNKI), and Chinese Biomedical Literature database (CBM) were searched from inception through September 2013. The search strategy was to use and combine the following key words: “problem-based learning”, “PBL”, “preventive medicine”, “epidemiology”, “medical statistics”, “health statistics”, “environmental health”, “nutrition science and food hygiene”, “occupational health and medicine”, “child and adolescent health”, “toxicology”, “hygienic toxicology”, or “social medicine”. In addition, references of all included articles were also identified by a manual search and studies matching the eligible criteria were retrieved.

Inclusion criteria

Studies included in the current meta-analysis have to meet the following inclusion criteria: (1) the study should compare the effectiveness of PBL group and traditional LBL group; (2) the courses of PBL pedagogy should be preventive medicine professional disciplines; (3) both randomized and non-randomized studies were considered, and (4) quantitative data about the effectiveness of PBL were available (i.e., they had to report a mean change in knowledge, attitudes or skills that occurred in response to an PBL intervention and was measured using a numeric scale such as an examination or test).

Data extraction and quality assessment

For each study, the following basic information was collected: the first author's name, publication year, the involved disciplines, number of participants in each group, educational background of the participants, mean scores of both groups and the outcome measure. Two of the authors extracted all data independently according to the inclusion criteria listed above and reached a consensus on all the items. If more than one article was published using the same population, we selected the most recent or most informative report.

Recognizing that many nonrandomized studies would be included, an adaptation of the Newcastle-Ottawa Scale was adopted to abstract information on methodological quality. The included studies were judged in terms of a proper method of randomization, allocation concealment, blinding, the comparability of the populations, the completeness of the data, and the assessment of outcome. The full score was 8 triangles, and a high-quality study was defined as a study with 5 or more triangles.

Statistical analysis

Standardized mean difference (SMD) for continuous data and odds ratio (OR) for dichotomous data were performed to estimate the pooled effects. Heterogeneity assumption was identified by the I statistics. A P-value ≤ 0.10 for the I test indicated a lack of heterogeneity among the studies, and then random-effects model (DerSimonian and Laird method) was used to calculate the summary standardized mean difference (SMD) or OR estimate of each study29. Otherwise, the fixed-effects model (the Mantel-Haenszel method) was used30. Sensitivity analyses were performed to assess the stability of the results, namely, a single study in the meta-analysis was deleted each time to reflect the influence of the individual data set to the pooled SMD. To evaluate the publication bias, Funnel plots and Egger's linear regression test was applied31. All analyses were carried out with Stata software (version 11.0; StataCorp LP, College Station, TX, USA), using two-sided P values.

Author Contributions

Conceived and designed the experiments: D.X., Z.Z., W.M. Performed the experiments: D.X., Z.Z., W.M. Analyzed the data: D.X., Z.L., C.H. Contributed reagents/material/analysis tools: D.X., Z.L., N.C., H.Z. Wrote the main manuscript text: D.X., Z.L. Reference collection and data management: D.X., Z.L. Statistical analyses and paper writing: C.H., T.N. Study design: D.X., Z.Z., W.M. Prepared figures 1–2: D.X. All authors reviewed the manuscript.