The Effect of Formative Program Evaluation on Continuous Program Improvement: A Case Study of a Clinical Training Program in Lao PDR.

This study aimed to evaluate the effect of the formative program evaluation on the continuous improvement of a clinical training program for Lao health professionals. The training program was conducted 4 times consecutively for total 48 health professionals, and the formative program evaluation was carried out during the whole process. To evaluate the satisfaction and the transfer of the trainees, the questionnaire survey, the focus group interview, and the trainees' medical records were used. After the end of each batch of the program, the evaluation data were analyzed, and its results were shared with the training management committee and the trainers, who, based on the results, reached a consensus on how to improve the program. The evaluation results and the comparison of them among the four batches of the program showed that there was a continuous increase of the satisfaction and the transfer of the trainees, especially in the early period of the program. The formative program evaluation which was conducted during the whole process of the clinical training program had a positive effect on the improvement of the program, especially in the early phase, by increasing the satisfaction and transfer of the trainees.

INTRODUCTION

The program evaluation is generally classified into three types: planning, summative, and formative evaluation. Even though distinctions between the different types of evaluation are sometimes blurred, differentiating their intent helps us in clarifying our understanding of evaluation process (1). Planning evaluation takes place before a program begins to give those involved in program development a precise understanding of the program, and it is sometimes referred to as "pre-formative evaluation" (2). Summative evaluation is the typical and most common type of evaluation, which is conducted at the end of a program to provide decision-makers with judgements about the program's overall merit or worth. However, although necessary, it often comes too late to be much help (3). On the other hand, formative evaluation occurs during the process of a program to provide those who are responsible ongoing information about whether things are going as planned and whether expected progress is being made. If not, this same information can be used to guide necessary improvements, before it is too late (4). Even though there is a great extent of literature on formative evaluation, most of them mainly focus on its conceptual framework, methodology and use. Surprisingly, the subsequent effect of using the findings of formative evaluation has not received systematic attention, and few researches demonstrate this by comparing data from the initial program with the final program to show whether there was an improvement in program implementation and impacts (5). This study aimed to evaluate the subsequent effect of the formative program evaluation based on a case study of a clinical training program in Lao People's Democratic Republic (PDR).

The main health care delivery system of Lao PDR is a government-controlled, public system which has a strong vertical structure with three levels: central, provincial, and district level (6). Even though all health care professionals in the country are required by law to continuously improve their knowledge and skills, the continuing professional development (CPD) training system in Lao PDR has not yet been functioning well (7). In 2012, under the support of Korea International Cooperation Agency (KOICA), faculties from Seoul National University (SNU) College of Medicine in Korea and University of Health Sciences (UHS) in Lao PDR launched the "Continuing Professional development training project to strengthen the capacity of provincial and district hospitals in Lao PDR" (8). At the beginning of the project, UHS faculties conducted a needs assessment survey in Luang Prabang province, which was the pilot area for this project. Based on the needs assessment results, Korean and Lao faculties worked together to develop a CPD training program and a handbook including 80 clinical topics on 5 major clinical specialties: internal medicine, surgery, obstetrics and gynecology, pediatrics, and emergency medicine. And then, UHS faculties provided a 1-week 'training of trainers' program for 30 medical faculties of the Luang Prabang provincial hospital. Also, the training management committee composed of provincial hospital executives and department chairs was established to monitor and manage the training effectively. Finally, the trained provincial hospital faculties provided a 10-week training program in the provincial hospital for the district hospital health professionals in Luang Prabang province. The training was conducted 4 times consecutively over 2 yr, and 12 medical professionals, 1 from each district hospital, were invited to each training program. Among the total 48 medical professionals, 35 of them were medical assistants, and the other 13 were medical doctors. The training was composed of 5 major clinical sections, and the trainees rotated each clinical section every 2 weeks. Lecture about clinical topics, observation of trainers' performance, and trainee's own medical practice with trainers' feedback were the main training methods.

MATERIALS AND METHODS

Formative program evaluation was conducted during the whole process to continuously improve the training program (Fig. 1). Kirkpatrick model was applied for the program evaluation covering level 1 (reaction) and level 3 (transfer) (9). Questionnaire survey and focus group interviews with the trainees were used to evaluate the reaction of the trainees. The questionnaire was designed by the co-work of faculties from SNU and UHS based on the literature review (1011). It was composed of 14 items, regarding the goals and objectives, relevance, organization, trainers' knowledge, trainers' communication, trainers' preparedness, training methods, discussion and interaction, practical session, handbook, facilities, schedule, applicability, and helpfulness of the training program. The trainees were asked to rate the items with a 5-point Likert scale (1, strongly disagree; 2, disagree; 3, neutral; 4, agree; 5, strongly agree) (Table 1). The survey was conducted every two weeks at the end of each clinical section of the training program. Focus group interview with the trainees was facilitated by the UHS faculties at the end of each 10-week training program.

Fig. 1

The time schedule of the clinical training and the formative program evaluation.

Table 1

The questionnaire form for the survey

Items	Rating Scale*
1) The program's goals and objectives were clearly stated	①---②---③---④---⑤
2) The material in the program was relevant to the clinical practice in the district level	①---②---③---④---⑤
3) The material in the program was well organized	①---②---③---④---⑤
4) The trainers were knowledgeable in the subject matter	①---②---③---④---⑤
5) The trainers were an effective communicator	①---②---③---④---⑤
6) The trainers were well prepared	①---②---③---④---⑤
7) The methods of the training were appropriate for this program	①---②---③---④---⑤
8) There were sufficient opportunities for discussion and interaction	①---②---③---④---⑤
9) There were enough practical sessions (bed side teaching) in the training program	①---②---③---④---⑤
10) The handbook will be helpful to me	①---②---③---④---⑤
11) The facilities were suitable	①---②---③---④---⑤
12) The schedule was suitable	①---②---③---④---⑤
13) I will be able to apply much of the material to the clinical practice in the district level	①---②---③---④---⑤
14) I feel that the program will help me do the clinical practice better in the district level	①---②---③---④---⑤

*1, strongly disagree; 2, disagree; 3, neutral; 4, agree; 5, strongly agree.

The transfer of the trainees was evaluated through the review of medical records written by the trainees. Total 354 medical records written by 44 trainees before and 3 months after the training program were collected. Global rating with a 5-point anchored scale (1, novice; 2, advanced beginner; 3, intermediate; 4, proficient; 5, expert) was used for the evaluation. A total of 25 trainers received a half-day workshop before assessing the medical records, and they practiced with some pilot medical records and received feedback during the workshop. The medical records were coded by the coordinator, and they were randomly distributed to the trainers without any personal and time information. The medical records were assessed twice from a different trainer. The difference of average scores between the medical records written before the training and those written after the training was calculated to evaluate the transfer of the trainees.

The evaluation data was collected under the responsibility of the training management committee and analyzed by the study team. The results were shared with the committee and the trainers after the end of each batch of the training program. The committee and the trainers reached a consensus on how to improve the training program through the group discussion based on the evaluation results. After the final end of the training program, the average scores of the survey results and the achievement of global rating scores of medical records among the four batches were analyzed statistically with the ANOVA using SPSS.

Ethics statement

The institutional review board of Seoul National University College of Medicine and Seoul National University Hospital exempted review of this study since it was an analysis of de-identified data (IRB No. 1507-026-686).

RESULTS

According to the results of the survey, the trainees were quite satisfied with the training program (Table 2). However, especially in the early period of the training program, the trainees were not so satisfied in such areas as the organization and schedule of the training program, and discussion, interaction, and practical sessions during the training program. From the second batch, there was continuous increase of the satisfaction of the trainees in all the items of the questionnaire, and most of the significant improvement happened between the first and the second batch (Supplementary Fig. 1).

Table 2

Results of the survey (n = 48, 5-point Likert scale*)

Items	1st Batch (n = 12)	2nd Batch (n = 12)	3rd Batch (n = 12)	4th Batch (n = 12)	F	Post hoc test (Tukey B)
1) Goals and objectives	4.00 ± 0.66	4.60 ± 0.49	4.58 ± 0.70	4.82 ± 0.39	22.23†	1<2=3=4
2) Relevance	4.33 ± 0.68	4.80 ± 0.40	4.80 ± 0.40	4.87 ± 0.34	16.08†	1<2=3=4
3) Organization	3.80 ± 0.71	4.35 ± 0.61	3.90 ± 0.73	4.37 ± 0.52	12.59†	1=3<2=4
4) Trainers' knowledge	4.30 ± 0.56	4.93 ± 0.25	4.90 ± 0.30	4.93 ± 0.25	43.66†	1<2=3=4
5) Trainers' communication	4.00 ± 0.58	4.70 ± 0.53	4.82 ± 0.39	4.82 ± 0.39	40.03†	1<2=3=4
6) Trainers' preparedness	4.00 ± 0.55	4.65 ± 0.55	4.55 ± 0.53	4.80 ± 0.40	27.74†	1<2=3<4
7) Training method	3.93 ± 0.71	4.72 ± 0.45	4.68 ± 0.47	4.82 ± 0.39	36.67†	1<2=3=4
8) Discussion and interaction	3.55 ± 0.65	4.18 ± 0.62	4.28 ± 0.52	4.28 ± 0.61	20.48†	1<2=3=4
9) Practical sessions	3.40 ± 0.83	4.23 ± 0.75	4.63 ± 0.49	4.43 ± 0.59	38.56†	1<2< 3, 1<2=4
10) Handbook	4.65 ± 0.58	4.98 ± 0.13	4.92 ± 0.28	4.92 ± 0.28	10.38†	1<2=3=4
11) Facilities	3.85 ± 0.63	4.52 ± 0.60	4.27 ± 0.71	4.53 ± 0.54	15.73†	1<2=3=4
12) Schedule	3.71 ± 0.62	4.48 ± 0.50	4.48 ± 0.50	4.48 ± 0.50	30.89†	1<2=3=4
13) Applicability	4.34 ± 0.62	4.58 ± 0.50	4.73 ± 0.45	4.80 ± 0.54	15.44†	1<2=3=4
14) Helpfulness	4.57 ± 0.53	4.68 ± 0.47	4.85 ± 0.36	4.90 ± 0.30	7.68†	1=2< 4, 1<3=4
Average score	4.02 ± 0.31	4.60 ± 0.28	4.60 ± 0.22	4.69 ± 0.20	86.53†	1<2=3=4

*1, strongly disagree; 2, disagree; 3, neutral; 4, agree; 5, strongly agree; †P<0.001 by ANOVA.

The focus group interviews provided similar but more in-depth view of the trainees' reactions (Table 3). Most of the comments about the training program were positive. However, especially at the first batch of the training, some trainees suggested that there should be more practice and interaction in the training program and the trainers should pay more attention and provide more explanation to the trainees. From the second batch of the training, comments about the program and the trainers became more positive, and there was less criticism about the issues that were previously mentioned. In the second batch, some trainees suggested that the training should be more applicable to the situation of district hospitals, which also seemed to be improved from the third batch of the training program.

Table 3

Results of the focus group interview with the trainees

	1st Batch (n = 12)	2nd Batch (n = 12)	3rd Batch (n = 12)	4th Batch (n = 12)
Strengths	I gained knowledge and skills	I will transfer knowledge to my colleagues	It was useful	It was useful
	It was closely related to the work in district hospitals	Trainers paid attention to the trainees	Trainers paid attention to the trainees	Trainers are providing clear explanation
	It will help me perform better	Trainers were cooperative	Trainers were cooperative	Trainers have good interpersonal skills
	It will improve the primary care in district hospitals	Training is very helpful for district hospitals	It covered theory and practice	Trainers are paying close attention to the lesson
			It was applicable to district hospital situation	Topics covered theory and practice
				I will be able to apply new skills
				I will transfer knowledge to my colleagues
				It will improve the primary care in district hospitals
Weaknesses or suggestions	Trainer should pay more attention on the training	Trainees should do more practice	The training place was inconvenient
	Trainer should provide more explanation	The training should be more applicable to the situation of district hospitals
	Practical session should be increased
	There should be more interaction
	Teaching time should be longer

The average global rating scores of the medical records which were written after the training program were higher than those written before the training, except the second batch (Table 4). And the achievement of the average global rating scores between the medical records before the training and those after the training increased from the first batch to the last batch, especially between the second and the third batch (Supplementary Fig. 2).

Table 4

Results of the medical record review (n = 44, 5-point anchored scale*)

Parameters	1st Batch (n = 10)	2nd Batch (n = 10)	3rd Batch (n = 12)	4th Batch (n = 12)	F	Post hoc test (Tukey B)
Before the training	2.24 ± 0.57	2.67 ± 0. 54	2.38 ± 0.21	2.87 ± 0.57	3.53†	1=2=3< 4, 1<2=3=4
After the training	2.50 ± 0.48	2.59 ± 0.36	3.02 ± 0.39	3.82 ± 0.40	24.52‡	1=2<3<4
Achievement	0.26 ± 0.50	-0.08 ± 0.41	0.64 ± 0.48	0.95 ± 0.62	8.16‡	1=2< 4, 2<3=4
P value of paired t-test	0.124	0.529	< 0.001	< 0.001

*1, novice; 2, advanced beginner; 3, intermediate; 4, proficient; 5, expert; †P=0.024 by ANOVA; ‡P<0.001 by ANOVA.

DISCUSSION

The purpose of this study was to evaluate the effect of the formative program evaluation, which was applied to continuously improve the training program. According to the evaluation results, there was continuous improvement of the reaction and the transfer of the trainees from the first batch to the last batch of the training program. We might have a close look at the ways how the formative program evaluation contributed to the continuous improvement of the training program.

At the first batch of the training program, the trainees were not so satisfied in some areas like organization and schedule of the training program, and discussion, interaction, and practical sessions in the training program. Some trainees thought the training activities were not enough for them, and there should be more practical sessions like bed side teaching or real practice rather than lectures. Trainees also suggested that the trainer should pay more attention to them and there should be more discussion and interaction between the trainer and the trainees. The review of the medical records written by the first batch trainees showed a minimal achievement of global rating scores. These results of the formative program evaluation were shared with the committee and the trainers at the end of the first batch of the training program. The committee and the trainers reached a consensus on how to improve the training program through the group discussion based on the evaluation results. They decided to increase the training activities, especially practical sessions like bed side teaching and real practice under supervision. And even during the lecture time, they agreed to foster more interaction and discussion between the trainers and the trainees. The training management committee also emphasized to the trainers that the trainees are not young students, but experienced medical professionals.

At the second batch of the training program, the reaction of the trainees was much improved than the first batch. The average scores of all the items in the survey were much increased, and there was no item which was rated below 4.0. Also the trainee's comments, especially about the trainers, in the focus group interview became more positive than before. However, the review of the medical records written by the second batch trainees did not show any achievement of global rating scores. After a group discussion based on the results of the survey and focus interview, the training management committee and the trainers reached on a consensus that there still should be more practical sessions for the trainees, which later could be applicable to the medical practice in the district hospitals.

At the third batch of the training program, the trainees were more satisfied with the practical sessions and more trainees appreciated the applicability of the training program. And the achievement of global rating scores of the medical records was much increased comparing to the first and the second batch. The program evaluation results of the fourth batch showed a minimal improvement of the reaction and transfer of the trainees.

There were several key factors which should be noted in this study on the formative program evaluation. First of all, two levels, level 1 (reaction) and level 3 (transfer), of the Kirkpatrick model were selected for the program evaluation for the reason that the reaction of the trainees should be closely related to the implementation of the training program and the transfer of the trainees could be a reliable parameter of the outcome or impact of the training program (12). Second, the survey for the evaluation of the trainees' reaction was conducted every two weeks during the training program to improve the reliability of the ratings. It was to minimize the dominating effect of the trainee's emotional experiences of both the peak and the end of the training program (13). Third, focus group interviews were added to seek more deeply the problems of the training program which were discovered by the survey results. By this, the quantitative evaluation using a Likert scale and qualitative evaluation using focus group interviews, both of which have their relative merits (14), were integrated into the formative program evaluation process. Finally, to evaluate the transfer of the trainees, medical records which were written 3 months after the training program were collected, analyzed, and compared with those written before the training. It was based on the evidence from the previous studies that most meaningful changes would take place over a longer period of time than just right after the training (1516).

There were some limitations of this study, too. We were not able to assign a control group and conduct a case-control study due to the small size of the trainees group. And the formative program evaluation did not cover the level 2 (learning) and level 4 (result) of the Kirkpatrick model because of the feasibility and the time limit of the evaluation process. Further study is needed for more delicate study design and concrete results. However, even though it was a pilot clinical training program in Lao PDR, the plan for the formative program evaluation was designed from the beginning of the project, and was shared with the all the participants. The whole evaluation process was conducted by the Lao health professionals under the guidance and support of the study team.

In conclusion, the results showed that the formative program evaluation contributed to the continuous improvement of the training program.