Does Pre-Survey Training Impact Knowledge of Survey Administrators and Survey Outcomes in Developing Countries? Evaluation Findings of a Training of Trainers Workshop for National AIDS and Reproductive Health Survey-Plus in Nigeria.

BACKGROUND: Although, Nigeria had conducted various national surveys followed by central and state level trainings for survey administrators, prior pre-survey trainings have not been systematically evaluated to assess their impact on knowledge gain and final outcome of the survey. A central training of trainers' session was organized for master trainers on the conduct of the 2012 National AIDS and Reproductive Health Survey.
OBJECTIVES: To evaluate the impact of training on the quality of conduct of a national research survey in the 36 states and the Federal Capital Territory in Nigeria.
METHOD: A total of 185 participants consisting of State AIDS Program Coordinators, Reproductive Health Coordinators, State Laboratory Scientists, Lead Supervisors and Counselor Testers were invited from the 36 states in Nigeria and the FCT for the central training of trainers in Abuja. The training lasted 5 days and the trainees were grouped into two on the basis of behavioral epidemiology and laboratory components. Training tools such as the developed protocol, training power point slides, practical sessions such as role plays, and usage of HIV rapid test kits were utilized during the training. The facilitators were drawn from Federal Ministry of Health (FMoH), universities and research Institutions as well as Non-Governmental Organizations (NGOs). The facilitators prepared and administered 25 structured questions for the behavioral group and 28 questions for the laboratory group at the beginning of the training to assess the participants' knowledge of HIV and the survey. The same questions answered by Trainees responded to the same questions prior to the commencement and at the end of the trainings. Scores were aggregated to 100 for each test. We conducted paired t-test to determine statistically significant differences between pre-test and post-test results at 0.05 significance level and ANOVA to determine if there were differences in knowledge level among different groups. RESULT: The overall mean pre-test and post-test scores were 64.0% and 77.4% respectively indicating a 13.4% knowledge gain above what it was at the beginning of the training. The mean pre-test score and post-test score for the Southern states (SN) were 64.7% and 80.3% while that of the Northern states (NN) were 63.5% and 75.3% representing a knowledge gain of 15.6% and 11.8% respectively. There was statistical significant difference in the post-test scores between the two regions (p=0.001) and in knowledge gained after the training (p=0.017). CONCLUSIONS AND PUBLIC HEALTH IMPLICATIONS: Comparison between the pre test and post test scores at the 5-day training showed a significant gain in knowledge of participants. The survey training contributed positively to the preparation and building of knowledge needed for the conduct of 2012 NARHS-plus.

Intorduction

Any organization determined to survive and prosper in the current challenging economy, must understand the imperative to invest in training and professional development in order to improve efficiencies in production as well as to acquire the greatest return in investment of human capital.[1] Furthermore, several authors have suggested that training is most extensive and important in any organization or establishment[2, 3]. Training is becoming increasingly complex, and tools to determine the proficiency of training programs and trainees are needed. Although emphasis has been placed recently on expanding assessment and demonstrating outcomes in proficiencies other than knowledge, evaluation of the depth and breadth of trainings’ knowledge base remains critically important, because this constitutes the foundation on which competence is built, and is an essential requisite for the development of sound reasoning skills.[4, 5] Therefore, training as a tool to acquire best practices in the proficient conduct of services cannot be over emphasized.

National AIDS and Reproductive Health Survey (NARHS) had been conducted three times in Nigeria (2003, 2005, and 2007)[6, 7, 8]. After 2005 NARHS survey, there was a need to include biomarker-HIV testing (biological) components into the survey and it captures ages 15-64 years, hence the nomenclature of NARHS-plus was therefore adopted in 2007[8]. The 2012 NARHS was the fourth since its inception. In the past, NARHS and NARHS-plus data collection started with central level training of States AIDS Program Coordinator (SAPC), Reproductive Health Coordinator (RHC), State Laboratory Scientists (SLS), HIV Counselor-testers (CTs), and Lead supervisors but the impact of the trainings on the participants have not been evaluated to assess whether there is need for such trainings or not, during the survey. The training of trainers (TOTs) workshop was aimed at teaching selected survey administrators on how to conduct and supervise the NARHS plus survey based on the complexity and enormity of the study especially the ‘plus’ component. The goal was to replicate same training at the state level to other supervisors, interviewers, and counselor-testers who served as research assistants in the collection of data on the field during the survey exercise.

The training was conducted using the engagement of participants in plenary discussions, presentations, group discussions, role plays, and question and answer sessions. The various sessions exposed the participants to issues of community engagement and participation in research, basics of HIV and AIDS, challenges of ensuring informed consent, reproductive health indices in Nigeria, good clinical and laboratory practice and management, data monitoring and management in research. The group interactions afforded participants the opportunity to learn from one another’s experience while the role plays simulated actual practical experiences in the field. The trainees were expected to give same training in their respective states. Based on the above, this evaluation study set out to assess the impact of five-day training on the quality of conduct of research survey, namely, NARHS-plus. The overall evaluation research questions were: does training of trainers at the central level have any impact on the level of knowledge that could predicate a successful conduct of NARHS-plus survey in Nigeria, and are there variations in the level of knowledge among participants from different states or different regions of the country?

METHODS

Study Design:

This study was an evaluation of training using a cross sectional design with structured questionnaire pre and post training.

Target population

The target populations were all State AIDS Program Coordinator (SAPC), State Reproductive Health Coordinator (SRH), State Laboratory Scientist (SLS) from all 36 states and the Federal Capital Territory (FCT) in Nigeria. In addition there were one Lead Supervisor (LS) and one Counselor Tester (CT) from each state and FCT.

Sampling technique

Total sampling of the target population was done giving 185 trainees invited for the central training. They were divided into two regions- North and South for convenience. Seventeen states from southern Nigeria had their training for 5 days followed by that for the FCT and 19 states from northern Nigeria for another 5 days.

Training methods

The training was conducted for 5 days. The five days training of trainers (TOTs) workshop for NARHS –Plus 2012 survey for SAPCs, RHCs, SLS, CTs and lead supervisors from the 36 states of Nigeria and FCT was conducted between 27th September to 7th October, 2012. The training sessions took place in Maraba, Nassarawa State for the Southern States and in Abuja for the Northern States.

The training methods were didactic, role plays and hands-on. Training included the use of training tools such as the training protocol, training slides, and engagement in practical sessions on the use of HIV rapid test kits. At the venue of training, the trainees were grouped into two on the basis of behavioral epidemiology (SAPC, SRH and LS) and laboratory components (SLS and CT). The facilitators were drawn from the Federal Ministry of Health (FMOH), academic institutions as well as supporting organizations. Facilitators prepared 25 and 28 structured questions on knowledge of HIV and the survey for the behavioral and laboratory tract respectively, scores were aggregated to 100. Trainees answered the same set of questions before (pre-test) and after (post-test) the trainings.

Data analysis

We conducted descriptive analysis, used summary statistics, paired t-test at 0.05 significance level to determine statistically significant difference between pre-test and post-test performances of the trainees while ANOVA was used to determine the differences in knowledge level among different groups.

RESULTS

One hundred and seventy (91.9 %) trainees participated in the pre-test while 177 (95.6 %) attempted the post-test, 158 attempted both pre and post-test. Also, the number of trainees that participated in the pre-test, post-test and took both during the training for the Southern Nigeria (SN) were 72, 82 & 68 and Northern Nigeria (NN) were 98, 95 & 90 respectively. Combining all the respondents irrespective of their zones, the overall pre-test and post-test score were 64.0% and 77.4% respectively indicating a 13.4% gain in knowledge over the baseline knowledge level (Table 1). The pre-test score and post-test score for the SN and NN were 64.7% and 80.3% respectively, and 63.5% and 75.3%. The knowledge gain was 15.6% and 11.8% for SN and NN respectively (Table 1).

Table 1

Attendees’ Performances in the Pre-test, Post-test and Differences between the tests

Regions	Pre-test (100)	Post-test (100)	Differences (100)
Northern States (n)	95	98	90
mean	63.52	75.31	11.78
SEM	1.14	1.15	1.26
Southern States (n)	82	72	68
mean	64.65	80.34	15.68
SEM	1.13	1.17	1.35
Both (n)	177	170	158
mean	64.0	77.42	13.42
SEM	0.80	0.85	0.93

n = number of participants/zone SEM= standard error of error

Comparing the performances in the two sets of training, Table 2 shows that the baseline knowledge was not statistically different among trainees from SN and NN (p=0.44), however there was statistical significance in the posttest scores between the two regions (p=0.001) and in knowledge gained (p=0.017).

Table 2

Comparison of mean scores in pre-test and post-test across the two regions using ANOVA

		Sum of Squares	df	Mean Square	F	Sig.
Pretest & Region	Between SN & NN	68.682	1	68.682	.598	.440
	Within SN & NN	20087.361	175	114.785
	Total	20156.044	176
Posttest & Region	Between SN & NN	2182.115	1	2182.115	12.13	.001
	Within SN & NN	30447.740	170	179.104
	Total	32629.855	171
Knowledge gain & Region	Between SN & NN	1224.840	1	1224.840	5.803	.017
	Within SN & NN	33348.362	158	211.066
	Total	34573.202	159

Figure 1 shows the mean scores of the attendees from each state in the two tests as well as the differences between the scores. The highest mean scores for the pretest, posttest and their differences were recorded in Delta (87.9%), Akwa Ibom (74.0%), and Lagos (72.4%) states respectively while the lowest mean scores were from Kwara (54.1%) -pretest, Sokoto (66.4%) - posttest and Sokoto (3.4%) knowledge gain. In Table 3, the scores in the pretest and the posttest were compared, the analysis of variability of the scores showed that the mean scores for the states were statistically significantly different (p<0.05) in both tests but there was no significant difference in knowledge gain of the attendees across all the states in the area of knowledge of HIV epidemiology, HIV counseling, AIDS- related issues, survey administration etc.

Figure 1

Distribution of scores in the pre-test, post-test and knowledge gained across the states

Table 3

Variability in performances of attendees across their states

		Sum of Squares	d	Mean Square	F	Sig.
Pretest * State	Between the States	6357.561	36	176.599	1.792	.009
	Within the States	13798.482	140	98.561
	Total	20156.044	176
Posttest * State	Between the States	6602.086	36	183.391	1.699	.016
	Within the States	14354.276	133	107.927
	Total	20956.363	169
Knowledge gain * State	Between the States	4743.494	36	131.764	.951	.555
	Within the States	16768.020	121	138.579
	Total	21511.514	157

While the highest mean scores for the pre-test was in the Counselor Testers’ group (67.2%); post-test- Supervisors’ group (79.1%); and knowledge gain was recorded in SAPCs’ group (17.6%). The lowest mean scores for the pre-test was SRHCs’ group (60.4%), post test was the Laboratory Scientists’ group (73.1%), and knowledge gain was also the Laboratory Scientists’ group (8.0%). The highest knowledge gain was recorded among the SAPC (17.6%), followed by the SRHC (16.7%). The overall mean pre-test and post-test scores were 64.0% and 77.4% respectively indicating a 13.4% gain in knowledge above baseline knowledge. (See Figure 2 and Table 4.)

Figure 2

Performance of attendees by their designations and regions

Table 4

Distribution of Pre-test and Post-test Scores and Knowledge gain across the categories of the attendees.

Designation		Pretest100	Posttest100	Difference100
Counselor/Testers	n	35	33	32
	mean	67.2442	77.7074	10.7161
	SE(Mean)	1.72945	1.33728	1.51283
Lab Scientists	n	37	37	34
	mean	65.7317	73.0701	7.9847
	SE(Mean)	1.69773	1.80256	1.74589
Supervisors	n	39	35	31
	mean	65.1282	79.0857	15.0968
	SE(Mean)	1.54080	1.68900	1.94292
SAPCs	n	35	34	33
	mean	62.0571	78.5882	17.5758
	SE(Mean)	1.87237	2.12200	2.28923
SRHCs	n	31	31	28
	mean	60.3871	76.0000	16.7143
	SE(Mean)	2.07684	2.41857	2.46073
Total	n	177	170	158
	mean	64.0	77.4	13.4
	SE(Mean)	.80438	.85406	.93123

Although there was no statistically significant difference in the pretest and posttest mean scores of the five groups of attendees, the mean scores of knowledge gained across the five groups were statistically significantly different (p<0.05) (see Table 5).

Table 5

Variability in performances of attendees across the various groups

		Sum of Squares	df	Mean Square	F	Sig.
Pre-test * Designation	Between Designations	1055.952	4	263.988	2.377	.054
	Within Designations	19100.092	172	111.047
	Total	20156.044	176
Post-test * Designation	Between Designations	852.966	4	213.241	1.750	.141
	Within Designations	20103.397	165	121.839
	Total	20956.363	169
Knowledge gain * Designation	Between Designations	2198.709	4	549.677	4.355	.002
	Within Designations	19312.805	153	126.227
	Total	21511.514	157

In an attempt to explore the details of the significant knowledge gain obtained in Table 5, we carried out the dependent sample t-test on the post-test and pre-test scores across different characteristics were as shown in Table 6. The differences between post-test and pre-test scores were statistically significant within the Southern attendees, North Attendees, the two zones combined, various designation groups irrespective of their zones.

Table 6

Analysis of Knowledge Gained (differences between the paired Posttest and Pretest scores) by the attendees

Categories	Paired Differences		95% CI of the Difference		t	df	Sig.
	Mean	SE(Mean)	Lower	Upper
Regions	South	15.685	1.349	12.991	18.379	11.621	67	0.000
	North	11.82	1.255	9.324	14.315	9.412	89	0.000
	Overall	13.483	0.931	11.644	15.322	14.479	157	0.000
North	Counselor/Testers	7.776	1.568	4.45	11.102	4.957	16	0.000
	Lab Scientist	5.452	2.258	0.707	10.197	2.414	18	0.027
	Supervisor	16	2.856	9.974	22.025	5.602	17	0.000
	SAPC	12.631	2.873	6.594	18.669	4.395	18	0.000
	SRHC	17.647	3.29	10.67	24.623	5.362	16	0.000
South	Counselor/ Testers	14.047	2.475	8.737	19.358	5.674	14	0.000
	Lab Scientist	11.192	2.581	5.655	16.728	4.336	14	0.001
	Supervisor	13.846	2.506	8.386	19.306	5.525	12	0.000
	SAPC	24.285	2.978	17.85	30.72	8.153	13	0.000
	SRHC	15.272	3.806	6.79	23.755	4.012	10	0.002
Overall	Counselor/Testers	10.716	1.512	7.63	13.801	7.083	31	0.000
	Lab Scientist	7.984	1.745	4.432	11.536	4.573	33	0.000
	Supervisor	15.096	1.942	11.128	19.064	7.77	30	0.000
	SAPC	17.575	2.289	12.912	22.238	7.678	32	0.000
	SRHC	16.714	2.46	11.665	21.763	6.792	27	0.000

Similar to the analysis shown in Table 6, we analyzed the paired (post-test and pre-test) differences among attendees from each state. The differences between post-test and pre-test scores were statistically significant (p<0.005) among the attendees from Abia, Adamawa, Bayelsa, Edo, Ekiti, Katsina, Lagos, Nassarawa, Niger, Ogun, Ondo, Oyo, Plateau, Taraba and Yobe states only. As shown in Table 7, most participants (51.5%) believed that the training sessions were very good.

Table 7

Evaluation of the training by the participants

s/n.	Item	Very Poor	Poor	Good	Very Good	Excellent
1.	Publicity for the training	3.0%	7.3%	27.4%	45.7%	16.5%
2.	Communication with participants prior to arrival	3.0%	7.8%	24.0%	41.3%	24.0%
3.	Preparation of participants for the training prior to arrival	3.0%	7.8%	31.9%	40.4%	16.9%
4.	Feeding	0.6%	13.9%	33.9%	29.7%	21.8%
5.	Accommodation and other logistics	8.3%	16.7%	23.2%	31.0%	20.8%
6.	Hospitality	1.8%	11.7%	31.9%	33.7%	20.9%
7.	Responsiveness to logistic challenges	2.4%	10.7%	34.9%	43.8%	8.3%
8.	Networking opportunity	0.0%	6.8%	44.1%	39.1%	9.9%
9.	Training Materials	1.2%	13.2%	40.1%	36.5%	9.0%
10.	Quality of the training	0.0%	2.4%	18.3%	51.8%	27.4%
11.	Facilitation (Conduct, Performance, Time Keeping)	0.6%	0.6%	26.2%	51.2%	21.4%
12.	Sessions and their contents	0.0%	1.2%	23.5%	55.9%	19.4%
13.	Arrangement of sessions	0.0%	2.4%	27.6%	52.9%	17.1%
14.	Overall, how would you rate the training?	0.6%	4.1%	24.6%	51.5%	19.3%

Discussion

This study was aimed at evaluating the impact of five-day training on the quality of conduct of research survey of NARHS plus in order to answer the research question; does training of trainers at the central level have any impact on the level of knowledge and successful conduct of NARHS survey in Nigeria since there has been no recent systematic analyses of the quality and comprehensiveness of training received in any national survey in Nigeria.

The study revealed that the participants have positive attitudes about training (training attitudes), since the training afforded them the opportunity to acquire additional knowledge in basic issues concerning HIV/AIDS and its counseling and testing, and build data collection skills to be able to be comported to conduct interviews as well as manage a research study of this magnitude. It also afforded them the opportunity to learn about national requirements and regulations. It is clear from the training results that recent trainees feel very well prepared (well trained and competent) in many areas, particularly in HIV survey. Pre-test level of knowledge, supported by the improved post-test scores (North and South) is reasonable and indicates a right selection of methods and training participants.

For participants, the training afforded them the opportunity to acquire new knowledge and build skills to be able to conduct interviews as well as manage a research study of this magnitude. It also afforded them the opportunity to learn about national health survey requirements and regulations. Despite the relatively small number of respondents, this training provides the only current and detailed assessment of training of HIV survey in Nigeria across a wide spectrum of learning and content areas. Nevertheless, the results have potentially important implications for HIV surveillance and education in Nigeria. The train-the-trainer program was effective in developing sustainable quality NARHS plus in Nigeria as demonstrated by the fact that there is increase in knowledge base of the participants. In addition, the central training helped build a cadre of trainers who will be able to do an effective step down training throughout the states of the federation as seen in the result.

The recent evaluation of learning strategies used by United Nations Children’s Fund (UNICEF) in resource-limited settings noted that training local professionals to train their colleagues is generally less expensive than sending national or international experts to conduct trainings[9]. In addition, the use of local trainers to train their peers has the advantages of building local capacity as well as ensuring the trainings have cultural relevance and application which will help to enhance learning. Thus, it is likely that this central training model will continue to be applied to assist in ensuring that there is uniformity in knowledge impartation at the state levels. However, efforts will be made to mitigate differences in quality through use of competency-based curricula, well-designed training programs and, when needed, implementation of performance and quality improvement methodologies.

In another study[10] on the effect of co-presenting training items during supervised classification learning of novel relational categories, in a test phase measuring learning and transfer, the comparison group significantly outperformed a control group receiving an equivalent training session of single-item classification learning. In a similar study on Family Physician (trainers and non-trainers) and their practices to see whether there were differences in trainers and non-trainers and in how their practices were organized and their services were delivered, Trainers scored higher on all but one of the items, and significantly higher on 47 items, of which 13 remained significant after correcting for covariates. Trainers (and training practices) provided more diagnostic and therapeutic services, made better use of team skills and scored higher on practice organization, chronic care services and quality management than non-training practices[11].

Limitations

The qualities of training at the state levels were not evaluated, to ascertain the same level of delivery as the one at the central training. The result was only based on increase knowledge base of the participants after the central training. Further evaluations needed to be done whether the people trained at the various states acquire same level of improvements as seen at the central training. The paper has demonstrated that the trainees need training as indicated by the knowledge gained noted to be significant. However, the paper did not demonstrate how the knowledge gained has improved the conduct of the survey or the quality of data collected during the survey because the survey itself is yet to be concluded. For same reason, it could not relate the knowledge acquired during the trainings to the quality of data generated on the field from the southern and northern zones. This will be assessed after the conclusion and dissemination of survey findings nationwide.

conclusions and Public Health Implications

This is the first report on the effect of a central training of trainers on the effective conduct of NARHS plus survey in Nigeria. However, it is important to follow-up on the training at the state level to see the impact of the knowledge gained at the central training and using the skills they have acquired. This information allows the organizers to determine future training needs, either by zones or otherwise. Results on the increase in knowledge base of this central training program in developing trainers are also significant, providing a basis of comparison for future programs. This finding is comparable to similar evaluations of TOT models, such as that conducted by UNICEF which found the TOT trainees going on to provide step down training to their colleagues.[9]. Although this was a central program, the lessons learned – in terms of factors contributing to program success and the ways in which challenges were addressed – may be applicable in the implementation of any such training program in the future.

In conclusion, our evaluation of this central training program demonstrates that a TOT-based central training program can be successfully endorsed for an effective conduct of surveys in Nigeria, with the ability to rapidly scale-up human capacity for both service delivery and training in a sustainable fashion.