Quantitative Evaluation of the Community Research Fellows Training Program.

CONTEXT: The community research fellows training (CRFT) program is a community-based participatory research (CBPR) initiative for the St. Louis area. This 15-week program, based on a Master in Public Health curriculum, was implemented by the Division of Public Health Sciences at Washington University School of Medicine and the Siteman Cancer Center.
OBJECTIVES: We measure the knowledge gained by participants and evaluate participant and faculty satisfaction of the CRFT program both in terms of meeting learning objectives and actively engaging the community in the research process. PARTICIPANTS: We conducted analyses on 44 community members who participated in the CRFT program and completed the baseline and follow-up knowledge assessments. MAIN OUTCOME MEASURES: Knowledge gain is measured by a baseline and follow-up assessment given at the first and final session. Additionally, pre- and post-tests are given after the first 12 sessions. To measure satisfaction, program evaluations are completed by both the participants and faculty after each topic. Mid-way through the program, a mid-term evaluation was administered to assess the program's community engagement. We analyzed the results from the assessments, pre- and post-tests, and evaluations.
RESULTS: The CRFT participants' knowledge increased at follow-up as compared with baseline on average by a 16.5 point difference (p < 0.0001). Post-test scores were higher than pre-test scores for 11 of the 12 sessions. Both participants and faculty enjoyed the training and rated all session well.
CONCLUSION: The CRFT program was successful in increasing community knowledge, participant satisfaction, and faculty satisfaction. This success has enhanced the infrastructure for CBPR as well as led to CBPR pilot projects that address health disparities in the St. Louis Greater Metropolitan Area.

Introduction

The benefits of community-based participatory research (CBPR) have been well established; however, there is limited information on developing the infrastructure and increasing community capacity to partner in CBPR projects. Participating in public health research training can prepare community members for collaborative work with academic researchers and empower them to act as equal partners in the research process (1, 2). In an effort to alleviate mistrust, foster community-academic relationships, and educate community stakeholders in St. Louis, the Division of Public Health Sciences at Washington University School of Medicine (WUSM), and the Siteman Cancer Center, began the community research fellows training (CRFT) program. Aligned with CBPR principles, this program has the potential to be mutually beneficial for the participants, training faculty, and the local health research community.

In 2010, a similar training program, Community Alliances for Research Empowering Social Change (CARES), was initiated on Long Island, New York (3, 4). This program was developed based on feedback from community members of the Suffolk County Minority Health Action Coalition at a mini-summit on Minority Health focused on CBPR (5). Participants newly introduced to this concept felt these approaches could really benefit their communities but requested training in research methods to address concerns about their limited knowledge about research and their ability to be equal partners in a process they did not know much about.

The CARES academic-community collaboration developed a training program for community members based on the standard Master of Public Health curriculum, designed to implement culturally appropriate ways to increase research literacy among community members (4). The CRFT program built upon the model created by the CARES training program. The St. Louis-based CRFT program expanded the goals of the CARES program, adding topics to the curriculum, based on CARESs participant evaluations and input from the community advisory board to make it culturally tailored and region specific. After the CARES program evaluation (3), there was discussion that the success of the CARES program was specific to the suburban region and small class size (6). The CRFT program more than doubled the program size and was conducted in urban St. Louis, as opposed to the former Long Island, New York suburban location, in order to assess the generalizability of this CBPR program’s approach. This paper provides a quantitative evaluation of the CRFT program. To this end, we measure the knowledge gained and evaluate participant and faculty satisfaction with the CRFT program to determine whether learning objectives and goals were met through this CBPR approach.

Background and Rationale

Community-based participatory research emerged from research traditions of 1980s and 1990s that focused on engaging stakeholders affected by the public health concerns at hand (7). Community engagement is a powerful instrument in bringing about positive social and community health change (8). Active community member engagement in the research process improves health outcomes, health promotion and prevention, and institution-community relationships (3, 4, 9–12). Community engagement requires a long-term process that builds trust, values contributions of all stakeholders, and generates a collaborative framework (13). CBPR is an effective vehicle to speed up the elimination of the mortality and morbidity disparities consistently seen among minorities, low-income, and other vulnerable populations (14). The success of CBPR is dependent on the strong formation of community-researcher relationships; in order to engage communities to collaborate with researchers to address identified health concerns, researchers must build trust and rapport with community members by maintaining a consistent presence (4, 12, 15, 16).

Several studies suggest that medical mistrust and negative encounters with health care personnel are closely linked to racial disparities in health (17, 18). Attempting to reduce racial disparities is complicated by medical mistrust among other barriers that reduce participants’ willingness to actively engage in medical research (19). CBPR has been shown to be effective in ameliorating or abating some of these issues by engaging underserved communities (9, 20) and has emerged as an evidence-based approach to address the complex issues that affect the health of marginalized populations using innovative and effective community–academic partnerships to address health disparities (21–23). Over the years, CBPR has become valued as an effective research strategy for improving community health and reducing health disparities (24–32). Community engagement can contribute to a more nuanced understanding of health problems, increasing the relevance of problems examined (33–35), improving the quality and outcomes of health promotion activities, disease prevention initiatives, and research studies (10, 31, 36).

Methods

Community research fellows training program

The CRFT program was a pilot project of the Program to Eliminate Cancer Disparities at the Siteman Cancer Center (SCC), Barnes Jewish Hospital, and WUSM. The goal of CRFT was to train community members to serve as the bidirectional conduit between WUSM/SCC and communities in St. Louis. A community advisory board (CAB) was formed to help guide all aspects of the CRFT program including recruitment and acceptance decisions, program implementation, selection of pilot projects, and evaluation of the program. The objectives of this training were to

Enhance community knowledge and understanding of research

Create a pool of trained community members who can serve on Institutional Review Boards and community research advisory boards

Develop CBPR pilot projects that address health disparities in the St. Louis Greater Metropolitan Area

Enhance the infrastructure for CBPR

Provide community members with skills to engage as equal partners in every phase of the research process.

The CRFT program sought to involve community members in research methods training designed to help them to become good consumers of research; understand the utility of research in improving health outcomes in their communities, increase their understanding of how to work with academic researchers, and develop skills that increase capacity for organizations and communities to engage in research. To this end, a 15-week training course, adapted from the CARES program, was designed and implemented (4). It is important to note that a community group was convened to consider the faculty rationale and strategy for implementing the program. Both the faculty and the community members were focused on ways to facilitate academic researchers and community members as co-equals. The advisory group reviewed the materials from the CARES program and believed that a similar program could benefit the community. The course is comprised of 25 topics, divided across 12 didactic training sessions and 3 experiential workshops, held weekly (Thursday evenings 6–9 p.m.) April–August 2013. Each session is a condensed 3 hour lay-friendly version of Master of Public Health (MPH) curriculum topics including health literacy, ethics, cultural competency, epidemiology, quantitative and qualitative research methods, chronic disease prevention, clinical trials, study design, program evaluation, and grant writing. Each session was led by one or two faculty members recruited to teach in the program by the Principal Investigator; Table 1 lists the session topics and learning objectives from the course syllabus given to participants at orientation.

Table 1

Session topics and learning objectives.

	Topics	Learning objectives
Session 1	Community health	Define Public Health Define Community Health Identify contributing factors that impact the health of a community Describe community health activities Describe the Healthy People 2020 initiative Assess the need for a program
Session 2	Research methods/data	Define research Describe the steps of the research process Identify and explain research methodology Identify appropriate research methods and techniques Define data Compare and contrast quantitative and qualitative data Compare and contrast primary data and secondary data
Session 3	Public health research/health disparities	Define public health research Identify and explain types of research Identify and explain types of research methods Explain why research is important Define health disparities Identify major health disparities in the St. Louis including those by gender, race/ethnicity, geographic location, and socioeconomic status Understand and provide example of causes of health disparities with respect to prevention, incidence, and mortality Discuss the social determinants of health Describe public health strategies and interventions for reducing health disparities
Session 4	Public health library resources/health literacy	Understand public resources available at Becker Medical Library Describe library sources useful for public health research Define health literacy Understand the limited literacy perspective Describe the association between literacy and health Describe health literacy on a national scale Discuss current WUSTL research on health literacy
Session 5	Cultural competency	Define cultural competency Describe the need for culturally competent research and practice based on a historical perspective Identify contributing risk factors for health disparities Identify skills associated with cultural competent practices Conduct cultural competency self-assessment Develop SMART goals for programs and projects Identify culturally competent evaluation approaches Understand the importance of evaluation
Session 6	Introduction to epidemiology/evidence- based public health/community-based prevention	Define epidemiology Identify major contributions of epidemiology Identify frameworks for understanding disease processes Compare and contrast observational studies vs. clinical trials Define evidence-based public health Discuss methods for community-based prevention
Session 7	Quantitative methods	Identify strengths and weakness of quantitative methods Describe strengths of mixed-methods approaches Describe stages of questionnaire design Identify sampling methods Understand usefulness of statistics in health research Understand p-values and odds ratios
Session 8	Community-based participatory research	Describe history and principles of CBPR Critically evaluate their own position within their community(ies) and their potential roles within CBPR projects Describe methods to ensure that CBPR research benefits all partners Lessons learned from CBPR projects CBPR efforts in St. Louis
Session 9	Research ethics I and II	Define research ethics and bioethics Compare and contrast clinical ethics vs. research ethics Identify examples of unethical practices in research Understand ethical theories and professional ethical duties Identify historical milestones in ethics Understand the Belmont Report Understand NIH – IRB Protocol Review Standards
Session 10	Qualitative methods	Define basic principles of qualitative research methods Describe the characteristics of qualitative research Describe the advantages and disadvantages of qualitative methods Understand and distinguish between different types of qualitative approaches Facilitate qualitative interviews and focus groups Understand the relationship between qualitative and quantitative research methods Discern when a qualitative research design is desirable
Session 11	Clinical trials and bio-banking	Understand clinical trials research Describe the role of clinical trials research in advancing medical practice Discuss the impact of minority participation in clinical trials research Define bio-repository Describe the type of research conducted from bio-repository data Discuss the risks and benefits of minority participation in bio-repositories
Session 12	Health policy research/human subjects certification I	Define health policy and health services research Identify and develop relevant well framed health policy research questions Describe public use and other common data sources for health policy research Participants will be certified in the conduct of human subjects research Conduct an informed consent process to recruit a participant in a research study Develop a humans subjects and HIPPA compliant research proposal
Workshop 1	Research synthesis/research evaluation	Describe the research process Identify the components of research design Develop a conceptual model Develop a research hypothesis Conduct a literature review Develop an evaluation plan for a research project or proposal Determine appropriate evaluation metrics and measures
Workshop 2	Human subjects certification II/history of healthcare in St. Louis	Participants will be certified in the conduct of human subjects research Conduct an informed consent process to recruit a participant in a research study Develop a humans subjects and HIPPA compliant research proposal Respond to the request for proposal (RFP) Development of pilot project ideas
Workshop 3	Family health history/grant writing	Understand importance of collecting and maintaining a family health history Understand the role of family health history in healthcare Complete a family health history chart Understand grant guidelines and requirements Understand the power of collaboration for grant writing Develop SMART goals and specific aims Define a project and develop a research plan Develop a collaborative grant proposal including background and significance, specific aims, preliminary studies, research design and methods

The information found in this table was previously provided in the Supplemental Section of Coats et al. (37) and can be found here: http://jre.sagepub.com/content/suppl/2014/12/10/1556264614561959.DC1/DS_10.1177_1556264614561959_T1.pdf

Twelve CAB members, 17 faculty members, and 10 research assistants were involved in the conception, planning, and implementation of the CRFT program (37). While participants are not compensated they do receive free training and resources; to further empower the participants and engage them in the academic process, throughout the CRFT program, we referred to our community members as “fellows” similar to those on fellowship at an academic institution (from this point forward, we will do the same here). Fellows were recruited through The St. Louis American (a local newspaper; 32%), E-mail (20%), radio (14%), community websites (2%), community newsletters (6%), flyers (8%), personal referrals (6%), and word of mouth (12%) during January–March 2013. Of the 62 applicants, 50 (81%) were accepted into the program, which was double the program target of 25. The inclusion criteria for the CRFT program required participants to be at least 18 years old and live or work in the St. Louis greater metropolitan area. Applications were reviewed by the CRFT CAB such that the participants were selected to be purposefully diverse in composition in terms of work, education, and life experiences.

The training uses multiple teaching approaches (large didactic interactive lectures, small group activities, group exercises, and small and large group discussions) to explain topics in ways that reach a variety of learning styles (37). Consistent with CBPR principles and the needs of adult learners, participants provided feedback on the most feasible day and time for the course. Pedagogically, each session is formatted to support active class participation that encourages adults to draw upon their experiences, group activities that require problem solving and direct application of the lecture material, supplemented by homework assignments that foster independence and self-direction through application of the material to their communities (38). Using CBPR approaches, this training program recognizes the contribution participants can make to the learning process and the diversity of the cohort foster example-based learning using culturally competent region specific scenarios. Further information about the program development, recruitment strategies, training structure, program implementation, pilot projects, and best practices can be found elsewhere (37).

The Human Research Protection Office at WUSM classified this study as program evaluation and non-human subjects’ research. Analysis was completed using SAS/STAT 9.4 (Cary, NC, USA). Of the 50 fellows enrolled in the CRFT program, 45 (90%) completed the 15-week training program and 44 (88%) completed both the baseline and follow-up assessments. The majority of the fellows who completed both the baseline and follow-up assessments were female (84%) and African-American/Black (86%). The fellows ranged in age from 28 to 72, with a mean age of 51. The cohort of fellows was comprised of community members (30%), those affiliated with community-based organizations (23%), health care workers (21%), government workers (11%), those affiliated with faith-based organizations (9%), and those in academia (7%). Education attainment ranged from junior high school to those with a graduate degree. Over half (57%) of fellows self-reported previously taking a research course (Table 2). The 6 fellows who did not complete the training and/or final assessment were similar to the 44 who completed the training and the final assessment; all were African-American/Black females, ranging in education attainment from a high school diploma to graduate degree. Their affiliation varied from faith-based organizations, community-based organizations, academia, and community members with no organizational affiliation. The mean age was slightly lower, 48 (range 36–57). For consistency, from this point forward, the sample will be the 44 fellows who completed both the baseline and follow-up assessments.

Table 2

Demographic characteristics of CRFT participants in evaluation sample (.

Characteristic	n	%
Gender
Female	37	84.1
Male	7	15.9
Race
African-American/Black	38	86.4
White	6	13.6
Education attainment
Graduate degree	23	52.3
Bachelor’s degree	7	15.9
Some college/associate’s degree	12	27.3
High school diploma	1	2.3
Junior high or some high school	1	2.3
Affiliation
Faith-based organization	4	9.1
Healthcare worker	9	20.5
Community-based organization	10	22.7
Academic	3	6.8
Government	5	11.4
Community member	13	29.6
Previously taken a research course
Yes	25	56.8
No	19	43.2
Age (years)
Mean	51.4
SD	10.6

Assessment of participant knowledge

The fellows’ baseline and follow-up assessments were linked using their CRFT ID numbers. Each assessment consisted of 29 identical open-ended questions created to measure the fellow’s knowledge of the CRFT training topics (Table 1). The baseline questionnaire also included the Patient Trust in Medical Researchers scale (39), an adaptation of the Computer Engaged Research Index, multiple levels of empowerment indices (40, 41), and two health literacy assessments, the Newest Vial Sign and the Rapid Assessment of Adult Literacy in Medicine (37, 42–45). This analysis only focuses on the initial 29 open-ended assessment questions, administered at both baseline and after the completion of CRFT, in order to assess the participants’ knowledge gain. For the full baseline and final questions with sample answers, see Coats et al. (37). A baseline and final score were computed for each individual by summing the individual question’s scores, with a possible score of 0, indicating an incorrect answer, 1 indicating a partially correct answer, or 2 indicating an essentially correct answer. Baseline and follow-up assessments were graded by two graders using a rubric to ensure consistency (37). We created a sum score for each individual for both the baseline and follow-up assessments. To assess the difference between the baseline and follow-up assessment, we performed a paired t-test after finding no evidence against normality using the Shapiro–Wilk test. We then looked at fellows’ responses to see if each individual question’s score increased, decreased, or remained the same from baseline to follow-up.

For each question, we examined the percent that scored “essentially correct” on the baseline assessment as compared to the percent that scored “essentially correct” on the follow-up assessment. We analyzed the data in a two-way contingency table, assessing the change from baseline to follow-up. To assess whether the contingency table is symmetric, we used McNemar’s test, with the null hypothesis that the questions are answered correctly or incorrectly at the same rate from baseline to follow-up.

Fellows took pre- and post-tests at each of the 12 didactic training sessions. These pre- and post- tests were developed by the CRFT team and approved by the teaching faculty member each week. The first eight sessions consisted of five multiple-choice questions on the pre- and post-tests. The final four sessions consisted of 10 multiple-choice questions on the pre- and post-tests. Questions assessed the learning objectives the faculty member intended to cover during the weekly session; pre and post assess the same content but use different items. Pre- and post-tests were scored by a team of CRFT research assistants using a SAS macro developed for this purpose (46). Due to the violation of normality assumptions, we used a non-parametric test, the Wilcoxon signed-rank test, to evaluate the score differences on pre-test and post-test for each session.

Evaluation of participant satisfaction

At the end of each of the 12 didactic training sessions, fellows completed evaluations. Evaluations consisted of six statements with Likert-scale response options and four free response questions; eight of the sessions had two parts, and therefore, two evaluations, making for a total 20 evaluation questions. We focus on the quantitative Likert-scale response questions for this analysis. Fellows were asked to rate the following statements from 1 to 5, with 1 being “Strongly Disagree” and 5 being “Strongly Agree” for evaluation questions 1–5, and 1 being “Poor” and 5 being “Excellent” for the sixth evaluation question:

The exercise learning objectives were met.

The information learned in this session was helpful.

I understood the concepts presented in this session.

The facilitator(s) were well organized.

The facilitator(s) seemed knowledgeable about the subject.

Overall, how would you rate this session?

The mean for each statement is computed, as well as a range of within-session means.

The CRFT CAB felt it was important to also assess satisfaction with teaching in the program. A main goal of the CRFT program is to create a bridge between the St. Louis community and researchers; to foster this connection, faculty satisfaction is imperative. Fifteen faculty members taught the 12 didactic training sessions and completed evaluations. The CRFT faculty evaluation consisted of seven Likert response items:

Community Research Fellows seemed well prepared for today’s training session

Community Research Fellows frequently took notes in the training session

Community Research Fellows contributed to discussions in the training session

Community Research Fellows provided comments that were insightful and constructive

Community Research Fellows asked insightful and constructive questions

Community Research Fellows listened attentively and seemed interested when I presented materials and information

Overall, how would you rate your experience teaching this session?

Items 1–6 have Likert-scale responses 1 to 5, with 1 being “Strongly Disagree” and 5 being “Strongly Agree” and item 7 has Likert-scale responses with 1 being “Poor” and 5 being “Excellent.” In addition, there were three yes/no items:

Would you be willing to teach again for the CRFT program in the future?

Did you learning anything from the Fellows during the training session?

Are you willing to collaborate with Fellows on a CBPR pilot project?

This analysis only examines the 10 close-ended questions on the faculty evaluation which also included 3–4 opened ended response questions depending on answers to the close-ended questions. The mean, SD, minimum and maximum for each of the 7 Likert response items were computed. The three dichotomous response questions were examined by the percent reporting yes to each question.

Results

Overall, there is evidence of knowledge gained, with the average score increasing from 20.6 on the baseline assessment to 37.1 on the final assessment (mean change of 16.5, range −7, 41). This corresponds to an absolute percent increase of 28.4% (range 12, 70.7%). Only three fellows (out of 44, 6.8%) decreased their scores from baseline to follow-up. The paired t-test for this knowledge change was highly significant (p < 0.001). Of the 29 questions, on average fellows answered about 7 (23.5%) questions essentially correctly at baseline (mean, 6.8; SD, 7.0; median, 4.0) and 16 (54.8%) questions essentially correctly at follow-up (mean, 15.9; SD, 6.7; median, 15.0). We then looked at fellows’ responses to see if each individual question’s score increased, decreased, or remained the same from baseline to follow-up (Table 3). At least 50% of the fellows increased their score for 12 (41.4%) of the 29 questions. The three questions with the greatest percent increase in score were defining bio-repository or biobank [36 (81.8%) increased their score from baseline to follow-up], defining the Belmont Report [28 (63.6%) increased their score from baseline to follow-up], and explaining the difference between primary and secondary data [26 (59.1%) increased their score from baseline to follow-up]. The three smallest differences were for naming two of the most common data sets used for health policy research [5 (11.4%) increased their score from baseline to follow-up], defining the term ethnography [11 (25.0%) increased their score from baseline to follow-up], and describing the health promotion planning model that is best to prevent and reduce substance abuse in African-American communities [12 (27.3%) increased their score from baseline to follow-up].

Table 3

Baseline to follow-up question score changes (.

	Score decreased n (%)	Score remained the same n (%)	Score increased n (%)	% CRFT fellows answered essentially correct		McNemar’s test
				Baseline	Follow-up	S statistic	p-Value
1. What is informed consent?	7 (15.9)	17 (38.6)	20 (45.5)	31.8	61.4	7.35	0.012*
2. What is the Belmont report?	2 (4.6)	14 (31.8)	28 (63.6)	9.1	47.7	17.00	<0.001*
3. What is Tuskegee experiment?	4 (9.1)	26 (59.1)	14 (31.8)	50.0	72.7	6.25	0.021*
4. Define health literacy	5 (11.4)	21 (47.7)	18 (40.9)	29.6	59.1	9.94	0.002*
5. Define evidence-based public health	2 (4.5)	21 (47.7)	21 (47.7)	27.3	56.8	13.00	<0.001*
6. Define cultural competency	3 (6.8)	20 (45.5)	21 (47.7)	31.8	70.5	15.21	<0.001*
7. What role does the Institutional Review Board play in research?	3 (6.8)	17 (38.6)	24 (54.6)	36.4	77.3	14.73	<0.001*
8. What is HIPAA?	7 (15.9)	14 (31.8)	23 (52.3)	29.6	70.5	13.50	<0.001*
9. Explain the difference between quantitative and qualitative research methods	2 (4.6)	16 (36.4)	26 (59.1)	27.3	63.6	12.80	<0.001*
10. What is the difference between primary and secondary data?	1 (2.3)	17 (38.6)	26 (59.1)	20.5	54.6	13.24	<0.001*
11. Explain the difference between community-based participatory research and traditional research	5 (11.4)	14 (31.8)	25 (56.8)	27.3	65.9	12.57	0.001*
12. What is epidemiology?	6 (13.6)	16 (36.4)	22 (50.0)	20.5	56.8	11.64	0.001*
13. What is a bio-repository or biobank?	2 (4.6)	6 (13.6)	36 (81.8)	6.9	79.5	30.12	<0.001*
14. What is a clinical trial?	4 (9.1)	18 (40.9)	22 (50.0)	13.6	43.2	11.27	0.001*
15. What is the mixed method approach?	1 (2.3)	19 (43.2)	24 (54.6)	20.5	68.2	21.00	<0.001*
16. Define the term ethnography	2 (4.6)	31 (70.5)	11 (25.0)	6.8	27.3	9.00	0.004*
17. What is the purpose of focus groups?	5 (11.4)	19 (43.2)	20 (45.5)	29.6	63.6	13.24	<0.001*
18. What is the overarching goal of healthy people 2020?	3 (6.8)	16 (36.4)	25 (56.8)	22.7	72.7	18.62	<0.001*
19. What type of information should you expect to get from a community health assessment?	7 (15.9)	23 (52.3)	14 (31.8)	27.3	47.7	6.23	0.023*
20. Describe the health promotion planning model that you believe is best to prevent and reduce substance abuse in an African-American community	8 (18.2)	24 (54.6)	12 (27.3)	11.4	22.7	2.27	0.227
21. What are the social determinants of health?	11 (25.0)	16 (36.4)	17 (38.6)	27.3	40.9	2.25	0.210
22. List three social determinants of health	11 (25.0)	19 (43.2)	14 (31.8)	54.6	61.4	0.47	0.648
23. What is research?	10 (22.7)	15 (34.1)	19 (43.2)	31.8	61.4	7.35	0.012*
24. Define racial health disparities?	4 (9.1)	17 (38.6)	23 (52.3)	25.0	72.7	21.00	<0.001*
25. What are the components of a SMART goal?	5 (11.4)	21 (47.7)	18 (40.9)	13.6	40.9	10.29	0.002*
26. What is an odds ratio?	5 (11.4)	25 (56.8)	14 (31.8)	2.3	27.3	9.31	0.003*
27. What is a p-value?	3 (6.8)	26 (59.1)	15 (34.1)	4.6	31.8	10.29	0.002*
28. List an effective method to advocate for a specific health issue in your community	9 (20.5)	20 (45.5)	15 (34.1)	43.2	56.8	2.00	0.238
29. Name two of the most common used data sets for health policy research	6 (13.6)	33 (75.0)	5 (11.4)	11.4	13.6	0.14	1.0000

*.

*.

For each of the 29 questions, we examined which fellows scored “essentially correct” on the baseline assessment as compared to scoring “essentially correct” on the follow-up assessment. Using McNemar’s test, we found that 24 of the 29 questions had statistically significant results, suggesting that there was evidence of improvement from baseline to follow-up (Table 3). This further suggests that there was not only overall score improvement but also question level improvement for the majority (82.8%) of the questions.

We then examined the sample stratified by whether or not the fellow had taken a previous research course. Twenty-five of the 44 fellows (56.8%) self-reported previously taking a research course. This group had a mean difference between the baseline and follow-up assessment scores of 13.4, or 23.1% absolute increase. The remaining 19 fellows (43.2%) had a mean difference between baseline and final assessment scores of 20.6, or a 35.5% absolute increase in score. The scores in both groups are normally distributed, with a Shapiro–Wilk p-value of 0.778 and 0.783, respectively. We performed a two-sample t-test and found that the score differences are statistically significant, with a p-value of 0.038, indicating that although both groups had significant overall increase of knowledge from the baseline to the follow-up assessment, fellows who had not previously taken a research course had a significantly higher increase in knowledge. This suggests that both fellows who had taken a research course previously and those who had not, gained knowledge from the CRFT program, and that the knowledge gained was more substantial for those who had never taken a research course. This further emphasizes that this course is well designed, both for those who have some previous knowledge about research, and for those who lack it completely.

Comparisons for the mean percent of correct scores on pre- and post-tests at each session showed that in 11 of the 12 sessions, post-test scores were higher than pre-test scores; one session, Public Health Research, had an average post-test score lower than the pre-test score. Based on the Wilcoxon signed-rank tests, sessions 6 (Epidemiology), 8 (CBPR), 9 (Research Ethics), 10 (Qualitative Methods), and 11 (Clinical Trials) post-test scores were significantly higher than pre-test scores (p  = 0.001, <0.001, <0.001, <0.001, and 0.028, respectively); the post-test score for session 3, Public Health Research, was significantly lower than the pre-test score (p  = 0.003) (Table 4).

Table 4

CRFT pre-/post-tests scores and score difference (post-test minus pre-test) by session.

Session	n	Pre-test percent		Post-test percent		Score difference		Wilcoxon signed-rank test
		Mean	SD	Mean	SD	Mean	SD	p
1. Community health	42	76.67	18.70	81.90	13.11	5.24	24.22	0.189
2. Research methods	43	80.00	21.82	85.12	16.96	5.12	28.32	0.229
3. Public health research	39	88.21	12.75	75.90	17.88	−12.31	23.67	0.003*
4. Health literacy	41	79.51	18.16	80.49	16.42	0.98	26.44	0.747
5. Cultural competency	36	85.56	16.29	90.00	13.09	4.44	21.97	0.269
6. Epidemiology	41	74.63	22.37	88.29	17.87	13.66	25.47	0.001*
7. Quantitative methods	40	55.50	21.95	64.50	23.75	9.00	31.36	0.067
8. CBPR	39	71.28	16.41	89.23	15.11	17.95	22.85	<0.001*
9. Research ethics	39	74.87	14.85	87.69	11.80	12.82	15.55	<0.001*
10. Qualitative methods	42	61.90	19.54	72.14	12.79	10.24	14.73	<0.001*
11. Clinical trials	38	77.37	13.69	81.32	11.19	3.95	10.54	0.028*
12. Health policy research	41	61.71	15.64	65.12	16.90	3.41	17.26	0.278

*.

*.

Overall, session evaluations were high, with all session mean evaluation scores between 4 and 5 on a five point Likert-scale, with 4 indicating “Good” or “Agree” and 5 indicating “Excellent” or “Strongly Agree” depending on the question (Figure 1). The average response to the first evaluation statement, assessing whether the learning objectives were met, is 4.4, with the individual session averages ranging from 4.2 to 4.6. The session with the highest mean in terms of this first statement was second session, part two, Data. The average response to the second statement, whether information learned during the session was helpful, is 4.5 with a mean range from 4.1 to 4.7. The session with the highest mean in terms of the second statement is session three, Public Health Research. The average response to the third statement, whether the fellow understood the concepts presented in the session, is 4.3 with the individual session averages ranging from 3.7 to 4.6. The session with the highest average response to this item is session 8, CBPR. The average response to the fourth statement, whether the facilitator was well organized, is 4.6 with the individual session averages ranging from 4.1 to 4.8. The session with the highest average response to this item is session 7, Quantitative Research. The average response to the fifth statement, whether the facilitator seemed knowledgeable, is 4.7 with the individual session averages ranging from 4.3 to 4.9. The session with the highest average response to this item is the second half of session four, Health Literacy. The sixth item, the overall session evaluation, has a mean of 4.5 with a range from 3.7 to 4.8. The session with the lowest overall evaluation is second session, part two, Data. This is intriguing, as this session also ranked the highest for the first item, evaluating if the exercise learning objectives were met, suggesting that fellows ranked sessions overall based on more than whether the learning objectives were met. The session with the highest overall mean evaluation is the fourth session, part two, Health Literacy. As seen above, this session also ranked as having the most knowledgeable session facilitator. This information is useful for evaluating the importance of faculty involvement in this program. Averaging all six items for each session, the mean within-session evaluation is 4.5 with a range from 4.1 to 4.7. This indicates that, on average, responses for all items for each session was between “Agree” and “Strongly Agree” or “Good” and “Excellent.”

Figure 1

Fellows’ evaluations. ***Overall mean evaluation score, averaging all six Likert-scale evaluation responses from each fellow.

Faculty evaluations are an integral part of the CRFT program’s evaluation. To note, 15 (100%) of the faculty members responded that they would be willing to teach again for the CRFT program. Thirteen (87%) responded that they learned something from the fellows during the training session and 15 (100%) responded that they are willing to collaborate with fellows on a CBPR pilot project. The mean overall faculty rating of their teaching experience is 4.8 (Table 5). This indicates that the majority of the faculty members rated the experience as “Excellent” on a Likert-scale from 1-“Poor,” to 5-“Excellent.” This further suggests a CBPR success.

Table 5

CRFT faculty evaluations. Likert-scale responses ranged from 1 to 5, with 1 being “Strongly Disagree” and 5 being “Strongly Agree” or 1 being “Poor” and 5 being “Excellent.”

Question	N	Mean	SD	Minimum	Maximum
Community Research Fellows seemed well prepared for today’s training session	15	4.67	0.49	4	5
Community Research Fellows frequently took notes in the training session	15	4.40	0.63	3	5
Community Research Fellows contributed to discussions in the training session	15	4.93	0.26	4	5
Community Research Fellows provided comments that were insightful and constructive	15	4.87	0.35	4	5
Community Research Fellows asked insightful and constructive questions	15	4.93	0.26	4	5
Community Research Fellows listened attentively and seemed interested when I presented materials and information	15	4.87	0.35	4	5
Overall, how would you rate your experience teaching this session?	15	4.8	0.56	3	5

Discussion

We assessed participant knowledge gain and evaluated participant and faculty satisfaction. The CRFT program evaluation results build upon and enhance the positive results demonstrated in the CARES program. The completion rate in the CRFT program was much higher than that in the CARES program; 90% of the fellows completed the 15-week training, as opposed to 68% in the CARES program, and 88% completed the baseline and follow-up assessments, as compared to 58% in the CARES program (3). Additionally, attendance rates were higher in the CRFT program, and the sample increased from 11 completing both the baseline and follow-up in the CARES program to 44 completing the baseline and follow-up in CRFT. Our findings suggest that the participants did gain knowledge through this program, as evidenced through the significant increase in score from baseline to follow-up assessment and the average increase in all but one of the pre-/post-tests. We also found that, while both participants who had previously taken a research course and those who had not experienced significant increases in assessment scores from the baseline to follow-up assessment, those who had never taken a research course experienced higher increases in scores on average. This suggests that the CRFT program is well designed, both for those who have some previous knowledge about research, and for those who lack it, and, as would logically follow, those who have no previously research knowledge will learn substantially more. We believe that it was important for the CRFT program to have a diverse group of fellows in terms of previous knowledge as some of the learning occurs in small groups with other classmates. Though we have a modest sample size (n = 44), we chose robust tests to evaluate all aspects of the CRFT program.

This quantitative evaluation analysis is just one component of the comprehensive (formative and summative) mixed-methods (quantitative/qualitative) evaluation of the CRFT program. The use of multiple items to assess participant knowledge and satisfaction allows for triangulation of results. For example, Session 3 Public Health Research was the only session to see a significant decrease in score on the post-test compared to the pre-test. However, in the session evaluation, participants said that the information learned in the session was helpful; more so than any other session (highest mean). This suggests a need to refine items on assessment instruments, specifically pre- and post-tests. Session pre- and post-tests consisted solely of multiple-choice items to help reduce participant burden and time of administration. However, we needed to increase the number of questions (from 5 to 10) to increase the variability of question types (easy, medium, hard) thus increasing variability in scores across participants; a major limitation is some participants felt like they were taking a test. Inconsistency between pre-test and post-test scores could be due to differences in items used on the assessments. In future work, we examine the use of the same items (in a different order) on the pre-/post-test.

The participant and faculty evaluations were similarly positive, with all averages above four on a five point Likert-scale, averaging between “Good” and “Excellent” or “Agree” and “Strongly Agree” depending on the question. The overall satisfaction, both from the participant and the faculty member’s perspective, is crucial in evaluating a CBPR program. Our assessment of the knowledge gained by fellows through the CRFT program confirms that the first, second, and fifth CRFT program goals have been met. Our evaluation of participant and faculty satisfaction suggests the fourth program goal has been met. In terms of the third program goal, the CRFT program has funded two pilot projects, led by participants in the program, and two more projects have been funded by other sources; we believe that CRFT has been instrumental in enhancing the CBPR infrastructure in St. Louis, opening the doors to collaboration in the future. CRFT pilot projects extend Fellows learning beyond the classroom setting to implementing CBPR projects in real community-based settings, demonstrating the ability of the course to create community–academic partnerships that address community driven research questions (37).

Conclusion

The CRFT program was successful in increasing participant knowledge in public health topics, such as epidemiology, CBPR, research ethics, and clinical trials, as evidenced by the increased scores from the baseline to final assessment and positive difference in pre- and post-tests administered after each session. At the initiation of this program, we set out to achieve five goals.

The first goal was to enhance community knowledge and understanding of research. From a quantitative perspective, there is evidence of knowledge gained with an improvement from 20.6 on the baseline assessment to 37.1 on the final assessment (mean change of 16.5) and the mean increase in post-test scores compared to pre-test scores. This demonstrates that the course content (Table 1), was at an appropriate level for the incoming fellows, despite the diversity in education background. Additionally, the evaluations completed after each session overwhelmingly suggest that the fellows felt that the learning objectives were met during each session, again demonstrating that the communities’ knowledge and understanding of research was enhanced by this program.

The second goal was to create a pool of trained community members who can serve on Institutional Review Boards and community advisory boards. While the quantitative data here suggest that the fellows, on average, would have the knowledge base to serve on these boards, the actual outcomes more clearly support this objective. CRFT alumni have formed their own Patient Research Advisory Board and are currently serving on at least five advisory boards and community partnerships across the WUSM medical center; five CRFT alumni current serve on the CRFT community advisory board.

The third goal was to develop CBPR pilot projects that address health disparities in the St. Louis Greater Metropolitan Area. The knowledge gained, as demonstrated here, was a springboard for 9 groups consisting of a total of 30 fellows, to submit brief proposals for pilot projects. Four of these were funded, and are currently underway in the St. Louis Greater Area (37).

The fourth goal was to enhance the infrastructure for CBPR. The gathering of community members and faculty for bidirectional learning was successful in developing CBPR infrastructure, and the positive participant and faculty evaluations suggest both groups enjoyed this approach to partnership development. Additionally, the four pilot projects demonstrate that the foundation laid by this course created a conducive environment for community members to become involved in the research process.

Finally, our fifth goal was to provide community members with skills to engage as equal partners in every phase of the research process. Our course covers the spectrum of the research process, from research methods, to quantitative and qualitative analysis, to research ethics, synthesis, and evaluation. We demonstrate here, through the increase in scores, that knowledge was gained in these areas, giving the participants the information needed and pilot projects provide an opportunity for this goal to be realized. Overall, the CRFT program was successful, as evidenced by the knowledge gained, the positive participant and faculty evaluations, and the movement toward initiating fellow-inspired community-centric pilot projects.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.