EVALUATION OF A TRANSDISCIPLINARY CANCER RESEARCH TRAINING PROGRAM FOR UNDER-REPRESENTED MINORITY STUDENTS.

AIM/
PURPOSE: This paper will describe the initial development of systems to evaluate research education activities of a U.S. academic Partnership to train minority students as cancer researchers and provide preliminary data from monitoring Partnership activities during the first six months.
BACKGROUND: There is increased focus on multidisciplinary/transdisciplinary research training programs. However, few training programs have included detailed evaluations to assess their outcomes and effectiveness.
METHODOLOGY: The Temple University/Fox Chase Cancer Center and Hunter College Regional Comprehensive Cancer Health Disparity Partnership (TUFCCC/HC Cancer Partnership, or the Partnership) is a recently-initiated center focused on training individuals from under-represented minorities (URMs) as cancer researchers. Evaluation of the training activities involves detailed specification of goals for each of the Partnership's Cores; objectives for addressing each goal; measures and indicators to determine progress towards each objective; and data sources to provide information for the measures/indicators. CONTRIBUTUION: This paper will provide important information for evaluation of training programs focused on students from URM populations and development of trans-disciplinary research education programs.
FINDINGS: Goals, objectives, measures, and data sources for evaluation of the Partnership's Research Education Core (REC) were developed jointly by personnel from the REC and the Planning Evaluation Core (PEC) in an iterative process. These measures capture the training activities led by the REC (e.g., number of seminars and workshops), scientific output by trainees (e.g., papers and grants), and ability of the program to meet trainees' goals and expectations. RECOMMENDATIONS FOR PRACTITIONERS AND RESEARCHES: Evaluation plans for transdisciplinary training programs need to be developed prior to program initiation. Evaluation measures should be jointly specified by training and evaluation personnel, then reviewed and revised in an iterative process. IMPACT ON SOCIETY: This program is intended to enhance diversity among cancer researchers and increase studies to address disparities in cancer care. FUTURE RESEARCH: The PEC will oversee the evaluation of Partnership training activities over the five year period and assist Partnership leaders in ensuring successful outcomes.

INTRODUCTION

In the United States (U.S.), racial/ethnic minority subgroups bear a disproportionate share of the cancer burden. To achieve health equity, it is essential to understand the factors that underlie and drive cancer health disparities. A well-trained, diverse scientific workforce is a key component in addressing health disparities; however, investigators from diverse backgrounds are underrepresented across a number of fields, including biomedical research.

The Temple University/Fox Chase Cancer Center (TUFCCC) and Hunter College (HC) Regional Comprehensive Cancer Health Disparity Partnership (TUFCCC/HC Cancer Partnership, or the Partnership) is a recently-initiated center focused on training individuals from under-represented populations including racial/ethnic minorities and first generation college students in cancer research. The overall goal of the Partnership is to establish an integrated collaborative research infrastructure between TUFCCC and HC that develops and supports cancer research, education and outreach to eliminate cancer health disparities in underserved populations. The TUFCCC/HC Partnership involves a stewardship model that broadens the scope of cancer training to create new pathways to careers; prepares students and junior faculty to participate in multidisciplinary endeavors; engages scientists more fully in science and society: and recruits from traditionally underrepresented populations.

This paper provides background information on the Partnership; describes the initial development of systems to evaluate Partnership research education activities to train cancer researchers; and discusses preliminary data from monitoring Partnership training activities during the first six months.

METHODOLOGY

PARTNERSHIP MEMBERS

The Partnership includes:

Hunter College (HC).

Hunter College is a comprehensive teaching and research institution serving approximately 23,000 students, of which 70% are ethnic/racial minorities, 28% are firstgeneration college students, and 42% are recipients of federal need-based aid.

Temple University/Fox Chase Cancer Center (TUFCCC).

Temple University (TU) is a comprehensive public research university with more than 39,500 students enrolled in over 400 academic degree programs. Fox Chase Cancer Center (FCCC), which is affiliated with the Temple University Health System, is a National Cancer Institute-designated Comprehensive Cancer Center and a member of the National Comprehensive Cancer Network (NCCN).

PARTNERSHIP STRUCTURE

The Partnership is comprised of five cores:

Administrative Core.

The goals of the Administrative Core are to build on an effective leadership and governance infrastructure to provide administrative, scientific and programmatic oversight and monitoring, including timely coordination and communication, logistical support and budgetary management to ensure the successful achievement in high-quality standards of excellence in the proposed aims and objectives for all cores, pilot/full projects and shared resources.

Biostatistics and Bioinformatics Shared Resource Core (BBC).

The goal of the BBC is to facilitate and enhance research in the Partnership by providing expertise in experimental design, statistical analysis, bioinformatics, epidemiology, genomics, data management and statistical computing.

Community Outreach Core.

The goals of Community Outreach Core are to provide a community-based participatory research framework for the Partnership and develop and implement effective community outreach activities and intervention initiatives to reduce cancer disparities among underserved African, Asian and Hispanic American populations in the Eastern region of U.S.

Planning and Evaluation Core.

The goals of the Planning and Evaluation Core are to evaluate the short-term and long-term process and outcomes for the overall Partnership and each Core; to assess funded and new research projects; and, to provide recommendations to Partnership leadership to ensure the accomplishments of the overall goal of the Partnership.

Research Education Core.

The goal of the Research Education Core is to support educational activities that complement and enhance the training of a diverse workforce to meet the nation’s biomedical, behavioral and clinical research needs to address cancer health disparities.

RESEARCH EDUCATION CORE SPECIFIC AIMS

The Research Education Core (REC) addresses the following specific aims:

Develop and implement a comprehensive cancer research education program that provides research enrichment opportunities for undergraduate and graduate students from under-represented populations.

The Research Education Core will offer: (1) an 8-week intensive Summer Cancer Research Institute (SCRI) for undergraduate and graduate students in biomedical, clinical, or population-based cancer research; (2) ongoing skills-based training through mentored research projects; and (3) a rich curriculum of cancer seminars, workshops, journal clubs, and grand rounds throughout the academic year.

Provide mentorship and research support for URM cancer fellows (postdocs, junior faculty, or early-stage investigators) to enhance career development and facilitate their transition to independent investigators.

The Core will offer an established environment to support the career development of selected cancer fellows, who will receive close mentorship and support for two years, through the implementation of mentored research projects supported by pilot project funds.

Facilitate academic-community research opportunities to support the conduct of culturally appropriate cancer research in partnership with underserved minority communities and populations.

The Core will increase access to educational and research opportunities that utilize multidisciplinary approaches and engage underserved minority communities in addressing cancer health disparity research questions through collaborative academic-community partnerships.

PARTNERSHIP TRAINING ACTIVITIES

Summer Cancer Research Institute (SCRI).

The SCRI is an 8-week research intensive program that enables students to participate in a cancer curriculum and gain hands-on training in cancer research on the TUFCCC campus. Students will be matched with mentors for their research training. Trainees will attend seminars and will participate in mentored cancer research projects and skill-building workshops. At the conclusion of the program, students will be required to prepare a project abstract and oral presentation to their peers and to faculty advisors at the SCRI Symposium. During the SCRI, trainees can take advantage of other seminars and unique events that may be temporally relevant and occurring campus-wide.

Enriched training activities.

During the academic year, trainees will select specific enriched training activities in accordance with their individualized training plan and appropriate for their current needs and future career goals. These activities include:

Cancer Seminars. TUFCCC sponsors several cancer seminar series including a campus-wide noon cancer seminar throughout the academic year, a monthly Distinguished Lecture Series, and a postdoctoral seminar series.

Journal clubs and working groups. There are a number of topic-specific journal clubs that meet on a regular basis. Examples include journal clubs in Immunology, Cancer Biology, and Genetics. Similarly, a number of working groups meet regularly to address common themes including an Inflammation Working Group and a Health Disparities Working Group. The Health Disparities Working Group meets quarterly to address issues of health disparities in the local communities. Meetings include presentations by faculty and community partners on issues related to study design and community participation.

Professional/skill development workshops. Trainees can participate in various workshops offered throughout the year. Prior workshop topics have included: presentation skills; grant-writing workshops; English as a Second Language course; staff management skills; and networking. Because communicating effectively to different audiences is askill that all trainees should develop, the Core will host workshops to enhance trainees’ communication skills and confidence in engaging with diverse audiences.

Scientific meetings. Trainees will be encouraged to present their research at scientific meetings. The Core will support trainees’ travel costs and meeting expenses. National meetings also offer training workshops and professional networking opportunities.

Mentored research projects.

Trainees will participate in mentored research projects, which offer key experiences in laboratory training, conducting experiments, analyzing data, and reporting out-comes. This will provide trainees with skills in conducting literature reviews, practicing safe protocols in the lab, and using standard laboratory equipment or procedures. In combination with the training activities described above, implementation of these research projects will enable trainees to gain hands-on experience in developing a project from the initial idea, obtaining the appropriate regulatory (human subjects or animal use) approvals, data collection, and analysis and interpretation. The pilot data obtained from this project can be used to support applications for external funding. Trainees will also have multiple opportunities to present their findings to various audiences.

Both graduate students and postdoctoral fellows will have opportunities to participate in workshops and training activities on public speaking, professional networking, and effective mentorship. A major strength of the program is leveraging our strong network of secondary peer mentors — postdoctoral fellows and research associates — who can serve as peer mentors with shared experiences. These peer mentors provide not only informal mentoring but also social and peer support to help students develop strategies for addressing challenges.

FACULTY MENTORS

Core faculty provide mentorship and guidance to trainees on mentored research project proposals, career development plans, and grant applications. To ensure that trainees have multiple sources of support, a “team mentoring” approach is used, in which each trainee will have at least two faculty mentors. In addition, Core faculty facilitate community engagement, where appropriate, to increase the translational relevance of cancer research projects to underrepresented communities.

A Mentoring Advisory Committee (MAC) has also been convened. The overall function and responsibilities of the MAC are to: a) identify and assist with pairing mentees with appropriate mentors; b) provide oversight and guidance on mentorship activities; c) provide programmatic assistance to Core trainees and mentors, as needed; and d) facilitate career development and networking opportunities across institutions.

EVALUATION DATA COLLECTION

A key element for collection of data to evaluate the Partnership research training activities is to remove barriers (i.e., decrease burden) for submitting needed evaluation measures. This increases both the quality and the completeness of collected information (Holden & Zimmerman, 2008; Patton, 2009). To minimize the evaluation data collection burden, the Planning and Evaluation Core (PEC) uses a concurrent, real-time framework. Rather than delayed gathering of evaluation measures at later time points, with the potential accompanying recall bias, this permits collection of needed evaluation information with enhanced efficiency and quality. In addition, information on satisfaction with/utility of Partnership services, workshops, seminars, and other supports and events is collected via web-based anonymous survey and sent directly to the PEC. This encourages researchers, clinicians, trainees, and community members to provide honest and critical feedback regarding Partnership training activities and facilitate rapid quality improvement to address concerns.

FINDINGS

The Planning and Evaluation Core (PEC) collaborated with the REC to evaluate career development, mentorship, and other research education activities. The aims of the PEC include:

Establishing a planning and evaluation structural system to monitor progress, improve activities, and assist in achieving the overall goals of the Partnership to address cancer health disparities;

Establishing a systematic process to solicit and review new research project proposals to increase the competitive research capacity among TUFCCC/HC partners and promote research initiatives in cancer care for underserved minority populations, and

Conduct ongoing process, outcome, and impact evaluation of Partnership Cores and research projects to ensure proposed short-term and long-term outcomes are met.

The PEC oversees the evaluation of Partnership training activities and assists Partnership leaders in ensuring successful outcomes. The framework for the evaluation of Partnership research training activities involves three components:

Identification of Goals that describe the main research training activities of the REC;

Specification of Objectives to determine whether each Goal is successfully accomplished;

Listing Measures and Indicators used to assess progress towards completion of each Objective;

To guide and coordinate this planning and evaluation process, it was important to identify measures that appropriately indicate the status of processes and achievement of associated outcomes; collect data in an-ongoing process; analyze and interpret collected information; identify barriers, challenges, and unmet needs; and provide feedback to improve performance, increase quality, and provide greater effectiveness and efficiency. This is an iterative process, involving initial identification and collection of evaluation measures, followed by ongoing analysis and review of these measures; discussion of the relevance, completeness, and value (i.e., the time required or other burdens involved in collecting a measure relative to its importance to the evaluation) of measures; and refinement of the evaluation measures and data collection processes. The initial goals, objectives, and measures/indicators for the REC evaluation are presented in Table 1.

Table 1.

Goals, objectives, and measures/indicators for the REC Evaluation

GOALS	OBJECTIVES	MEASURES/INDICATORS
Develop and implement a comprehensive cancer research education program	• Establish admissions process for cancer research education program• Provides research opportunities for undergraduate and graduate students	Summer Cancer Research Institute [SCRI]:• Number of trainee applications and trainees selected for 8-week SCRI• Demographic characteristics of applicants and selected trainees• Seminars, workshops, journal clubs, and other training sessions offered in SCRI• Surveys examining achievement of educational goals and satisfaction among trainees• Trainee publications, presentations• Career milestonesEnriched Training Activities:• List of cancer seminars• List of journal clubs and working groups• List of workshops and skill development training events offeredMentored Research Projects:• Number of mentors participating• Number of trainees• Demographic characteristics of trainees• Satisfaction of trainees with group events and individual-level mentoring programs• Scientific meetings/conferences attended by trainees• Trainee publications, presentations• Career milestones
Provide mentorship and research to enhance career development and facilitate transition of URM cancer fellows to independent investigators.	• Support cancer fellows at the postdoctoral, junior faculty, or early-stage investigator level	• Number of mentors available/participating• Demographic characteristics of selected fellows & research interests• Number of seminars, workshops, journal clubs, and other training/mentoring sessions offered• Surveys examining achievement of educational goals and satisfaction among trainees• Fellow publications, presentations• Scientific meetings/conferences attended• Grant proposals submitted, grants funded• Career milestones
Facilitate CBPR opportunities to support the conduct of culturally appropriate cancer health disparities research with underserved minority communities and populations.	• Provide educational offerings/events to support academic-community research collaborations	• Number of REC training sessions & workshops• Attendance by researchers and community members at training sessions & workshops• Number of academic-community collaborative projects• Publications and presentations• Scientific meetings/conferences attended• # of grant applications submitted involving academic-community partnerships; # of grants funded• Researcher/community partner satisfaction survey

Using this approach, the PEC evaluates both processes and short- and long-term outcomes. In addition, the PEC evaluates trainee-reported utility of career mentoring activities on development of competencies in team-based science and cancer communication skills. This includes use of a Self-Assessment of Interdisciplinary Research Competencies instrument (Gebbie et al., 2008). Responses to this self-assessment instrument are collected at initiation of the training program; self-reported deficiencies are used to individualize career development and mentoring programs for each trainee. This instrument is re-administered each year to assess changes in self-assessed “team science” competencies.

Results of the 6 month evaluation of the Partnership research education activities, based on the measures/indicators indicated in Table 1, will be presented at the conference.

CONCLUSIONS

This paper describes the structures and processes for evaluation of research training activities in the TUFCCC/HC Regional Comprehensive Cancer Health Disparity Partnership, a newly-initiated center funded by the U.S. National Institutes of Health (NIH) that focuses on training individuals from under-represented minorities (URMs) as cancer researchers. The Partnership involves a unique collaboration between two organizations with a shared vision of advancing cancer equity among underserved and underrepresented minority populations through training and research. This program is intended to enhance diversity among cancer researchers and increase studies to address disparities in cancer care. A strength of the Partnership is a dedicated Planning and Evaluation Core (PEC) to monitor progress towards achieving research education goals and objectives; identify challenges in realizing these outcomes; and provide recommendations to successfully address these challenges.

Over the next five years, the Partnership will educate students from under-represented populations in cancer research. The evaluation of these research education activities will help ensure that students, mentors, and community members benefit from their participation with the Partnership and that the Partnership’s aims are successfully completed.