The REDIH experience: an emerging design to develop an effective training program for graduate students in reproductive science.

BACKGROUND: A training program in Reproduction, Early Development, and the Impact on Health (REDIH) was initiated in 2009 by researchers specializing in biomedical, clinical, population health, and ethics research from seven collaborating universities in Quebec and Ontario, and Health Canada. This paper reports the findings from the first three years of the 6-year program.
OBJECTIVES: The objective of the REDIH program is to provide increased opportunities for excellent training in reproduction and early development for graduate students and fellows, in order to build research, clinical, regulatory, decision-making, and industry capacity in Canada.
METHODS: A mixed methods approach was used to evaluate the REDIH training program, so as to combine the strengths of both qualitative and quantitative studies. A total of four focus groups (two with mentors and two with trainees) were run during the June 2012 REDIH meeting. Surveys were administered directly after each training module. The W(e)Learn framework was used as a guide to design and evaluate the program and answer the research questions.
RESULTS: The data from the analysis of the focus group interviews, in corroboration with the survey data, suggested trainees enjoyed and benefited from the REDIH experience. Trainees provided several examples of new knowledge and skills they had acquired from REDIH sessions, regarding reproductive and early developmental biology, and health. A few trainees who had been in the program for over a year provided examples of knowledge and skills acquired during the REDIH session that they were using in their place of work. Next steps will include following up on REDIH graduates to see if the program has had any impact on trainees' employment opportunities and career development.
CONCLUSION: Trainees and mentors concluded that the curricular design, which focuses on modules in 2-day learning sessions over a 6-year period, with opportunities for application in the workplace, enabled the sessions to be tailored to the outcomes of the formative evaluation. By sharing our experiences with REDIH, we hope that others can benefit from this unique emerging design, which focuses on the flexibility and receptivity of the mentors, and results in a program that lends itself to curriculum modification and tailoring as learners' needs are solicited and addressed.

Introduction

Quality education programs are typically characterized by extensive upfront planning and design work, requiring a considerable investment of time and specialized resources.1 A well-designed program is also the culmination of clear learning objectives, relevant content, effective instructional methods, a comfortable learning environment that reflects both learning goals and learners’ developmental needs, and is one that incorporates continuous evaluation to ensure constant improvement.2

The Reproduction, Early Development, and the Impact on Health (REDIH) program, with its biannual modular curriculum, research, and application components, has developed a balance between structure and flexibility, which evolved as a result of using an emergent program design.3 The emergent design is facilitated by the modular approach to curriculum and a strong focus on formative evaluation, with ongoing feedback to trainees, program mentors, and administrators. As the trainees shared their opinions on what would make the program more useful for them, the program designers made adjustments to accommodate the trainees’ preferred learning styles, desire for more practical and specific content, and greater opportunities for application. One key finding from the evaluation of the REDIH program after 3 years in operation was the importance of program facilitators and the design team responding to emerging learning needs. Moreover, this training program benefits from being conducted over a 6-year period. The luxury of time provides the opportunity to implement changes in response to trainees’ feedback, in particular their efforts at application of content in the workplace.

Among increased calls for more robust training experiences and outcomes, the design of a learning experience is emerging as a critical cornerstone of quality.1,4 Ye t our experience with the REDIH program suggests that quality training experiences must also be responsive to participant learning needs as they emerge throughout the program. In the REDIH program, perceptions of quality were strongly linked to fit between participants’ experiences, needs, wants, perceived competencies, a formal evaluation process, the facilitators’ respect for participants, and responsiveness to the evaluation feedback.

Effective evaluation should assess both acquisition of knowledge and skill and its transfer to the workplace, with evidence-based findings to support these changes. The most important purpose of evaluation is not to “prove but to improve” the education program.5 A number of program evaluation models and frameworks that focus on the transfer of knowledge to the workplace have emerged, shifting the evaluation focus from outcomes (evidence) to process (practice).2,6–11 Process evaluation enables practice developers to evaluate as they develop, making evaluation an integral embedded part of practice development rather than an “add on” activity.12 There have been a number of evaluation strategies shifting the focus from proving to improving, utility-led evaluating, and participatory evaluation, all with the basic premise of understanding and improving the learning process.

In this study, the focus is on understanding the learning process and knowledge transfer in a complex health care environment. Therefore, the W(e)Learn framework, which was specifically designed to be used to guide the design, delivery, and evaluation of health care education programs, was used.2

The W(e)learn framework

The W(e)Learn framework2 was originated from the demand driven learning model (DDLM) and the DDLM companion evaluation tool as a quality standard to evaluate the process and outcomes of a program, gain an understanding of stakeholders’ (eg, learners, designers) experiences and perspectives with the process, and identify lessons learned in order to provide recommendations for future eLearning initiatives.13,14 The DDLM demonstrates that carefully designed educational programs with the appropriate blend of factors can help achieve desired outcomes and act as a mechanism for managing complex social systems.15

The W(e)Learn Framework is grounded in social constructivist learning theories and has sophisticated and tested guidelines to support instructional design. The framework also provides well defined learning outcomes and their connection to instructional components. In addition, the W(e) Learn framework has embedded flexibility to guide the design and evaluation of learning programs.16

Since publication, W(e)Learn has been used in numerous studies.16,17–20 The W(e)Learn framework and the companion tools have been used jointly to develop and evaluate programs.19 For example, in Canada, they have been used in provincial level eLearning programs, including ePhysician Health (http://ephysicianhealth.com) and eWorkplace Health (http://eworkplacehealth.com). In the United States, a team at the University of North Carolina at Chapel Hill has used the W(e)Learn toolkit over four sequential terms to implement an eLearning program on patient safety as part of the Agency for Healthcare Research and Quality TeamSTEPPS® initiative (http://teamstepps.ahrq.gov).16

The W(e)Learn framework supports collaborative online and blended learning. The “(e)” in brackets suggests that web technologies can potentially bridge approaches that integrate a strong focus on both collaboration and effective learning experiences. It acknowledges that different blends of factors (eg, delivery modes, theories, pedagogical approaches, media, environments, and communication strategies) are necessary to meet the needs of various learners. Designed to be applicable regardless of the blend of these factors, W(e) Learn’s deliberate versatility makes it useful for developing online, blended, or face-to-face learning programs.16

W(e)Learn is grounded in a community orientation. Learning communities and communities of practice are two types of communities supported within W(e)Learn. In a learning community, a group of learners comes together for a set period of time. It is marked by joint learning tasks and a sense of “we are in this together to accomplish something”. Through interactions, it is possible for a learning community to become a community of practice – a group of people who learn together, have a common orientation to work (context, profession, skills and knowledge), and want to stay connected.2 The emphasis on health care education, emerging design, and learning community make W(e)Learn the logical choice for a framework to guide the design, delivery and evaluation of the REDIH program (Figure 1).

Figure 1

W(e)learn.

Note: Reproduced with permission from Casimiro L, MacDonald CJ, Thompson TL, Stodel EJ. Grounding theories of W(e)Learn: a framework for online interprofessional education. J Interprof Care. 2009;23(4):390–400.11

Purpose

The purpose of this paper is to share the results of the first 3 years of the REDIH 6-year program. Both trainees and mentors pointed out there appeared to be something very special about the REDIH program. Trainees and mentors concluded that the curricular design, which focuses on modules in 2-day learning sessions over a 6-year period with opportunities for application in the workplace, enabled the sessions to be tailored to the outcomes of the formative evaluation. By sharing our experiences with REDIH, we hope that others can benefit from this unique emerging design focusing on the flexibility and receptivity of the mentors and resulting in a program that lends itself to curriculum modification and tailoring as learners’ needs are expressed.

Summary of the REDIH program

The objective of the REDIH program is to provide increased opportunities for excellent training in reproduction and early development for graduate students and fellows, in order to build research, clinical, regulatory, decision-making, and industry capacity in Canada. REDIH builds on three successful, collaborative, and interdisciplinary research networks that were previously funded by Strategic Initiative research program grants from the Canadian Institutes of Health Research (CIHR). REDIH itself is funded by the CIHR Strategic Training Initiatives in Health Research program. Considerable added-value is provided by cross-program training in ethical/legal/social issues, semiannual group meetings with a broad range of training modules, off-site workshops, and training opportunities provided in research groups and in clinical settings. Details on REDIH can be found at http://redih.ca.

Objectives of the REDIH program evaluation

The basic principle is that learning modules and the overall REDIH program are evaluated by the learners (trainees) who participate in them, as well as by the mentors. Various quantitative and qualitative data collection methodologies are used throughout the course of the REDIH program to allow participants’ voices to be heard and to obtain objective evidence regarding the participants’ perceptions of the learning resources and the impact of the learning program on the specified outcomes.

The following overall research questions were used as a guide for the evaluation of the REDIH program:

How did trainees react to the learning experiences?

Did trainees acquire new knowledge and skills regarding reproductive and early developmental biology and health?

Was there a change in trainees’ attitudes toward the value and use of fostering and strengthening interdisciplinary, interpillar Canadian research teams? (Pillars in this context refer to the four CIHR research pillars, which are biomedical, clinical, health systems and services, and the health of populations).

Did trainees transfer knowledge into their research projects, and interactive teaching? Did this result in:

(a) increased exposure to research addressing current and future clinical needs, and ethical, legal, and social issues that address emerging issues and critical thinking; and

(b) support for becoming independent investigators?

Did participating in the learning program help trainees become more effective professionals? If so, in what ways?

Focus group protocol questions (see Supplementary materials A) were designed for both the trainees and mentors based on the results and recommendations arising from the previous 2 years of the research project.

Methodology

Research approach

A mixed methods approach was used to evaluate the REDIH training program for the reason of combining the strengths of both qualitative and quantitative studies.21 In this mixed methods approach, quantitative and qualitative data were collected concurrently to obtain a full understanding of the research questions. This method offsets the weaknesses and complements the strengths of the quantitative and qualitative research approaches.22–24 The mixed methods approach was used as no single explanation can account for the feasibility of the program (pluralism), as these training modules are established from a complex real-world practice. In this study, mixed methods were useful to triangulate the data from the surveys with the data from the focus group interviews. In addition, the focus group data enabled us to obtain rich data and delve deeper into the issues that emerged from the survey questions. Together, the quantitative and qualitative data provided complete data to address the research questions.

Participants

Semiannual training sessions for mentors (university faculty and federal government research scientists; experienced researchers) and trainees (MSc, PhD students, and postdoctoral fellows) were the main venues for the diverse training modules (workshops in presentation skills, peer review and writing skills, knowledge translation, regulatory issues, careers in industry, and ethics, to name a few). Trainees were selected for REDIH through a competitive process, after already having been accepted for graduate or postdoctoral training by mentors who are members of REDIH. Trainees receive a scholarship of approximately 12,000 Canadian Dollars per year for 2 years. In the first year of the REDIH program, there were eleven trainees. Every year thereafter, an additional eleven trainees were selected. Some trainees wanted to attend REDIH sessions after their 2 year funding period ended. Therefore, at each REDIH session, we now have approximately 25 trainees and approximately the same number of mentors.

Trainees and mentors from the participating universities (medical and health science faculties) were brought to a central location for two biannual face-to-face training sessions held with trainees and mentors. Mentor and trainees travel expenses were covered by the REDIH project. The sessions ran from 9 am to 5 pm for 2 days, with a networking dinner held at a local restaurant for all trainees and mentors during the first evening. The trainees do not receive any credits or fellowship at the end of the program. Several of the MSc and PhDs go on to do PhDs and postdoctoral fellowships. Some trainees use REDIH as a venue to network and find an advisor for their next degree or fellowship.

Qualitative data collection

A total of four focus groups (two with mentors and two with trainees) were run during the June 2012 REDIH meeting. There were a total 21 trainees involved in the two simultaneously run trainee focus group interviews (ten in one focus group and eleven in the other). The trainee focus group participants comprised three postdoctoral fellows, ten PhD students, and eight master’s students from seven universities in Ontario and Quebec (Carleton University, Western University, University of Montreal, University of Ottawa, McMaster University, Laval University, and McGill University). The two mentor focus group interviews comprised fifteen participants (one with eight and one with seven) from the same seven universities, as well as Health Canada, and a fertility clinic. Relevant information from the two mentor focus groups was used to supplement and corroborate the two trainee’s focus groups.

Qualitative analysis

Each focus group was audiotaped and then transcribed verbatim. The transcripts were checked for accuracy by the researcher listening to the audio recording (mp3 file and comparing it to the transcribed text. Open coding of the text was then performed by hand. After a preliminary list of codes was developed, the transcripts were coded a second time. The coding process consisted of grouping the common codes together to form themes and subthemes based on the W(e)Learn framework. The coding was reviewed several more times to ensure that no new codes emerged from the data.

Quantitative data collection and analysis

The constructs of the W(e)Learn framework (content, delivery, service, structure, and outcomes) guided the data analysis of the REDIH Training Module Assessment surveys. The surveys were administered directly after each training module. Descriptive statistics and response frequencies were used to assess the trainees’ experiences with the two learning modules and the Ottawa Reproductive Biology Workshop meeting that trainees attended.

Qualitative findings

The findings from the four focus groups (two with trainees and two with mentors) are chronicled in the ensuing sections. The findings are organized under facets of the W(e)Learn framework: content, media, structure, service, and outcomes.

Content

During the 2 day June 2012 REDIH meeting, the first day comprised a combination of lectures and discussions by mentors who run clinics supplemented with tours of their labs, clinics, and a hospital neonatal intensive care unit. A social dinner was held during the evening for both mentors and trainees. The second day was a more traditional REDIH meeting day involving student research presentations (oral and poster sessions), presentations by, and a panel discussion with, mentors, and informal discussions over lunch. These events were implemented to provide mentors and trainees with time to network.

Laboratory day visit

Trainees unanimously agreed that the tour day (including a 3-hour lecture on assisted human reproduction, a tour of the Ottawa Hospital ultrasound facility incorporating a discussion on prenatal diagnosis with imaging and genetic testing, a tour of the Ottawa Hospital neonatal intensive care unit, and a tour of the infertility lab) was interesting, informative, and a privilege. A trainee PhD attending his first REDIH meeting shared how impressed he was with the day planned for them:

There was a 3-hour talk…I thought it was totally interesting. It made me think ‘oh maybe this is something I want to do’. Going through all the ultrasound pictures was very interesting. All around it was an amazing day.

For one PhD trainee, the 3-hour talk was the highlight of the day:

A lot of clinical information that you read is in papers. Having a presentation from the director of a clinic, I could actually take those numbers and appreciate them. I really liked having her talk about her clinic and what they do.

A master’s trainee enthusiastically described her perception of the lecture:

I knew a little bit about assisted reproductive technologies, but not really the extent of it. It was nice to get a really great detailed description of how it works. I really enjoyed that.

A PhD trainee discussed how she was able to gain an understanding of specific procedures as a result of the ‘Assisted Human Reproduction’ lecture. She spoke appreciatively about things she learned during the lecture and confirmed the value of this component of the REDIH program:

[…] when you think of injecting sperm into the egg you [would think you] just push [the needle] through. But actually you have to create negative pressure before … [The presenter] went into detail and then you actually saw the video. You really got the sense of doing it yourself and what it actually entails. I thought it was really well done. I can see why people would want to continue this practical aspect of the training.

Ultrasound and neonatal intensive care unit tour

The trainees all agreed that the session in the ultrasound lab was interesting, relevant, and meaningful. The trainees commented on how they were able to make a connection between the effects of their research in the lab and patient care. One PhD trainee voiced her excitement about the visit:

The session with Dr X was fantastic. She was engaging and she was clearly excited about what she does. To show us the kids who, without the research we do, wouldn’t have lived 20 years ago just really hit home.

Several other trainees commented on how the day helped them make a connection between research and practice:

I like the fact that, in the morning, we actually got to visualize all of these technical things that you hear about and read about in papers […] but to solidify that with an actual visit [to the clinics]. It was good to have that transition and be able to apply what you do in the bigger picture to what happens on the scale of actually doing it.

Correspondingly, a PhD trainee revealed how she liked the fact that Dr X discussed a lot about the tests behind the prenatal screening and the information you can gather from the tests:

I feel this aspect touched upon the whole screening process and what ultrasounds at specific weeks can tell you.

A master’s trainee also made a link between the research and the clinical side of reproductive medicine.

In the morning they talked about [assisted human reproduction] and in the evenings we saw triplets. We looked at the problems they had and at the issues of triplets. It was helpful.

Similarly, a second master’s trainee agreed that the discussion at the ultrasound clinic was interesting and she too commented on the benefit of linking the research to the practical side of things:

We had Dr X going through details of prenatal screening. Bringing us into the neonatal ICU [intensive care unit] was definitely a tremendous experience. You got to see it all.

A master’s trainee reinforced the tremendous amount of learning that occurred during the day:

We saw ultrasounds and then she explained in detail what you would see in this disorder. What [the ultrasound] could potentially tell you. She taught us a lot.

A PhD agreed and added:

I have listened to Dr X talk a couple of times, but I have never seen the real babies. So it was good to put into perspective what it means. You can learn more in the real world than just the basic science.

One PhD trainee described having the opportunity to visit the neonatal intensive care ward as a “privilege”:

She was apologizing to us for having to visit in small groups and wait. Are you kidding? It was a privilege.

A master’s trainee pointed out that the nursing staff in the intensive care unit were very accommodating. When describing the entire day, trainees agreed they also enjoyed the lunch provided.

A PhD trainee pointed out that she acknowledged a connection between what they learned during the laboratory visit day and a REDIH session held at a previous meeting on ethics:

I am sure within the context of what we learned today, ethics can definitely play a huge role with deciding to keep/freeze/thaw [eggs] and all those factors.

Similarly, a master’s trainee pointed out implications for patient care between the presentation in the morning and the afternoon clinic:

She talked about how it may be beneficial to freeze embryos, and then bring them back so that you can use them in another cycle. I thought it really brought in the patient side of things. I don’t think we have really been focused towards that as of yet.

Tour of the infertility lab

The tour of the infertility lab at the end of the day received mixed reviews. The main criticism was around the organization of the day. A few participants wanted more time in the lab and some wanted more explanation and samples provided. However for several trainees, the lab visit was worthwhile. One master’s trainee shared:

[….] it was really nice to visualize where it happens and even the morphology of the sperm. I didn’t realize you had to identify so many things before you factored in what you were looking for. I deal with embryos, so it was nice to see the sperm side.

Finally, a PhD trainee appeared to gain an appreciation for the extreme organization required in running an effective infertility lab:

I really enjoyed the visit to the clinic so we could see what goes on. Everything is put in place and there is a huge calendar of people doing things. The organization is really military-like.

Transition from training to career

Trainees unanimously agreed that the presentation on how to write a curriculum vitae (CV) was beneficial. One master’s trainee said that she applied for a job a few months ago and needed help putting together her CV. In her words, this session was “really, really helpful.” Correspondingly, a PhD trainee shared that he thought the module was extremely useful:

Everyone has a general idea of what a basic CV is, but they were able to tell us for a research position, for industry etc. It will be useful, not just for tomorrow or next week, but for the rest of our life and career development.

Trainees stated that they were grateful to have the opportunity to write their questions about developing their CV down on a piece of paper and submit them anonymously:

You can really get your thoughts out better when you’re writing them down. Plus you have anonymity.

The presenter grouped the trainees’ questions into three general themes and then addressed them. Several trainees commented on the effectiveness of this presentation style: “That worked really well. She almost assessed every question and put it into categories.”

Media

Trainees’ responses with regard to Structure fell into two themes: “student presentations” and “mentor presentation styles”. These two themes are discussed in the ensuing sections.

Student presentations

During a previous REDIH meeting, trainees were instructed to create a presentation about their research for a “lay audience”. However, the mentors providing feedback were not informed of the instructions given to the trainees and judged them as if they were making a scientific presentation. Trainees recommended that the same information be shared with both trainees and mentors so that the feedback they received aligned with the format in which they prepared and delivered their presentation. One PhD trainee stated:

It wasn’t that we didn’t have the information last time, it was that [the mentors] didn’t. They didn’t know what we were told so they were judging us based on a scientific presentation, when we were instructed to talk to a lay audience.

At the recent REDIH session, trainees confirmed that mentors provided feedback that was consistent with the instructions provided on how to prepare their presentation. In the words of one PhD trainee:

I did an oral. It was for a nonexpert audience. I still have to improve on it, but it was good to try and do that. I didn’t do an oral in the last meeting, but I heard the evaluation sheet was unrelated to the actual presentation requirements. This time it was nice because it was just strengths, weaknesses, and comments. It was great and I got a lot of good feedback.

Trainees reported there were still some “issues” with their presentations. One trainee reported that the desk holding the projector, the microphone, and the screen were all so close together that they could not see their slides. One trainee suggested that, if they had a collar microphone, you could improvise.

For the most part during this REDIH session, there were no discrepancies reported between the expectations communicated to the trainees and the feedback provided to trainees by mentors. A few trainees still suggested that more details on what it means to do a “layperson” presentation would be helpful. One master’s trainee shared, “I get to dumb it down a little bit, but to what extent are you thinking?” A second master’s trainee also commented on the confusion created by trying to understand what was expected in a laypersons’ presentation:

For example, in my results I didn’t show any data. I just said what my data was. In my reviews someone said they would have liked to see more details and other people said that I use too many scientific words […]

A second PhD student shared:

My review from one [mentor] was that I actually listened to instructions, where everyone else wanted me to use more data. They are close, but they need to define it better.

Another PhD trainee expressed what his interpretation of communicating to a layperson means:

My experience with explaining things to a layperson is family and friends and they kind of get things when you explain it to them theoretically. But as soon as you pull out the graph, that is when they get all confused. It wasn’t too clear what they were expecting in terms of science.

A PhD trainee explained that, to her, a lay audience implies a grade seven science education. She suggested the mentors needed to set up the scenario more:

Say “imagine you are asked to speak at a fundraising event about your research”, what would you say? If they would rather have us speak like we are at a conference that is different. Which is why when I gave my presentation in Quebec City I felt like a grade seven could understand it, but scientists would think it was too simple.

In conclusion, some trainees suggested that more details and agreement on what a “layperson” presentation means would be helpful. They suggested that mentors need to define layperson presentation so that everybody listening or evaluating will know upfront what to expect.

Mentors’ presentation styles

Related to the delivery of the REDIH program, several trainees commented on how much they enjoyed not only the content of the presentations, but how effective the presenters were at delivering their presentations. Trainees described how one presenter went into detailed explanations using real life examples. They raved about how the presenter in the ultrasound discussion was enthusiastic and made connections between theory and practice. Trainees referred to how, in the module on CVs, the presenter had an actual CV that she used for a sample. One master’s trainee explained:

I liked the fact that we had the actual CV and then she commented expressively on it. We know what we should include and why. I liked the specific examples because they really stick in your head, and you get the idea of what a good CV is.

Finally, trainees in all sessions noted how receptive presenters were to questions and how effective they were at answering them. One master’s trainee commented, “I also really enjoyed that the presenter gave us time to think of questions, and they answered them very effectively.” In a similar vein, a PhD trainee stated, “They answered all questions and there were a lot who asked questions that I hadn’t thought of so I really appreciated it.”

Structure

Trainees’ responses with regard to structure fell into four themes: notice; organization; travel time; and presentation criteria. These four themes are discussed in the ensuing sections.

Notice

Trainees suggested that it would be helpful to have more notification of when the meetings are going to take place so they can plan their schedules. The recommendation to schedule the REDIH dates in advance has been suggested by trainees in previous REDIH sessions, where one trainee complained that she had to cancel a trip to Boston that she had planned for months due to the REDIH dates not being arranged in advance. A PhD trainee shared:

I know it is hard to organize, but when we get information two weeks before it can be a crunch to get a presentation and a poster together. I think they do too much last minute prep. We only heard two weeks ago and the agenda was even later.

A master’s trainee agreed and stated, “Just more organization, just a heads up”. A second master’s trainee confirmed that she would like more notice of upcoming meetings and expectations: “Right now there are a lot of conferences so when you put REDIH in last minute it can mess with your schedule.”

Organization

In a similar vein, trainees reported that they would appreciate some organization of how they are to get between sites during the laboratory day visits. A PhD trainee commented on the need for more organization of travel arrangements and room numbers: “We didn’t know how we were getting between sites. Everyone was confused. We were just wandering around.” Several trainees said they did not know how they were going to get between sites and suggested organizing travel for trainees would be appreciated.

Travel time

Although all trainees enjoyed the laboratory tour day, they did have a few ideas for how the structure of the day could be improved. Several trainees suggested the amount of time spent traveling between sites could be reduced. In the words of one PhD trainee:

The only thing I would change would be the amount of travel time it took going from the clinic, back to the hospital, and back to the clinic […] for a 15-minute tour. If we could have had the tour in the morning perhaps in the middle of the talk to break it up? The talk was fantastic but it was just a really long 3 hours to sit there.

Trainees suggested that the two 40-minute trips between sites was not an efficient use of time. A second PhD suggested structuring the tour more:

It was very short. I know the clinic is very busy. But it was just a bunch of scopes and you toured. I think it would have been nice to say this is where we do ICSI [intracytoplasmic sperm injection]. The embryos travel here to the incubator. Or here’s where we receive samples from the patient. This is the procedure room. Just more explanation of what was happening at each station.

Another trainee agreed that the tour could be more structured and suggested, “Or let us see any dummy sample like the ones that are not so important but the ones that are maybe frozen.”

One PhD trainee was disappointed that they did not have more time in the lab. She shared, “A bit more time in the lab [would be good] so we are not rushing through. Give more time to the practical part.” Another PhD trainee stated he was really interested in assisted reproductive technologies and in-vitro fertilization and was very excited about going to the clinic but was disappointed with the way the time was utilized:

I was a little disappointed that we didn’t get the full out tour. “This is who we are and this is what we do.” Maybe it was just a timing issue and how long it took to get there but that would be something I would recommend for next time.

When asked what the least rewarding aspect of REDIH was, one trainee response related to the structure of the REDIH program. He said, “Schlepping around from place to place. I have to say it was a real bummer. I felt like we were all like lost puppies.” A second trainee agreed and said, “Loss of time, we were the last ones going and we had to rush through [the neonatal lab].”

On a more positive note, trainees commented on how grateful they were to have the information on some of the modules sent to them via email. They reflected on how, in the last REDIH meeting, they would have appreciated receiving the grant proposal that was being reviewed by the mentors. Once again, it seems the mentors took the feedback to heart and this time they sent materials out to both mentors and trainees. A master’s trainee acknowledged this effort:

I also like that this time they planned a little bit more and gave us the presentation before hand for this session. They actually made the point to have it sent to us right away so we could actually have it as it was unfolding.

Presentation criteria

Lastly, when asked if anything about REDIH was not working for them, one PhD trainee stated, “Defining the presentations a little more so that everyone gives the same presentation – that is not working.” Trainees agreed that explicit instructions on what is meant by “layperson” presentations would be helpful.

Service

Trainee responses with regard to service fell into three themes: feedback; time with mentors; and receptive to feedback. These three themes are discussed in the ensuing sections.

Feedback

In the past REDIH meeting, trainees complained that they had put a lot of time and effort into creating their posters and that no mentors and very few trainees came to visit them. They reported that they did not feel the time and effort that they put into designing their posters was worth it. Also, trainees said at the last REDIH meeting that the time the posters were displayed was so short that trainees did not have the chance to visit each other’s posters. Finally, trainees objected that the posters were displayed during a social hour, so the posters were competing with the food being served. Moreover, many of the trainees missed the food because they were committed to presenting their posters. One PhD trainee shared:

I am glad that, during this session, they assigned people to come to your poster. In Quebec City, I was next to where the food came out, so we just stood next to my poster and ate and drank. This time, one of my evaluators found me and said, “come and do your poster now, because I have to go later.” She really took it seriously, which was so nice.

Several other trainees agreed that, during this REDIH session, the mentors delivered with regard to providing constructive feedback. One master’s trainee stated, “Yes, I got [the feedback] right after the presentation. They just came and handed it to you. They tell you the main points when they give you your comments, and it was really nice to hear.” Trainees were very appreciative of the constructive feedback they received on their presentations. One trainee stated, “They were very enthusiastic about giving a bunch of feedback. It was nice!” A PhD trainee explained he has made a presentation at four REDIH meetings and this was the first time he ever received feedback:

It was absolutely helpful. I am going to an international conference in a week, presenting the same presentation so it was great. I love the fact that they actually spend 10 minutes explaining it to you. They were very nice about it and we got to keep the review. I hope it continues.

Finally, another PhD trainee compared his experiences doing poster sessions between the last REDIH meeting and this meeting.

I was doing a poster. I didn’t have anyone come and see my poster because I was off in the corner [at the last REDIH session]. It was nice [at this REDIH session] they gave the mentors four trainees to go and see. Even if they aren’t interested in it, they will come and give you a rundown of your poster. It is nice to know that at least one person will see you.

Mentors obviously took the trainees’ feedback seriously and, during this REDIH meeting, each mentor was assigned to visit and provide feedback on a trainee’s poster. However, one PhD trainee suggested this setup seemed forced:

I think the poster session could have been a little bit more organic. I know people were asked to make sure that principle investigators (PI) were visiting posters. But it felt patronizing when [the mentor] would come along with a paper and say, “I have you… go.” A little bit of finesse would have been a little less ridiculous.

One PhD trainee explained that there was some confusion on the schedule about when they were to be at their poster. A couple of trainees reported that they were called back to their posters when they did not realize they were supposed to be at them. One PhD trainee explained:

It wasn’t really clear when we were supposed to be at our poster. [The agenda] said poster viewing and in the afternoon it said poster presentations so I think a lot of us thought that we could go and see everybody else’s poster and in the afternoon we had to be at ours. So I was at someone else’s poster and one of my judges came over and was like “you need to come over here”.

When trainees were asked to share the most rewarding aspect of the REDIH program, one PhD trainee listed several attributes he found helpful, including having the opportunity to present his research, learning how to interact with PIs, learning how to answer questions, and different ways to disseminate information, including a poster and a presentation for a lay audience. A master’s trainee revealed that receiving feedback on how to do an effective presentation was her favorite REDIH output. She shared:

The presenter was a clinician scientist. She said, “I want you to present your work in one sentence – like you’re in an elevator with me. Go through your actual presentation. You have to make it real, use worldly terms and provide the big picture.” It was great to think of it that way.

Time with mentors

At the very first REDIH meeting 3 years earlier, the mentors all sat together during lunches and dinners. Trainees complained that there was too much segregation between mentors and trainees. They communicated that they wanted an opportunity to network with the mentors and “pick their brains”. Since that time, mentors have made a concerted effort to eradicate the segregation including arranging assigned seating at dinners, choosing a seat by a trainee, inviting trainees to join them at meal time, and ensuring mentors provide feedback on trainees’ poster presentations and oral presentations. Some of these strategies have been more successful than others but trainees consistently report that they definitely recognize and appreciate the efforts made by the mentors.

A PhD trainee said that insisting alternate seating with PIs and trainees worked: “[…] because we were forced to converse. It can be hard to break out of your little bubble for both PIs and students. So forcing us to be physically next to each other did work.” Trainees also reflected on the speed dating activity that had taken place at the previous REDIH meeting:

I think that really helped with our comfort level at this meeting. Because we had the opportunity to sit down and talk to a number of PIs in a much more laidback atmosphere. So now I feel much more comfortable approaching any of the PIs and talking to them. I think we should probably do that every other meeting but with a bit more time.

A mentor also commented on the fact that the isolation between mentors and trainees had been eradicated:

I am a mentor who has been with the program since the beginning. I have seen the changes occurring and the increased interaction between the students and mentors. There has been improvement and that makes it interesting for mentors as well as for trainees. Overall, I perceive that it is a very good program and it should continue past the end of the program [grant funding].

At this REDIH meeting, as with the last, trainees had no complaints about the time they had to network with mentors. For example, one PhD trainee stated, “Dinner was nice! The conversation was quite varied so we got to know [the mentors] personally as well as professionally.” At this REDIH meeting, unlike most of the previous meetings, having enough time to network with the mentors was not an issue with trainees.

Finally, a PhD trainee wanted to communicate his appreciation to the mentors for being so personable. This is very different from the first REDIH session where trainees criticized the mentors for not making an effort to interact with them. In the words of one PhD student:

I would just like to say thank you to the PIs for being so open to all of us… they don’t have to be. They could just join their own little group and be very happy talking to one another. But instead they’re very open. I would like to say thank you for that.

Receptive to constructive feedback

Both trainees and mentors emphasized how being receptive and responding to trainee feedback has created an emerging design to the REDIH program and contributed to the program’s success. One mentor involved in REDIH for 3 years commented on this unique design:

I have been in [the REDIH program] from the start. I think we have done a terrific job of continual reevaluation and improvement based on the evaluations we got after each of the sessions. I have never been involved in a program like this before where we look at what we have done and what we could do to make it better. That is really impressive.

One PhD trainee attending his fifth session over the 3 years reported that, “REDIH just keeps getting better. It’s really clear that the mentors are listening to what we say. They […] are trying to improve our experience. I really enjoyed it.” Similarly, a mentor spoke of continuous revisions of the REDIH program based on the trainee evaluations. “Constant reevaluation and taking feedback seriously […] that is one of the things that is really working well. To keep coming up with modules they seem to appreciate and when they say it is useless – out it goes!” A second PhD trainee agreed that the overall experience gets better each REDIH session. She also commented on how receptive the mentors are to constructive feedback.

One of the many examples provided with regard to mentors responding to trainees’ feedback was the response to the organization of lunch discussions. At the last REDIH meeting in Quebec City, trainees and mentors were interspersed at tables over lunch and asked to discuss and then report on a set of questions. In order to allow participants to catch the last train out of town at noon on the second day of the REDIH meetings, the first day of the meeting was extended to a 12-hour day. Much of the trainee resistance to the structured lunch was because they needed and wanted some time to relax during this long day. During the recent REDIH session, the days were more balanced, and also the lunches were less formal with no follow-up presentation requirement. Trainees appeared grateful that their voices had been heard and they were allowed more time to enjoy their lunch and have informal discussions with trainees and mentors. One trainee stated:

I thought the lunches this time were way better, everyone talking about the same thing so you don’t have to pick and choose which one you go to. Everything was less structured which makes it easier to carry on from the morning. It was a really great conversation.

Similarly, a mentor provided his perception of the unique emergent design in the REDIH program:

[Trainees] have never been that spoiled before by telling us what they want and expecting us to redesign the program every 6 months. It is incredible the response of this group to the demand. Although they are making a critique every time, we are readjusting to things they like and they don’t like.

Finally, another mentor commented on the great opportunity the trainees have at each REDIH meeting to anonymously communicate their reactions to the REDIH session and have an impact on the future of the program through constructive feedback: “This is your chance if there is something that is pressing or that you want to know a little bit more [about] or want to see things going in a different direction, say so.”

Website

A PhD trainee suggested that REDIH could be improved by putting the sessions online so they could refer back to the content at their leisure: “It would be beneficial to have what we did in the past to refer to. You remember them but it would be great to be able to go back. I feel like the website isn’t really being used.”

Lab visits

The REDIH program has funds to support trainees to visit trainees and mentors at each other’s university. Although there were rumors that these funds were available, 2.5 years into the program, no trainees had taken advantage of this opportunity. During the last REDIH meeting, trainees shared that the information on how to access the funds had not been communicated. There was confusion on how much money was available, who was eligible to apply, and what the criteria and procedures were for accessing the funds. The REDIH staff responded to this concern and communicated this information to trainees. During the focus group at this REDIH meeting, when asked if anyone had visited another mentor’s lab, no one had, but an exchange was planned for trainees from Western to visit trainees and mentors at McGill the next day. When asked if they were being funded to go, a master’s trainee explained, “Yes, and it wasn’t hard at all.” The trainees were very excited about visiting the lab and learning different in-vitro fertilization techniques.

Travel arrangements

With regard to the structure of REDIH, a PhD trainee felt there could be better communication in order to make travel to the REDIH meetings more convenient. She shared that she drove from Montreal to Ottawa in the morning. She thought there were probably other people driving as well. She suggested that providing a list of names, locations and emails of people attending the meeting could facilitate communication and organizing travel: “It would be nice to carpool and get to talk to people.” Trainees suggested providing a resource for carpooling and transportation.

Outcomes

Trainee responses with regard to outcomes of REDIH fell into five themes: knowledge transfer; networking; confidence; enrichment; and learning community. These five themes are discussed in the ensuing sections.

Knowledge transfer

When trainees were asked if they have had the opportunity to use the information presented in previous sessions in their work situation, there were indications that some knowledge transfer is beginning to take place as a result of the REDIH program. One PhD trainee stated that his goal during the speed dating activity was to explore a possible supervisor for a postdoctoral position. He reported that he learned a lot of ideas and tips from the mentors during the activity and stated, “I definitely took that home.” Once again, the trainees confirmed that the session on publishing a paper provided by two mentors who are the editors of one of the top journals in the field of reproductive biology was not only interesting, but provided practical information that several trainees have implemented in their work situation. One PhD trainee stated, “Just hearing some of the ideas they had on how to start a paper […] gives you an idea of ‘oh that’s how I should start.’ That was helpful.” In a separate interview, another PhD trainee also revealed that he too is using the information from the publishing workshop. He stated, “There was a workshop on how to write a paper, I am using it right now.” A master’s trainee stated that he has used the information learned during a previous module on grant review.

Networking

When asked what the best outcome from their involvement in REDIH has been, as in past sessions, trainees identified “networking” as the biggest advantage. One master’s trainee shared, “Networking with everyone and how it’s a one-to-one ratio [with the mentors]. There’s always a mentor to talk to and they’re so open to talking to you.” A PhD trainee agreed but emphasized the benefits of networking were not limited to mentors but were also between and among the other trainees. “We didn’t know each other but maybe in the future we’ll be able to collaborate and this is something I feel is very nice also.” Another PhD trainee attending her third meeting reported what she appreciated most was that REDIH is a small, intimate group. She stated:

You really get to know the PIs. Unlike a conference where there could be hundreds or thousands of people, we really get the opportunity to sit down and talk to people one on one. They remember who you are and they actually get to know you.

A PhD trainee attending his first REDIH meeting agreed and stated the thing he was most impressed with was the intimacy between the trainees and the PIs. “I feel like I’ve really gotten involved with the reproductive biology community. I didn’t feel like that before even though I’ve been working in [the reproductive community] for the last five years. I think it’s been really excellent.” Similarly, a postdoctoral trainee attending his first meeting confirmed that REDIH provided an opportunity to network: “A thing that came up a lot was [mentors] talking about how important networking is. Coming here is by far the most networking I’ve experienced. So that’s pretty cool.” A master’s trainee in her second year of the program supported the networking concept and shared, “This program provides a chance to meet the trainees and mentors.” Finally, a mentor also acknowledged the importance of REDIH providing an opportunity for trainees to network:

The best thing about this program is the opportunity for the trainees to interact among themselves and particularly for me the possibility of them going to other labs to either learn things or to teach the techniques that they have learned in their own lab.

Confidence

When asked to share the outcomes of the REDIH program, one of the most common answers provided by mentors and trainees was that REDIH helped develop confidence in the trainees. One mentor commented:

This program helps build confidence. My student is interacting with each of you and reading your papers. She can sit down and have a glass of wine with you and talk about soccer and that breaks that wall down. She has really increased her confidence and I can see her relaxing more as she presents.

A second mentor also reported that he attributed the REDIH program with helping to build confidence in his trainee (student): “My student certainly wouldn’t have the opportunity to mix among such outstanding scientists in such an intimate way [without REDIH]. She has really benefited from that and I can see it in her confidence and in her developments.” A master’s trainee also commented on how REDIH contributed to her confidence building. In her words, “…being part of a small group all researching one aspect of the same thing really creates a community. I wouldn’t feel awkward anymore approaching someone that knew a technology. I think [REDIH] builds a sense of confidence.” A second PhD trainee commented on how the efforts made by the mentors helped trainees to overcome their shyness and have the confidence to ask questions: “[The mentors’] efforts are definitely apparent. They are giving us the steps, and the skills so that shyer people are asking the questions they need to be asking and want to be asking, whereas they wouldn’t have in the past.” Likewise, a master’s trainee commented that the efforts made by the mentors to interact with trainees helped her overcome her shyness:

The REDIH mentors are getting better and better. Last night it was a wonderful dinner and we had a lot of interaction. I wasn’t shy because I know a lot of mentors after 2 years whereas in first year I only knew my supervisor. Before, the mentors were very passive and that is why there was no communication between us. Last night was totally different and they were very friendly, which I liked.

Enrichment

From the mentors’ perspective, one of the most beneficial outcomes of REDIH is the enrichment that the program provides to the trainees’ graduate studies program. Mentors discussed at length how the experiences trainees had during REDIH provided an opportunity that definitely put them at an advantage. Moreover, they expressed how they would like all graduate students in their labs to have the same experiences and opportunities. One mentor put it this way, “I think the best thing is enrichment. I think we could say that just about any trainee in REDIH gets aspects that they could not or do not get at their home institution.” Another mentor agreed and provided his perspective on how REDIH enriches trainees’ graduate experience:

To cover things that are not being discussed elsewhere like career orientation, ethical aspects, problems we have in our everyday lives as scientists. Nowhere else can they have such a good view of what is ahead compared to a REDIH meeting. For me that is very good and something different. It is value added for trainees.

Similarly, a PhD trainee confirmed how much she appreciated experiencing the practical clinical side of reproductive medicine. She explained that this is her third REDIH meeting and that this session really brought them into the real world:

In the past we spent a lot of time listening to the mentors’ research and each other’s research, but in my graduate program we have a lot of opportunity to talk about our research. But we don’t have much opportunity for real-world experience and to see what it is actually like when you are not just at your bench working with embryos. I thought it was really well done.

Learning community

Trainees and mentors at the sixth REDIH meeting all expressed contentment at being part of the REDIH program. Both mentors and trainees discussed how the program continually improved, and those who had been involved in REDIH for a long time pointed out changes and improvement that had taken place as a result of being receptive and responding to constructive feedback. In particular, all participants noted how the segregation between mentors and trainees had been eradicated. Trainees commented on how they appreciated the mentors hearing their voices and responding to feedback. Trainees noted that efforts made by mentors to integrate with the trainees, provide feedback, and eliminate workshops that were not the right fit for trainees’ needs and adding workshops that they requested would benefit them.

As a result of the integration between mentors and trainees and the connections developing between and among trainees, it appears that trainees feel safe to express their opinions and concerns. Learning is enhanced through a commitment to the collective good by learning through and with others. Knowledge is shared and expertise resides not only with the facilitators but also across the REDIH community. Trainees said they experienced a sense of connectedness, cohesion, trust, and interdependence. Finally, trainees praised the REDIH mentors for being very nurturing and accepting and making everyone feel comfortable. The data suggested that because the mentors used feedback mechanisms throughout the REDIH experience and demonstrated a willingness to listen and act on trainee feedback, the general and consistent atmosphere of receptivity, openness, and flexibility has enhanced the overall learning experience.

Survey results

Details of the survey results for the three modules can be found in Tables S5.1–S5.15 of Supplementary materials B. The survey can be found in Supplementary materials C. Twenty REDIH trainees completed the evaluations for career planning/curriculum vitae, 19 finished the in-vitro fertilization module evaluations, and 17 completed the neonatal unit visit evaluations.

Career planning/curriculum vitae

Overall, all participants were very pleased with this module. They found it to be very informative and interactive. Scores for all the constructs were very high. Only the constructs of media and outcomes had considerable non-applicable responses.

Structure: With regard to the structure of the module, almost all responses from the participants were positive. Mean responses ranged from 4.68 to 4.90 out of 5.

Content: Most trainees indicated either “always” or “often” for all the items related to content of the module. Very few trainees considered the items “not applicable” or “undecided.” Mean responses ranged from 4.65 to 4.85 out of 5.

Media: Most responses related to media were positive. Results ranged from 4.59 to 4.90 out of 5.

Service: Participants felt that the service provided by the facilitator was fine. Mean responses ranged from 4.55 to 4.91 out of 5.

Outcomes: Many of the responses to the outcomes of the module were positive. However, many items were also considered not applicable: ‘I have acquired proficiency in new techniques (5), I will initiate new ideas and/or projects (5), and I will apply new skills in the workplace (4)’. Mean scores ranged from 3.67 to 3.90 out of 4.

Open-ended questions: Most responses to the open-ended questions reflected the feedback provided from the focus group interviews. Generally, the participants enjoyed the module and found it very interesting and informative, especially those who would be graduating soon and applying for work. Participants found the handout very useful and liked the opportunity to write down questions.

In-vitro fertilization

Trainees were very satisfied with the module. One trainee commented, “Very good, do it again!” and, “Very interesting and relevant.” They appreciated learning the exact process of assisted reproductive technology (ART) and different positions in the ART lab.

Structure: With regard to the structure of the module, most responses from the participants were either “often” or “always.” With regard to the structure of the module, most responses from the participants were positive: relevant material; relevant to audience; well planned; and content aligned with the learning objectives. Mean scores ranged from 4.35 to 4.94 out of 5.

Content: Items ranged from 4.72 to 4.82 out of 5.

Media: Most responses related to media were positive. Mean scores ranged from 4.27 to 4.94 out of 5. Some trainees felt that the, ‘opportunities for self-reflection (4)’ was not applicable.

Service: Participants felt that the service provided by the facilitator was fine. One of the items was considered by the trainees to be not applicable: ‘suggestions and complaints were quickly responded to by facilitators (11)’. Mean scores ranged from 4.63 to 4.94 out of 5.

Outcomes: Many of the responses to the outcomes of the module were positive. Mean scores ranged from 3.08 to 3.83 out of 4.

Open-ended questions: Most responses to the open-ended questions reflected the feedback provided from the focus groups. Generally, participants really enjoyed the visit. However, there were several comments from participants indicating that they would have liked to have more time for the clinic visit and less traveling between sites.

Neonatal visit

Trainees were very satisfied with the module. One trainee commented, “Very interesting to see how clinicians diagnose pregnancy difficulties” and, “Such an honor to tour the neonatal intensive care unit. Dr X is an amazing person and researcher.”

Structure: With regard to the structure of the module, almost all responses from the participants were either “often” or “always.” With regard to the structure of the module, almost all responses from the participants were positive: relevant material; relevant to audience; well planned; and content aligned with the learning objectives. Mean scores ranged from 4.41 to 4.93 out of 5.

Content: Items ranged from 4.60 to 4.82 out of 5.

Media: Most responses related to media were positive. Mean scores ranged from 3.92 to 4.94 out of 5. Some trainees felt that the, ‘learning activities included realistically complex tasks similar to those I face in the workplace (5)’ was not applicable.

Service: Participants felt that the service provided by the facilitator was fine.

Outcomes: Many of the responses to the outcomes of the module were positive. Mean scores ranged from 3.00 to 3.76 out of 4.

Open-ended questions: Most responses to the open-ended questions reflected the feedback provided from the focus groups. Participants really enjoyed the visit. Many indicated that there was nothing needed to improve the session.

Discussion and conclusion

This study highlights that all five dimensions of the W(e) Learn framework (content, media delivery, structure, service, and outcomes) are intertwined throughout the design, delivery, and evaluation of a training program. It is the collective impact of these dimensions that leads to a cohesive learning experience. Second, the structure dimension of W(e)Learn emphasizes the importance of understanding participants’ needs and motivations. In this study, the ability to use supporting applications and troubleshoot problems was a prerequisite for a positive learning experience. The flexibility of the W(e)Learn framework to guide both the design and evaluation of a reproductive biomedicine experience illustrates the dynamic intersections between theory and best practices. When “theoretical frameworks inform actions, and actions modify theories so that future actions grow out of what we have learned by experience and reflection, the entire system is energized.”25

This study also demonstrated that evaluation is critical for program design, improvement, and long-term success.26 An essential component to improving the health care system will require health care educators to continually gather and analyze information from teaching sites and develop pedagogical solutions to identified problems that result in improved health care education and practice. This process will involve preceptors taking advantage of professional development opportunities, keeping up-to-date with medical education literature and best practices, and utilizing emerging technologies, teaching tools and strategies. The findings about the value of emergent design in this study were serendipitous in nature; nonetheless, they help set the stage for a more in-depth look at emerging design strategies.

The data from the analysis of the focus group interviews and corroborated by the quantitative survey data suggested trainees enjoyed and benefited from the REDIH experience. Trainees provided several examples of new knowledge and skills they had acquired from REDIH sessions regarding reproductive and early developmental biology and health. A few trainees who had been in the program for over a year provided examples of knowledge and skills acquired during the REDIH session that they are using in their place of work. Next steps will include following up on REDIH graduates to see if the program has had any impact on their employment opportunities and career development.

This study suggests that implementation of sustainable high quality education is facilitated through systematic integration of evaluation into the learning process.27 As participants shared the challenges and successes of participating in this reproductive medicine experience, insights into the characteristics of an immersive, robust learning culture emerged. Such a culture is characterized by a true integration of delivery partners: (A) administration; (B) evaluators; (C) facilitators; and last but not least, (D) the trainees participating in the program.1,4 Furthermore, this study highlights the unique way in which evaluation can coexist on multiple levels and can impact the design of a training program for graduate students.

Our findings suggest that designing courses that are responsive to emergent needs requires: (A) keeping your finger on the pulse of the learning experience by creating opportunities to collect and act on feedback; and (B) a modular flexible course design that enables modifications between and during sessions.4 Perceptions of quality were strongly linked to a fit between participants’ experiences, needs, and wants; a formal evaluation process; a flexible modular course design; and project administrators’/facilitators’ responsiveness to ongoing evaluation feedback.