A new Mentor Evaluation Tool: Evidence of validity.

BACKGROUND: Mentorship plays an essential role in enhancing the success of junior faculty. Previous evaluation tools focused on specific types of mentors or mentees. The main objective was to develop and provide validity evidence for a Mentor Evaluation Tool (MET) to assess the effectiveness of one-on-one mentoring for faculty in the academic health sciences.
METHODS: Evidence was collected for the validity domains of content, internal structure and relationship to other variables. The 13 item MET was tested for internal structure evidence with 185 junior faculty from Schools of Dentistry, Medicine, Nursing, and Pharmacy. Finally, the MET was studied for additional validity evidence by prospectively enrolling mentees of three different groups of faculty (faculty nominated for, or winners of, a lifetime achievement in mentoring award; faculty graduates of a mentor training program; and faculty mentors not in either of the other two groups) at the University of California San Francisco (UCSF) and asking them to rate their mentors using the MET. Mentors and mentees were clinicians, educators and/or researchers.
RESULTS: The 13 MET items mapped well to the five mentoring domains and six competencies described in the literature. The standardized Cronbach's coefficient alpha was 0.96. Confirmatory factor analysis supported a single factor (CFI = 0.89, SRMR = 0.05). The three mentor groups did not differ in the single overall assessment item (P = 0.054) or mean MET score (P = 0.288), before or after adjusting for years of mentoring. The mentorship score means were relatively high for all three groups.
CONCLUSIONS: The Mentor Evaluation Tool demonstrates evidence of validity for research, clinical, educational or career mentors in academic health science careers. However, MET did not distinguish individuals nominated as outstanding mentors from other mentors. MET validity evidence can be studied further with mentor-mentee pairs and to follow prospectively the rating of mentors before and after a mentorship training program.

Introduction

Mentorship plays an essential role in enhancing the success of junior faculty. Faculty with mentors report increased productivity, more satisfaction with time spent at work, greater sense of self-confidence about advancement and promotion and ability to be promoted [1-6]. Conversely, previous research has shown that failed mentorship may contribute to mentees not obtaining grant funding and leaving academic careers, among other negative outcomes [7,8]. The National Institutes of Health devoted 2.2 million dollars to create the National Research Mentoring Network dedicated to, among other goals, mentor training and development of mentoring best practices. As a result, an increasing number of academic institutions have implemented faculty mentoring programs [9-13].

Mentor effectiveness is dependent on multidimensional factors and requires more than having a mentor with ideal qualities [14-17]. Assessing mentor effectiveness can help institutions provide feedback to mentors to improve mentoring relationships and in the most extreme cases, identify those pairings that are not working to allow mentees to seek new mentors. The first step in developing such a mentor assessment instrument is to identify the characteristics of effective mentors. Several investigators have performed such research [14,18-21] and identified the following traits as desirable: expertise in their research field, available to their mentees, interest in the mentoring relationship, ability to motivate and support mentees, and advocacy for their mentees.

Several evaluation tools have been proposed to measure mentor effectiveness and competency; however, these instruments have limited utility as they are relevant for specific types of mentors, specific populations, or have not been rigorously validated [14-17]. For example, Berk et al. designed two different scales to evaluate mentors, the Mentorship Profile Questionnaire and the Mentorship Effectiveness Scale [14]. The Mentorship Profile Questionnaire is aimed at research mentors and assessed the nature of the mentor-mentee relationship and specific quantitative outcome measures such as number of publications or grants [14]. The Mentorship Effectiveness Scale, a 12 item Likert rating scale, assessed more subjective aspects of the relationship and qualities of the mentor [14]. Mentees who were nominated by their mentors tested the Mentorship Effectiveness Scale, but the investigators did not perform psychometric testing to provide evidence of validity for either scale. Schafer et al [22] developed a medical student mentoring evaluation tool called the Munich-Evaluation-of-Mentoring-Questionnaire which focused exclusively on medical students’ satisfaction with their mentors. This instrument was tested for reliability and validity, and it was found to be a reliable and valid instrument. Similarly, Medical Student Scholar-Ideal Mentor Scale was developed and provided validity evidence for the score for use in assessing mentors for medical student research projects. [23]

The Clinical and Translational Science Awards (CTSA) mentoring working group identified five mentoring domains and six mentoring competencies in which clinical and translational science mentors could be evaluated [16,17]. The five mentoring domains were: meeting and communication; expectations and feedback; career development; research support; and psychosocial support [17]. The six competencies included communication and relationship management, psychosocial support, career and professional development, professional enculturation and scientific integrity, research development, and clinical/translational investigator development [16]. Based on these domains, they developed the Mentoring Competency Assessment, a 26 item instrument to appraise the effectiveness of clinical and translational (C&T) science mentors [24]. They tested the reliability of their instrument as well as construct validity by performing confirmatory factor analysis of the instrument against the six domains of competencies for C&T mentors [24]. However, the CTSA mentoring group focused on C&T clinician and scientist mentors’ evaluation and they did not include clinician educator mentors in their study. Dilmore et al [15] also focused on C&T science mentors by administering the Ragins and McFarlin Mentor Role Instrument [25] to C&T science mentees; they concluded that it had good reliability and validity evidence in capturing multiple dimensions of the mentoring relationship. Furthermore, Jeffe et al. shortened the 33-item Ragins and McFarlin Mentor Role Instrument (RMMRI) and 69-item Clinical Research Appraisal Inventory (CRAI) to be an easily administered tool to longitudinally follow the progress of junior researchers enrolled in the Programs to Increase Diversity Among Individuals Engaged in Health-Related Research (PRIDE) [26]. These investigators used iterative process of exploratory principal components analysis to reduce the number of items in the RMMRI from 33 to nine and CRAI from 69 to 19 items. The shorter versions of RMMRI and CRAI were able to retain the psychometric properties of the longer version of the instrument.

The instruments described above are limited by either insufficient validity evidence or are useful only with a limited population of mentors (C&T mentors or medical student mentors). None was used to assess mentoring performance of diverse health sciences faculty mentors of clinicians, educators or researchers. To that end, our objective was to develop and provide validity evidence for a Mentor Evaluation Tool to evaluate the effectiveness of one-on-one mentoring whether mentors are health science researchers, clinicians, educators and/ or career mentors.

The construct for the Mentor Evaluation Tool is to measure mentor effectiveness. According to Healy and Welch, mentorship is “an activity in which more senior or experienced people who have earned respect and power within their fields take more junior or less experienced colleagues under their care to teach, encourage and ensure their mentees’ success”. [27] The National Research Mentoring Network defines mentoring as: “A mutually beneficial, collaborative learning relationship that has the primary goal of helping mentees acquire the essential competencies needed for success in their chosen career. It includes using one’s own experience to guide another person through an experience that requires personal and intellectual growth and development” [28]. We acknowledge that other mentoring models that incorporate team mentoring, peer mentoring, and distant and web-based mentoring are also important to mentee success. [7] However, as a dyadic mentoring relationship is often a key component of many mentoring programs, we chose to focus on this context of mentorship as we developed our tool. We focused on the following domains for the tool: expert in the field, accessible to their mentee, interest in the mentoring relationship, ability to support the mentee in career and research. Evidence for validity of an assessment tool consists of five areas: content, response process, internal structure, relationship to other variables and consequential [29]. We focused our psychometric study on the content, internal structure and relationship to other variables domains.

Furthermore, literature review revealed that some academic institutions are utilizing mentorship evaluation tools for selecting good mentors and for academic promotion. The Mentoring Function Scale and the two dimensional scales are used to assess teaching staff mentoring in nursing school and clinical nursing staff mentoring in clinical placement [30]. At the China Medical University, mentors to medical students who performed well earned two credit points out of a maximum of 10 toward their annual teaching evaluations which were used toward academic promotions [31]. Similarly, at the University of Toronto, mentorship activities were noted in the promotion portfolio as part of the faculty’s annual performance review. In addition awards were given to faculty who demonstrated excellence in mentoring [6]. At the University of California San Francisco, excellence in mentorship is being recognized and annual awards are given to mentors in research and in medical education. Mentorship activities are part of the portfolio for academic promotion. A mentorship assessment tool with validity evidence such as the MET therefore is essential to provide objective data on a mentor’s capabilities as a mentor.

Materials and methods

Development of the Mentor Evaluation Tool (MET): Content validity and internal structure evidence

The relevant literature was reviewed to identify mentoring best practice and qualities of effective or admired mentors as well as existing mentor evaluation instruments [14,18,32-38]. Based on the literature review and extensive discussion among the research team and a panel of mentoring experts to identify consistent themes, we initially developed an 18-item mentor evaluation instrument which was pre-tested with 20 School of Dentistry faculty in 2009 (Fig 1). Based on those results, the Mentor Evaluation Tool (MET) was refined to 13-items with a seven-point bidirectional scale. Five items were eliminated from the initial set of 18 due to low variability or near universal endorsement (i.e. ceiling effects) by the mentoring experts developing the instrument. Items were mapped to the CTSA domains.

Fig 1

Mentor Evaluation Tool development.

Internal structure: Factor analysis

To test for internal structure of the instrument, a study was conducted in 2010. Junior faculty from all four professional schools at the University of California San Francisco (UCSF) (Schools of Dentistry, Medicine, Nursing, and Pharmacy) were invited to participate (Fig 1). From these data, we determined the reliability and conducted exploratory factor analysis. The final MET from the 2010 study was used for this current validity study performed in 2014.

MET validity study including relationship to other variables: 2014 study

Design

We designed a study to determine if the MET is able to distinguish individuals identified as outstanding mentors from other mentors. Our hypothesis was that individuals nominated as outstanding mentors and/or formally trained in a skills-based program as mentors would be rated higher by their mentees than those in neither group.

Participant characteristics for the 2014 study

Three different groups of faculty mentors were recruited from the four professional schools at UCSF: Dentistry, Medicine, Nursing, and Pharmacy. The first group consisted of UCSF Lifetime Achievement in Mentoring Award (LAMA) faculty mentors who received or had been nominated for a highly competitive campus-wide mentoring award. The second group consisted of faculty who completed the UCSF Clinical and Translational Science Institute (CTSI) Mentor Development Program (MDP), a five month 25-hour mentor training course designed to improve mentoring knowledge and skills [9]. The last group consisted of a randomly selected group of UCSF faculty mentors not in either of the other two groups (non-LAMA/MDP mentors). All faculty mentors in each of the three groups were asked to provide the names of three to five mentees who could be contacted to complete the MET. They were initially contacted in June 2014 and if they did not reply to our request, they were contacted twice more until August 2014. Their mentees could be trainees or junior faculty who were clinicians, educators, and/or researchers. The inclusion criteria for the mentees were that their mentors identified them as mentees and the mentors were LAMA recipients or nominees, MDP graduates, or non-LAMA/MDP mentors at UCSF. The UCSF institutional review board reviewed and approved the study as an exempt study.

Procedures

All 30 LAMA recipients and nominees (from 2007 to 2015), and the 76 faculty who had completed the CTSI MDP (from 2007 to 2015) were invited to participate in this research project. One hundred faculty members who were non-LAMA/MDP mentors were randomly selected and invited to enroll. If the mentors agreed to participate in the study, then they were requested to provide some basic demographic information (e.g. gender, number of years being a mentor), and the names and email addresses of three to five of their mentees. The rationale for choosing three to five mentees per mentors was to include as many diverse mentors as possible. New faculty mentors may only have three mentees while experienced mentors may have more than five mentees. The study investigators solicited mentees’ participation via email. If mentees agreed to enroll in the study, they completed the MET anonymously through a web-based survey tool (Qualtrics™ Provo, UT) which included a consent form that mentees were required to complete prior to starting the 13 item scale, overall satisfaction item, and demographic information (S1 Appendix). Mentors and mentees were contacted up to three times to invite and remind them to participate in the study.

Statistical analysis

For the 2014 study, confirmatory factor analyses (CFAs) were performed using the structure of 5 mentorship domains developed by CTSA [17] as well as using a single domain. In addition, the mean MET score was calculated for the 13 items. As long as eight or more items were non-missing, the mean MET score was calculated. The seven-point scales were rated (scored) as: strongly disagree (-3), disagree (-2), slightly disagree (-1), neither disagree or agree (0), slightly agree (1), agree (2), and strongly agree (3). Relationship to other variables was first assessed by comparing the mean MET score to a single overall ordinal assessment item included at the end of the questionnaire with Spearman correlation and linear regression. Additional evidence was assessed by performing one-way nonparametric analysis of variance (NP ANOVA) to compare the mean rank MET score and the single overall assessment item among the three mentor groups with a stepdown Šidák procedure for alpha-corrected pairwise comparisons. Overall significance was set at alpha = 0.05 and 95% confidence intervals (CIs) were estimated. To account for the possibility that mentors with more experience mentoring scored better, nonparametric analysis of covariance (NP ANCOVA) was performed adjusting for number of years mentoring. Analyses were performed with SAS Version 9.4 (Cary, NC).

Results

Content validity of the 13 item MET

To test content validity of the 13 item MET compared to other evaluation tools, we mapped the questions in our instrument to the five mentoring domains developed by the CTSA mentoring working group [16,17]. (Table 1) We also provided references of other mentor evaluation tools that supported the items included in our instrument (Table 1). MET items assessed mentor’s competencies and abilities as outlined in the five mentoring domains.

Table 1

Content evidence for the 13-item Mentor Evaluation Tool.

Questions	Mentor Domains*	Other references
My mentor is accessible	Meeting and communication	Berk et al.2005[14]
		Fleming et al.2013[24]
My mentor is an active listener	Expectations and feedback	Berk et al.2005[14]
	Meeting and communication	Fleming et al.2013[24]
My mentor demonstrates professional expertise	Research support	Berk et al.2005[14]
		Fleming et al.2013[24]
My mentor encourages me to establish an independent career	Career development	Fleming et al 2013[24]
My mentor provides useful critiques of my work	Expectations and feedback Research support	Berk et al.2005[14]
		Fleming et al.2013[24]
My mentor motivates me to improve my work	Research support	Berk et al.2005[14]
		Fleming et al.2013[24]
My mentor is helpful in providing direction and guidance on professional issues	Career development	Berk et al.2005[14]
		Fleming et al.2013[24]
My mentor acknowledges my contributions appropriately	Expectations and feedback	Berk et al.2005[14]
	Career development	Fleming et al.2013[24]
My mentor takes a sincere interest in my career	Psychosocial support	Fleming et al 2013[24]
My mentor helps me to formulate clear goals	Career development	Fleming et al 2013[24]
My mentor facilitates building my professional network	Career development	Berk et al.2005[14]
		Fleming et al.2013[24]
My mentor provides thoughtful advice on my scholarly work	Research support	Berk et al.2005[14]
	Expectations and feedback	Fleming et al.2013[24]
My mentor is supportive of work-life balance	Psychosocial support	Fleming et al 2013[24]
Overall, I’m satisfied with my mentor	All

*Mentor domains = Meetings and communication, Expectations and feedback, Career development, Research support and Psychosocial support. [17]

Internal structure evidence from 2010 study

In the 2010 study, 149 respondents out of 840 junior faculty members who were invited to participate in the pilot study reported having a mentor and completed the initial MET. Exploratory factor analysis (EFA) with principal components of the 13 items showed 1 major factor with eigenvalue above 1.0 accounting for 87% of variation. The standardized Cronbach’s coefficient alpha for the 13-item scale was 0.94, which indicated high scale reliability. In the content validity study of 2014, 158 respondents reported having a mentor and completed the MET. Repeated principal components EFA indicated that with principal components of the 13 items with 1 major factor resulted in one eigenvalue above 1.0 accounting for 70% of variation. The standardized Cronbach’s coefficient alpha for the 13-item scale was 0.96 and all 13 items had standardized correlations with the mean MET score of at least 0.70. The confirmatory factor analysis (CFA) results supported a model with a single factor for two of the three criteria. The CFI was 0.89 and SRMR was 0.05, but the RMSEA was 0.15. However, the CFA for the five domains in Table 1 only fit slightly better with CFI of 0.92 and SRMR of 0.04, but RMSEA of 0.14 –also meeting two of the three criteria.

Relationship to other variables: 2014 study

In the 2014 study, 61 out of 206 invited mentors (S1 Dataset), agreed to participate in the study for an overall response rate of 29.6% (Fig 2). Fifty-eight percent (n = 158) of invited mentees responded and completed the MET (Fig 2). Two-thirds (67%) of mentees were faculty members (n = 106), 23% were residents and 10% were students (Table 2). In addition, over half had been working with the mentor for more than three years (n = 102, 65%) (Table 2). Most mentees found the mentor on their own (n = 97, 61%) rather than being assigned by their department or by other methods.

Fig 2

Recruitment flow diagram.

Table 2

Characteristic of the mentees (N = 158) count (Percent).

	LAMA (n = 45)	MDP (n = 67)	Non-LAMA/MDP (n = 46)
Level of mentees
Faculty	30 (67)	40 (60)	36 (78)
Resident	8 (18)	21 (31)	7 (15)
Students	7 (15)	6 (9)	3 (7)
Number of years with your mentor
<1 year	0 (0)	5 (7)	4 (9)
1–2 years	2 (5)	10 (15)	2 (4)
2–3 years	3 (7)	17 (25)	10 (22)
>3 years	38 (88)	35 (52)	29 (64)
Missing data	2	0	1
Mentor was assigned or found by the mentee
Assigned	5 (12)	19 (28)	9 (20)
I found myself	30 (70)	38 (57)	29 (64)
Other	8 (19)	10 (15)	7 (16)
Missing data	2	0	1

MDP = Mentors who completed the Mentor Development Program, LAMA = awardees and nominees of the Lifetime Achievement in Mentorship Award, and All Non-LAMA/MDP = Mentors who are not awardees or nominees of the LAMA or MDP.

The mean mentoring score correlated well with the single overall mentoring item (Spearman r = 0.58, both P<0.0001). There was no statistically significant difference in the single overall assessment item (NP ANOVA P = 0.054) or the mean MET score (NP ANOVA P = 0.288) among the three different groups of mentors. The mean MET scores were relatively high for all three groups (more than halfway between “agree” and “strongly agree”): mean (95% CIs) for LAMA was 2.7 (2.6–2.8), for MDP graduates was 2.6 (2.4–2.7), and for non-LAMA/MDP was 2.6 (2.3–2.9). The LAMA group had less variability than the MDP group, which had less variability than the non-LAMA/MDP. In fact, the mean MET score for the non-LAMA/MDP group was so variable that the range in that group was the full-scale range (i.e. -3 to 3). In the MDP group, the range of the mean MET score was -0.8 to 3.0, while in the LAMA group the range of the mean MET score was 1.7 to 3.0. After adjusting for the number of years of mentoring, the mean MET score still did not differ by group (NP ANCOVA).

Discussion

The main objective of this study was to provide evidence of validity for the Mentoring Evaluation Tool (MET). MET has been tested with mentors who were researchers, clinicians, educators, and/or career mentors. The 13-item MET had good evidence of content validity based on development of the items and good evidence of internal structure based on the exploratory and confirmatory factor analyses. The MET score did not distinguish among the three groups who distinguished themselves from one another by reputation and/or preparation in mentoring even when adjusting for years of experience in mentoring. We conclude that the evidence for content and internal structure validity support the use of the single factor MET for faculty from diverse health sciences disciplines serving as research and/or career mentors. We suggest further research is needed to establish additional validity evidence.

Furthermore, some academic institutions are utilizing mentorship evaluation tools for selection of good mentors and for academic promotion [6,30,31]. Currently at the University of California San Francisco, excellence in mentorship is being recognized by annual awards given to mentors in research and in medical education. In addition, mentorship activities are part of the portfolio for academic promotion. However, currently we are not using the Mentor Evaluation Tool to obtain objective data of the mentor’s effectiveness. The future plan is to use Mentor Evaluation Tool to assess and evaluate faculty’s mentoring capabilities and use this data as part of the academic promotion process.

There are several possible explanations for why our instrument did not distinguish among the three mentor groups. First, mentors selected their mentees to be contacted for the study. Perhaps mentors chose mentees who would evaluate them favorably thus narrowing the differences among the ratings. Since there is no accurate data set on mentor and mentee pairing for faculty of School of Dentistry, Medicine, Nursing, and Pharmacy, it was necessary for the mentors enrolled in the study to provide information about their mentees. Second, mentors and mentees from all three groups were likely more interested in mentoring and more likely to agree to participate in the voluntary survey; the mentees often had long-term relationships with them. Third, there is a long held notion that trainees are reluctant to share negative experiences and to rate their mentors critically [17,39]. Even though the data from the MET was collected anonymously, mentees may have been hesitant to rate their mentors negatively. Fourth, the halo effect from the mentees could have inflated the score of their mentors. The halo effect has been described previously as a favorable perception of one mentor characteristic which carries over to relatively positive scores for other mentor qualities [14,15,25]. Fifth, MDP participants may have been encouraged by their department or division chairs to enroll in the MDP to improve their mentoring skills and their mentees completing the MET may have included those mentored before they completed the program. Finally, non-LAMA/MDP participants may have included mentors who received mentor training other than the MDP at the University of California San Francisco (UCSF). Therefore, the overall scores of non-LAMA/MDP may have been higher than expected compared to LAMA or MDP mentors.

We recognize that the 13-item measure had a very high reliability. As noted by Tavakol and Dennick, when reliabilities exceed 0.90 there may be redundancy in the items [40]. We felt that the items formed a coherent set, reflecting important domains of mentoring, and that the instrument was short. All 13 items had standardized correlations with the mean MET score of at least 0.70. We did not try to reduce the 13 items but this could be a line for future work.

There were some limitations to this study. The mentors and mentees were recruited from a single health sciences institution that may limit the generalizability of these findings. It is possible that mentors selected the mentees with whom they had their best mentoring relationships to complete the survey, and thus biased the result toward favorable scores. However, there is no evidence or reason to believe this potential bias would apply differentially among the three groups. The response rate of the mentor and mentees were lower than we expected despite three reminders over two months (June to August 2014) and even personal communication with some of the mentors. Mentees were also reminded to complete the survey three times within one month of receiving their information from their mentors. We used a binary, hierarchical definition of mentorship and did not survey other types of mentors such as peer mentors or team mentors. Rating of these mentors may have been different and this should be explored in future studies. In addition, we did not conduct focus group among the mentees of three groups of mentors. This information may have provided more insight into mentee assessment of their mentors and may have better differentiated among the three groups.

The strength of our study is that MET obtained evidence for content and internal structure validity domains by mentees of mentors from the Schools of Dentistry, Medicine, Nursing and Pharmacy. These mentors were clinicians, educators, researchers, and/or career mentors. In addition, the MET is a short instrument that should facilitate its acceptance and use by mentees.

Conclusion

The Mentor Evaluation Tool (MET) has evidence of validity for content and internal structure and we recommend its use for examining performance of mentors by mentees in the academic health sciences. The MET provides a way to study mentoring further including examining mentor characteristics, mentee preferences and effectiveness of mentoring training programs. This tool has flexibility not previously found in other instruments and should become a valuable resource as more institutions develop mentoring programs.

Mentor Evaluation Tool.

(PDF)

Click here for additional data file.

MET validation.

(PDF)

Click here for additional data file.

25 Feb 2020

PONE-D-19-35869

A New Mentor Evaluation Tool: Evidence of Validity

PLOS ONE

Dear Mr. Yukawa,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by 29.03.2020. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Slavko Rogan

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements:

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Review «A new mentor evaluation tool: Evidence of validity”

This study is about the development and validity evidence for a Mentor Evaluation Tool assessing one-on-one mentoring for faculty form Schools of Dentistry, Medicine, Nursing and Pharmacy. It is an important contribution to mentorship evaluation for faculty in academic health professions.

Here some comments:

- Line 41: Delete period (typo)

Good introduction to topic and overview of literature, which lead to the comprehensible objective of this study.

- Line 94: delete one space (typo)

- Line 105 – 115: In this section i.e. the before lines 105-115 you mentioned tools which have limited utility (only relevant for specific types of mentors or a specific population or not rigorously tested for validity). Regarding the example in these lines it seems not clear what you criticize. Is it that it is only focusing on C&T science mentees? Should be clearer.

- Line 149-150: As the last sentence of this section is related to what was written before I would add “therefore” => “….such as the MET therefore is essential ….”

- Line 161: how many participants and how many of each group? Specify what you mean with “groups”. The reader probably does not have them in mind anymore, so maybe list them in parenthesis?

- Line 191: Is the 2014-study the present study you describe in the abstract and under “Design” (Material and Methods)? – this is somehow confusing throughout the manuscript and should be clarified from the beginning (which is/are pilot studies and which is the present study?). In this respect your manuscript is not easy to read and clarification would be a help for the reader.

- Line 191.192: remove double space to be consistent within your manuscript.

- Line 201-202: Why did you chose this method that faculty mentors could choose the mentees to be contacted? Why not randomly chose the mentees out of the whole group of each mentor. I think this needs explanation as it could be a bias, i.e. mentor could choose the mentees he/she anticipates most to rate positively. (Later in the discussion you refer to this point as a limitation).

Why did you choose 3-5 Mentees? What are your thoughts/ calculations about participant number.

- Line 213-214: Did the mentees provide their consent that their names and email-addresses were provided?

- Line 215: was this done anonymously? You answer this question only in the discussion.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

29 Mar 2020

Dear Editor and Reviewers;

We appreciate your thorough review of our manuscript, “A New Mentor Evaluation Tool: Evidence of Validity” PONE-D-19-35869, and for your insightful suggestions and feedback. Please find our responses to your comments below. The new line numbers indicate where we made the changes according to your suggestion on the marked up version of the manuscript.

Delete period. Extra period has been deleted. Line 41

Delete one space. The space has been deleted. Thank you. Line 94

In this section i.e. the before lines 105-115 you mentioned tools which have limited utility (only relevant for specific types of mentors or a specific population or not rigorously tested for validity). Regarding the example in these lines it seems not clear what you criticize. Is it that it is only focusing on C&T science mentees? Should be clearer.

Thank you for your comment. A new sentence was added to clarify that the investigators focused on C&T clinician and scientist mentors’ evaluations. Line 104-106

As the last sentence of this section is related to what was written before I would add “therefore” => “….such as the MET therefore is essential ….”

The last sentence of this section was revised as you suggested. Line 152

How many participants and how many of each group? Specify what you mean with “groups”. The reader probably does not have them in mind anymore, so maybe list them in parenthesis?

Thank you for your suggestion. We have included the number of mentors in each group and clarified the three groups. We contacted all 30 mentors who have received the Lifetime Achievement in Mentorship Award, all 76 mentors who have completed the Mentor Development Program and 100 mentors who have not received the Lifetime Achievement in Mentorship Award or completed the Mentor Development Program. Line 199-207 and 219-221

Is the 2014-study the present study you describe in the abstract and under “Design” (Material and Methods)? – this is somehow confusing throughout the manuscript and should be clarified from the beginning (which is/are pilot studies and which is the present study?). In this respect your manuscript is not easy to read and clarification would be a help for the reader.

We revised the Material and Methods section to clarify which studies were used in the development of the Mentor Evaluation Tool and which part is the current validity study. Line 167-170 and 191-193

Remove double space to be consistent within your manuscript.

Double space was removed between the previous line 191 and 192. Line 189-190

Why did you chose this method that faculty mentors could choose the mentees to be contacted? Why not randomly chose the mentees out of the whole group of each mentor. I think this needs explanation as it could be a bias, i.e. mentor could choose the mentees he/she anticipates most to rate positively. (Later in the discussion you refer to this point as a limitation).

Thank you for the suggestion. Ideally we would have used this approach, but the UCSF School of Medicine does not have a comprehensive database of mentor / mentee pairs for which to draw from so we had to rely on the mentors to provide the names of their mentees. The study design insured that mentors would not know which mentees evaluated them so as to limit any possible bias. Line 229-231

Why did you choose 3-5 Mentees? What are your thoughts/ calculations about participant number.

The rationale for choosing 3-5 mentees per mentors was to include as many diverse mentors as possible. New faculty mentors may only have 3 mentees while experienced mentors will have more than 5 mentees. We wanted to include a wide range of mentors, those who have been a mentor for several years, and those with less experience being a mentor. Line 225-227

Did the mentees provide their consent that their names and email-addresses were provided?

Mentees consented before they started the evaluation of their mentors. This will be included in the method section. Line 229-231

Was this done anonymously? You answer this question only in the discussion.

Mentees consented anonymously. We will include this in the Material and Method section as well as in the discussion section. Line 229

Primary contact author:

Michi Yukawa, MD, MPH

Michi.Yukawa@ucsf.edu

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

27 Apr 2020

PONE-D-19-35869R1

A New Mentor Evaluation Tool: Evidence of Validity

PLOS ONE

Dear Dr. Yukawa,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by 20 May. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Slavko Rogan

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The author addressed all my previous comments satisfactorily. I therefore recommend publication of this manuscript on the condition that 3 grammatical errors are corrected:

- line 170: change "the relevant literatures were reviewed" to "the relevant literature was reviewed"

- line 196: change "rated more higly" to "rated higher"

- line 227: change "greater than five mentees" to "more than five mentees"

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: A New Mentor Evaluation Tool- Evidence of Validity.docx

Click here for additional data file.

14 May 2020

Dear Editors and Reviewers; May 9, 2020

Thank you very much for your thoughtful comments and review of our manuscript “A New Mentor Evaluation Tool: Evidence of Validity” PONE-D-19-35869R1. We have responded to all of your suggestions below. The new line numbers correspond to the manuscript with track changes.

Reviewer #1: The author addressed all my previous comments satisfactorily. I therefore recommend publication of this manuscript on the condition that 3 grammatical errors are corrected:

Line 170: change "the relevant literatures were reviewed" to "the relevant literature was reviewed"

Thank you very much for your suggestion. The sentence was changed. (Line 188)

Line 196: change "rated more higly" to "rated higher"

Thank you very much for your edit. The sentence was changed. (Line 214)

Line 227: change "greater than five mentees" to "more than five mentees"

Thank you very much for your suggestion. The sentence was changed. (Line 247)

Reviewer #2:

Your introduction would be further strengthened by underscoring the negative consequences of poor mentors. That is the catalyst of creating an evaluation tool, which measures the effectiveness of a mentoring relationship. It could inform further mentor training.

We appreciate your suggestion. We added the following sentence to the Introduction: “Conversely, previous research has shown that failed mentorship may contribute to mentees not obtaining grant funding and leaving academic careers, among other negative outcomes (7,8).” (Lines 59-64)

L42 – Numbers under 10 are usually spelled out (this is throughout the document).

Thank you very much for your correction. All numbers under 10 now are spelled out throughout the manuscript.

L127-131 – The definition of mentorship described here is outdated. It is no longer an older, wiser, more experienced singular individual within the same industry. The more contemporary approach is a team of diverse mentors which may include someone senior to you but also those junior to you and at your level. If you are using the binary hierarchical definition of mentorship you should recognize that this is a limitation and it should be stated as such.

Thank you very much for your comment. We agree that team mentoring, peer mentoring etc. are very important to mentee success. We have added the following verbiage to better incorporate your suggestion, including a more recent definition of mentorship: The National Research Mentoring Network defines mentoring as: “A mutually beneficial, collaborative learning relationship that has the primary goal of helping mentees acquire the essential competencies needed for success in their chosen career. It includes using one’s own experience to guide another person through an experience that requires personal and intellectual growth and development” (28). We acknowledge that other mentoring models that incorporate team mentoring, peer mentoring, and distant and web-based mentoring are also important to mentee success.(7) As a dyadic mentoring relationship is often a key component of many mentoring programs, we chose to focus on this context of mentorship as we developed our tool. (Lines 138-149).

Also, as you suggest, we have added this issue to the discussion of study limitations. (Line 405-408)

For maximum impact, the research done here needs to be triangulated. Surveys are great but they miss answering the “why” question. That’s where interviews and focus groups could fill in the gap and would either affirm or disaffirm the survey data. This might also differentiate between the three groups.

Thank you very much for your suggestion. We agree that focus groups conducted among the mentees of three different groups of mentors would have provided more insightful data and may have differentiated among the three groups. We have added this as another one of our study limitations and something to explore in the future. (Line 408-411)

L356 – You mentioned the survey was sent out three times over two months. You need to list the two month duration in your methods section as well.

Thank you very much for your correction. We have added this to the methods section. (Line 227-229 and 402)

Submitted filename: Response to reviewers_5_2020.docx

Click here for additional data file.

26 May 2020

A New Mentor Evaluation Tool: Evidence of Validity

PONE-D-19-35869R2

Dear Dr. Yukawa,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Slavko Rogan

Academic Editor

PLOS ONE

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: Thank you for making the suggested modifications. I think it strengthens your manuscript. Well done.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

5 Jun 2020

PONE-D-19-35869R2

A New Mentor Evaluation Tool: Evidence of Validity

Dear Dr. Yukawa:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Slavko Rogan

Academic Editor

PLOS ONE