Finding the Sweet Spot: Investigating the Effects of Relationship Closeness and Instrumental Activities in School-based Mentoring.

School-based mentoring programs are popular prevention programs thought to influence youth development; but rigorous evaluations indicate that these programs often have small effects on youth outcomes. Researchers suggest that these findings may be explained by (a) mentors and mentees failing to develop a close relationship and (b) mentors not setting goals or focusing on specific skills necessary improve outcomes. We assessed these explanations using data from approximately 1360 mentor and mentee pairs collected through a national study of school-based mentoring (called, "The Student Mentoring Program"). Specifically, we tested the influence of mentee-reported relationship quality and mentor-reported use of goal-setting and feedback-oriented activities on academic, behavioral, and social-emotional outcomes. Results suggested that youth reported relationship quality was associated with small to medium effects on outcomes. Moreover, goal-setting and feedback-oriented activities were associated with moderate to large effects on outcomes. We also found significant interactions between relationship quality and goal-setting and feedback-oriented activities on youth outcomes. We conclude that there appears to be a "sweet-spot" wherein youth outcomes are maximized. The results of this study suggest a need for school-based mentoring programs to monitor and support mentors in developing a close relationship while also providing opportunities to set goals and receive feedback.

Introduction

Youth mentoring programs pair supportive adults with young people to help children navigate the developmental transition from childhood to adulthood (DuBois & Keller, 2017; Rhodes, 2005). The rationale for youth mentoring is grounded in developmental science research, where researchers found that children who have positive relationships with supportive adults fair better than those that do not (Bowen & Chapman, 1996; Garmezy & Masten, 1986; Parra, DuBois, Neville, Pugh‐Lilly, & Povinelli, 2002). Young people who develop relationships with supportive adults (e.g., teachers, coaches, family friends, parents) achieve higher grades, engage in less risky behaviors, and report higher levels of life satisfaction (Jiang, Lyons, & Huebner, 2016; Lyons, Huebner, & Hills, 2013; Malecki & Demaray, 2003). However, it is unclear what makes a relationship cohere and what makes it supportive or not, particularly in the context of structured programs that comprise the majority of efforts at formal mentoring. Moreover, it is unclear how such support translates to influence on positive youth development.

This uncertainty may occur because there is variability in what is meant by the term mentoring, what is thought to be necessary for mentoring to occur, and variability in the context, structures, and goals of mentoring program. Some have argued that the closeness of the mentor–mentee bond is the defining feature and central ingredient of successful mentoring relationships and outcomes (Rhodes & DuBois, 2008), whereas others suggest that relationship quality may be tangential to other goal‐focused or prevention focused activities (Cavell & Elledge, 2014). However, there is relatively little understanding of how both of these hypothesized mechanisms (i.e., relationship quality and goal focused activities) may interact with each other to simultaneously influence outcomes of mentoring programs. It seems that by understanding how these mechanisms correspond to outcomes, researchers and practitioners may be able to engineer program supports to improve impact. Toward this end, the purpose of this study is to test the relative and combined influence of relationship quality, goal setting, and feedback on mentee outcomes in the context a large‐scale randomized controlled trial of school‐based mentoring.

Theoretical Models of Mentoring

Mentoring programs attempt to facilitate positive interactions between children and adults in order to promote healthy academic, behavioral, and social‐emotional development (Rhodes, 2005). However, the nature of these interactions and mechanisms through which these outcomes are thought to occur differs from program to program. In some programs, mentors emphasize interactions that are meant to foster a stronger, closer relationship. In other programs, interactions are designed to teach youth skills necessary for healthy youth development. Researchers have described these differences in terms of two models of mentoring: developmental and instrumental. Developmental mentoring is a model of mentoring that emphasizes a close, long‐lasting relationship as the primary mechanism through with mentors support mentee development. Instrumental mentoring is a complementary or alternative framework that emphasizes skill development and goal‐directed activities. In the subsections below, we review the theoretical and empirical support for these different mentoring models.

Developmental Mentoring

Significant resources and funding have been aimed at developmental mentoring programs, many of which are based Rhode's model of youth mentoring (Rhodes, 2005). The model posits that mutuality, trust, and empathy in the mentoring relationship are prerequisites to the mentee's social‐emotional, cognitive, and identity development, which all contribute to positive outcomes (Rhodes, 2005). According to Karcher, Kuperminc, Portwood, Sipe, and Taylor (2006), developmental mentoring involves mentors “playing games or engaging in other forms of recreational activity with their mentees, as well as talking with them about mutual interests… [with the goal to] establish conditions in the relationship that can facilitate the youth's social, cognitive, and emotional development” (p. 714).

Empirical studies of youth mentoring programs have shown an inconsistent association between the length and strength of the mentoring relationships. Some empirical studies suggest a positive association between the length and quality of the relationship with outcomes (Grossman, Chan, Schwartz, & Rhodes, 2012; Grossman & Rhodes, 2002). Bayer, Grossman, and DuBois (2015), for example, found a positive association between the strength of the relationship and youth outcomes. Despite these findings, Cavell and Elledge (2014) point out that the connection between relationship quality and outcomes are correlational and effect sizes are typically small (i.e., Cohen's d between 0.12 and 0.18; Bayer et al., 2015). Consistent with this critique, Lyons and McQuillin (2018) found a negative quadratic relation between youth‐reported relationship quality and outcomes in school (i.e., grades and behavior). The authors concluded that relationship quality may be a necessary but not sufficient mechanism to improve outcomes in school.

Instrumental Mentoring

As in developmental mentoring, instrumental mentoring emphasizes rapport, working alliance, mutual consent, regular contact, positive interaction, and a focus on the mentee. Instrumental programs do not necessarily emphasize the duration of the relationship or the closeness of the bond; rather the focus is typically on a goal directed activity or structured skill building experience that is specifically designed to target an outcome (e.g., school grades, positive behavior, emotional regulation; McQuillin, Terry, Strait, & Smith, 2013). Thus, instrumental models of mentoring specify the mechanism of change as the skills and increased competence as opposed to the closeness of the bond or the endurance of the relationship (DuBois, Portillo, Rhodes, Silverthorn, & Valentine, 2011; Nakkula & Harris, 2013). In fact, there is some evidence that brief‐instrumental (i.e., goal focused) programs are capable of producing positive effects. McQuillin and Lyons (2016), for example, found moderate, positive changes in student grades and behavior after middle school youth participated in a brief (i.e., 12‐week), goal‐focused, mentoring program. There is concern, however, that goal focused, or instrumental mentoring programs may detract from the construct of mentoring. Rhodes and DuBois (2008) articulated this, writing:

One area of concern is the increasingly broad range of activities—such as tutoring, after‐school, and service learning programs—that are argued to constitute mentoring. Underlying this trend seems to be the perspective that any program in which adults are brought into contact with young people may count as providing mentoring regardless of the nature or time frame of the relationships that are involved… In effect, mentoring programs have moved in a direction that is in danger of trivializing what research indicates is at the very heart of the intervention: a caring adult‐youth relationship.(p. 257)

Mentoring Context

As Rhodes and DuBois (2008) suggest, the context (e.g., after‐school or service learning programs) and programmatic goals may influence the focus of what occurs within the youth–adult relationships. In some contexts (e.g., academic settings), it may be more appropriate to focus on developing academic skills while other contexts may require mentors to have different foci. In this study, we discuss some of these considerations for mentoring programs that occur in the school context. We focus on mentoring in the schools, called school‐based mentoring (SBM), because SBM has become one of the most popular site‐based contexts in which mentoring services are provided to young people (Garringer, McQuillin, & McDaniel, 2017).

Some researchers suggest that some SBM programs may be incompatible with the developmental model of mentoring (McQuillin et al., 2013). Because the SBM relationship is often characterized by shorter relationships, briefer meetings, and less frequent contact in comparison to mentoring programs that occur in other contexts (e.g., through community‐based volunteer organizations; Bernstein, Rappaport, Olsho, Hunt, & Levin, 2009; DuBois et al., 2011; Herrera, Grossman, Kauh, & McMaken, 2011), there may be fewer opportunities to develop a close mentoring relationship. Thus, the context and structures within the school environment may attenuate the influences of the relational mechanisms thought to be active in mentoring programs. In response to these findings, some researchers reasoned that the contextual limitations of the school environment may prohibit long‐lasting, close youth‐adult relationships and have suggested that school‐based programs emphasize goal‐directed activities as the primary mechanism for producing change (Cavell & Elledge, 2014; McQuillin, Smith, & Strait, 2011).

Gaps in Research

Most evaluation research indicates that on average SBM programs produce small positive effects on some outcomes, but there is considerable variability in the effects (Wheeler, Keller, & DuBois, 2010). As a result of these findings, researchers have investigated the relations between two theoretical mechanisms through which mentors are thought to influence youth outcomes – the quality of the mentor–mentee relationship and the skills that mentors teach youth (Bayer et al., 2015; McQuillin & Lyons, 2016). Rhodes (2005), for example, described the process of a close mentoring relationship as leading to skills development (i.e., cognitive, social‐emotional, and identity development). In addition, others have suggested that mentors may have larger effects when they engage in goal‐oriented interactions in the context of a supportive youth–adult relationship (Karcher, Herrera, & Hansen, 2010). However, studies have not examined the relative role of these processes as it relates to youth outcomes. This is notable because both instrumental and developmental models of mentoring discuss the youth–adult relationship as well as skill development as components within the model. Yet, no study has explored how both of these processes influence the outcomes targeted by SBM programs (i.e., academic, behavioral, and social‐emotional).

In addition to the theoretical gaps in the research, there are methodological challenges associated with estimating the effect of a treatment as a function of variables that are not easily randomized (e.g., the quality of the mentoring relationship). New methods have been developed to approximate the causal effects in these scenarios (e.g., propensity score matching, complier average causal effects models). Although these approaches work well for categorical data, they are not easily applied to questions that involve estimating treatment effects as a function of a continuous third variable (e.g., what is the estimated treatment effect as a function of relationship quality?) (Hill, 2011).

The Current Study

The current study was conducted to examine the effects of an SBM program as a function of youth‐reported relationship quality and two indicators of instrumental activities (i.e., goal‐setting and constructive feedback) as reported by the mentor on youth's academic outcomes (i.e., school‐reported grades and standardized test scores). In this study, we test the relative role of developmental and instrumental activities in an SBM context to understand how relationship quality and instrumental activities influence youth outcomes. Although these data have been published elsewhere (e.g., Bernstein et al., 2009; Lyons & McQuillin, 2018), the following analyses are new.

Based on previous research related to relationship quality and outcomes (Bayer et al., 2015), we hypothesized that (a) effects would increase as a youth reported a higher‐quality relationship with their mentor and (b) instrumental activities would significantly be associated with youth outcomes. Because SBM program are considered a nonspecific, prevention intervention, we assess these relations across three domains commonly targeted by mentoring programs: academic (measured by grades and standardized test scores), social‐emotional, and behavior (measured by misconduct and delinquency reports as well as school bonding). However, given the exploratory nature of these analyses, we did not specify the form of the relations between instrumental and relationship quality on youth outcomes.

Method

Participants

Participant data comes from the Impact Evaluation of the Student Mentoring Program (Bernstein et al., 2009). Teachers and other school personnel referred youth who they deemed as in need of mentoring. In total, approximately 2670 youth were recruited for the study and 1360 were randomly assigned to participate in the Student Mentoring Program (see Fig. 1 for participant flow; note: sample sizes are rounded to the nearest tens place in accordance with the Institute for Education Science's reporting guidelines). Less than 1% of students were not randomly assigned, but automatically placed in the treatment group because school personnel identified them as in “extreme need of mentoring services”. The youth who were not randomly assigned were excluded from analyses. The participants were 47% male and the average age was 11 years, 2.4 months. Forty‐one percent of the sample identified as Black or African American, 31% identified as Hispanic, 22% identified as White, and the remainder identified as another ethnicity. Eighty‐six percent of participants were eligible free or reduced lunch.

Figure 1

Participant flow. Note: Per Institute for Education Sciences guidelines, sample sizes were rounded to nearest tens place. Thus, marginal and overall n's do not necessary sum together

The mentors were mostly women (72%) and had an average age of 32. Sixty‐six percent of mentors were White, 29% were African American, and less than 10% were American Indian or Alaskan Native, Asian or Native Hawaiian or other Pacific Islander. Eighty‐two percent mentors reported having at least some college or post‐secondary training; 31% of mentors were college students. About one‐third of the mentors were married or living with a partner. For additional information on the mentors or mentees, see Bernstein et al. (2009).

Measures

Although we review the measures used in this study below, a more detailed description of the measures used in the Student Mentoring Program can be found in Appendix C of the original report (Bernstein et al., 2009).

Covariates

A number of widely used measures were adapted for use by the Student Mentoring Program and used as covariates in these analyses. The adapted measures included the select subscales measuring prosocial behavior (i.e., parental relationships) from the National Longitudinal Study of Adolescent Health (Bernstein et al., 2009), the Self Perception Profile for Adolescents (Harter, 1988), the School Bonding scale (Hawkins, Guo, Hill, Battin‐Pearson, & Abbott, 2001), and SAMHSA's Monitoring the Future measure of delinquency and misconduct (Johnston, O'Malley, & Bachman, 2000). Covariates included a 10‐item scale for pro‐social behaviors (e.g., “I volunteer to help others through church, mosque, temple, or synagogue.”; α = .69), a four‐item future orientation scale (e.g., “How important is it for you to graduate from high school?”; α = .76), an eight‐item misconduct scale (e.g., “During the past month did you punch, kick, or hit someone?”; α = .72), and an eight‐item scholastic efficacy and school bonding scale (e.g., “I do well at my classwork”; α = .74). All items were rated on a scale from 1 (none) to 4 (a lot). In addition, data on recent (i.e., over the past 30‐days) delinquent behavior was measured using a seven‐item scale (e.g., “during the past month did you carry a weapon, such as a club, knife, or gun?”; α = .74), which ranged from 1 (none) to 6 (10 or more times).

Relationship Quality

Evaluators of the Student Mentoring Program developed measure of mentee‐reported relationship quality (Bernstein et al., 2009). In this study, we used a composite score of four‐items reflecting youth‐perceptions of the bond between the mentor and mentee (e.g., “I feel that I can trust my mentor”). Youth reported relationship quality on a four‐point Likert scale ranging from 1 (not true at all) to 4 (very true). To facilitate interpretation of model parameters, we converted relationship quality scores to z‐scores (mean = 0 and SD = 1) and reported measures of relationship quality in terms of normalized percentiles (i.e., 50th percentile corresponds to a z‐score of 0). Percentiles allow for a standardized measure of relative position and are more easily interpreted compared to raw scores. In this study, Cronbach's alpha equaled .59.

Instrumental Activities

Two instrumental activities were measured in this study: goal‐setting and constructive feedback and both were collected based on the mentor‐report. Mentors were asked to select the three most common activities they did with their mentees. Mentors endorsed goal‐setting based on the single‐item, “Set goals or standards for the student or helped the student set goals for him or herself.” Likewise, feedback was endorsed based on a single item, “Provided the student with constructive criticism about his or her behavior.” Mentors endorsed both measures of instrumental activities of these items were measured dichotomously (1 = indicating this was one of the three most common activities they did; 0 = indicating this was not something they did regularly).

Outcomes: School Records

School records were obtained for each mentee. The records included student grade level and demographic information, class performance in math, English language arts, science, and social studies as well as performance on state‐wide assessments and attendance. Grades were coded on a five‐point scale and correspond with A, B, C, D, and F marks. Additionally, state‐wide assessments in math and English were coded as proficient or not proficient. Disciplinary actions were also obtained and categorized as harassment, non‐compliance, property related offenses, drug‐related offenses, truancy, or violence. These categories were then condensed into the broad bands of delinquency (i.e., violence, drug‐related infractions, and truancy) and misconduct (i.e., non‐compliance, harassment, and property offenses) with total frequency recorded. Finally, school bonding following the completion of the mentoring program was also included as an outcome.

Procedures

The Student Mentoring Program was an SBM service provided to youth in elementary through high school. Youth were randomly assigned to receive a school‐based mentor for one school‐year. To be eligible to participate in this study, mentoring programs were required to serve students with few positive adult role models and needed to target academic and social needs of the students. According to Bernstein et al. (2009), “87% of mentors reported meeting on a one‐to‐one basis with their students, averaging 4.4 meetings per month with meetings lasting 1.1 hours on average” (p. 32). Mentor training and support services were provided by local community agencies. Data were collected from school‐, mentor‐ and youth‐report. Mentor and youth participants completed surveys in the fall and then again in the spring.

Data Analysis

To estimate the effects as a function of a continuous third variable, we follow the analytic procedures outlined by Hill (2011). This analytic approach is grounded in the potential outcomes framework wherein the theoretical outcome for each participant is conditional on their potential treatment assignment (i.e., their potential outcome). In this study, we obtain potential outcomes using a Bayesian Additive Regression Trees (BART) model. BART is a Bayesian, nonparametric model in which predictions are obtained from a recursive partitioning procedure in which data are split into homogeneous subgroups. The BART model is suited to estimate potential outcomes because it is robust to violations of the ignorability assumption. That is, BART allows for a large number of covariates to be included in the model, which decreases the likelihood that the model is misspecified. In addition, BART does not assume linear relations between covariates and outcomes and is designed for high dimensional data.

In the current study, the model used to estimate the treatment effect is shown in Equation (1).

In this model, the estimated effect for individual i (denoted as TE) was calculated based on the difference between two potential outcomes. Each potential outcome was calculated conditional on a set of covariates (denoted by x ), treatment assignment (denoted as TX) and mentee‐reported relationship quality (denoted as r ). As shown, the potential outcome for assignment to control was conditional on a mentee‐reported relationship quality equal to zero (i.e., R = 0). The standardized difference in these potential outcomes were calculated to estimate the student's effect for a given level of relationship quality measured in standard deviation units (i.e., Cohen's d). This effect is called the treatment‐on‐treated effect because treatment effects are estimated for only those assigned to the treatment condition.

Potential outcomes were estimated conditional on 26 covariates. Covariates included demographic characteristics (e.g., age, free or reduced lunch status), academic performance at baseline (e.g., school‐reported grades in math, science, English, and social studies), and baseline behavioral performance (e.g., self‐reported misconduct, self‐reported delinquency). To examine the potential moderating effects of the two instrumental skills measured: goal‐setting and constructive feedback, estimated treatment‐on‐treated effects as a function of relationship quality independently for two subgroups of the data: (a) youth who had mentors that reported the use of the instrumental skill (i.e., goal‐setting or feedback) and (b) those who did not. After we estimated the treatment‐on‐treated effects for each outcome, a lowess line was fit to the data. The results show the relations between the estimated treatment effects as a function of relationship quality and instrumental activities. We also conducted multiple regression analyses to assess (a) the linear associations between relationship and instrumental activities and outcomes and (b) possible statistical interactions between relationship quality and instrumental activities on outcomes.

Results

Descriptive statistics for the covariates and outcome variables are reported in Table 1. Approximately 53% (n = 520) of mentors reported that they engaged in goal setting with their mentees. Likewise, approximately 52% (n = 510) of mentors reported that they provided constructive feedback to youth. BART models were fit conditional on all covariates as described above. Next, estimated treatment‐on‐treated effects for outcomes were plotted as a function of student‐reported relationship quality and mentor‐reported instrumental skills (i.e., goal‐setting and constructive feedback). To assess the functional form of this relationship, a lowess line was fit to the data (Fig. 2 shows treatment‐on‐treated effects by constructive feedback and Fig. 3 shows treatment‐on‐treated effects by goal‐setting).

Table 1

Descriptive statistics

Variable	Mean	SD
Dependent variables (measured at Time 2)
School‐reported social studies grade	3.350	1.090
School‐reported English grade	3.341	1.039
School‐reported math grades	3.171	1.089
School‐reported science grade	3.328	1.092
School‐reported standardized reading test score	0.489	0.500
School‐reported standardized math test scores	0.443	0.497
Self‐reported school efficacy & bonding	3.047	0.587
Self‐reported misconduct	3.200	0.606
Self‐reported delinquency	3.869	0.271
Instrumental activities (measured at Time 2)
Goal setting (1 = set goals; 0 = did not set goals)	0.5181	0.500
Feedback (1 = gave feedback; 0 = did not give feedback)	0.5282	0.500
Covariates (measured at Time 1)
Age	11 years 2.4 months	1 years 5.3 months
Gender	1.528	0.499
Household composition (1 = two parents; 0 = not two parents)	0.567	0.496
Number of misconduct	0.563	2.218
Any misconduct	0.159	0.366
Behavioral referrals	0.219	0.414
Delinquency	0.253	0.908
School‐reported social studies grade	3.480	1.043
School‐reported English grade	3.455	1.012
School‐reported math grades	3.296	1.046
School‐reported science grade	3.494	1.026
School‐reported standardized reading test score	0.503	0.500
School‐reported standardized math test scores	0.514	0.500
Self‐reported school efficacy & bonding	3.153	0.545
Self‐reported misconduct	3.300	0.563
Self‐reported delinquency	3.892	0.232
Self‐reported home and community involvement	2.869	0.529
Self‐reported future plans	3.821	0.423
Self‐reported school efficacy	2.972	0.544
Self‐reported school bonding	3.086	0.636
Self‐reported peer relationships	2.749	1.015
Self‐reported parental relationships	3.073	0.741
Self‐reported other adult relationships	2.297	0.836
Self‐reported social responsibility	2.821	0.635

Figure 2

Cohen's d treatment‐on‐treated effects for mentoring relationship and instrumental skill “feedback”. RQ = Relationship Quality.

Figure 3

Cohen's d treatment‐on‐treated effects for mentoring relationship and instrumental skill “goal‐setting”. RQ = Relationship Quality.

Next, multiple regression analyses were conducted to examine the strength and direction of relations between relationship quality, instrumental skills, and treatment‐on‐treated effects. Regression results for outcomes as a function of relationship quality and constructive feedback are presented in Table 2. Main effects of relationship quality ranged from near zero (e.g., Math grades, β = −.064) to moderate and positive (e.g., State test scores in reading, β = .341). Main effects of constructive feedback ranged from large and negative (e.g., Social studies grades, β = −.917) to near zero (e.g., State test scores in math, β = .088). Effect sizes for interactions between relationship quality and constructive feedback ranged from moderate and negative (e.g., State test scores in math, β = −.352) to small and positive (e.g., Science grades, β = .032).

Table 2

Regression results for feedback

Dependent variable	Predictors	β	SE	t‐value	p‐value
School bonding	Intercept	.178	0.002	79.397	<.001
	RQ	.131	0.008	16.346	<.001
	Feedback	−.289	0.003	−88.727	<.001
	RQ x Feedback	.061	0.012	5.277	<.001
Delinquency	Intercept	.377	0.004	87.855	<.001
	RQ	.396	0.015	25.793	<.001
	Feedback	−.181	0.006	−29.031	<.001
	RQ x Feedback	−.203	0.022	−9.169	<.001
Misconduct	Intercept	−.040	0.012	−3.262	.001
	RQ	.135	0.044	3.065	.002
	Feedback	.046	0.018	2.551	.011
	RQ x Feedback	.221	0.063	3.498	<.001
Social Studies grades	Intercept	.604	0.003	232.510	<.001
	RQ	.045	0.009	4.851	<.001
	Feedback	−.917	0.004	−242.739	<.001
	RQ x Feedback	−.068	0.013	−5.070	<.001
Science grades	Intercept	.351	0.002	187.183	<.001
	RQ	−.003	0.007	−0.509	0.61
	Feedback	−.495	0.003	−181.437	<.001
	RQ x Feedback	.032	0.010	3.289	0.001
Math grades	Intercept	.144	0.002	58.244	<.001
	RQ	−.064	0.009	−7.285	<.001
	Feedback	−.115	0.004	−32.073	<.001
	RQ x Feedback	−.052	0.013	−4.120	<.001
English grades	Intercept	.197	0.003	73.448	<.001
	RQ	.027	0.010	2.862	0.004
	Feedback	−.833	0.004	−213.300	<.001
	RQ x Feedback	−.046	0.014	−3.324	<.001
Standardized reading test scores	Intercept	.012	0.002	5.950	<.001
	RQ	.341	0.007	49.180	<.001
	Feedback	.050	0.003	17.700	<.001
	RQ x Feedback	−.232	0.010	−23.170	<.001
Standardized math test scores	Intercept	−.047	0.002	−26.770	<.001
	RQ	.309	0.006	49.570	<.001
	Feedback	.088	0.003	34.910	<.001
	RQ x Feedback	−.352	0.009	−39.250	<.001

RQ = Relationship Quality.

Regression results for outcomes as a function of relationship quality and goal‐setting are presented in Table 3. Main effects of relationship quality ranged from moderate and negative (e.g., Math, β = −.273) to small and positive (e.g., Social studies grades, β = .111). Main effects of goal‐setting ranged from large and negative (e.g., Social studies grades, β = −.668) to small and positive (e.g., State test scores in math, β = .252). Effect sizes for interactions between relationship quality and goal‐setting ranged from small and negative (e.g., Social studies grades, β = −.183) to moderate and positive (e.g., Math grades, β = .332).

Table 3

Regression results for goals

Dependent variable	Predictors	β	SE	t‐value	p‐value
School bonding	Intercept	.143	0.002	57.386	<.001
	RQ	.129	0.009	14.802	<.001
	Goals	−.406	0.004	−113.559	<.001
	RQ x Goals	.009	0.013	0.677	0.498
Delinquency	Intercept	.360	0.004	91.160	<.001
	RQ	.545	0.014	39.330	<.001
	Goals	−.375	0.006	−65.900	<.001
	RQ x Goals	−.521	0.020	−25.830	<.001
Misconduct	Intercept	−.087	0.007	−11.731	<.001
	RQ	.324	0.026	12.460	<.001
	Goals	.304	0.011	28.520	<.001
	RQ x Goals	−.367	0.038	−9.691	<.001
Social Studies grades	Intercept	.361	0.002	145.020	<.001
	RQ	.111	0.009	12.740	<.001
	Goals	−.668	0.004	−186.630	<.001
	RQ x Goals	−.183	0.013	−14.410	<.001
Science grades	Intercept	.171	0.002	79.827	<.001
	RQ	.054	0.008	7.220	<.001
	Goals	−.306	0.003	−99.184	<.001
	RQ x Goals	−.069	0.011	−6.257	<.001
Math grades	Intercept	.034	0.003	9.746	<.001
	RQ	−.273	0.012	−22.568	<.001
	Goals	.133	0.005	26.794	<.001
	RQ x Goals	.332	0.018	18.819	<.001
English grades	Intercept	−.431	0.003	−155.920	<.001
	RQ	−.113	0.010	−11.700	<.001
	Goals	.252	0.004	63.330	<.001
	RQ x Goals	.160	0.014	11.300	<.001
Standardized reading test scores	Intercept	−.066	0.001	−44.710	<.001
	RQ	.079	0.005	15.370	<.001
	Goals	.212	0.002	100.130	<.001
	RQ x Goals	.231	0.008	30.790	<.001
Standardized math test scores	Intercept	.143	0.001	103.611	<.001
	RQ	.098	0.005	20.295	<.001
	Goals	−.250	0.002	−126.041	<.001
	RQ x Goals	−.068	0.007	−9.727	<.001

Discussion

This study examined how mentee‐reported relationship quality and mentor‐reported goal‐setting and feedback‐oriented activities influence the strength of the mentoring relationship and instrumental activities contributed to academic, behavioral, and social‐emotional outcomes. Previous research indicates that the strength of mentoring relationships plays a key role in programs achieving desired outcomes in SBM programs (Bayer et al., 2015; Lyons & McQuillin, 2018). In addition, instrumental activities (i.e., constructive feedback and goal‐setting) have also been considered key mechanisms for promoting youth development, and programs that emphasize instrumental activities show promise (McQuillin & Lyons, 2016). However, the relative merit of these mechanisms (i.e., developmental vs. instrumental) is not well understood. Thus, it is important to understand how developmental mentoring (i.e., focus on the relationship) and instrumental mentoring (i.e., focus on goals and feedback) combined may influence youth outcomes in mentoring programs.

We used data from the largest experimental study of SBM, the Student Mentoring Program, to examine the interactions between instrumental and developmental aspects of mentoring. The findings showed that a combination of goal‐ and feedback‐oriented approaches along with the development of a close relationship had the largest effects for student outcomes and especially behavioral outcomes. These results demonstrate the importance of the mentoring relationship, but also the need to incorporate instrumental activities within SBM programs. Based on these findings, we suggest that there may be a “sweet‐spot,” wherein, even in school‐based relationships, outcomes may be achieved by balancing relationship development and instrumental activities; though, in isolation, these approaches may not be sufficient to produce helpful or noticeable effects. We draw three primary conclusions from our results.

First, for most outcomes, mentee‐reported relationship quality has a positive association with treatment effects; but the effect sizes vary depending on the outcomes. For school grades, effect sizes generally ranged from near zero to small (β's near .1); whereas, for behavioral outcomes (i.e., delinquency and misconduct), the effect size falls in the small to moderate ranges with parameter estimates. This result is consistent with prior research on relationship quality and youth mentoring that suggests that relationship quality has a positive association with outcomes (Bayer et al., 2015).

Second, we found that instrumental activities (i.e., providing feedback and setting goals) generally has a positive association across academic, behavioral, and social‐emotional outcomes; but this association also varies depending on the outcome. For behavioral outcomes, the association between instrumental activities and treatment effects generally fell within the small to moderate range (β's from .13 to .4). In contrast, the association between instrumental activities and treatment effects and most academic outcomes fell between the moderate to large range. Although these effect sizes are consistent with other experimental studies of manualized instrumental mentoring programs (McQuillin & Lyons, 2016), this result is significant because it demonstrates the association of instrumental activities within highly variable mentoring programs.

Third, we found evidence of synergistic effects between relationship quality and goal setting and feedback that were especially pronounced for behavioral outcomes. Specifically, mentees who reported a high‐quality relationship and who had mentors that regularly engaged in goal‐setting or feedback were estimated to have the greatest treatment effects compared to those who did not. These findings are consistent with the notion that the relationship matters in mentoring programs (Rhodes, 2005), but also provide support for the merit of the mentoring relationship serving as a vehicle for outcome focused activities in mentoring (Cavell & Elledge, 2014). Balancing developmental and instrumental activities may be particularly important in SBM programs due to the time‐limited nature of the mentoring relationship. Additionally, instrumental activities provide a basis for researchers and practitioners to understand what occurs in mentoring relationships, thereby making it easier to identify the mechanism for change (McQuillin & Lyons, 2016). When assessing developmental mentoring outcomes, the activities that the mentor and mentee engage in together are often unclear. Generally, hybrid models of mentoring that include developmental and instrumental activities may prove most beneficial for students in SBMs.

Although data from the impact evaluation of SMP data found null effects (Bernstein et al., 2009), we found positive effects across academic, behavioral, and social outcomes after accounting for the relationship quality and instrumental activities that occurred between mentors and mentees. Our findings suggest that, when accounting for relationship quality and instrumental activities, the SMP actually did benefit some of the students who participated. However, most mentoring relationships did not achieve the level of relationship quality and instrumental activities necessary to produce a moderate to large average intent‐to‐treat effect. We hope this study is enlightening for mentoring researchers who have traditionally theorized that mentoring relationship as the key mechanism for change. These results suggest a need to renew emphasis on infusing instrumental activities in mentoring relationships in SBM programs.

Limitations

Though researchers designed the methods that were employed in the current study to control for plausible alternative explanations in quasi‐experimental data, the study was non‐experimental in nature, and therefore causal inferences must be tempered. We see an urgent need to use experiments to test how relationship‐focused and instrumental activities independently and/or synergistically influence mentoring outcomes in SBM programs. Although the statistical methods employed in this study are more robust to violations of the ignorability assumption, it is possible that treatment‐on‐treated effects are biased due to unmeasured third variables related to treatment assignment and the outcomes. In other words, it could be that some unmeasured aspect of the mentor, mentee, or program influenced both instrumental activities as well as academic performance and behavior.

Additionally, measures of instrumental and relationship quality were not derived from previously validated instruments. The items on the relationship quality scale were adapted from other measures of the mentoring relationship quality and the revised scale was not previously validated. In addition, instrumental activities were not randomly assigned, were based on mentor report, and were measured dichotomously. Thus, the measure of instrumental activities can only be considered a crude estimate of the instrumental activities in the relationship. Mentors may have only reported salient aspects of the work that they did with their mentees as there were no systems in place for researchers to systematically monitor dosage or timing of instrumental activities. In future studies, researchers should consider formally tracking mentor‐mentee interactions to better understand how mentor interactions influence the developmental and instrumental aspects of mentoring (see discussion of textured assessments in McQuillin, Lyons, Clayton, & Anderson, 2018).

As a result of the limited monitoring of dosage and implementation, the sequencing of instrumental and relationship skills in the mentoring relationship was not clear. It is possible that if the relationship is developed first, there may be an upward spiral in which the relationship drives instrumental activities, which in turn continues to the drive the relationship as well as engagement in instrumental activities. We hope for future research that can test this possibility.

Implications

Our findings demonstrate the need for mentor training and support to focus on more than just relationship quality. Mentors should also receive training on instrumental activities and skills that directly relevant to outcomes considered important by programs and the youth who participate in them. This type of training will serve a dual purpose; on one hand, mentors will be provided with skills to use, but additionally, programs will have a better idea about how to evaluate outcomes. Currently, the content and activities that take place between a mentor and mentee are often ambiguous, but a focus on instrumental skills and activities will facilitate the process of evaluating mentors and mentors based on the implementation of target skills.

The quality of youth adult relationships influences youth school‐related outcomes (i.e., behavior and academic outcomes in school), but the quality of mentoring relationship is not the only thing that matters. How mentors teach and support youth in developing instrumental skills related to increasing performance at school also has an important role in producing positive effects. The results of this study suggest that mentoring programs should carefully attend to the interactions between the quality of the mentoring relationship and instrumental activities. In order to do this, future studies must examine causal mechanisms and experimentally manipulate instrumental activities within traditional developmental mentoring program to evaluate the processes through which mentoring programs produce positive school‐related outcomes.

Conflict of Interest

The authors declare that they have no conflict of interest.