Comparing artificial intelligence and human coaching goal attainment efficacy.

The history of artificial intelligence (AI) is filled with hype and inflated expectations. Notwithstanding, AI is finding its way into numerous aspects of humanity including the fast-growing helping profession of coaching. Coaching has been shown to be efficacious in a variety of human development facets. The application of AI in a narrow, specific area of coaching has also been shown to work. What remains uncertain, is how the two compare. In this paper we compare two equivalent longitudinal randomised control trial studies that measured the increase in clients' goal attainment as a result of having received coaching over a 10-month period. The first study involved human coaches and the replication study used an AI chatbot coach. In both studies, human coaches and the AI coach were significantly more effective in helping clients reach their goals compared to the two control groups. Surprisingly however, the AI coach was as effective as human coaches at the end of the trials. We interpret this result using AI and goal theory and present three significant implications: AI coaching could be scaled to democratize coaching; AI coaching could grow the demand for human coaching; and AI could replace human coaches who use simplistic, model-based coaching approaches. At present, AI's lack of empathy and emotional intelligence make human coaches irreplicable. However, understanding the efficacy of AI coaching relative to human coaching may promote the focused use of AI, to the significant benefit of society.

Introduction

Since its inception in the 1950s, artificial intelligence (AI) has seen several periods of growth and decline, casting doubt on its actual versus claimed efficacy [1]. Lately, renewed interest in AI has led to numerous novel applications of this technology, including in healthcare and helping professions such as psychology and coaching [2-4]. In this paper, coaching is defined as a one-on-one structured conversation between a coach and client with the aim of facilitating sustainable change for the individual and potentially other stakeholders [5]. Coaching is a late entrant to the application of AI, and AI’s role and efficacy in coaching remain largely under-researched.

Coaching is an important helping profession. It is a fast-growing multi-billion dollar per year industry [6] and has grown substantially both in practice and research in the last decade [7]. Numerous coaching meta-studies have made a clear case for its efficacy [8-13]. There is a strong link between successful coaching outcomes and the relationship and bond between the coach and client with convincing evidence that the coach-client relationship is the most significant factor in coaching success [14-16]. Nevertheless, the current limitations of AI, especially relating to true human intelligence and emotions [17], cast doubt over the ability of an AI coach to currently compete with a human coach.

Recent studies on the application of AI in psychology, however, have suggested that AI could be effective in certain domains of promoting human wellbeing. Fulmer et al. [18], for example, used an AI agent based on cognitive behavioral therapy (CBT) to reduce self-identified symptoms of depression and anxiety in college students. They concluded that AI could serve as a cost-effective and accessible therapeutic agent. Greer et al. [19] found that young adult cancer patients had reduced anxiety compared to a control group after using a positive psychology-based AI coach for four weeks. These findings suggest that, while AI lacks true human intelligence and emotions, positive outcomes are possible even in practices that have traditionally relied on a strong human connection. This might potentially also be the case for coaching.

One of the primary focal areas of coaching and what sets it apart from other helping professions, is assisting clients with goal attainment [20, 21]. Understanding the efficacy of AI coaching compared to human coaching in the domain of goal attainment therefore seems like a reasonable starting point for AI coaching research. This leads us to ask the following research question: In a similar setting, how does AI coaching compare to human coaching efficacy in relation to client goal attainment?

In this paper, we investigate this question by presenting a comparison of the two studies on goal attainment coaching: the first involving human coaches and the second an AI coach. We interpret the results in terms of the current state of AI and goal theory. We also discuss the way these results may pave the way for aspects of coaching to be made more widely available and the implication for coaches and the coaching industry. Given the continued growth of coaching as a helping profession and its proven efficacy, understanding how AI could play a role in scaling and democratizing this service is an important research area.

Current capabilities of artificial intelligence

AI has seen several false starts mostly because of exaggerated claims of progress and ability that inevitably led to disappointment. An example is Marvin Minsky, the father of AI who back in 1967 stated that “Within a generation … the problem of creating artificial intelligence will substantially be solved” [22]. AI has experienced a few “winters” where these types of exaggerations led to withdrawal of funding and the collapse of interest in AI research and development [1]. However, the recent resurgence in AI interest appears to be more sustainable as AI is focused on specific specialist areas in line with current AI capabilities, and shows promise in areas such as decision-making processes [23].

AI is defined as “the broad collection of technologies, such as computer vision, language processing, robotics, robotic process automation and virtual agents that are able to mimic cognitive human functions” [24 p4]. However, in order to understand AI’s realistic capabilities, it is important to distinguish between three types of AI: (i) Artificial narrow intelligence (ANI) refers to systems that can perform a specific task in a narrow context, such as a self-driving car; (ii) Artificial general intelligence (AGI) refers to systems that have intelligence similar to human intelligence; and (iii) Artificial super intelligence (ASI) refers to systems that can outperform human intelligence [25-27]. AGI and ASI do not currently exist and by acknowledging this fact, it creates a more realistic picture of what AI and specifically ANI can accomplish [28].

For the foreseeable future AI entities will remain unconscious machines that can at best support humans in complex, specific tasks [17]. This implies that ANI systems will be highly specialised and skilled in specific tasks and may even outperform humans in these narrow focus areas [29]. Perhaps instead of waiting for true AGI, multiple narrow AI applications could be interconnected to collaboratively perform tasks in a synergistic manner, possibly with utility beyond what a singly ANI application could do [17]. AI is not yet poised to completely replace humans; however, the improved ability of AI and its increased use in the helping professions suggest we need to investigate more closely the relationship between AI and human interaction. In the highly human-centric context of coaching, the human-AI relationship becomes critical.

Human-AI interactions and relationships in the context of coaching

An indication of the growing importance of human-AI interaction is the emergence of studies augmenting current human-computer interaction (HCI) theory dedicated to human-centered AI (HCAI) [30] and human-AI interaction (HAII) [31]. The focus of AI development seems to be shifting away from pure scientific and academic exploration to useful applications that also consider human factors [30]. Human factors include the creation of AI systems that have social benefits and consider the ethical implications of AI. It also includes the consideration of the role of humans in the AI ecosystem and awareness of the need for a more human-centred approach [32, 33]. The advancement of AI, combined with the focus on placing humans at the centre, have led to new development of AI roles, ranging from being purely assistive to helping with team collaboration [34, 35]. The fast-growing area of AI-assisted decision-making, for example, requires clear boundaries on human versus AI authority and accountability. This is observed in the healthcare industry context, where decisions on patient care and diagnosis can have life or death consequences. As healthcare professionals team up with AI, there is a real danger that the “third wheel” effect (additional, potentially redundant or confusing opinions) may decrease combined human and AI effectiveness [23].

The present study is not focused on augmented human plus AI interaction since the AI coach used operates autonomously from a human coach. However, the AI coach’s sole task is to interact with (coach) a human client. Therefore, the interaction and especially the relationship between the AI and human remain important. There are several suggested ways to create AI coaches that focus on strong human-AI relationships.

Of primary concern is the need for the AI to have social ability, demonstrate credibility and context awareness and be proactive in assisting clients [36]. It is also important that the AI coach strives to embody the aspects that make human coaching effective, including demonstrating trust, empathy, transparency, predictability, reliability, ability, benevolence, and integrity. To create a strong AI-human relationship these aspects can be operationalized as suggested in Table 1 (see Terblanche [4] for a detailed discussion).

Table 1

AI design practices to support strong coach-coachee relationships.

Coach attribute	AI design consideration
Trust	• Avoid the ‘uncanny valley’ effect [37]• Communicate data privacy agreement [38]• Create consistent AI personality [39]• Reduce security and privacy concerns [40]
Empathy	• Use a human name and human-like conversational cues [41]• Remember user’s likes, dislikes and preferences across sessions [40]
Transparency	• Reveal non-humanness [42]• Practice self-disclosure [43]• Showcase purpose and ethical standards [44]
Predictability	• State possible behaviour change due to continuous learning [42]• Find a balance between a predictable personality and sufficient human-like variation [45]• Use conversational context in interactions [46]
Reliability	• Fail gracefully [42]• Monitor chatbot performance and reliability [42]• Provide confirmation messages [40]
Ability	• Use established theoretical models (e.g. goal attainment) [47, 48]• Use personalisation and avoid generic responses [49]
Benevolence	• Communicate positive intent [42]• Demonstrate a positive attitude and mood [40]
Integrity	• Clearly communicate limitations [42]• Clarify purpose in the introductory message [50]

Potential benefits and ethical challenges in AI coaching

The application of AI in the helping professions and in coaching specifically holds numerous potential benefits. In the related field of psychology AI offers new modes of treatment, the ability to reach currently excluded populations, improve patient response and free up limited resources such as highly trained psychologists [51]. These same advantages apply to coaching.

The benefits of coaching are well researched and several meta-studies have shown that coaching can help people with various aspects including: performance and skills; wellbeing; coping; work attitudes; goal-directed self-regulation; improved work/life balance; psychological and social competencies; self-awareness and assertiveness, increased confidence; developing relationships, networks and interpersonal skills; adapting to change more effectively; helping to set and achieve goals; role clarity; and changing behaviors [9, 10, 13]. However, not everyone has access to a coach, especially in less affluent societies. In Africa, for example, the average cost of an organisational coach is approximately 100 USD per session, which puts it out of reach of many [52]. The problem is not only cost. There is a dire shortage of skilled coaches in many parts of the world. Of the more than 40,000 coaches registered with the International Coaching Federation, fewer than 2,000 are in Africa [53]. It seems that currently most of humanity is excluded from the benefits of professional coaching, even though there are calls for coaching to be viewed as a social process that could benefit currently marginalised groups [54]. AI potentially holds the key to expanding the reach of coaching. The ability of AI to scale and provide basic coaching services at a vastly reduced cost could overcome these current limitations, possibly democratizing coaching to the significant benefit of society.

The use of AI in coaching raises ethical concerns. These include prevention of harm, lack of guidance on developing ethical AI, respect and protection of client autonomy, transparency in the use of algorithms, bias, and data ownership [4, 51]. For AI coaching to be widely accepted and trusted, these ethical challenges must be addressed by stakeholders [36].

Goal theory

An important theoretical foundation of this paper is goal theory as applied in coaching. Goal theory is well established and widely used due to a history of empirical research and application. It is in essence an approach explaining the need to establish goals as an intrinsic motivation where a relationship exists between goal difficulty, level of performance, and effort involved [55]. Goal theory is supported by five principles regarding goal setting: clarity (specific and clear); challenge (sufficiently difficult); commitment (buy-in from onset); feedback (regular stock-taking on progress); and complexity (not too complex) [56]. Goals are “internal representations of desired states or outcomes” [57 p388]. Goal setting and attainment have been shown to have a positive effect on workplace performance [56]. Goal attainment has also been linked to positive emotions and increased wellbeing [58-60].

Various factors influence peoples’ goal attainment success. A study by Klein and Fishbach [61] showed that disrupting the expectations of goal attainment may lead to reduced satisfaction and lower goal evaluation, even though the goal is eventually achieved. Other factors that could influence goal attainment include the experience of power whereby people who feel less powerful are less motivated to reach their goals when the goal seems far away [62]. People from cultures where personal honor is important may also delay their goal pursuit if they receive a threat to their moral reputation, such as being called a liar [63].

Certain actions can enhance goal attainment, including the writing down of goals, measuring goals and having specific time frames attached to them, and making a public commitment to someone regarding the goal [55]. There are various types of goals, for example, proximal and distal goals [64].

Goal theory is used extensively in coaching as an underlying mechanism to facilitate self-regulation [64]. During coaching, individuals with the help of their coach set goals, develop and execute action plans, monitor progress and change either goals or action plans based on feedback and progress [65]. Coaching in particular provides the monitoring function that helps to translate goals into actions, which in turn leads to progress [66].

Several empirical studies have shown that coaching is effective in improving goal attainment. In a randomized control trial (RCT) study, Grant et al. [67] found that four coaching sessions over a 10-week period led the intervention group to a significantly higher level of goal attainment compared to the control group. Zimmerman and Antoni [68] analyzed 33 coaching dyads using longitudinal multilevel analyses and found that clients experienced increased goal attainment. Losch et al. [69] compared individual coaching, self-coaching and group training and found that individual coaching was effective and superior in helping leaders achieve their goals.

As goal theory is intrinsic to coaching and since coaching has been shown to improve goal attainment, it emerged as an appropriate theory to investigate an impact of AI coaching on goal attainment in the present study.

Coach maturity

With the coaching industry growing rapidly, a diverse range of people are attracted to become coaches for reasons ranging from the promise of increased freedom, balanced lifestyle, self-control and a reprieve from corporate politics, bureaucracy and pressure [70]. Coaching is an unregulated industry, which implies that coaches enter the profession with various levels of training and experience [71], ranging from no training at all to post-graduate degrees [72]. The result is that coaches practice at different levels of coach maturity [73]. The notion of coach maturity is an important consideration given that AI has taken over the jobs of some people [74], suggesting that humans coaches who operate at a low level of complexity may be rivalled by AI coaching.

Megginson and Clutterbuck [73] distinguished among four levels of coach maturity. At the lowest level coaches follow a “models-based” approach where they are typically more interested in following a set, mechanistic process rather than exploring the complexities of the client’s world. They are “doing coaching to the client”. This type of coaching is typical of novice coaches who rely on the coaching skills and techniques they had been trained in initially [72]. On the second level, “process-based” coaches follow a slightly more flexible approach using an expanded but still limited set of tools and techniques. They are “doing coaching with the client”. On the third level, “philosophy-based” coaches apply a broader mind-set to the client’s situation and practice reflection before and after coaching sessions. The top level, “systemic eclectic” is acquired through much experience and allows a coach to exhibit a sensitive, intelligent approach to the client situation and utilize the most appropriate approach give the context. They are part of the system in which the coaching occurs. This coach maturity model was summarized by Drake [75 p143]: “…as novices they learn the rules, as intermediates they break the rules, as masters they change the rules and as artisans they transcend the rules”.

This coach maturity model is testimony to how humans can integrate knowledge and apply learning across domains, allowing navigation of complex situations. While AI is currently incapable of this, the fact that ANI can perform specific tasks on a level of human competency and beyond [29] suggests that the lowest level of coach maturity (models-based) is potentially within the ability of a well-designed narrow AI system.

Methodology

The two studies we compare were both longitudinal RCT designs. The studies were designed with the CONSORT guidelines for RCT research in mind and these guidelines were adhered to as far as possible [76]. Both studies were conducted over ten months with different groups of participants. Study 1 ran from October 2017 to July 2018 and consisted of a control group and a human coach group where participants received coaching from a human coach. Study 2 ran from November 2019 to August 2020 and consisted of a control group and an intervention group where participants received coaching from an AI chatbot coach. The second study was conducted after the first one because the AI coach was only created in 2019 after the completion of the first study. The same data collection instruments were used in both studies.

The research was approved by the ethics committee of a London-based University, project reference UREC/19.1.5.6. Written informed consent was obtained from all participants in both studies as per the requirements of the ethics approval.

Participants

Participants in both studies were recruited via email from a business school in the United Kingdom. Their fields of study included business management, economics, marketing, tourism, events management and logistics. Participants in Study 1 were randomly allocated into two different groups: Control 1 (n = 105) and Human coaching (105). For Study 2, participants were allocated into Control 2 (n = 134) and AI coaching (n = 134) groups. In total over the two studies, 327 participants successfully submitted data over all eight time-points which were used for the data analysis. See Table 2 for group numbers, demographic distribution, and mean scores of the dependent variable used in this study.

Table 2

Goal attainment means of the four participant groups across the eight time-points.

	Control 1	Human coaching	Control 2	AI coaching
n	85	94	73	75
Gender	77% female	64% female	60% female	54% female
Age	23.04 (18–53)	22.56 (18–51)	23. 81 (18–46)	21.57 (18–48)
Goal attainment 1	25.95 (12.50)	27.66 (11.65)	27.22 (12.78)	26.06 (12.17)
Goal attainment 2	27.68 (14.54)	27.03 (11.22)	27.03 (12.30)	24.89 (12.19)
Goal attainment 3	29.31 (14.59)	31.62 (11.56)	30.53 (13.25)	30.05 (11.31)
Goal attainment 4	31.26 (15.42)	34.64 (13.16)	32.56 (14.60)	33.04 (13.58)
Goal attainment 5	32.25 (14.93)	35.93 (13.78)	31.35 (14.68)	34.10 (12.46)
Goal attainment 6	33 (14.95)	37.32 (13.71)	32.30 (15.27)	35.64 (16.15)
Goal attainment 7	33.47 (14.79)	41.29 (14.58)	34.59 (15.04)	39.36 (17.18)
Goal attainment 8	37.74 (17.49)	41.15 (15.43)	35.01 (16.04)	41.11 (17.25)

Note. The table shows number of participants, distribution of females, mean age (min-max), mean scores of goal attainment and (standard deviation within brackets).

Capturing the placebo effect in research requires subjects to not be aware whether they receive treatment or not [77]. In these two studies we were unable to investigate the placebo effect as coached students were given access to a human coach or the AI coach (Vici) almost immediately after the start of the experiment. The students in the control group were also aware of not having access to human or AI coaching. To address outcomes expectancy [78], the control groups received a fact sheet that provided information about goal attainment, psychological wellbeing, resilience and perceived stress at the start of the trails. They were also asked to think of and specify goals they wanted to achieve over the next ten months. In RCT studies there is a danger of a nocebo effect where participants in the control group have negative outcome expectations because they are aware of not receiving the intervention. In the present study we believe this was managed based on the research of Colloca et al. [79], who had found that a higher number of exposures to trial conditioning correlated to longer duration of nocebo responses. In our study, control group participants were only “conditioned” (made aware of not receiving coaching and given information sheets) at the start of the trials. The relatively long duration of the trials (10 months) likely also helped to diminish the nocebo effect.

Procedure

Participants conducted a survey over eight time-points using an online survey platform. The first survey was a baseline survey before the participants had been allocated to any of the conditions. The baseline survey collected demographic data, dependent variables and the participants were asked to specify two goals that they aimed to work on over the phase of the project. These goals were supposed to be something challenging that was either new or something that has been difficult for them to achieve in the past. After the baseline survey had been completed, the participants completed a monthly survey for six months where the participants were asked to rate the success and the difficulty of their two goals. The monthly survey was distributed for Control group 1, Control group 2, and the AI coaching group. The participants of the Human coaching group received their survey after they had conducted their monthly coaching session. The eighth survey was distributed to all the participants three months after the last monthly survey.

Each student had unique login details to their survey, allowing the participants to be reminded of their goals and allowing the administrators of the survey to send out reminders to the participants when submissions were missing.

Human coaching

Students were coached by professional coaches trained and qualified in a relational model of coaching by a UK-based institution. Their qualifications included at least the Ashridge Executive Coach Accreditation and the European Individual Award (EIA) by the European Mentoring & Coaching Council at Senior Practitioner level. The 105 coaches were on average 50 years old and had at least three years of business coaching experience. Coaches and students were matched randomly with each coach assigned to one student only. Students had six one-hour coaching sessions over a period of six months, one session per month. All sessions were conducted via Skype. There was no prescription on the topic or content of the coaching sessions. Coaches and participants had complete freedom to decide how they wanted to use the session, which topics and goals they wanted to set and pursue and which homework tasks between sessions needed to be completed. All participants had to participate in all six coaching sessions to remain part of the study.

AI coaching

Applying the principle of narrow AI to coaching suggests the creation of a form of artificial narrow intelligence (ANI) that can perform one specific coaching task, rather than an attempt to create a machine replica of a human coach. The AI coach used in this study, Coach Vici was based on expert system (ANI) principles using chatbot technology. The sole purpose of the chatbot was to help participants with goal attainment. Expert systems are considered a form of narrow AI and are described as complex software programs based on specialized knowledge, able to provide acceptable solutions to individual problems in a narrow topic area [80, 81]. Chatbots in turn are computer programs that interact with users via natural language either through text, voice, or both [82].

Vici is a custom-developed text-based chatbot deployed on the Telegram instant messaging platform. The chatbot was developed using the Designing AI Coach (DAIC) framework that recommends merging aspects of strong human coaching relationship with chatbot design best practices and using proven, evidence-based coaching theories as foundation [4]. In line with these recommendations, Vici was designed to facilitate goal attainment according to goal theory [55]. Vici had two types of text-based conversations with users. In the first type of conversation, Vici helped users to specify realistic goals by questioning them on the importance, feasibility and impact of their stated goals. Vici then helped users to commit to achievable actions that would help them reach their goals. In the second type of conversation, users would check in with Vici to report on their goal and action progress, reflect on obstacles that prevented them from progressing and changing their actions plans if necessary. These conversations assisted users to monitor the progress of their goals and actions. Vici also helped users to distinguish between proximal (< 6 months) and distal (> 6 months) goals [83]. Vici was available 24/7 to the experimental group and they could use it as often as they wanted, but at least once a month. A detailed analysis of the AI coach usages is presented in the Results. Fig 1 shows examples of interactions with Coach Vici.

Fig 1

Measures

Goal attainment

Grant et al. [67] developed a goal attainment measure which was adapted for the purpose of this study. The goal attainment measure contained self-reported scores of how successful the participants perceived they had been in achieving their goals and how difficult they perceived their goals. The successful score was measured on 11 points, where each point represented 0%, 10%, 20%, etc. up to 100%. The difficulty score was measured using a seven-point Likert scale ranging from ‘Very easy’ to ‘Very difficult’.

The overall goal attainment score was then calculated by multiplying the success and the difficulty score for each goal separately and dividing the scores for the two different goals to create an average goal attainment score.

Results

To assess the implications of coaching on goal attainment, a 4x8 Mixed Factorial ANOVA was conducted using the four different groups (Control 1, Control 2, Human coach group and AI coach group) as grouping variable and their eight self-reported measures of goal attainment as dependent variables. A power-analysis using G*Power 3.1.9.7 [84] was conducted to determine the effect size required to identify a statistically significant interaction between four groups over eight time-points. A Mixed Factorial ANOVA with a within-between interaction of 327 participants, a power of 0.95 and alpha level of 0.05 indicated that effect of the coaching intervention would have to be above η = .014 to identify a significant interaction.

The Mixed Factorial ANOVA indicated a statistically significant interaction of group and time, f (13.18, 1296.36) = 2.35, p = .004, ηρ2 = .023. To break down this interaction, the development of goal attainment was first analysed within each group using separate Repeated Measures ANOVA over the eight time-points. The Repeated Measures ANOVA indicated that all groups had a significant development of Goal Attainment over time. Both control groups behaved similar, and the trend of the development remained similar over the two time-points. The first control group reported and effect size of ηρ2 = .16, p < .001 and the second control group showed an effect size of ηρ2 = .11, p < .001. Both control groups significantly developed their goal attainment from baseline to time-point 4 (p = .005 and p = .003 respectively), but kept it at a stable level for the remaining time-points.

Furthermore, the effects size over the eight time-points of the two experiment groups were remarkably higher compared to those of the control groups (Human coach group = ηρ2 = .265, p < .001 and AI coach group = ηρ2 = .269, p < .001). The first significant development of goal attainment was identified at time-point 4 for both experiment groups (p = .01 and p = .004 respectively), but the development kept significantly increasing over time-point 7 and time-point 8 for both experiment groups.

Bonferroni corrected multiple comparisons were used to test the difference between each of the four groups at each time-point. The tests indicated that a significant difference between the Human coach and Control group 2 could be identified at time-point 5 where the human experiment group had a significant higher goal attainment score (p = .047) than Control group 2. This effect increased with time and both the Human coach group, and the AI coach group showed significant effects compared to those of the control groups at time-point 7 (Human coach group–Control 1, p = .002; Human coach group–Control 2, p = .008; AI coach group–Control 1, = .022; AI coach group–Control 2 = .048). However, the goal attainment of the two experiment groups never significantly differed between each other throughout the experiment.

The effect size of ηρ2 = .023 which was higher than the critical effect size (> η = .014) according to the power analysis indicated that the sample size for this study was sufficient. These findings further indicate that the participants successfully increased their goal attainment over the time of the study. As shown in Fig 2, receiving coaching had a positive impact on the development of goal attainment for the participants, but both formats of coaching appeared to have had very similar effect.

Fig 2

We also analyzed the usage of the AI coach in terms of usage frequency of the chatbot to identify any potential within-group differences in Study 2. We were able to identify a significant difference in development of goal attainment when splitting the frequency of usage into two equal groups based on their median usage (6 AI coaching sessions, t (73) = -2.24, p = 0.028, d = 0.52. The lower usage group had an average increase on goal attainment of 17.62 (sd = 32.50) compared to 37.62 (sd = 34.16) in the higher usage group. This suggests that more frequent use of the AI coach led to higher goal attainment.

To understand the nature of goals across the two studies and four groups, two of the authors independently analysed the first goal in both studies to assess the theme of the goal, the type of outcome (concrete or vague goal) and whether the goal was proximal (<6 months) or distal (>6 months). The inter-rater reliability of the analysis indicated a very high similarity between the reviewers on all three categories, with Cohen’s kappa of κ = .95, p < .001 for the theme of the goal, κ = .94 for the outcome and κ = .91, p < .001 for the timeline of the goal.

The themes of the goals that were identified related to the participants’ studies (38%), self-development (22%), career (18%), health and wellbeing (16%), other (2%), finances (2%) family (1%) and property or car (1%). Most of the goals were concrete and measurable (60%), for example, “To gain overall mark of 75% in study year 1” and 58% of the goals were long-term focused (>6 months). Furthermore, the proportion of themes, type or proximal within the four different groups did not significantly differ among each other.

The type of outcome and the proximity of the goals were added as covariates into the model, but neither had a significant impact on the development of goal attainment ((outcome, f (7, 2037) = 1.46, p > .05, η = .005) and proximity, f (7, 2037) = 2.06, p > .05, η = .007)). The themes were analysed separately due to the large variety of the themes, but no significant differences between any of the themes were found on the development of goal attainment, f (7, 334) = .80, p > .05, η = .017). These findings indicate that the individual differences in the goals among the participants did not impact the development of goal attainment over time in this study.

Discussion

This paper investigates the performance of an AI chatbot coach relative to human coaches in terms of client goal attainment by comparing two longitudinal RCT studies, the second being a replication study of the first. In both studies the experimental groups who had received either human coaching (Study 1) or AI coaching (Study 2) had significantly higher goal attainment than the control groups. A surprising result is that the AI coach rivalled the human coaches in participant goal attainment with a similar outcome at the end of the study after ten months (Human coach group = ηρ2 = .265, p < .001 and AI coach group = ηρ2 = .269, p < .001). Using goal theory, we attempt to explain this result and we discuss three important implications: (i) the possibility of democratizing coaching; (ii) the way AI coaching could enlarge the need for human coaches; and (iii) a warning to coaches to enhance their coaching praxis.

Goal theory states that there is a higher level of goal attainment if goals are clear, there is buy-in from the onset, regular feedback is provided on progress, and the goals are sufficiently challenging and not too complex. Practically this translates to writing down goals, having specific time frames associated with a goal, measuring progress and making a commitment to someone about completing the goal [55]. Most coaching approaches have at their core the notion of goal attainment. In fact, this is what sets coaching apart from other helping professions [64] and explains why human coaches were able to help participants increase their goal attainment. Goal theory and how it is typically practised by human coaches was used to create the AI coach, Vici. The AI coach helped participants write down their goals by typing it into the application, asked questions to test for the feasibility and level of realism of the goals, and went even further by helping participants create an action plan to reach their goals. Human coaches would be able to engage in a more complex and nuanced discussion about goals and one would therefore expect human coaches to outperform the AI coach who employs a scripted conversation. Although the AI coach lacked nuanced intelligence, it had an advantage over human coaches due to the rigorous and consistent way it executed goal theory. The human coach could decide which aspects of goal theory they implemented in each session and in fact, some of the coaches may have not been well versed in goal theory. For example, in the human coaching sessions the coaches may occasionally have forgotten to ask about goal progress or potentially did not keep an explicit record of goal progress as it is up to each coach to decide how to manage the coaching intervention. The AI coach, however, was programmed to always enquire about goal progress and keep a record for reference to share with the participant. Furthermore, human coaches may have asked participants to verbalize their goals instead of writing them down, whereas the AI coach required the participants to type their goals into the application as writing down a goal has been shown to increase goal attainment [55]. It seems therefore that the rigor and mechanistic execution of goal theory by the AI coach and its inability to deviate from a set process (which could in fact detract human coaches) compensated for its lack of human intelligence.

The results (Fig 2) show that the AI coach trailed the human coaches slightly throughout the eight measurements up to the last time-point (T8). Measures were taken monthly between T1 and T7 with a final follow-up measure (T8) after three months. Between T7 and T8, human coach participants did not receive any more coaching, whereas the AI coaching participants were free to keep using the AI coach. This could explain why goal attainment of human coaching participants declined between T7 and T8, but kept increasing for the AI coach group to ultimately equal the human coach group. The convenience and constant availability of the AI coach probably assisted in its performance relative to human coaches.

An important predictor of coaching success is the readiness of the coachee [85]. Due to the randomised nature of this study, we can assume that in both studies participants were equally open and ready for coaching. Being perceptive to coaching relates to a person’s state at a particular time of day such as their energy levels, mental alertness, and general physical state. In the human coaching group, sessions were scheduled in advance and because two people are involved in the logistics, one can assume that at times appointments were honoured despite the coach or participant not being in an optimal state for the engagement, potentially negatively affecting the efficacy of that session. In the AI coaching group, the participants had complete freedom to decide when to have a conversation with the chatbot, which may have contributed to a more optimal engagement. While we did not explicitly measure these variables, we suggest that this convenience factor may have helped the AI coach to perform well compared to human coaches. Additionally, the AI coach was available 24/7 and participants could use it as often as they chose.

The results indicate that participants who used the AI coach more often had higher goal attainment. Human coaching is expensive and therefore participants in Study 1 only had one session per month. There was no extra cost associated with additional AI coaching sessions. This underscores two of the main advantages of AI coaching–its scalability and cost effectiveness compared to human coaching. The superior availability and use of the AI coach compared to the human coaches could therefore also explain why the AI coaching group performed so well relative to the human coaching group.

The implications of these results are three-fold. Firstly and most importantly, this presents the possibility of democratization of coaching. A number of coaching efficacy meta-studies have shown coaching to be effective in helping people develop, grow and achieve their goals. Coaching is however reserved for a select few due to its cost and the availability of coaches, especially on low-income geographies such as Africa. Even in organizations, individual coaching is usually reserved for the managers and senior leaders. The results from this comparative study suggest that AI coaching, when implemented to have a specific focus in line with the current capabilities of narrow AI, is an affordable and scalable alternative to certain aspects of human coaching. The benefits of coaching could therefore be made available to many more people than is currently the case.

The second implication relates to the coaching industry. Many people are concerned that AI threatens their job security [86]. Coaches may therefore rightly be concerned that AI coaches, such as Vici, pose a threat to their livelihood. The opposite may in fact be true. If AI can help democratize coaching, more first-time users of coaching services would be exposed to the benefits of coaching. Due to the limited abilities of AI, at some point users of AI coaching services may have the need for more advanced and intelligent human coaching. We believe this broadening awareness of and exposure to coaching through AI could in fact create more opportunities for human coaches. Human coaches should view AI coaching as an opportunity, not a threat, in line with the findings of a recent study [74].

The final implication relates to coaches and their praxis. The efficacy of the AI coach in this study suggests that coaches who operate at a low level of coach maturity [73, 75] could be replaced by AI coaches. Therefore, human coaches need to evaluate their coach maturity and invest the necessary resources to improve their coaching knowledge and skills to ensure that they offer their clients a valuable and relevant service. Humans currently and for the foreseeable future will outperform AI in terms of context awareness, transference of learning and higher order complex sense-making [17]. Coaches should ensure that they embody these uniquely human forms of intelligence in their coaching praxis.

Limitations and future research

Participants in both studies were undergraduate students. This implies that the results may not generalise to other populations; however, the effects observed are still valid given that similar groups of participants were used in both studies. Measurements were performed by means of self-scores by participants, which may introduce the possibility of self-score bias. These limitations are offset to some degree by the relatively large sample size and the longitudinal, RCT research design.

In terms of future research, other narrowly focused AI coaches, who specialise in one specific coaching aspect such as wellness, self-awareness or emotional intelligence, should be created and used in a replication study similar to this goal-attainment AI coach. This would help us understand what other coaching aspects can be automated. Should some of these other coaching aspects yield positive results in an AI implementation, the possibility of creating a composite AI coach consisting of an amalgamation of these narrow AI capabilities should be researched. While general AI is not yet possible, perhaps the sum of numerous narrow AI coaching capabilities could create a synergetic AI coaching effect.

Conclusion

Uniquely human characteristics such as emotional intelligence and empathy allows human coaches to build bonds with their clients that no AI can currently rival. This comparison study however shows that AI coaches that focus on a narrow aspect of coaching and are based on fundamental, proven theories may very well rival human coaches in that specific coaching aspect. While AI coaches will not out-perform human coaches as a whole any time soon, these specific applications of coaching could democratize coaching and make its benefits available to a much wider audience, while at the same time potentially growing the demand for human coaches through exposing more people to the benefits of coaching.

(XLSX)

Click here for additional data file.

(DOCX)

Click here for additional data file.

13 Dec 2021

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Stefano Triberti, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

3. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: 1. More emphasis should be put on the procedure enacted to counter the placebo effect. By informing the control group of the coaching benefits, we may incur a nocebo effect. How do you account for that?

2. The meeting with the human coach is once per month. However, nothing is said for the chatbot that can be consulted 24/7. Is there a positive correlation between goal attainment and frequency of usage?

3. There is a typo in "An interesting notion it that ..."

Reviewer #2: This manuscript compares two equivalent longitudinal RCT studies that measured the increase in clients' goal attainment as a result of receiving coaching over a 10-month period.

In my opinion, the manuscript has the potential to make a nice contribution to the study of Artificial Intelligence, however, it needs radical implementations and, in this current form, it is not acceptable for publication to PLOSONE Journal.

I reported below some suggestions for future submission:

Despite the introduction include some well-written sections, it could be interesting to understand in which contexts AI is used today.

The authors cited articles that are currently under review. For future submission, I discourage citing articles that it is not yet published (unless the articles are “in press”).

The authors stated that AI will remain unconscious machines that can at best support humans in complex, specific tasks.

However, we know that is expected that AI will participate more and more in decision-making processes. In this sense could be interesting to discuss in the paper the effect of AI on human relationships. For example, this article explores the effect of AI on health decision-making. Triberti, S., Durosini, I., & Pravettoni, G. (2020). A “Third Wheel” Effect in Health Decision Making Involving Artificial Entities: A Psychological Perspective. Frontiers in public health, 8, 117.

The authors describe the Goal theory. It is important to learn more about the use of goal theory in coaching, also presenting studies on its efficacy and clarifying its relationship with AI in coaching.

Additionally, I have several doubts about the experimental design:

- The authors declared that they conducted an RCT. However, it is not explained the criteria and the research protocol of the research design (es. did the authors use the CONSORT checklist?). More information is needed.

- It is not clear why the authors carried out the data collection in different years and why they compared two different control groups. If there is a reason for this, it must be specified in the text.

- It is not clear how the meetings with human coaching were carried out. What topics were covered during the meetings? Did all participants participate in all meetings? What requests were made to the participants after the meeting? It is necessary to better clarify how it was carried out, also using tables or figures.

- The same with the AI ​​chatbot. It is not clear what content is offered. It could be useful to present examples.

- The chatbot was also searchable 24/7 by users. This is a big difference from the experimental group with human coaching. This difference could have an impact on the results.

- It is also useful to report how many times participants have consulted AI coaching (if available) and understand how this can affect the results (e.g., use this variable as a covariate).

- Additionally, people have many different goals. The authors describe very briefly this aspect in the manuscript. I think it would be useful to describe accurately in the paper the goals classification and consider them in the statistical analyses.

Finally, in the discussion, it is important to accurately discuss the practical implication of the results. It would also be useful to underline how human characteristics (e.g., emotional intelligence) make the human figure irreplaceable in the relationship with clients.

For these reasons, despite I consider the study itself an interesting contribution to the study of Artificial Intelligence, the manuscript is not acceptable for publication in its current form.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

12 May 2022

PLOS ONE

Author’s response to reviewers’ comments for:

PONE-D-21-31265: Comparing Artificial Intelligence and Human Coaching Goal Attainment Efficacy

Dear Editor and Reviewers

Thank you kindly for affording us the opportunity to review our paper. We found the comments and suggestions very helpful and hope we have addressed your concerns in full.

This is very important, first-of-kind research for the coaching community – researchers, practitioners and purchasers of coaching services – as it may pave the way for more mainstream use of AI in the coaching. We would therefore be honoured to share this with the world through your respected and high impact journal.

Thank you

# Editor’s comments Response from author

1 Clarity in methodology

RESPONSE: The methodology has been amended to clarify the questions of the reviewers. Please refer to detailed comments below.

2 Depth of theoretical background - importantly, while the background is rich in info for what regards coaching, it is very poor for what regards human-Artificial Intelligence interaction literature

RESPONSE: The literature review was extensively edited to include two new sections on AI and coaching:

Human-AI interactions and relationships in the context of Coaching

Potential benefits and ethical challenges in AI coaching

3 Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming.

RESPONSE:Referencing was changed from APA to Vancouver and other stylistic changes also made to comply with PLOS ONE requirements

4 We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly.

RESPONSE: Anonymised data is included in the submission as Supporting Information files

5 In your revised cover letter, please address the following prompts:

If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

RESPONSE: See #4 above

6 Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well. RESPONSE: Added anonymous reference to a UK University (for blind review)

The research was approved by the ethics committee of a London-based University, project reference UREC/19.1.5.6. Written informed consent was obtained from all participants in both studies as per the requirements of the ethics approval.

# Reviewer #1 comments Response from author

1 More emphasis should be put on the procedure enacted to counter the placebo effect. By informing the control group of the coaching benefits, we may incur a nocebo effect. How do you account for that?

RESPONSE: Addded the following in Methodology

In RCT studies there is danger of a nocebo effect where participants in the control group have negative outcome expectations because they are aware of not receiving the intervention. In the present study we believe this was managed based on the research of Colloca et al. (2010). They found that a higher number of exposures to trial conditioning correlated to longer duration of nocebo responses. In our study control group participants were only “conditioned” (made aware of not receiving coaching and given information sheets) at the start of the trials. The relatively long duration of the trials (10 months) likely also helped to diminish the nocebo effect.

2 The meeting with the human coach is once per month. However, nothing is said for the chatbot that can be consulted 24/7. Is there a positive correlation between goal attainment and frequency of usage?

RESPONSE: Yes there is. Added the following in the Results section

We also analyzed the usage of the AI coach in terms of how many times the AI coaching chatbot was used to identify any potential within-group differences in the second study. We were able to identify a significant difference in development of Goal Attainment when splitting the frequency of usage into two equal groups based on their median usage (6 AI coaching sessions, t (73) = -2.24, p = 0.028, d = 0.52. The lower usage group had an average increase on Goal Attainment of 17.62 (sd = 32.50) compared to 37.62 (sd = 34.16) in the higher usage group. This suggests that more frequent use of the AI coach led to higher goal attainment.

Also added this to the Discussion section:

Additionally, the AI coach was available 24/7 and participants could use it as often as they chose. The results indicate that participants who used the AI coach more often had higher goal attainment. Human coaching is expensive and therefore participants in study 1 only had one session per month. There was no extra cost associated with additional AI coaching sessions. This underscores one of the main advantages of AI coaching – it’s scalability and cost effectiveness compared to human coaching. The superior availability and use of the AI coach compared to the human coaches could therefore also explain why the AI coaching group performed so well relative to the human coaching group.

3 There is a typo in "An interesting notion it that ..."

RESPONSE: This sentence has been reworded

# Reviewer #2 comments Response from author

1 Despite the introduction include some well-written sections, it could be interesting to understand in which contexts AI is used today.

RESPONSE: The literature review was extensively edited to include two new sections on AI:

Human-AI interactions and relationships in the context of Coaching

Potential benefits and ethical challenges in AI coaching

2 The authors cited articles that are currently under review. For future submission, I discourage citing articles that it is not yet published (unless the articles are “in press”).

RESPONSE: Thank you for this remark. These references have been removed.

3 The authors stated that AI will remain unconscious machines that can at best support humans in complex, specific tasks.

However, we know that is expected that AI will participate more and more in decision-making processes. In this sense could be interesting to discuss in the paper the effect of AI on human relationships. For example, this article explores the effect of AI on health decision-making. Triberti, S., Durosini, I., & Pravettoni, G. (2020). A “Third Wheel” Effect in Health Decision Making Involving Artificial Entities: A Psychological Perspective. Frontiers in public health, 8, 117.

RESPONSE: Thank you for the reference to this helpful paper. It was included in the new expanded literature review that also covers various aspects of AI in these two sections:

Human-AI interactions and relationships in the context of Coaching

Potential benefits and ethical challenges in AI coaching

4 The authors describe the Goal theory. It is important to learn more about the use of goal theory in coaching, also presenting studies on its efficacy and clarifying its relationship with AI in coaching.

RESPONSE: More literature added about efficacy studies of coaching and goal attainment and links to AI as follows:

Several empirical studies have shown that coach is effective in goal attainment. In a RCT study Grant et al. (2009) found that four coaching sessions over a 10-week period led the intervention group to a significantly higher level of goal attainment compared to the control group. Zimmerman and Antoni (2018) analyzed 33 coaching dyads using longitudinal multilevel analyses and found that clients experienced increased goal attainment. Losch et al. (2016) compared individual coaching, self-coaching and group training and found that individual coaching was effective and superior in helping leaders achieve their goals.

We conclude that goal theory is well understood, pragmatic, intrinsic in coaching and that coaching has been shown to improve goal attainment. It is therefore an ideal foundation theory for use in a narrow AI application.

5 The authors declared that they conducted an RCT. However, it is not explained the criteria and the research protocol of the research design (es. did the authors use the CONSORT checklist?). More information is needed.

RESPONSE: Added the following under Methodology

The CONSORT guidelines for RCT research design were followed and adhered to as far as possible (Schulz, 2010).

6 It is not clear why the authors carried out the data collection in different years and why they compared two different control groups. If there is a reason for this, it must be specified in the text.

RESPONSE: Added the following under Methodology:

The second study was conducted after the first study because the AI coach used was only created in 2019 after the completion of the first study.

7 It is not clear how the meetings with human coaching were carried out. What topics were covered during the meetings? Did all participants participate in all meetings? What requests were made to the participants after the meeting? It is necessary to better clarify how it was carried out, also using tables or figures.

RESPONSE: Added more detail under “Human coaching” in the methodology section

Coaches and participants had complete freedom to decide how they wanted to use the session, which topics and goals they wanted to set and pursue and which homework tasks between sessions needed to be completed. All participants had to participate in all six coaching sessions to remain part of the study.

8 The same with the AI chatbot. It is not clear what content is offered. It could be useful to present examples.

RESPONSE: Added four screen shots (figure 1) to illustrated Coach Vici’s conversation

9 The chatbot was also searchable 24/7 by users. This is a big difference from the experimental group with human coaching. This difference could have an impact on the results.

RESPONSE: Added the following in the Results section

We also analyzed the usage of the AI coach in terms of how many times the AI coaching chatbot was used to identify any potential within-group differences in the second study. We were able to identify a significant difference in development of Goal Attainment when splitting the frequency of usage into two equal groups based on their median usage (6 AI coaching sessions, t (73) = -2.24, p = 0.028, d = 0.52. The lower usage group had an average increase on Goal Attainment of 17.62 (sd = 32.50) compared to 37.62 (sd = 34.16) in the higher usage group. This suggests that more frequent use of the AI coach led to higher goal attainment.

Also added this to the Discussion section:

Additionally, the AI coach was available 24/7 and participants could use it as often as they chose. The results indicate that participants who used the AI coach more often had higher goal attainment. Human coaching is expensive and therefore participants in study 1 only had one session per month. There was no extra cost associated with additional AI coaching sessions. This underscores one of the main advantages of AI coaching – it’s scalability and cost effectiveness compared to human coaching. The superior availbailty and use of the AI coach compared to the human coaches could therefore also explain why the AI coaching group performed so well relative to the human coaching group.

10 It is also useful to report how many times participants have consulted AI coaching (if available) and understand how this can affect the results (e.g., use this variable as a covariate). See point above

11 Additionally, people have many different goals. The authors describe very briefly this aspect in the manuscript. I think it would be useful to describe accurately in the paper the goals classification and consider them in the statistical analyses.

RESPONSE: Added more stats regarding goals in the Results section:

To understand the nature of goals across the two studies and four groups, two of the authors independently analysed the first goal in both studies to assess the theme of the goal, the type of outcome (concrete or vague goal) and whether the goal was proximal (<6 months) or distal (>6 months). The inter-rater reliability of the analysis indicated a very high similarity between the reviewers on all three categories, with Cohen’s kappa of κ = .95, p < .001 for the theme of the goal, κ = .94 for the outcome and κ = .91, p <.001 for the timeline of the goal.

The themes of the goals that were identified relate to the participants’ studies (38%), self-development (22%), career related (18%), health and well-being (16%), other (2 %), financial related (2%) family related goals (1%) and property or car related (1%). Most of the goals were concrete and measurable (60%), for example, “To gain overall mark of 75% in study year 1” and 58% of the goals were long term focused (> 6 months). Furthermore, the proportion of themes, type or proximal within the four different groups did not significantly differ between each other.

The type of outcome and the proximity of the goals were added as covariates into the model but neither had a significant impact on the development of goal attainment ((outcome, f (7, 2037) = 1.46, p > .05, ηp2 = .005) and proximity, f (7, 2037) = 2.06, p > .05, ηp2 = .007)). The themes were analysed separately due to large variety of the themes but no significant differences between any of the themes were found on the development of goal attainment, f (7, 334) = .80, p > .05, ηp2 = .017). These findings indicate that the individual differences in the goals among the participants did not impact the development of goal attainment over time in this study.

12 Finally, in the discussion, it is important to accurately discuss the practical implication of the results. It would also be useful to underline how human characteristics (e.g., emotional intelligence) make the human figure irreplaceable in the relationship with clients.

RESPONSE: Changed the Conclusion section to include this argument. However I want to emphasise that AI coaching can still be effective even without this human connection. This I believe is the novel contribution of this research study:

Uniquely human characteristics such as emotional intelligence and empathy allows human coaches to build bonds with their clients that no AI can currently rival. This comparison study however shows that AI coaches that focus on a narrow aspect of coaching and are based on fundamental, proven theories may very well rival human coaches in that specific coaching aspect. While AI coaches will not out-perform human coaches as a whole any time soon, these specific applications of coaching could democratize coaching and make its benefits available to a much wider audience, while at the same time potentially growing the demand for human coaches through exposing more people to the benefits of coaching.

Submitted filename: Comments from reviewers and author response.doc

Click here for additional data file.

8 Jun 2022

Comparing Artificial Intelligence and Human Coaching Goal Attainment Efficacy

PONE-D-21-31265R1

Dear Dr. Terblanche,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Stefano Triberti, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

10 Jun 2022

PONE-D-21-31265R1

Comparing artificial intelligence and human coaching goal attainment efficacy

Dear Dr. Terblanche:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Stefano Triberti

Academic Editor

PLOS ONE