Introducing the QACP: development and preliminary validation of an instrument to measure psychotherapist's core competencies.

The movement towards the conceptualization, description and evaluation of psychotherapists' competencies has been widely developed in the last years and has relevant implications for psychotherapy, training, and continuous education. In Italy, this movement has been supported by the Committee for Psychotherapists' Competencies established in 2010 by FIAP (Italian Federation of Psychotherapy's Associations) and CNSP (National Association of Psychotherapy's Training Institutes) and has involved more than 1000 psychotherapists from different approaches, by means of conferences, expert meetings, workshops, and focus groups. One of the outcomes of this process has been the development of a new self-assessment tool for core competencies (i.e., those that are shared by therapists from all modalities): the QACP (Questionario per l'Autovalutazione delle Competenze dello Psicoterapeuta). The present study aims to present the process of development and the preliminary proofs of the validity of this tool. Construct and knowngroup validity of the questionnaire were examined, and reliability was estimated by computing the internal consistency reliability coefficients for both the overall and the subscale scores. Overall, the instrument showed satisfactory psychometric characteristics. The limitations of the study and the results are discussed and directions for further research are proposed.

Introduction

Mental health is widely recognized as a key factor that deeply impacts biological and social health in any cultural context. Protecting and promoting mental health also protects and promotes physical health, social health, and prosperity (Herman & Jane-Llopis, 2012; Jane-Llopis, Barry, Josman, & Patel, 2005; Patel, Swartz, & Cohen, 2005; WHO, 2004). Psychotherapy plays a central role in public and private care for mental disorders and has a positive cost-benefit ratio: of the 91 studies analysed by Jeremy et al. (1999), 90% showed lower utilization of medical care following psychotherapy or other psychoeducational interventions.

Like any other profession, psychotherapy must define its identity and the services it provides. Unfortunately, finding a shared definition of psychotherapy is not easy, and the process of creating a clear identity, both from a scientific and social perspective, is still ongoing, with wide variations in different countries (Lunt, et al., 2001, Lunt, 2008; Van Deurzen, 2001; Waller, 2001; Young, 2011). Against this background, the movement towards the definition of psychotherapists’ competencies seems to be a crucial step, both scientifically and politically relevant. Furthermore, the movement towards competencies in psychotherapy is fostering the dialogue among the different approaches and new ways of conceptualizing professional training and continuous education (Falender & Shafranske, 2012; Fouad & Grus, 2014; Hatcher, et al., 2013; Horn, DeMers, Lightfoot & Webb, 2019; Plakun, Sudak, & Goldberg, 2009; Rief, 2021; Rodolfa & Schaffer, 2019; Taylor & Neimeyer, 2017; Wise & Reuman, 2019). A central question arising from this process is how to assess psychotherapist’s competencies (Barber, Sharpless, Klostermann & McCharty, 2007; Barlow & Brown, 2020; Fairburn & Cooper, 2011; Koddebush & Hermann, 2019; Kuhne, Meister & Maas, 2020; Manring, Beitman & Dewan, 2003).

This paper presents a tool of self-evaluation of the core competencies, the QACP (Questionario di Autovalutazione delle Competenze dello Psicoterapeuta). It is the result of 10 years of dialog and research in the Italian community of psychotherapists, promoted by the FIAP (Italian Federation of Psychotherapy’s Associations) and the CNSP (Association of the Italian Training Institutes of Psychotherapy), and supported by the SIPSIC (Italian Society of Psychotherapy).

Definition of competence

According to Kaslow (2004), ‘competence refers to an individual’s capability and demonstrated ability to understand and do certain tasks in an appropriate and effective manner consistent with the expectations for a person qualified by education and training in a particular profession or specialty thereof’ (p. 775). By definition, professionals are specially educated and trained individuals who belong to a community that shares goals, criteria, methods, and deontology, and who are able to clearly state, express, and show their competencies (Francesetti, 2013). Therefore, each profession must be able to define, educate and offer services within a framework of conceptualized competencies. Competency in clinical psychology practice has been defined by Epstein and Hundert (2002) as the ‘habitual and judicious use of communication, knowledge, technical skill, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served’ (p. 226). Kaslow (2004) refers to ‘competencies’ which she defines as ‘elements of competence’ (p. 775) that can be observed and measured. This approach has attracted the interest of various health care disciplines, resulting in the development of a competency-based movement since the 1990s (Athay & Orth, 1999; Hoge, Paris & Adger 2005; Spencer & Spencer, 1993). This interest has led to the development of competency-based frameworks in several clinical disciplines, including professional psychology (Barnett, Doll, Younggren & Rubin, 2007; BPS, 2005; EACLIPT Task Force on ‘Competences of Clinical Psychologists’, 2019; Hatcher, et al., 2013; Humphreys, Crino & Wilson, 2018; Kaslow, 2004; Rodolfa, et al., 2014; Rubin et al., 2007; Sumerall, Lopez & Ohelert, 2000) and psychotherapy (APA, 2006; Hughes, 2014; Sperry, 2010a, 2010b; Koddebusch & Hermann, 2018; Nelson et al., 2007; Rief, Schramm & Straus, 2021; Roth, Hill & Pilling, 2009).

In Europe, two representative professional bodies have independently developed a comprehensive description of the psychotherapist’s competencies: the European Federation of Psychological Associations (EFPA) and the European Association for Psychotherapy (EAP) (EAP, 2012; 2013; Plantade-Gipch, Van Broeck, Lowet, Karayianni & Karekla, 2020; Young, Szyszkowitz, Oudijk, Schultess & Stabingis, 2013). The two descriptions have similarities and differences, but they are mainly convergent and overlapping (Aherne et al., 2018). In the United States, Sperry (2010a, 2010b) has developed a network that examines the core competencies of various disciplines: psychiatry, psychology, marital and family therapy, counselling and social work. Sperry identifies six core areas of competencies: i) conceptual foundations; ii) the relationship; iii) intervention planning; iv) intervention implementation; v) intervention evaluation; vi) cultural and ethical sensitivity. These core domains of competencies are strongly interrelated and can be broken down into individual components reflecting specific knowledge and skills. For example, the core competence of intervention planning consists of five essential clinical competencies: i) perform a comprehensive diagnostic assessment; ii) develop an accurate diagnosis according to nosographic systems; iii) develop an effective clinical case formulation; iv) develop an effective treatment plan; v) draft an integrative clinical case report. According to Sperry (2010b), the interconnection between the six domains of competencies can be represented through three concentric circles. The construction and maintenance of an authentic therapeutic relationship which provides the context for the development of a working alliance occupies a central position. Surrounding it is the middle circle representing the basic intervention processes of psychotherapy: intervention planning, intervention implementation, and intervention evaluation and termination. The outer circle includes conceptual foundation as well as culturally and ethically sensitive practice, which directly influence the other competencies in the inner circles, providing a contextual framework for their application.

Relevance of competencies for psychotherapy

The APA (2006), the European and Italian bodies (EAGT, 2013; FIAP, 2012; Young, et al., 2013) and many authors (Benjamin, 2001; Halonen et al., 2002; Humphreys et al., 2018; Sumerall et al., 2000) affirm that a cultural shift towards competencies in psychotherapy is necessary and will improve the profession on many levels. They advocate a movement within psychotherapy towards an emphasis on the acquisition and lifelong maintenance of competence as a primary goal, from graduation to retirement. Focusing on psychotherapist’s competencies in education, training, accreditation and credential processes, continuous education and supervision supports the integration of knowledge, working methods, attitudes, research findings and ethical and cultural sensitivities of the professionals, the definition of the common ground and shared language in the profession and professional social identity.

A competency-based approach to education and training improves the relationship between learning goals, curricular design, outcome measures, and program improvement; it allows trainers to better define what graduates are expected to accomplish; it facilitates better alignment between instructional methods and objectives; it ensures breadth of training; it provides a structure to keep pace with current trends; it ensures the competence of graduates; and it improves accountability of training programs (Benjamin, 2001; Halonen et al., 2002; Hatcher, et al., 2013; Kenkel & Peterson, 2009; Simonds, Behrens & Holzbauer, 2017; Sumerall et al., 2000).

Competency-based education provides psychotherapists with a clear definition of the profession that encompasses practice, science education, social relevance and the public interest; increases the likelihood that the profession can continue to self-govern; and provides the public and policy makers with a clear understanding of what a psychotherapist is and does (Benjamin, 2001; Halonen et al., 2002). This orientation creates an educational experience that provides an opportunity for self-assessment and lifelong learning (Kaslow, 2004).

Assessment of competencies

It seems time for psychotherapy to adopt not only a culture of competence, but also a culture of competence assessment (Horn, et al., 2019; Leigh, et al., 2007; Roberts, Borden, & Christiansen, 2005) that builds upon a long history of competence assessment within and outside psychology (APA, 2006; Bourg et al., 1987; Bourg, Bent, McHolland, & Stricker, 1989; Bowden & Masters, 1993; Burke, 1990; Callan, Peterson, & Stricker, 1986; Kaslow, 2004; Kaslow et al., 2004; Lucia & Lepsinger, 1999; Mentkowski & Associates, 2000; Peterson, 2004; Peterson et al, 1992; Peterson, Peterson, Abrams, & Stricker, 1997; Roberts et al., 2005; Rychen & Salganik, 2001; Sumerall, Lopez, & Oehlert, 2000). Assessment of competencies enhances learning, evaluates progress, helps determine the effectiveness of the curriculum and training program, builds the social identity of psychotherapists and protects the public (Kaslow, 2004; Kaslow et al., 2004).

Assessment of competencies is a complex and multifaceted process that should involve a multi-trait, multimethod, and multi-informant process (APA, 2004, 2006; Kaslow, et al., 2007a). Each evaluation method has limitations and strengths (Kaslow et al., 2009), and there is some consensus that ‘no single method is able to provide a comprehensive assessment of all aspects of […] competence’ (Muse & McManus, 2013, p.495). For example, although external evaluation, such as observer-based evaluation, provides an external view of one’s performance by taking the context into account (Falender & Shafranske, 2007; Fletcher & Bailey, 2003; Kaslow et al., 2009), it has several limitations, including the wide heterogeneity of evaluations (Kuhne et al., 2020).

Among the many possible approaches, there are procedures and tools for self-assessment (Kaslow et al., 2009; Mathieson, Barnfield, & Beaumont, 2009; Sburlati & Bennet-Leavy 2014; Ularntinon & Friedbaeg, 2016). Following self-assessment, therapists can identify their competencies limitations, make plans to improve them by engaging in appropriate professional development study, supervision and consultation (Ericsson, 2009), and monitor progress toward their competence goals (Belar et al., 2001; Caverzagie, Shea & Kogan, 2008; Kaslow et al., 2007b; Kaslow et al., 2009; Sburlati, Bennet-Leavy, 2014).

A competence self-assessment tool provides psychotherapists with an opportunity to navigate their professional path and can be particularly useful during training. Even though self-assessed, it can provide an important feedback to both trainees and trainers to orientate their efforts towards the achievement of fundamental training goals (Yan, 2020). In addition, identifying possible deficiencies in the training group, allows trainers to adopt the best strategies in order to compensate for any weaknesses that may arise (Demyan, Abraham & Bui, 2018; Kaslow et al., 2018; Vacha-Haase et al., 2019) including adjusting the training program (Schwartz-Mette, 2009) and, at the same time, allows trainees to become aware of their own difficulties and therefore to be able to focus on areas that need strengthening (Brown & Harris, 2013; Eva & Regehr, 2005; Kaslow et al., 2007b; Panadero, Jonsson & Botella, 2017).

The use of a self-assessment tool is particularly important to avoid possible fears produced by an external judgement on aspects involving trainees’ personal dimensions. It can also acknowledge the achieved and raise awareness of one’s strengths in clinical practice. Another important aspect is that the evaluation of core competencies - common to all modalities - promotes awareness of the common ground of psychotherapy and so the dialogue and mutual respect in the community of psychotherapists.

Given the prevalence of problems of professional competence identified among therapists and trainees (Gaubatz & Vera, 2006; Hunter & Schwartz-Mette, 2021; Theriault & Gazzola, 2010), paired with the ethical obligation of therapists to provide competent care (American Counseling Association, 2014; American Psychological Association, 2010), continuous assessment and evaluation of one’s professional work are greatly needed. Supervision is the main method to address this need. A self-assessment questionnaire on professional competence, which obviously does not replace the indispensable function of supervision, can be a valuable support. It can provide feedback and an opportunity to reflect on one’s professional practice, on possible weaknesses and the need for further training. It is also a support to avoid selfreferentiality, into which a psychotherapist can fall even after years of professional practice (Di Maria & Formica, 2009). Despite the reported advantages, there are some limitations to self-assessment including the low accuracy of self-reported judgments (e.g., Davis et al., 2006) influenced by some evaluation biases (Karpen, 2018; Ruscio, 2007). This raises a number of issues and research findings show mixed conclusions: therapists may be more or less accurate in assessing their level of competence (Dunning, Heath & Suls, 2004; Karpen, 2018; Mathieson et al., 2009; Walfish, McAlister, O’Donnell & Lambert, 2012). It has been reported that less competent therapists overestimated their level of competence when their own ratings were compared to ratings made by objective observers (Brosan, Reynolds & Moore 2008); on the other hand, more competent therapists may underestimate their abilities when compared to supervisors’ ratings (Mc- Manus, Rakovshik, Kennerley, Fennell & Westbrook, 2012). However, other research shows that self-assessment of trainees in clinical psychology is accurate compared to supervisors’ evaluations (Hitzeman, Gonsalvez, Britt & Moses, 2020).

Templates, procedures and tools have been proposed to support self-assessment of psychotherapists’ competencies (Belar et al., 2001). Self-assessment practices are often non-standardized and one way to standardize selfassessment would be to use reliable and valid instruments to assess therapists’ competencies. The vast majority of assessment tools are observer-rated, i.e., the therapist is rated by a supervisor, either in vivo or from a recording, so there is a lack of validated tools for self-assessment (Fletcher & Bailey, 2003).

Another important point is that research has mainly focused on clinical psychology competencies, whereas studies on psychotherapists core competencies are underdeveloped. Therefore, further research seems relevant to develop and validate standardized tools for self-assessment of psychotherapists’ competencies. In light of this, the development and testing of standardized and validated tools for self-assessment seem to be a promising way to explore and measure competencies in psychotherapists’ professional communities and training programs.

Although we are aware of the conflicting evidence mentioned above regarding the efficacy of self-assessment in this area, we believe it is worthwhile to commit ourselves to the provision to the psychotherapy community of a previously non-existent self-assessment instrument that is specific to the assessment of core competencies, independent of psychotherapy modalities, and whose validity and reliability have been adequately tested. The goal, even given the less than unanimous evidence for the usefulness of this type of approach, is not to replace assessment based on direct session observation or assessment conducted by supervisors, but to flank these approaches and, in particular, to provide psychotherapy trainees with an instrument that is easy to use and provides immediate feedback about the perceived acquisition of competences that are fundamental to work in this field. With this goal we developed the QACP: Questionario di Autovalutazione delle Competenze dello Psicoterapeuta (Self-Assessment Questionnaire of Psychotherapist’s Competencies).

The QACP development process

The QACP is the result of a long journey aiming to create a culture of competence in Italy. It was created through a three macro-phases process, referring to precise time frames, each of them has been important step towards a competence-based culture in the Italian community of psychotherapists. The starting point is the year 2011, when FIAP (Italian Federation of Psychotherapy Associations) and CNSP (Association of the Italian Training Institutes of Psychotherapy) created a joint Committee for Competencies of Psychotherapists. As an expression of the two associations, the Committee was simultaneously interested in the processes relevant to psychotherapy training and in promoting a respectful and scientifically fruitful dialogue between the different modalities.

In the first phase (2011-2014), the document on the core competences of the EAP (European Association for Psychotherapy), based on the description of 13 domains of competence, was translated into Italian and modified according to Italian culture and legislation.

The core competence document was discussed in expert meetings, focus groups and workshops and gradually modified according to the process in which a large part of the Italian community was involved. The most significant results were: first, the cross-cultural sharing, the increase of awareness on competencies in psychotherapy and the co- creation of a common language and ground; second, the process has had a positive impact on the political and academic network that supports the two associations and psychotherapy in Italy, promoting dialogue between different groups and strengthening the willingness to work together for common goals.

The second phase (2015-2018) was focused on the development of the description of the specific competencies for different approaches. The aim was, therefore, to build several taylor-made documents representing the different modalities. Seven working groups were formed for the different areas: Body-oriented, Cognitivist, Gestalt, Integrative, Psychodynamic, Rogersian, and Systemic-family therapy.

A constructive dialogue, through expert meetings and intra- and inter-paradigmatic focus groups, enabled the production of seven documents, each representative - in epistemological, theoretical and clinical terms - of the different paradigms. Each document maintains the structural basis of the core competencies document: each approach added its specific competencies to the thirteen domains of the core competencies already validated in the first phase.

The process of creating the new documents can thus be described both in linear terms, maintaining in this phase a structural continuity with the previous document, and as an extension through the conscious appropriation and description of the differences and definition of the specific identity of the different approaches.

The third phase (2018-2020), aimed to create a tool for the self-evaluation of core competencies that could be used by both psychotherapists and trainees enrolled in the training programs. After reviewing the relevant literature, the Committee combined the description of the 13 competence’s domains created in the first phase and the model created by Len Sperry. The two models were compared and integrated, and the domains were reduced to five basic areas of competence.

The whole process has deeply fostered the culture of competencies in the Italian Psychotherapy’s community and produced a common document on core competencies (2012), seven documents on specific competencies (2018) and a self-evaluation questionnaire on core competencies (2020).

Study goals

The goals of the present study are: i) to develop a new instrument for the self-assessment of psychotherapists’ competencies; ii) to study the psychometric characteristics and validity of the instrument, administered to a sample of psychotherapists and psychotherapy trainees.

Materials and methods

Participants

A total of 778 individuals participated in the research, of whom 540 trainees (69.4%) and 238 psychotherapists (30.6%). 81.7% of the sample were females, representing 74.3% of the therapists and 84.2% of the trainees respectively. Among the therapists, 57.6% declared work experience of 5 years or more. With respect to trainees, they are fairly balanced by year of course, with percentages of 28.2%, 23.3%, 24.8%, and 23.7% for the first, second, third, and fourth year of the course, respectively.

Procedure

Following the development of the QACP, we invited psychotherapists from different modalities to review the items to evaluate their suitability in measuring core psychotherapy competencies. Then, in order to further study the face validity of the instrument we invited psychotherapy trainees and therapists from different modalities to fill in the questionnaire and provide feedback on their perceptions of the instrument and its adequacy to measure competencies.

Finally, to investigate the psychometric characteristics of the instrument and to provide evidences of its validity to measure psychotherapy core competencies we recruited a larger sample of therapists and psychotherapy trainees.

Participants were recruited by inviting psychotherapy associations and training institutes belonging to FIAP and CNSP to invite their members, trainees, and trainers to participate in the study. More than 160 psychotherapy training institutes and 23 national associations were invited to share the invitation among their members. The invitation letter included an information sheet describing the aims and procedures of the research project and a weblink. Through the link, participants were able to access the informed consent form. Participants who signed up for the research were then directed to the online questionnaire. No information was collected about the origin of the research participants (i.e., which school they received the invitation from), as indicating affiliation with a training institute would have compromised the perception of anonymity of the research and would jeopardised the willingness of individuals to participate in the research.

The research was conducted in accordance with the ethical code of the Italian Association of Psychology and the 1964 Declaration of Helsinki and its subsequent amendments. All participants were informed of the nature and purpose of the study. They were assured of the confidentiality of the data and were informed that participation in the study was voluntary, that participants were not remunerated, and that no particular risks to participants were anticipated. Individual consent to participate was obtained before data collection.

Instruments

Description of the instrument: subscales and items

The QACP assesses the core competencies of psychotherapists, both trainees and professionals. It is an instrument for self-assessment of the competencies that are common to all the psychotherapeutic approaches, i.e., the core competencies.

As mentioned above, it is based on the document of core competencies developed by FIAP - CNSP (www.fiap.it) and on Sperry’s model of competencies (2010b).

The questionnaire is structured in 5 areas resulting from the integration of the two contributions: i) assessment and case formulation; ii) therapeutic relationship; iii) implementation of the intervention; iv) evaluation and conclusion of therapy; v) ethics and cultural sensitivity.

These 5 areas are divided into 15 subscales with a total of 63 items. Each subscale contains some items related to relevant competencies. Below we report the list of the assessed competencies:

Assessment and case formulation

Diagnostic assessment and competence to identify client’s problem. Highlighted aspects:

Competence in collecting and describing the onset of client’s problem, its development, its severity and relational/ existential background.

Competence in assessing the risk of self-harm, suicide and violent acts.

Competence in recognising the need for other professional consultations, for the involvement of other services and the competence to activate a network.

Competence in making a diagnosis according to a nosographic system (DSM or ICD or PDM).

Diagnostic assessment and competencies in making hypotheses about client’s problem. Highlighted aspects:

Competence in identifying the biographical pathways allowing to make sense of client’s discomfort.

Competence in evaluating the factors that caused client’s discomfort.

Competence in assessing the predisposing factors: biological vulnerabilities, traumatic events, family’s relationships, transgenerational and existential backgrounds.

Competence in identifying and assessing both environmental and personal factors that reinforce client’s distress.

Competence in identifying the implicit client’s needs. Cultural assessment. Highlighted aspects:

Competence in considering cultural and social resources and barriers.

Identification of therapeutic goals. Highlighted aspects:

Competence in identifying therapeutic goals.

Competence in identifying the focus and specific therapeutic interventions.

Awareness of potential obstacles and possible ruptures that might be encountered during therapy.

Competence in case formulation. Highlighted aspects:

Competence in gathering all the assessment elements in a coherent whole, in order to create a therapeutic plan.

Competence to collect all the meaningful elements in order to identify therapeutic goals, the place and the network of therapy, the focus and specific interventions. the conclusion of therapy.

Therapeutic relationship

Competence in supporting the therapeutic relationship. Highlighted aspects:

Competence in expressing genuine interest and empathy towards the client.

Competence in recognising client’s and therapy’s progresses.

Competence in accepting the difficulties of the client and deal with them constructively.

Competence in adjusting the interventions according to client’s progresses.

Competence in attuning to the different intellectual, emotional, bodily and behavioural levels of expression of the client in order to create a climate of acceptance and understanding.

Empathic interest and emotional and bodily tuning towards the client as fundamental elements of the therapeutic bond.

Definition of the contract and of the boundaries of the therapeutic relationship in order to preserve it from possible ruptures. Highlighted aspects:

Competencies in co-creating a clear and defined therapeutic contract with the client as part of the treatment.

Competence in detailing the therapeutic contract, in terms of the duration of therapy as a whole, duration of individual sessions, their frequency and the financial issues.

Competence in explaining the methodology of psychotherapeutic intervention in a way that is understandable to the client.

Competence in maintaining the boundaries of the setting in extra-session contacts, such as emails, letters, text messages, phone calls, visits and unexpected meetings.

Competence in recognizing and repairing relationship’s ruptures. Highlighted aspects:

Competence in managing those situations in which the client feels anxiety or discomfort towards diversity and alterity.

Competence in managing those moments when the client loses confidence in the therapy and expresses the intention to make contact with other professionals.

Competence in managing interventions that may create confusion, embarrassment, shame, anxiety or offence.

Implementation of therapeutic interventions

Knowledge of techniques and working methods of the reference model. Highlighted aspects:

Competence in self-assessing of knowledge and mastery in using techniques and working methods and awareness of their specific usefulness.

Competence in recognising when a particular technique or working method is useful and appropriate.

Competence in evaluating the limits and the effectiveness of a particular technique or working method and the competence to adjust it to a specific client.

Competence in paying attention to client’s experience regarding the application of a technique or working method and competence to orient and shape the intervention on the basis of this experience.

Maintaining a focus during therapy. Highlighted aspects:

Competence in identifying a focus of therapy.

Competence in being flexible enough to modify the focus in itinere in relation to what emerges from the client during the therapeutic process.

To recognise and resolve aspects that interfere with therapy. Highlighted aspects:

Competence in being in contact with emotional resonances emerging from the relationship with the client.

Competence in being able to understand the effect that the interventions - both explicit and implicit - have on the client.

Competence in managing the moments in which the client does not respect the agreed therapeutic contract.

Competence in not considering client’s resistance as an attempt to boycott therapy.

Competence in not giving advice or trying to solve clients’ problems on behalf of them.

Competence in repairing ruptures in the therapeutic alliance.

Assessment and conclusion of the therapeutic process

Monitoring the therapeutic process and prevention of drop out. Highlighted aspects:

Competence in monitoring the progress of the therapeutic process in its various phases, also in order to prevent any early interruptions that could more or less seriously compromise therapy’s effectiveness.

Competence in acknowledging both the progress, with the relative results already obtained, and any emerging criticality, with particular reference to those capable of deteriorating the therapeutic alliance.

Awareness of the role of supervision as an essential support tool in the therapeutic process.

Process of concluding the therapy. Highlighted aspects:

Competence in understanding the relevance of the phase of therapy’s conclusion.

Competence in acknowledging the affective dimension involved in the process of concluding therapy.

Competence in supporting the assimilation of the results achieved and the maintenance of them after the conclusion of the therapy, through their valorisation and through follow-up sessions.

Ethics and cultural sensitivity

Competence in knowing how to operate within an ethical framework. Highlighted aspects:

Competence in knowing and applying the professional, ethical and deontological guidelines that regulate the profession of psychotherapist, respecting the rules of privacy required in the relationship with the client.

Competence in managing the ethical dilemmas which could emerge also in the confrontation with other professional entities.

To operate in the presence of social and cultural differences and to contribute to the development of the discipline. Highlighted aspects:

Competence in recognising and accepting the cultural and social differences between therapist and client.

Competence in being open and keeping up to date with the emerging points of view that animate the scientific and cultural debate in the field of mental health, as well as with the contributions and indications coming from the most current research in the field of psychotherapy.

Plan of analysis

The first step in the validation procedure involved the examination of content validity. Content validity indicates the extent to which the items included in the instrument address important aspects of the construct being measured (Polit & Beck, 2006). The content validity of the instrument was assessed using the following procedure: experienced therapists (with at least 10 years of experience) were invited and given a content validation form. The form required each expert to rate the relevance of each item to the corresponding competency dimension. Responses ranged from 1 (‘The item is not relevant to the dimension being measured’) to 4 (‘The item is highly relevant to the dimension being measured’). A content validation index (CVI-I) was then calculated for each item by counting the number of experts who rated the item 3 or 4 and then dividing by the total number of experts involved. A total scale CVI (S-CVI) was then calculated for each dimension by averaging all the I-CVIs included in the dimensions. The CVI should be above 0.8 for the single items and above 0.90 for the scales.

The instrument’s face validity was assessed by asking a sample of therapists and student therapists to evaluate an advanced draft of the questionnaire (i.e. a version of the questionnaire with a different response format and 8 slightly differently worded items), particularly with regard to the clarity and comprehensiveness of the items in measuring therapists’ competencies. The questionnaire was administered to 298 therapists and trainees. In addition, respondents were asked to provide overall feedback in the form of an open-ended response regarding the suitability of the questionnaire for measuring psychotherapists’ competencies.

Finally, to examine the psychometric characteristics of the final version of the instrument, we conducted statistical analyses on the validation sample.

The analyses were conducted using SPSS Statistics 27 and Mplus 8. First, descriptive statistics and Pearson’s correlations were computed for the individual items and subscales of the questionnaire. Then, the reliability of the instrument was estimated using Cronbach’s alpha. A reliability coefficient was computed for each of the subscales included in the instrument, except for the cultural assessment area which was measured using a single item.

To assess the construct validity of the QACP we then examined its factorial structure: to do this, we followed a two-step procedure. Due to the small sample size to item ratio of the instrument, we first performed exploratory factor analyses (EFAs), using the ML extraction method, for the subscales included in the instrument. In particular, EFAs were performed for all dimensions except for the Cultural assessment and Case formulation subscales, which were measured with only 1 and 2 items, respectively. EFA results were considered positive, supporting the expected instrument factor structure, if one factor emerged for each dimension, explaining more than 50% of the variance (Streiner, 1994). To further investigate the factor structure of the instrument, we then conducted three confirmatory factor analyses in which we tested the congruence of the empirical data with the expected factor structures based on theoretical considerations. Three competing models, shown in Figure 1, were tested. All tested models include 5 dimensions, corresponding to the theoretically expected main competencies areas. The first model predicts that these five dimensions are correlated with each other, while the second model predicts the presence of a superordinate factor, which was assumed to influence each specific area of competence. Finally, the third model is a bifactor model, and hypothesizes the presence of one general, non-specific competence factor and several corresponding specific factors. To assess the fit of the confirmatory models we used the global fit indices suggested by Hu and Bentler (1999): root mean square error of approximation (RMSEA), standardized root mean squared residual (SRMR), and the comparative fix index (CFI). We interpreted these indices following the guidelines proposed by Hu and Bentler (1999), according to which the model fit is adequate when CFI >0.95, SRMR <=0.10 indicating meaningful differences; Burnham & Anderson, 2002), with lower values indicating the better fitting model (Kline, 2005).

Figure 1.

QACP factor structure models.

As a further test of the validity of the instrument, we analysed the known-groups validity of the instrument for known groups by conducting t-tests to compare the mean scores obtained by psychotherapists and trainees for each subscale included in the instrument. Consistent with the findings of previous studies that examined the influence of experience on therapist competencies (e.g., Orlinsky & Ronnestad, 2005; Stein & Lambert, 1995; Walsh, Roddy, Scott, Lewis, & Jensen-Doss, 2019), we expected experienced therapists to have higher levels of perceived competencies than trainees. Finally, we focused on the subsample of trainees and conducted independent sample t-test to test for differences in mean scores on the different subscales of the test for trainees attending different course years. In this case, we expect a higher level of competence in trainees who are more advanced in their training than in trainees at the beginning of their training, as already noted by several authors (e.g., Dennhag & Ybrandt, 2013; Evers, Schroder-Pfeifer, Moller, & Taubner, 2022; Messina, et al., 2018).

Results

Content and face validity

Content validity was examined by administering a content validation form to 6 experienced psychotherapists. On the basis of their responses I-CVI and S-CVI scores were computed. All QACP items showed adequate content validity, with I-CVI values always greater than 0.8, ranging from 0.88 to 1. S-CVI scores were also always adequate, ranging from 0.93 to 1. The instrument’s face validity was assessed by asking a sample of therapists and student therapists (N=298) to rate the items of the questionnaire in terms of their clarity and completeness in measuring therapists’ competencies. Only 0.5% of respondents felt the questionnaire was not clear enough and 4.6% felt it was incomplete in terms of the constructs measured. When asked about general feedback on the questionnaire, only 5 respondents gave an answer that included negative evaluations. These evaluations were specifically related to the incompleteness of the instrument (N=2, 0.01%) and the excessive length of the instrument (N=3, 0.01%).

Descriptive statistics and reliability

Preliminarily, we examined the frequency of responses and descriptive statistics for all the items included in the instrument. The results showed that the items performed adequately, with satisfactory distribution of the response categories frequencies. We then calculated the scores for the subscales by averaging the responses for the items included in each subscale. The descriptive statistics and reliability for each subscale are presented in Table 1.

Table 1.

Descriptive statistics for questionnaire subscales.

Subscale	Min	Max	Mean	SD	Skewness	Kurtosys	Cronbach’s α
Diagnostic assessment	0.00	10	6.93	1.62	–0.47	1.07	0.65
Clinical assessment	0.75	10	7.05	1.53	–0.40	0.89	0.86
Cultural assessment	0.00	10	7.64	2.56	–1.10	0.53	-
Treatment assessment	0.00	10	6.30	1.90	–0.15	0.02	0.81
Case formulation	0.00	10	6.64	1.96	–0.34	0.18	0.89
Therapeutic relationship	3.00	10	7.70	1.38	–0.22	–0.42	0.73
Contract and boundaries of the relationship	0.25	10	7.08	1.70	–0.32	0.03	0.90
Repairing ruptures	0.67	10	6.38	1.79	0.16	–0.29	0.86
Intervention implementation	0.67	10	6.72	1.90	–0.34	–0.08	0.69
Focus on the process	0.50	10	7.22	1.54	–0.26	0.52	0.89
Interfering factors	1.67	10	6.80	1.51	0.10	–0.38	0.84
Therapy process monitoring	0.00	10	7.03	1.57	–0.27	0.45	0.80
Therapy conclusion	0.00	10	7.08	1.72	–0.45	0.76	0.77
Ethics	1.43	10	8.17	1.16	–0.71	2.01	0.84
Differences and updating	1.50	10	7.42	1.63	–0.47	0.09	0.83

As can be seen from the table, the subscales have a wide range of variation in terms of potential (0-10) and appear to be normally distributed, with the exception of the cultural assessment subscale (single item), which has a slight negative asymmetry, and the ethics subscale, which is leptokurtic.

The reliability of the instrument was examined by calculating Cronbach’s α values for all subscales included in the questionnaire, except for the cultural assessment, which was measured with a single item. Table 1 shows that each subscale achieved an adequate or more than adequate level of reliability (mean α=0.81, median=0.84). Table 2 shows the correlations between the QACP subscales. As expected, the correlations between the subscales are always present and significant and of moderate to high strength.

Table 2.

Pearson’s correlation between QACP subscales.

	1	2	3	4	5	6	7	8	9	10	11	12	13	14
1. Diagnostic assessment	1
2. Clinical assessment	0.673[**]	1
3. Cultural assessment	0.190[**]	0.186[**]	1
4. Treatment assessment	0.629[**]	0.721[**]	0.208[**]	1
5. Case formulation	0.618[**]	0.673[**]	0.269[**]	0.793[**]	1
6. Therapeutic relationship	0.575[**]	0.713[**]	0.253[**]	0.699[**]	0.626[**]	1
7. Contract and boundaries of the relationship	0.464[**]	0.563[**]	0.249[**]	0.603[**]	0.564[**]	0.643[**]	1
8. Repairing ruptures	0.421[**]	0.534[**]	0.261[**]	0.599[**]	0.598[**]	0.638[**]	0.536[**]	1
9. Intervention implementation	0.573[**]	0.725[**]	0.210[**]	0.763[**]	0.718[**]	0.717[**]	0.659[**]	0.604[**]	1
10. Focus in the process	0.520[**]	0.689[**]	0.247[**]	0.707[**]	0.652[**]	0.731[**]	0.781[**]	0.608[**]	0.805[**]	1
11. Interfering factors	0.446[**]	0.556[**]	0.279[**]	0.570[**]	0.512[**]	0.665[**]	0.583[**]	0.683[**]	0.593[**]	0.652[**]	1
12. Therapy process monitoring	0.585[**]	0.662[**]	0.283[**]	0.688[**]	0.685[**]	0.650[**]	0.615[**]	0.638[**]	0.705[**]	0.725[**]	0.589[**]	1
13. Therapy conclusion	0.555[**]	0.609[**]	0.267[**]	0.689[**]	0.660[**]	0.655[**]	0.594[**]	0.570[**]	0.702[**]	0.676[**]	0.623[**]	0.693[**]	1
14. Ethics	0.511[**]	0.486[**]	0.310[**]	0.480[**]	0.484[**]	0.540[**]	0.559[**]	0.482[**]	0.528[**]	0.509[**]	0.512[**]	0.562[**]	0.506[**]	1
15. Differences and updating	0.445[**]	0.493[**]	0.205[**]	0.497[**]	0.446[**]	0.429[**]	0.415[**]	0.389[**]	0.444[**]	0.448[**]	0.477[**]	0.484[**]	0.494[**]	0.470[**]

*P<0.05; **P<0.01.

Factor structure

As stated in the method section, EFAs were performed for all dimensions except for the cultural assessment and case formulation subscales. EFA results were in line with theoretical expectations. For all dimensions, only one common factor was extracted after Kaiser criterion and scree plot examination. The percentage of variance explained by the extracted factors was always above 50%, ranging from 51.3% for the implementation of the intervention dimension to 76.4% for the ruptures dimension (average: 64.5%). We also checked whether each item loaded significantly with the dimension to which it belonged.

To further investigate the factor structure of the instrument, we then conducted three confirmatory factor analyses (see Figure 1). Specifically, we compared a model with 5-correlated-factors, a second order model with a general superordinate competence factor, and a bifactor model according to which a general therapist competence factor exists together with five specific orthogonal competencies factors. For the 5-correlated factors model overall fit was good, with all goodness of fit criteria indicating adequate fit (χ2=334.46, df=80, CFI=0.95, RMSEA=0.076, SRMR=0.036) and all regression weights statistically significant at P<0.05.

After checking the adequacy of fit of all tested models (see Table 3), we examined the Akaike’s information criterion (AIC) values to compare them: model 3 (bifactor) had the lowest AIC value, with a difference of 46.5 from model 1 and 54.7 from model 2. Thus, the factorial structure of model 3 is most compatible with the data collected.

Table 3.

Fit indices for factor structure models depicted in Figure 1.

Model	Description	Model χ2	df	P value	CFI	RMSEA	SRMR	AIC
1	Five-correlated-factors	334.458	80	<0.001	0.947	0.076	0.036	21563.645
2	Second-order model: superordinate and subordinate factors	352.655	85	<0.001	0.944	0.076	0.038	21571.842
3	Bifactor model: general factor and specific factors	287.970	80	<0.001	0.956	0.069	0.033	21517.157

CFI, comparative fix index; RMSEA, root mean square error of approximation; SRMR, standardized root mean squared residual; AIC, Akaike’s information criterion.

Criterion validity

As a further test of the validity of the instrument, we conducted t-tests for independent samples to check for significant differences between the levels of the subscales and the main dimensions investigated by the QACP. Table 4 shows the results of the tests carried out. All subscales and main dimensions investigated by the instrument differed significantly in the two sub-samples, with the therapist sub-sample always recording higher levels of perceived competence. The only exception was the cultural assessment dimension, where no significant differences emerged. It is important to note that the effect size, calculated as of Cohen’s is medium to large for all subscales (average d=0.79).

Table 4.

Descriptive statistics and independent sample t-tests for therapists and trainees subsamples.

	Therapists		Trainees		t	P	Cohen’s d
	M	SD	M	SD
1. Diagnostic assessment	7.58	1.64	6.73	1.56	4.89	<0.001	0.59
2. Clinical assessment	8.04	1.28	6.74	1.46	7.72	<0.001	0.93
3. Cultural assessment	7.81	2.63	7.58	2.55	0.27	0.79	0.03
4. Treatment assessment	7.53	1.66	5.90	1.80	7.28	<0.001	0.88
5. Case formulation	7.95	1.72	6.21	1.84	7.94	<0.001	0.96
6. Therapeutic relationship	8.56	1.13	7.41	1.34	7.31	<0.001	0.88
7. Contract and boundaries of the relationship	7.90	1.64	6.78	1.63	6.17	<0.001	0.75
8. Repairing ruptures	7.32	1.79	6.04	1.67	5.52	<0.001	0.67
9. Intervention implementation	8.17	1.35	6.21	1.80	9.30	<0.001	1.12
10. Focus in the process	8.21	1.27	6.86	1.47	7.94	<0.001	0.96
11. Interfering factors	7.56	1.45	6.53	1.45	6.05	<0.001	0.73
12. Therapy process monitoring	7.98	1.43	6.69	1.49	7.44	<0.001	0.90
13. Therapy conclusion	8.29	1.36	6.63	1.62	9.02	<0.001	1.09
14. Ethics	8.60	1.00	8.03	1.18	4.25	<0.001	0.51
15. Differences and updating	7.98	1.52	7.23	1.63	3.50	0.001	0.42
Assessment and case formulation	7.69	1.53	6.62	1.37	7.59	<0.001	0.76
Therapeutic relationship	7.93	1.28	6.75	1.30	8.38	<0.001	0.91
Implementation of the intervention	7.98	1.22	6.49	1.41	9.73	<0.001	1.10
Evaluation and conclusion of therapy	8.13	1.33	6.65	1.44	8.93	<0.001	1.05
Ethics and cultural sensitivity	8.29	1.12	7.63	1.19	4.73	<0.001	0.56
Total competence score	7.86	1.42	6.75	1.22	8.72	<0.001	0.87

M, mean; SD, standard deviation.

As a final step of analysis, t-tests were conducted focusing on the trainees’ subgroup. The theoretical expectation is that trainees who are at an advanced stage of their training will show higher scores on the subscales of the QACP. Specifically, the group of first- and second- year pupils was compared to the group of third- and fourthyear pupils (see Table 5). Again, testing revealed significant differences between the two groups, with the group of more advanced trainees perceiving higher average levels of competence in all areas investigated by the instrument, with the exception of the Cultural assessment and Differences and updating subscales. As expected, the differences between the two groups were smaller than those between therapists and trainees, with an average medium effect size (average Cohen’s d=0.52).

Table 5.

Descriptive statistics and independent sample t-tests for I-II and III-IV year trainees subsamples.

	I-II years		III-IV years commercial		t	P	Cohen’s d
	M	SD	M	SD
1. Diagnostic assessment	6.35	1.63	7.13	1.40	–5.18	<0.001	0.51
2. Clinical assessment	6.29	1.51	7.22	1.27	–6.73	<0.001	0.66
3. Cultural assessment	7.43	2.52	7.70	2.61	–1.04	0.300	0.10
4. Treatment assessment	5.35	1.79	6.47	1.64	–6.57	<0.001	0.65
5. Case formulation	5.66	1.93	6.78	1.61	–6.25	<0.001	0.63
6. Therapeutic relationship	7.06	1.32	7.74	1.32	–4.81	<0.001	0.52
7. Contract and boundaries of the relationship	6.36	1.69	7.14	1.49	–4.47	<0.001	0.49
8. Repairing ruptures	5.67	1.54	6.39	1.71	–3.98	<0.001	0.44
9. Intervention implementation	5.35	1.71	7.09	1.44	–9.71	<0.001	1.10
10. Focus in the process	6.41	1.48	7.30	1.32	–5.49	<0.001	0.64
11. Interfering factors	6.32	1.43	6.76	1.44	–2.64	0.009	0.30
12. Therapy process monitoring	6.31	1.53	7.05	1.37	–4.25	<0.001	0.51
13. Therapy conclusion	6.23	1.72	7.00	1.45	–3.97	<0.001	0.49
14. Ethics	7.90	1.31	8.15	1.05	–1.78	0.043	0.21
15. Differences and updating	7.07	1.76	7.38	1.48	–1.65	0.100	0.19
Assessment and case formulation	6.22	1.37	7.06	1.26	–6.49	<0.001	0.64
Therapeutic relationship	6.40	1.27	7.10	1.26	–5.15	<0.001	0.55
Implementation of the intervention	5.96	1.34	7.05	1.26	–7.46	<0.001	0.84
Evaluation and conclusion of therapy	6.27	1.52	7.00	1.28	–4.39	<0.001	0.52
Ethics and cultural sensitivity	7.49	1.29	7.76	1.09	–1.99	0.047	0.23
Total competence score	6.35	1.22	7.19	1.07	–7.42	<0.001	0.73

M, mean; SD, standard deviation.

Discussion

The usefulness of assessing the competencies of psychotherapists and trainees in psychotherapy has been widely recognised in the past (APA, 2006; Kaslow, 2004; Kaslow et al., 2004). For this reason, the availability of a standardised instrument to measure therapists’ core competencies, transversal to the different psychotherapy approaches, may represent a useful advance for trainees and training institutes to monitor and evaluate the achievement of specific training objectives.

In this study, we presented the process of developing and validating a relatively short and easy-to-fill self-report questionnaire to assess the therapeutic competencies required for clinical work. The instrument consists of different sections and assesses 5 core competence areas for the psychotherapist’s work, namely assessment and case formulation, therapeutic relationship, implementation of the intervention, evaluation and conclusion of therapy, ethics and cultural sensitivity.

The instrument has demonstrated adequate face and content validity and appropriate psychometric functioning. This result was not to be taken for granted: in particular, the fact that the instrument shows content validity is in a way an empirical confirmation of the validity of an approach to the conceptualisation of psychotherapists’ competences independent of specific therapeutic modalities, and in line with the theoretical proposals that have emerged in both American and European psychotherapy communities (Aherne et al., 2018; Sperry, 2010a, 2010b).

In order to assess the construct validity of the instrument, exploratory and confirmatory factorial analyses were carried out, from which it emerged that the most appropriate dimensional structure is that which foresees the presence of a general dimension of therapeutic competence to which are added dimensions of specific competencies, corresponding to the five theoretically hypothesised areas of competence, which in turn are coherent with the Sperry (2010a) model of core competencies. The factorial structure that was found to be most consistent with the data, i.e., the one showing the presence of both a general competence dimension and specific therapeutic competencies, is particularly noteworthy because, while it supports the conceptualization of separate competence dimensions and is thus consistent with the work of relevant authors in the field (EAP, 2012, 2013; Plantade- Gipch, et al., 2020), it also indicates that it is useful to consider the acquisition of psychotherapeutic competencies as a unified process. In terms of criterion validity, the instrument was shown to be able to detect the expected differences in competencies both between therapists and psychotherapy trainees and between trainees at different training levels. With respect to these findings, our results are consistent with the literature indicating a positive effect of both experience (Orlinsky & Ronnestad, 2005; Stein & Lambert, 1995; Walsh, Roddy, Scott, Lewis, & Jensen-Doss, 2019) and training (Evers, Schroder-Pfeifer, Moller, & Taubner, 2022; Messina, et al., 2018) on therapeutic competencies, supporting the instrument’s ability to monitor progress in core competencies acquisition.

Conclusions

The availability of a competencies’ self-assessment tool provides psychotherapists and trainees with a further opportunity to monitor their training and professional development. Although in self-assessment mode, the tool provides a different perspective on the progress of the training process, which is in addition to the subjective and unstandardized evaluations of the trainees and those of the trainers or supervisors, providing feedback on the achievement of training goals. Useful in identifying strengths and possible weakness in training. Indeed, the tool allows trainees to become aware of their own difficulties and thus better focus on areas that need to be strengthened. One of the strengths of the instrument is that it is easy to administer and does not use language specific to a particular therapy modality. This favours the possibility of administering the instrument to therapists and students with different theoretical backgrounds, allowing for the development of a possible common ground for comparison between schools. As for the individual it also allows to self-monitor, through repeated administrations, the changes occurred over the years of training and/or professional practice, a practice that is known to increase the likelihood to achieve the expected learning outcomes (Harkin, et al., 2016).

It is also important to recognize the limitations associated with the use of a self-assessment tool: the presence of possible bias in responses experienced by respondents at conscious, or unconscious, level, which could undermine the usefulness of the tool itself, leading respondents to answer in ways that are socially desirable or not entirely honest. We believe that this limitation, while unavoidable, is not detrimental to the usefulness of the instrument for all therapists and trainees who are interested in receiving feedback and increasing their awareness on possible limitations or difficulties in the practice of their profession. In this regard, respondents are supported by the mode of anonymous self-assessment which prevents possible fears of external judgment. The way in which the tool is constructed will also allow recognition of skills achieved and increase awareness of strengths in clinical practice.

Further steps will be taken to validate the tool, in particular to test for concordance between the assessment of core competencies identified with the QACP instrument and those obtained through different (non-self-report) instruments and/or through clinical expert assessment of trainees’ work. Additional investigations may follow this initial study: namely, to examine how trainees from different modalities acquire basic skills and to study the extent to which the development of basic skills (those assessed by the QACP) and specific skills (specific to each modality) are correlated. Specific skills could be assessed by means of existing tools, if available, or that will be developed by the different modalities, to assess trainees’ adherence and competence in applying the specific techniques and ways of working of the model they are learning.

Beside the validation of the QACP, this study has also promoted the growth of the culture on competence in the Italian community of psychotherapists, that ten years of work have produced. According to the mentioned research and guidelines (Kaslow, 2004; Rief, 2021; Roberts et al., 2005; Rubin et al., 2007), this culture improves the quality of the services that psychotherapists offer to their clients, contributing indirectly to the growth of health - social, mental and physical - in our Country. An element that could be even more important in a pandemic moment like the one we live in.