The reliability and validity of the clinical competence evaluation scale in physical therapy.

[Purpose] To examine the internal consistency, criterion-related validity, factorial validity, and content validity of the Clinical Competence Evaluation Scale in Physical Therapy (CEPT). [Subjects] The subjects were 278 novice physical therapy trainees and 119 tutors from 21 medical facilities. [Methods] The trainees self-evaluated their clinical competences and the tutors evaluated trainee competences using the CEPT. Overall trainee autonomy was evaluated using a visual analog scale (VAS) for self-evaluation and the trainees were also evaluated by their tutors. The content validity of the CEPT was examined by asking if the CEPT could evaluate the competence of novice physical therapists on a four-point scale.
[Results] Cronbach's alpha of the CEPT was 0.96 for the trainees and 0.97 for the tutors. The correlation coefficient between the total score of the CEPT and whole competence by VAS was 0.83 for the trainees and 0.87 for the tutors. Factor analysis identified two factors, "the specialty of the physical therapist" and "the essential competence of a health professional". Ninety percent or more of the trainees and the tutors answered that the CEPT could sufficiently evaluate the competence of novice physical therapists.
[Conclusion] The CEPT is a reliable and valid scale for clinical competence evaluation of novice physical therapists.

INTRODUCTION

Recently, actions to improve the quality of health professionals have often been discussed in Japan. In medical or nursing education, many actions including the improvement of postgraduate programs, and postgraduate medical education1), clinical training for newly graduated nursing personnel2), and the development of the core curriculum3) have been carried out. Because of the decline in educational quality due to the rapid increase in the number of undergraduate programs, the change of the needs of clients, the advances in medical treatment, the changes in social background and the diversification of the role of physical therapists, the clinical competence required for physical therapists is increasing4). Taking this into consideration, undergraduate and postgraduate physical therapist educational programs require further enhancement to provide patients with quality health care.

The results of questionnaires about continuing education for physical therapists in medical facilities5) indicate that educational targets have not been established, that the original education system was carried out independently by each facility, that the curriculum contents vary between facilities. Furthermore, the evaluation scales for novice physical therapists used in some facilities were originally developed in separate facilities. A standardized evaluation scale has not been developed. The educational target for medical continuing education is clinical training3). Nursing continuing education has an educational target, educational guidelines and four stages of evaluation that follow the educational target6). The American Physical Therapy Association has an educational target7), evaluation scales for entry level positions8), the postgraduate educational target set as Core Values Professionalism in Physical Therapy9), and the evaluation scales in continuing education10).

Proper evaluation is indispensable for conducting systematic education11). For novices, the initial years of practice are the time for the continued development of professional identity, knowledge base, clinical reasoning, and decision-making skills12). We investigated an evaluation scale for the continuing education for novice physical therapists. In a previous study13), we developed the Clinical Competence Evaluation Scale in Physical Therapy (CEPT) and confirmed its intra-rater reliabilities. For the development of the CEPT, we reported the qualitative study of the primary goals of continuing education for physical therapists14) by content analysis of semi-structured interviews with 15 physical therapists that had experience in staff training. The results obtained from this content analysis had a high inter-rater agreement. In the previous study, the intra-rater reliability of the self-evaluation and evaluation by the tutors ranged from moderate to high13).

The purpose of this study was to examine internal consistency, criterion-related validity, factorial validity, and content validity of the CEPT.

SUBJECTS AND METHODS

The participants in this study were physical therapists belonging to the 21 medical facilities within the Kanto area. These facilities had ten or more full-time physical therapists each, and participants were intentionally selected so the results might not be partial. Novice physical therapists with less than three years’ experience participated in the trainees group and the tutors who were guiding the novice physical therapists participated in the tutors group. We obtained written informed consent from all the 21 medical facilities to their inclusion in this study as an institution. We explained the purpose and contents of this study to individual participants orally or in written documents. The questionnaires were filled out anonymously. The return of a questionnaire constituted a subject’s written informed consent to participation in the study. This study was approved by the Epidemiologic Research Ethics Committee of Gunma University Faculty of Medicine (No.21-31).

The CEPT is comprised of 53 items in seven domains (knowledge of clinical physical therapy, decision-making skills, clinical skills, communication skills, attitudes of health professionals, self-learning abilities, and self-management). All items are assessed on a four-point scale (1 point: much instruction and advice needed, 2 points: some instruction and advice needed, 3 points: instructions from tutors are unnecessary, and being an autonomous practitioner, and 4 points: being able instructions from tutors are unnecessary, and having a high level of competence to serve as a good example to other novice physical therapists and physical therapy students). The total score of the CEPT ranges from 53 to 212. The clinical competences of the trainees were self-evaluated by the trainees and evaluated by the tutors using the CEPT. Overall trainee autonomy as physical therapists was evaluated using a visual analog scale (VAS) for self-evaluation and for evaluation of the trainees by their tutors. The content validity of the CEPT was examined by questioning if the CEPT could evaluate the competence of novice physical therapists on a four-point scale (strongly disagree, disagree, agree, strongly agree).

Internal consistency of the CEPT was assessed using Cronbach’s alpha for self-evaluation by the trainees and the evaluation by their tutors. The relationship between the total score of the CEPT, physical therapist experience period of the trainees (months), and the overall competence by VAS were analyzed using Pearson’s correlation coefficients and a significance level of 5%. Furthermore, the factorial validity of the CEPT for self-evaluation by the trainees was evaluated via exploratory factor analysis. A generalized least squares estimation and direct oblimin rotation on the factors was performed because the item distributions did not conform to a normal distribution. The number of factors was determined using a screen plot. All statistics were calculated using SPSS version 19.

RESULTS

Participants included in the study were 278 trainees and 119 tutors. The mean period of physical therapist experience (standard deviation) of the trainees and the tutors was 16.4 (10.5) months, 91.8 (33.6) months, respectively. Mean total score (standard deviation) of the CEPT of the trainees and the tutors was 126.3 (20.9), 137.8 (23.2), respectively.

Cronbach’s alpha of the CEPT was 0.96 for the trainees and 0.97 for the tutors. The correlation coefficient between the total score of the CEPT and physical therapist experience period of the trainees was 0.33 for the trainees and 0.46 for the tutors. The correlation coefficient between the total score of the CEPT and overall competence by VAS was 0.83 for the trainees and 0.87 for the tutors.

The 53 items were reduced to two factors, explaining 45.6% of the variance in the CEPT. Table 1 displays the factor loading based on a generalized least squares estimation and direct oblimin rotation of the two factors. The first factor included three domains: knowledge of clinical physical therapy, decision-making skills and clinical skills. The second factor included three domains: attitudes of health professionals, self-learning abilities and self-management. The domain of communication skills was included in both factors. The two factors identified were called “the specialty of the physical therapist” and “the essential competence of the health professional”.

Table 1.

Items of the Clinical Competence Evaluation Scale in Physical Therapy (CEPT) and their factor loading determined by exploratory factor analysis

In the content validity of the CEPT, 90% or more of the trainees and the tutors answered that the CEPT could sufficiently or almost correctly evaluate the competence of novice physical therapists (Table 2).

Table 2.

Distribution of the answers to “I think the CEPT can evaluate the clinical competence of the novice physical therapists”

	Trainees (n=278)	Tutors (n=119)
Strongly agree	11.2%	7.6%
Agree	82.4%	84.0%
Disagree	6.1%	8.4%
Strongly disagree	0.4%	0.0%

DISCUSSION

The purpose of this study was to examine the validity of the CEPT. To ensure the content validity of the CEPT, the development of the CEPT was based on a qualitative study14) of semi-structured interviews with expert physical therapists. High intra-rater reliabilities of the self-evaluation and evaluation by tutors were found. Generally, Cronbach’s alpha scores > 0.80 are excellent15) and the results of this study were 0.8 or more for both trainees and tutors, showing that CEPT has internal consistency. The physical therapist experience period of the trainees and the overall competence by VAS were assessed to examine the criterion-related validity of the CEPT, since there are no gold standard scales for evaluating the clinical competence of physical therapists. The total score of the CEPT showed poor or moderate correlation with the physical therapist experience period of the trainees, but a high correlation with the overall competence by VAS. Generally, clinical competence will improve with acquisition of clinical experience. Furthermore, since the correlation between the total score of the CEPT and the overall competence by VAS was high, the CEPT had high criterion-related validity.

In exploratory factor analysis of the CEPT, the first factor consisted of three domains (knowledge of clinical physical therapy, decision-making skills and clinical skills). These domains were called “the specialty of the physical therapist” and they show a physical therapist’s special competence. The second factor consisted of three domains (attitudes of health professionals, self-learning abilities and self-management) and was called “the essential competency of the health professional”, the level of competence necessary for all health professionals16). Communication skills are considered necessary in the specialty of physical therapy, and they are important skills for all health professionals. The elements of the three domains included in the first factor are important for gathering client information, but the elements of the three domains included in the second factor are important for relationships with clients, their family and the other health professionals. The results of our factor analysis indicate that the CEPT has high factorial validity and may be used to measure clinical competence in physical therapy continuing education.

Finally, since 90% or more of the trainees and tutors answered that the CEPT could sufficiently evaluate the competence of novice physical therapists, the content validity of the CEPT was established. The results of our previous studies and this study together show that the CEPT is an appropriate tool for evaluating the clinical competence of novice physical therapists in physical therapy continuing education.

This study had limitations in criterion-related validity as the physical therapist experience period and whole competence by VAS were not necessarily evaluating the competence of physical therapists. Future longitudinal studies are needed to provide insights into the difficulty of the items in the CEPT and the clinical competences of novice physical therapists.