Reliability of the OSCE for Physical and Occupational Therapists.

[Purpose] To examine agreement rates between faculty members and clinical supervisors as OSCE examiners. [Subjects] The study subjects were involved physical and occupational therapists working in clinical environments for 1 to 5 years after graduating from training schools as OSCE examinees, and a physical or occupational therapy faculty member and a clinical supervisor as examiners. Another clinical supervisor acted as a simulated patient. [Methods] The agreement rate between the examiners for each OSCE item was calculated based on Cohen's kappa coefficient to confirm inter-rater reliability.
[Results] The agreement rates for the behavioral aspects of the items were higher in the second than in the first examination. Similar increases were also observed in the agreement rates for the technical aspects until the initiation of each activity; however, the rates decreased during the middle to terminal stages of continuous movements.
[Conclusion] The results may reflect the recent implementation of measures for the integration of therapist education in training schools and clinical training facilities.

INTRODUCTION

In Japan’s super-aging society, social demands for rehabilitation are increasing. In line with this, the numbers of psychical therapist (PT) and occupational therapist (OT) training schools have rapidly increased to 248 and 182, respectively, and the total numbers of certified PTs and OTs were 110,000 and 65,000, respectively, as of 2013. The number of therapists is particularly excessive in the field of physical therapy, as more than 10,000 novices are certified on an annual basis. This has resulted in a high proportion, approximately 25%, of novice PTs and OTs with less than 3 years of clinical experience. The rapid increase in the number of novice therapists has consequently reduced overall years of experience of therapists in clinical environments, suggesting a decrease in the quality of clinical services. Under these circumstances, it may be necessary to expand the range of therapists’ activities. Also, in order to improve the conditions of their employment, which have become unfavorable over the past years, it may be crucial to enhance novice therapists’ abilities, while continuously training experienced therapists and nurturing specialists.

Diverse approaches to post-graduate training have already been implemented in clinical environments; however, such approaches have not been unified, and appropriate methods for assessing novice therapists’ clinical skills, as well as educational methodologies, have not yet been fully examined. Considering this situation, the unification of post-graduate and clinical education methods, based on the results of objective assessment of clinical skills in the early stages, may be vital for novice therapists’ development.

At our university, physical and occupational therapies are regarded as a domain of therapeutics, or a clinical science, and specialized education for students, in other words, “education to nurture specialists with clinical skills”, has been provided, focusing on clinical demands, since our department was founded in 2004. In short, our education focuses on clinical skills. However, in the current therapist education system, students’ clinical skills are assessed by supervisors of clinical training facilities, rather than the faculty members of training schools. In therapist education to nurture “clinical professionals”, this is a very important issue. Our faculty members are also engaged in clinical treatment, while continuously discussing appropriate methods for training students with therapists working in clinical environments. At our university, the Objective Structured Clinical Examination (OSCE) system has been adopted, with a view to specifying standards for clinical education in training schools and standardizing methods of assessment by clinical supervisors1). The OSCE is a method of clinical skill assessment, which was proposed by Harden2) in 1975, and it has been reported to be appropriate for assessing learning achievement levels in the psychomotor and emotional domains, which are difficult to evaluate with written examinations.

Aiming to establish an OSCE-based education system, we have conducted the OSCE before clinical training since 2005. The examination has also been conducted after clinical training since 2008, when the OSCE items were adopted as those for practice (so that students can systematically learn about each item, and assimilate the learning with due consideration for the behavior and skills needed in clinical environments as a preparatory step for clinical training).

We reported an association between OSCE scores, academic achievement levels, and clinical training outcomes, as well as inter-rater agreement rates (0.767 to 0.935) between faculty members in previous studies3,4,5). The present study focused on the necessity of confirming consistency between OSCE scores, representing the results of assessment by faculty members and clinical training outcomes assessed by clinical supervisors, in order to adopt the OSCE in post-graduate education. The objectivity of assessment refers to the consistency of results between different raters, and, therefore, it may be important to enhance the objectivity of OSCE-based assessment in terms of accountability of the assessment results. Accordingly, this study examined agreement rates between faculty members and supervisors of clinical training facilities as OSCE examiners.

SUBJECTS AND METHODS

The study subjects were PTs and OTs working in clinical environments for 1 to 5 years after graduating from training schools as OSCE examinees, and a physical or occupational therapy faculty member and a clinical supervisor as raters. Another clinical supervisor acted as a simulated patient. Before examination, the examiners (the faculty member and clinical supervisor) and simulated patient agreed upon the latter’s condition, in addition to the points of assessment.

The OSCE consisted of items to assess skills to assist and guide patients performing the following level 3 activities: elevating the upper body and sitting on the edge of the bed; standing up; wheelchair-to-bed transfer; dressing; toileting; and walking. It was conducted before (examination 1) and after (examination 2) learning sessions consisting of 60-minute lectures regarding each item, practice, and practical guidance in clinical environment, focusing on the applicability of the OSCE items in clinical environments.

The OSCE was conducted in 3 rooms (stations), in each of which a task was presented, and the examinees made the rounds of these stations to perform all tasks, following instructions. At each station, 2 examiners (a physical or occupational therapy faculty member and a clinical supervisor) and 1 simulated patient (another clinical supervisor) were present. The time allowed for performance of each task was 5 minutes, and, immediately after task completion, 2 minutes of feedback was provided. Subsequently, the examiners and simulated patient discussed and assessed the examinees’ performance. The tasks were not previously shown to the examinees.

For statistical analysis, Predictive Analytics Software (PASW) Statistics 18.0 was used. To confirm inter-rater reliability, the agreement rate for each item was calculated based on Cohen’s kappa coefficient. After calculation, means were adopted as data regarding the behavioral and technical items.

RESULTS

The agreement rates between the examiners for the behavioral aspects of transfer, dressing, standing up, and toileting were high in the examination. In contrast, those for elevating the upper body and sitting on the edge of the bed and walking were low. In examination 2, the agreement rate was 0.7 or higher for all items. For the technical aspect of transfer, the agreement rate was the highest, 0.5, in examination 1, followed by toileting, standing up, dressing, and walking, with rates of 0.3 to 0.4. The agreement rate was the lowest for elevating the upper body and sitting on the edge of the bed, 0.25. In examination 2, the agreement rates were generally higher than those in examination 1; 0.4 to 0.6 for transfer, standing up, toileting, and walking, and 0.3 for elevating the upper body and sitting on the edge of the bed and dressing (Table 1).

Table 1.

Intra-class correlation coefficients for the faculty members and clinical supervisors

	Sitting upon the bed	Standing up	Transfer: wheelchair-to-bed transfer	Toiletingactivity	Dressingactivity	Walking
Preparatory items
1st	0.534	1.000	0.860	0.910	0.738	0.399
2nd	1.000	1.000	0.911	0.867	1.000	0.745
Technical items
1st	0.245	0.429	0.520	0.430	0.380	0.341
2nd	0.294	0.566	0.630	0.487	0.286	0.469

The agreement rate was significantly low for the technical aspects of the following sub-items: elevating the upper body and sitting on the edge of the bed, being able to motivate the patient to rise and enhance his/her activeness, being able to confirm sufficient space for the lateral position, being able to appropriately guide the lower body toward the lateral position and being able to give instructions at appropriate times throughout the activity and dressing, being able to take appropriate safety measures, being able to confirm appropriate procedures before placing the patient’s arms in a sleeve, being able to provide appropriate guidance for placing the patient’s paralyzed arm in a sleeves, being able to provide appropriate guidance for placing the patient’s non-paralyzed arm in a sleeve (while ensuring a stable sitting position), being able to provide appropriate guidance/assistance for the elbow and shoulder when placing the paralyzed arm in a sleeve, and prepare for pulling an upper garment over the head and being able to correct the shoulder line and bottom edge of the garment on the paralyzed side. Furthermore, the agreement rates for the technical aspects remained high until the initiation of each activity, and, decreased during the middle to terminal stages of continuous movements (Table 2).

Table 2.

Agreement rates between the examiners in the OSCE level 3 (examinations 1 and 2)

Sitting up on the bed	1st	2nd
Preparatory items
● Appropriately groomed	1.000	1.000
● Being able to appropriately greet	0.446	1.000
● Being able to provide appropriate explanations before assessing movements to elevate the upper body and sit on the edge of the bed	−0.149	1.000
● Being able to appropriately confirm the patient’s consent	1.000	1.000
● Being able to appropriately remove the bed rail by the patient’s leg	0.463	1.000
● Being able to remove and keep blankets and clothes on the bed in order	0.383	1.000
Technical items
● Being able to motivate the patient to rise and enhance his/her activeness	0.267	0.007
● Being able to confirm sufficient space for the lateral position	0.285	−0.048
● Being able to roughly determine the patient’s remaining functions	0.474	0.565
● Being able to provide appropriate explanations regarding procedures from elevating the upper body to sitting on the edge of the bed	0.450	0.436
● Being able to appropriately guide the head upward	0.092	0.364
● Being able to appropriately guide the upper body toward the lateral position	0.022	0.718
● Being able to appropriately guide the lower body toward the lateral position	0.110	0.182
● Being able to give instructions at appropriate times throughout the activity	0.178	−0.004
● Being able to stand in an appropriate position for guiding the patient	0.162	0.371
● Being able to take preventive measures against falls, while maintaining an appropriate sitting position	0.414	0.353
Transfer: wheelchair-to-bed transfer	1st	2nd
Preparatory items
● Appropriately groomed	1.000	0.645
● Being able to appropriately greet	1.000	1.000
● Being able to provide appropriate explanations before assessing transfer	1.000	1.000
● Being able to appropriately confirm the patient’s consent	0.450	1.000
Technical items
● Being able to appropriately place the wheelchair by the bed	0.130	0.645
● Being able to adjust the height of the bed to ensure safety	0.743	0.564
● Being able to advise or assist the patient to raise the foot plates	0.083	1.000
● Being able to advise the patient to participate in the activity as much as possible	0.593	0.805
● Being able to provide guidance for moving the buttocks forward and sitting on the anterior part of the wheelchair seat	0.394	0.273
● Being able to place the non-paralyzed foot slightly ahead of the paralyzed one	0.031	0.434
● Being able to support the patient’s knee on the paralyzed side by maintaining it between the distal parts of both thighs	0.586	0.732
● Being able to give advice for bending the trunk forward, while placing the hands by the waist or on the knees	0.450	0.583
● Being able to support the patient’s chest and abdomen with the forearms before standing up	0.700	0.538
● Being able to give instructions to stand up together at an appropriate time	0.911	0.563
● Being able to guide/assist the patient to turn his/her buttocks to the bed, and slowly sit on it	0.901	0.596
● Being able to correct the posture when sitting on the edge of the bed, and make it stable	0.705	0.829
Dressing activity	1st	2nd
Preparatory items
● Appropriately groomed	1.000	1.000
● Being able to appropriately greet	1.000	1.000
● Being able to provide appropriate explanations before assessing dressing	1.000	1.000
● Being able to appropriately confirm the patient’s consent	−0.048	1.000
Technical items
● Being able to take appropriate safety measures	0.419	−0.253
● Being able to confirm the maintenance of a stable sitting position	0.574	0.554
● Being able to confirm the reach movements of the non-paralyzed upper body	0.384	1.000
● Being able to confirm appropriate procedures before placing the arms in the sleeves	0.532	−0.065
● Being able to provide appropriate guidance for maintaining the paralyzed arm straight by the trunk, using the weight of the upper body	0.160	0.680
● Being able to provide appropriate guidance for placing the paralyzed arm in the sleeves	0.220	0.000
● Being able to provide appropriate guidance for placing the non-paralyzed arm in the sleeves (while ensuring a stable sitting position)	0.211	−0.078
● Being able to provide appropriate guidance/assistance for the elbow and shoulder when placing the paralyzed arm in the sleeves, and prepare for pulling an upper garment over the head	0.450	−0.182
● Being able to provide appropriate guidance/assistance for drawing the garment over the head (while maintaining the pelvis in a neutral position)	0.516	0.974
● Being able to correct the shoulder line and bottom edge of the garment on the paralyzed side	0.333	0.227
Standing up	1st	2nd
Preparatory items
● Appropriately groomed	1.000	1.000
● Being able to appropriately greet	1.000	1.000
● Being able to provide appropriate explanations before assessing movements to stand up	1.000	1.000
● Being able to appropriately confirm the patient’s consent	1.000	1.000
Technical items
● Being able to confirm the appropriate height of an adjustable bed	0.471	1.000
● Being able to confirm the position of the patient’s buttocks, and move them forward with guidance/assistance	0.093	1.000
● Being able to move the patient’s legs at an appropriate angle, confirm the appropriate width between his/her feet and heels’ condition on the floor, and correct them with guidance/assistance	0.455	0.241
● Being able to correct the posture, alignment, and center of balance with guidance/assistance	0.130	0.669
● Being able to confirm and give advice for anterior pelvic tilt and participation as a preparatory step for leaving the bed	0.538	0.739
● Being able to support the patient’s pelvis with the hands and his/her trunk with the chest, while standing with the knees bent when leaving from the bed	0.193	0.364
● Being able to move the patient’s knees forward using the legs when leaving from the bed	0.492	0.323
● Being able to bend the patient’s head and trunk forward, and shift the center of balance at a safe speed with guidance/assistance	0.773	0.353
● Being able to support the patient’s knees, and ensure appropriate hip, knee, and trunk extension with guidance/assistance after leaving from the bed	0.637	0.336
● Being able to correct the alignment with guidance/assistance to maintain a stable standing position	0.505	0.630
Toileting activity	1st	2nd
Preparatory items
● Appropriately groomed	0.645	1.000
● Being able to appropriately greet	1.000	1.000
● Being able to provide the patients with appropriate explanations before toilet training	1.000	1.000
● Being able to appropriately confirm the patient’s consent	1.000	0.463
Technical items
● Being able to confirm appropriate environments for the activity	0.621	0.776
● Being able to provide appropriate explanations regarding procedures	0.669	0.000
● Being able to provide guidance for appropriately standing up from the bed	0.405	0.441
● Being able to provide guidance for a change of direction and stable standing position	0.114	0.389
● Being able to confirm the appropriate posture when pulling down clothes	−0.015	0.476
● Being able to give advice for smoothly and appropriately handling clothes	0.409	0.554
● Being able to provide guidance for appropriately sitting on the toilet seat	0.385	0.719
● Being able to provide guidance for the maintenance of an appropriate sitting position	0.281	0.448
● Being able to give advice for appropriately cleaning the anus/genitals	0.321	0.506
● Being able to provide guidance for appropriately standing up from the toilet seat	0.551	0.296
● Being able to confirm the appropriate posture when pulling up clothes	0.486	0.700
● Being able to give advice for smoothly and appropriately handling clothes when pulling up them	0.912	0.391
● Being able to provide guidance for a change of direction and sitting on the edge of the bed	0.409	0.639
Walking	1st	2nd
Preparatory items
● Appropriately groomed	0.645	1.000
● Being able to appropriately greet	−0.048	1.000
● Being able to provide appropriate explanations before performing gait analysis	0.000	−0.019
● Being able to appropriately confirm the patient’s consent	1.000	1.000
Technical items
● Being able to take safety measures	0.299	0.180
● Being able to ensure appropriate walking environments	−0.038	0.463
● Being able to confirm appropriate clothes, shoes, and aids for walking	0.284	0.349
● Being able to advise the patient to participate in the activity as much as possible	−0.048	0.369
● Being able to confirm the ability to move in a standing position	0.000	0.542
● Being able to appropriately instruct to start walking	0.753	0.780
● Being able to appropriately provide guidance/assistance during the swing phase on the paralyzed side	0.741	0.441
● Being able to appropriately provide guidance/assistance during the initial contact to loading-response phases on the paralyzed side	0.593	0.545
● Being able to appropriately provide guidance/assistance during the mid-stance phase on the paralyzed side	0.094	0.450
● Being able to appropriately provide guidance/assistance during the terminal stance phase on the paralyzed side	0.000	0.567
● Being able to confirm appropriate walking rhythms	0.577	0.535
● Being able to appropriately guide the patient after walking	0.831	0.411

DISCUSSION

In recent years, the OSCE has been used as an educational approach to the objective assessment of clinical skills, mainly in the field of medicine6,7,8). Compared to conventional written examinations, the OSCE enables examiners to assess clinical skills in the psychomotor, emotional, and cognitive domains, and clarify points to improve7). Saitoh et al. reported that the adoption of the OSCE in medical education has been effective at training medical students through the development of necessary basic skills in both technical and behavioral aspects, and it also enables educators to guide students in the appropriate integration of knowledge, skills, and behavior9). In fact, the OSCE adopted for clinical education in training schools has provided a certain effect3,4,5); however, education methods in training schools and clinical training facilities have not yet been unified.

Considering this situation, the present study examined agreement between faculty members and clinical supervisors of OSCE-based assessment results, with a view to adopting the OSCE system in post-graduate education.

In examination 1, the agreement rates between the examiners for transfer, dressing, standing up, and toileting were high in the behavioral aspect, while those for elevating the upper body and sitting on the edge of the bed and walking were low. In contrast, in examination 2, a high agreement rate of 0.7 or more was obtained for all items, suggesting improvement. In the technical aspects, the agreement rates were generally lower in both examinations 1 and 2 than those of behavioral aspects. The rate was particularly low, approximately 0.3, for elevating the upper body and sitting on the edge of the bed and dressing. Comparison of examinations 1 and 2 revealed the agreement rates were generally higher in the latter than in the former.

The agreement rates for the behavioral aspects of elevating the upper body and sitting on the edge of the bed, transfer, dressing, and standing up increased in examination 2. Nakahara et al. reported that, in the field of rehabilitation, including physical and occupational therapies, clinical supervisors frequently point out students’ problems in the emotional domain, such as insufficient sociability and inappropriate basic behavior10). The high agreement rates between the faculty member and clinical supervisor in the present study suggest that our OSCE-based assessment system may have a sufficient inter-rater reliability. On the other hand, the agreement rate for walking was lower than those for other items. This may be explained by a tendency for this activity to be regarded as PTs’ specialty, which may have influenced the results of this study as both PT and OT were the subjects. This supposition is supported by the markedly low agreement rate of the sub-item, being able to provide appropriate explanations before performing gait analysis, which may directly represent a specialty difference between the examiners.

For the technical aspects, the agreement rates were generally lower than those of the behavioral aspects. Markedly low rates of 0.2 to 0.3 were observed for elevating the upper body and sitting on the edge of the bed, and dressing. Particularly, for the sub-items of the former (being able to motivate the patient to rise and enhance his/her activeness, being able to confirm sufficient space for the lateral position and being able to give instructions at appropriate times throughout the activity) and the latter (being able to take appropriate safety measures and being able to confirm appropriate procedures before placing the arms in the sleeves), which represent the technical and partly behavioral aspects of preparation and communication skills, agreement was rarely obtained. Conversation rhythms, as well as the tone and volume of voice, are important factors for communication skills; therefore, the agreement rates for items representative of their technical aspects may have been influenced by examiners’ subjectivity more markedly than those of other items related to assistance and guidance. Similarly, the low agreement rate for elevating the upper body and sitting on the edge of the bed, an activity requiring continuous movements, may be explained by differences in from the method taught to perform this activity, taught in schools, in which the movements are separately analyzed. In short, in the OSCE, it may be necessary to consider the influence of examiners’ sense of values and the consequent subjectivity of their assessment. To address this issue, paradigm-based training for examiners and discussions regarding possible problems may be needed. To improve the inter-rater agreements, items that involve a mix of contents should be subdivided. In addition, in low agreement rate items, clear criteria for each rating should be decided by discussion among the rates.

The results of this study, may reflect the recent implementation of measures for the integration of therapist education in training schools and clinical training facilities. Rehabilitation medicine is a practical system, in which medical services are provided based on learning achievements. Therefore, in therapist education aiming to nurture learning specialists, it may be important to establish OSCE-based assessment systems, while reviewing the significance of OSCE-based education from the viewpoint of rehabilitation medicine, to devise new approaches, rather than simply adopting medical education systems. Further studies may be necessary to develop systems for the adoption of the OSCE in the training of novice therapists working in clinical environments and to provide lifelong education.