Difficulty levels of OSCE items related to examination and measurement skills.

[Purpose] The difficulty levels of level-2 OSCE (examination and measurement skills) items were examined, with a view to providing reference data for the determination of students' skills. [Subjects] A total of 284 graduates of physical (PT) and occupational (OT) therapy classes of 2011 (59 and 40), 2012 (46 and 36), and 2013 (61 and 42, respectively) were studied, with PT or OT faculty members as OSCE examiners and a simulated patient. [Methods] Scores for 11 level-2 OSCE items were compared between before and after clinical training.
[Results] Scores markedly increased after clinical training. On comparison among the items, scores for sensory examination were the highest, and those for interviews were the lowest.
[Conclusion] The results of this study indicate the necessity of considering an appropriate combination of different difficulty levels when adopting OSCE-based educational approaches.

INTRODUCTION

In Japan’s super-aging society, the numbers of physical (PT) and occupational (OT) therapists have rapidly increased in recent years to meet the increasing social demand for rehabilitation. The rapid increase in the number of therapists has consequently reduced overall years of experience of therapists in clinical environments, suggesting a decrease in the quality of clinical services. Under these circumstances, it is necessary to give more importance to postgraduate education in workplaces, in addition to improving school education systems. However, at present, postgraduate education is provided by associations or prefectural societies or individually, and it is difficult for therapists to independently organize systems to provide such education in each workplace. Furthermore, although a number of technical seminars for clinical therapists are being held, their contents are not applicable to some actual situations, as most participants in such seminars are healthy individuals. Considering such a situation, school education should focus on the development of basic abilities to obtain more specialized knowledge and skills after graduation, while postgraduate education should address their application. In line with this, it is important for faculty members to participate in clinical practice to clarify therapists’ actual skills, and to adopt educational approaches based on technical items necessary for clinical training. On the other hand, for clinical supervisors and PTs and OTs in charge of postgraduate education, it is essential to accurately recognize the content students learn at schools. Therefore, systems to provide standardized and comprehensive skill education throughout the period between pre- and postgraduation, involving faculty members, clinical supervisor, and PTs and OTs in charge of postgraduate education, are needed.

Under these circumstances, we have been engaged in specialized education for students, in other words, “education to nurture specialists with clinical skills”, since our department was founded within the university in 2004, focusing on clinical demands and regarding physical and occupational therapies as a “domain of therapeutics” or a clinical science. In short, the central goal of our education is the development of clinical skills. In the current therapist education system, however, students’ clinical skills are assessed by supervisors of clinical training facilities, rather than the faculty members of training schools. In therapist education to nurture “clinical professionals”, this should be a critical issue. To address this, our faculty members participate in clinical treatment, and continuously examine appropriate methods to train students with therapists engaged in clinical practice. Similarly, the university has adopted the Objective Structured Clinical Examination (OSCE) system, aiming to specify clinical education standards for training schools and to standardize methods of assessment by clinical supervisors1). The OSCE is a method of clinical skill assessment, proposed by Harden2) in 1975, and has been reported to be appropriate for assessment of learning achievement levels in the psychomotor and emotional domains, which are difficult to evaluate with written examinations3). The standardization of assessment and clinical skill education methods is also likely to improve the quality of clinical rehabilitation services.

In our department, OSCE items have been used as learning content since 2005 to standardize OSCE-based skill education methods, and such an educational approach has been adopted not only before but also after clinical training since 2007 to establish OSCE-based skill education systems. Furthermore, since 2011, the OSCE has been regarded as a course of study (subjective clinical skill training), and the details of OSCE-based learning in our department have been observed by clinical supervisors to increase the consistency of their assessments. Using the OSCE to link school education and clinical training together, we aim to integrate schools and clinical training facilities, which has been difficult for all training schools throughout Japan up to the present.

This study may have significance in standardizing methods to assess clinical skills in therapist education. Our previous studies revealed associations among OSCE scores, academic achievements, and clinical training outcomes4,5,6,7,8). In the present study, the difficulty levels of level-2 OSCE (examination and measurement skills) items were examined, with a view to clarifying appropriate teaching steps and challenging learning areas, as well as providing reference data for the determination of students’ skills when they perform different OSCE items.

SUBJECTS AND METHODS

A total of 284 graduates of physical (PT) and occupational (OT) therapy classes in 2011 (59 and 40), 2012 (46 and 36), and 2013 (61 and 42, respectively) were involved as OSCE examinees.

The OSCE was conducted in 3 rooms (stations), in each of which a task was presented, and the examinees made rounds of these stations to implement all tasks, following instructions. At each station, 2 examiners (a physical or occupational therapy faculty member and a clinical supervisor) and 1 simulated patient (another clinical supervisor) were present. The time to implement each task was 5 minutes, and, immediately after its completion, 2 minutes of feedback was provided. Subsequently, the examiners and simulated patient discussed, and assessed the examinees’ performance. The OSCE tasks were not previously shown to the examinees.

For statistics, Predictive Analytics Software (PASW) Statistics 18.0 was used. For comparison of scores for level-2 OSCE items (11) between before and after clinical training, a paired t-test was conducted, and, for comparison of scores between the items, one-way analysis of variance and Tukey-Kramer’s multiple comparisons were performed.

RESULTS

Scores for the following level-2 OSCE items markedly increased after clinical training: forearm circumference measurement (FC), upper-limb tendon reflex testing (U-TR), upper-limb range of motion measurement (U-ROM), upper-limb manual muscle testing (U-MMT), lower-limb range of motion measurement (L-ROM), lower-limb manual muscle testing (L-MMT), vital sign measurement (vital), leg length measurement (leg length), lower-limb tendon reflex testing (L-TR), and interviews. There were no significant differences in scores for sensory examination (sensory) between before and after clinical training, as both were high (Table 1).

Table 1.

Comparison of scores for level-2 OSCE items between before and after clinical training

	Sensoryn=31	Forearm circumferencen=50	U/E DTRn=24	U/E ROM-testn=73	U/E MMTn=56	L/E ROM-testn=56	L/E MMTn=63	Vital checkn=64	Length of lower limbn=42	L/E DTRn=53	Therapist interviewn=56
Before clinical training	84.8 ± 8.5	79.4 ± 10.4	78.1 ± 11.4	73.6 ± 9.8	71.3 ± 10.1	71.2 ± 9.2	71.2 ± 11.3	69.4 ± 11.2	69.2 ± 10.9	65.6 ± 16.2	50.0 ± 6.3
After clinical training	88.3 ± 7.6	87.4 ± 9.0	86.6 ± 5.7	86.1 ± 10.5	77.1 ± 11.6	79.9 ± 9.3	82.2 ± 7.7	85.2 ± 10.4	74.9 ± 12.1	82.2 ± 12.3	68.8 ± 11.2
Before clinical training vs. After clinical training		**	**	**	**	**	**	**	**	**	**

*p<0.05; **p<0.01

On comparison of scores among the items, before clinical training, scores for sensory were the highest, followed by those for FC, U-TR, U-ROM, U-MMT, L-MMT, L-ROM, vital, leg length, L-TR, and interviews, in this order. Scores for sensory were markedly higher than those for U-MMT, L-MMT, L-ROM, vital, leg length, L-TR, and interviews. Scores for FC were significantly higher than those for L-MMT, L-ROM, vital, leg length, L-TR, and interviews. Scores for U-TR were markedly higher than those for leg length, L-TR, and interviews. Scores for U-ROM were significantly higher than those for L-TR and interviews. In contrast, scores for interviews were markedly lower than those for U-MMT, L-MMT, L-ROM, vital, leg length, and L-TR.

After clinical training, scores for sensory were the highest, followed by those for FC, U-TR, U-ROM, vital, L-TR, L-MMT, L-ROM, U-MMT, leg length, and interviews, in this order. Scores for sensory were markedly higher than those for L-ROM, U-MMT, leg length, and interviews. Scores for FC were significantly higher than those for L-ROM, U-MMT, leg length, and interviews. Scores for U-TR were markedly higher than those for L-ROM, U-MMT, leg length, and interviews. Scores for U-ROM were significantly higher than those for L-ROM, U-MMT, leg length, and interviews. Scores for vital were markedly higher than those for L-MMT, leg length, and interviews. Scores for L-TR were significantly higher than those for leg length and interviews. Scores for L-MMT were markedly higher than those for leg length and interviews. In contrast, scores for interviews were significantly lower than those for L-ROM, U-MMT, and leg length (Table 2).

Table 2.

Comparison of scores among level-2 OSCE items

Before clinical training
	Sensory	Forearm circumference	U/E DTR	U/E ROM-test	U/E MMT	L/E MMT	L/E ROM-test	Vital check	Length of lower limb	L/E DTR	Therapist interview
Sensory
Forearm circumference
U/E DTR
U/E ROM-test
U/E MMT	**
L/E MMT	**	**
L/E ROM-test	**	**
Vital check	**	**
Length of lower limb	**	**	**
L/E DTR	**	**	**	**
Therapist interview	**	**	**	**	**	**	**	**	**	**
After clinical training
	Sensory	Forearm circumference	U/E DTR	U/E ROM-test	Vital check	L/E DTR	L/E MMT	L/E ROM-test	U/E MMT	Length of lower limb	Therapist interview
Sensory
Forearm circumference
U/E DTR
U/E ROM-test
Vital check
L/E DTR
L/E MMT
L/E ROM-test	*	**	**	*
U/E MMT	**	**	**	**	**
Length of lower limb	**	**	**	**	**	**	**
Therapist interview	**	**	**	**	**	**	**	**	**	**

*p<0.05; **p<0.01

DISCUSSION

In recent years, the OSCE has been introduced as an education approach for objective assessment of clinical skills mainly in the field of medicine9,10,11). Compared with conventional written examinations, the OSCE enables examiners to assess clinical skills in the psychomotor, emotional, and cognitive domains, and clarify points to improve10). Saito et al. reported that it is effective to adopt the OSCE in medical education in order to train medical students by developing necessary basic skills in both technical and behavioral aspects, and that it enables educators to guide students toward the appropriate integration of knowledge, skills, and behavior1).

Rehabilitation medicine is a practical system, in which medical services are provided based on learning achievements. For learning specialists, therefore, it may be important to review the significance of OSCE-based skill education from a viewpoint of rehabilitation medicine, and to systematize it through new approaches, rather than simply adopting medical education systems. The OSCE for PTs and OTs consists of 3 levels, aimed at improving students’ clinical skills: level 1, communication and caregiving skills; 2, examination and measurement skills; and 3, analytical and therapeutic skills. The feedback provided before and after clinical training enables students to subjectively recognize their clinical skills, and develop learning plans to improve their knowledge, behavior, and skills step by step. Such a practical learning process provides a basis for OSCE-based therapist education.

The present study examined the difficulty levels of level-2 OSCE items related to examination and measurement skills as basic physical and occupational therapy techniques needed in clinical environments, involving frequent contact with patients. It also aimed to clarify the effect of education based on these items on clinical training.

On comparison of level-2 OSCE scores between before and after clinical training, the scores for all 10 items increased after clinical training, and the increases in the scores were marked, except for sensory examination, for which the scores were already high before clinical training. Such improvement in the scores for all items may be explained by sufficient cooperation between the school and person in charge of clinical training, as clinical training was provided in a medical institution attached to a university.

The difficulty levels of vital sign measurement and lower-limb tendon reflex testing decreased, and the scores for them increased after clinical training, presumably as a result of repeatedly practicing these tasks, as they tend to be performed relatively frequently during clinical training; such repetition has a similar effect on motor learning in a clinical intervention. In contrast, the scores for interviews were the lowest before and after clinical training, although slight improvement was observed in the latter; the scores were particularly low for its subitems “showing empathy and understanding” and “accurately recognizing patients’ chief complaints”. Supportive listening is occasionally insufficient during clinical training, as some students desire to hear only what they look for. Such a tendency is likely to lead to difficulty in establishing a trust-based relationship with patients, and indicates that students do not appropriately recognize medical services as a framework for patients. In order to address this, it may be crucial to enhance students’ awareness of the importance of nonverbal communication through education, in addition to using simulated patients.

On comparison of scores among the OSCE items, while the difference between high and low scores were marked before clinical training, they became less marked after clinical training, confirming that clinical training appropriately covered all the level-2 OSCE items adopted in the education, and contributed to overall skill improvement. In short, in the school, systems to provide education for basic clinical physical and occupational therapy skills may have been well organized as an outcome of improving the quality of skill education, or integrating school programs, clinical training, and OSCE-based skill education. Furthermore, in such systems, students may have learned the association among courses of study with different contents, and improved their ability to reflect learning achievements on behavior and skills during clinical training with feedback focusing on behavior, skills, and knowledge.

In this study, it was demonstrated that therapist education in the school and clinical training facility had been gradually integrated. On the other hand, the results highlighted the necessity of considering an appropriate combination of different difficulty levels when adopting OSCE-based educational approaches. Further studies may be necessary to examine the difficulty levels of level-3 OSCE items (analytical and therapeutic skills) and standardize clinical skills, with a view to establishing appropriate skill education systems.