A core competence-based objective structured clinical examination (OSCE) in evaluation of clinical performance of postgraduate year-1 (PGY₁) residents.

BACKGROUND: Clinical competency certifications are important parts of internal medicine residency training. This study aims to evaluate a composite objective structured clinical examination (OSCE) that assesses postgraduate year-1 (PGY(1)) residents' acquisition of the six core competencies defined by the Accreditation council for Graduate Medical Education (ACGME).
METHODS: Six-core-competency-based OSCE was used as examination of the clinical performance of 192 PGY(1) residents during their 3-month internal medicine training between 2007 January and 2009 December. For each year, the reliability of the entire examination was calculated with Cronbach's alpha.
RESULTS: The reliability of six-core-competency-based OSCE was acceptable, ranging from 0.69 to 0.87 between 2007 and 2009. In comparison with baseline scores, the summary scores and core-competency subscores all showed significant increase after PGY(1) residents finished their 3-month internal medicine training program.
CONCLUSION: By using a structured development process, the authors were able to create reliable evaluation items for determining PGY(1) residents' acquisition of the ACGME core competencies.

Introduction

Competency assessment has been an obligation and an ongoing challenge for those institutions responsible for the training and certification of physicians. Such assessment has had the practical function of establishing minimal professional standards that ensure the basic fitness of future physicians. The introduction of in-training assessment using clinicians to assess the performance of residents is a task. The assessment method must provide reliable results.

The primary goals of internal medicine residency training program are to produce competent practitioners. In comparison with other conventional assessment methods, the Objective Structured Clinical Examination (OSCE) is a multidimensional practical examination of clinical performance of residents. Initially, experience with the OSCE has been somewhat limited in the United States, Canada, United Kingdom and Australia.1, 2, 3 However, this evaluation method has emerged elsewhere, in Asia, including Japan, Korea and Taiwan, as the premiere method for assessing clinical competence and uncovered some competency deficits that are missed by other methods.

Nevertheless, wide variation exists in the scoring schedules of OSCE. Recently, experts suggested that a more constructed core competency-based modification of the scoring system of OSCE is mandatory to appropriate evolution of clinical performance of residents.4, 5, 6, 7

In the present study, we had three goals: (1) to determine the reliability of the six-core-competency-based OSCE in sequential testing of PGY1 residents; (2) to compare core-competency acquisition before and after the internal medicine training program of PGY1 residents; and (3) to determine the usefulness of the information gained from PGY1 resident clinical performance.

Methods

PGY1 program

The outbreak of the severe acute respiratory syndrome (SARS) epidemic that occurred during 2003 exposed serious deficiencies in Taiwan’s medical care and public health care systems, as well as its medical education system. The Department of Health, Executive Yuan of Taiwan ROC has made efforts in promoting the “Project of Reforming Taiwan’s Medical Care and Public Healthcare System” since the spread of SARS was controlled. The reform of the medical care system aimed to provide better holistic medical treatment to people. The strategies and methods are strengthening the improvement of resident education and quality of medical care. A project titled “Postgraduate General Medical Training Program” was announced by the Department of Health in August 2003. Through this project, each doctor in his/her first year of residency (including internal medicine, family medicine, surgery, pediatrics, dermatologist, ophthalmologist, etc.) is required to fulfill 3 months of an internal medical training course, along with 36 hours of basic courses. In the past, there was no such program in Taiwan to provide general medical training for medical students after graduation. Therefore, the goal of this program is to ensure that all PGY1 residents have acquired Accreditation Council for Graduate Medical Education (ACGME) core competence in internal medical care.

OSCE setting

A comprehensive 6-station OSCE was administered to 192 PGY1 residents between 2007 January and 2009 December at Taipei Veterans General Hospital (Taipei VGH). Taipei VGH is a regional medical center that provides primary and tertiary care to active duty and retired military members, and their dependents. Taipei VGH serves as the primary teaching hospital for its internal medicine residency program. The examination was conducted on two consecutive Friday afternoons during the internal medicine training program of all PGY1 residents.

The OSCE consisted of 6 clinical problems (Table 1 ); each clinical problem consists of six core competencies defined by the ACGME (Table 2 ). The content of each clinical problem is included in Table 1. The OSCE has neither a written component nor technical skills stations, but is entirely performance-based. At some stations, standardized patients were used to mimic the clinical problems of actual patients. A faculty rater graded each PGY1 resident according to a given set of 12 predetermined items (using 2 items to evaluate each aspect of the ACGME core competencies) presented in the form of a checklist. The checklists for all 6 OSCE stations have similar items to cover ACGME core competency. The items that PGY1 presented completely were scored as “1” and those not presented were scored as “0”. All faculty raters attended serial training sessions that included extensive instructions on how to use the checklist in practice rating sessions. At each OSCE station, the raters acted as passive evaluators and were instructed not to guide or prompt the PGY1 residents.

Table 1

The content of objective structured clinical examination (OSCE) stations of postgraduate year-1 (PGY1) residents

Name of OSCE station	2007	2008	2009	Aspects of competence/skills assessed
Use of clinical information	○	○	○	PC- interviewing; physical examination
				MK- investigatory and analytic thinking
Organization and orderliness	○	○	○	P- sensitiviting to cultural, age, gender, and disability issues
				ICS- creation of therapeutic relations with patients
Patient safety and ethical issues	●	●		SBP- understanding interaction of their practice with the larger system; advocating for patient within the health care system
				PBLI- analyzing own practice for needed improvement
Creation of therapeutic relations with patients		●	●	SBP- knowledge of practice and delivery system
				PBLI- using evidence from scientific studies (EMB); using of information technology
Providing patient-centered care	●			P- respect and altruism
				ICS- listening
Counseling and educating patients and family	●	●		MK- knowledge and application of basic science
				PC- couseling and educate patients and families; preventive health service; informed decision-making
Decision-making (clinical judgment)			●	SBP- understanding interaction of their practice with the larger system; advocating for patient within the health care system
				PBLI- analyzing own practice for needed improvement
Clinical differential diagnosis			●	MK- knowledge and application of basic science
				PC- counseling and educate patients and families; preventive health service; informed decision-making
Improvement of quality of clinical care		●		ICS- listening
				P- respect and altruism
Using evidence-based practice	○			PBLI- use evidence from scientific studies (EMB); use of information technology
				SBP- knowledge of practice and delivery system
Interaction with whole medical system			●	ICS- listening
				P- respect and altruism
Reliability (Cronbach’s alpha) of OSCE	0.72	0.69	0.87

○ = the station implemented for the year; ● = the station implemented and standardized. Patients used for the year; ICS = interpersonal and communication; MK = medical knowledge; P = professionalism; PBLI = problem-based learning and improvement; PC = patient care; SBP = systems-based practice; each OSCE station assessed two domains of ACGME core competencies; each domain (skill) of ACGME competency was equally assessed twice in 6-station OSCE of every year.

Table 2

Factor analysis of six core competence-based checklists in objective structured clinical examination (OSCE)

Domain of ACGME competence and skills	Factor 1	Factor 2	Factor 3	Factor 4	Factor 5	Factor 6
Patient care skills
Interviewing	0.40
Counseling and educating patients and families	0.43
Physical examination	0.43
Preventive health service	0.55				0.31
Informed decision making	0.34
Interpersonal and communication skills
Creation of therapeutic relations with patients		0.87
Listening		0.65
Professionalism skills
Respect and altruism			0.59
Sensitivity to cultural, age, gender, and disability issues			0.80
Practice-based learning and improvement skills
Analyzing own practice for needed improvement				0.73
Using evidence from scientific studies (EMB)				0.68		0.41
Using information technology				0.71
Systems-based practice skills
Understanding interaction of their practice with the larger system					0.57
Advocate for patient within the health care system					0.68
Knowledge of practice and delivery system					0.74
Medical knowledge skills
Investigatory and analytic thinking						0.62
Knowledge and application of basic science						0.71

Data are eigen values (0–1): correlation coefficients between items assessing specific skills of each ACGME competency with each factor.

At each station, the summary scores were the sum of all the checklist items, and the six core competency subscores were the sum of specific items for each competency. When presented, all scores were translated into 100 percentages. Finally, each PGY1 resident got summary scores and core competence subscores of the 6 OSCE stations. The “borderline group method” was used to set the standard of “pass”. Each station’s “pass” score was the mean of the scores of PGY1 residents whose OSCE scores were rated “borderline”.9, 10

Analyzing the reliability of the scoring system

Preliminary studies with senior residents permitted us to identify and eliminate unreliable and ambiguous items, resulting in an instrument consisting of 12 items, with 2 items assessing each of the six core competencies (Table 2).

To evaluate the reliability of the OSCE checklist as measure of their competency, we conducted a factor analysis of the scoring system (items) of OSCE. Beginning with a principal component analysis (PCA),11, 12 we identified six components with eigen values greater than 1.0, suggesting six factors underlying the score of PGY1 OSCE (Table 2). A correlation table of scores for each station was calculated, and used to exclude stations that correlated poorly with all other stations from further analysis.

Study protocols and reliability (internal and external)

In 2007, we tested 60 PGY1 residents; in 2008, 72 and in 2009, 60. Furthermore, we assessed internal and test/retest (external) reliability of the six-core-competency-based OSCE by computing Cronbach’s alpha coefficients. Inter-rater reliability was assessed by percent agreement and kappa statistical analysis. In a subgroup of PGY1 residents (n = 70), the same clinical problems with identical OSCE setting were used to test their clinical performance before and after finishing the 3-month internal medicine training program. For this group of PGY1 residents, the difference in their pre-course and post-course OSCE scores mainly came from the sequential time-point of assessment. Actually, the raters and standardized patients were unchanged between pre-course and post-course OSCE. Accordingly, the Cronbach’s alpha coefficients of test/retest (external) reliability reflect the stability of the OSCE.

Data analysis

Data were shown as mean ± standard deviation (SD). Statistical analysis was performed using SPSS version 13.0 (SPSS Inc., Chicago, IL, USA). Analysis of variance was used to compare the means between summary OSCE scores and core-competency subscores, McNemar and Chi-square test were used to analyze the subscores and pass rates between different summary scores and core-competency subscores. Paired 2-samples t test was used to compare the performance of OSCE between pre-and post-course of 3-month PGY1 internal medicine training period. Inter-rater reliability was analyzed with Kappa statistics. Factor analysis was assessed with PCA. Spearman-Brown prophecy formula was sued to calculate the number of OSCE stations needed to reach the desired level of reliability (0.8). An α of <0.05 was accepted as statistically significant.

Results

The internal reliability of each year’s six-core-competency-based OSCE had excellent agreement (Table 1, Cronbach’s alpha in 2007 = 0.728, in 2008 = 0.69, in 2009 = 0.87). The overall 3-year internal reliability of summary OSCE scores and core-competency subscores were also acceptable (Table 3 ). In addition to professionalism aspects of core-competency, the overall 3-year inter-rater reliability (Kappa statistics) on the summary OSCE scores and core-competency subscores was also good (Table 4 ).

Table 3

Reliability (internal consistency) of OSCE scores derived from PGY1 residents

	aScore (%)	Range (%)	Reliability: (Cronbach’s alpha)
Average of total summary OSCE score	74.9 ± 12.1	41.8–81.9	0.73
Patient care subscore	83.1 ± 17.9	49.9–87.1	0.673
Medical knowledge subscore	76.8 ± 14.9	50.1–82.9	0.672
Practice-based learning and improvement subscore	82.1 ± 11.2	61.9–82.1	0.642
Interpersonal and communication skills subscore	84.1 ± 12.9	54.2–88.1	0.709
Professionalism subscore	70.2 ± 12.7	44.7–78.1	0.683
Systems-based practice subscore	76.5 ± 19.7	34.7–80.1	0.702

Data were expressed as mean ± SD.

Score = (obtained points/maximum possible points) × 100%; (n = 192).

Table 4

Inter-rater reliability of OSCE scores derived from PGY1 residents

	Percent agreement	Kappa
Average of total summary OSCE score	0.91	0.732
Patient care subscore	0.93	0.71
Medical knowledge subscore	0.88	0.74
Practice-based learning and improvement subscore	0.90	0.77
Interpersonal and communication skills subscore	0.83	0.69
Professionalism subscore	0.89	0.79
Systems-based practice subscore	0.92	0.68

Two raters for each OSCE station; (n = 192).

In Fig. 1 A, a significant difference in the performance between different aspects of core competency (F = 49.8, p < 0.05) was noted. PGY1 residents had the highest pass rate in the aspect of interpersonal and communication skills (81%), whereas the lowest pass rate was noted in the aspect of professionalism (53%). Interestingly, the corresponding data in Table 3 confirmed that the highest subscores were noted in the aspects of interpersonal and communication skills, whereas the lowest subscores were observed in the professionalism aspect.

Fig. 1

(A) The overall pass rate (pass students/total students × 100%) and (B) the degree of the improvement of core-competence-based PGY1 OSCE. PC: patient care; MK: medical knowledge; PBLI: practice-based learning and improvement; ICS: interpersonal and communication skills; P: professionalism; SBP: systems-based practice.

The eigen values (with variance accounted for presented in parentheses) of the six factors resulting from this PCA of our study were 6.28 (28%), 2.93 (39%), 2.17 (47%), 2.25 (54%), 1.4 (59%) and 1.4 (63%), respectively. Additionally, the resulting scree plot of the eigen values revealed that the leveling off to a straight horizontal line occurred after the sixth eigen value. Then, we performed a principal axis factor analysis with oblique rotation to a preselected six-factor solution. The factors that emerged from this analysis clearly reflect each of the six core competencies of residency (Table 2). Intriguingly, items from each competency loaded on their intended factor and did not load with items from different scales of competencies. In other words, the factor analysis resulted in a nearly perfectly clean structure, providing evidence that our theoretically derived items in checklist of PGY1 OSCE were distinct and evident in the reflection of core competence of their clinical performance.

In a sub-group of PGY1 residents who received both the pre-course and post-course six core-competency-based OSCE, the test/retest (external) reliability was borderline (Table 4). However, we clearly observed significantly increased OSCE scores and subscores after the internal medicine training program. Meanwhile, the pass rates were markedly improved in all aspects of core competencies in PGY1 residents (Fig. 1B). The most improved aspects were patient care and professionalism (Table 5 ). In contrast, the performance in systems-based practice was not significantly improved after the training program.

Table 5

Comparison of performance of PGY1 residents between pre-course and post-course of a 3-month internal medicine training program

	Pre-course	Post-course	Reliability: (Cronbach's alpha)
Average of total summary OSCE score	74.9 ± 12.3	79.9 ± 17.8∗	0.602
Patient care subscore	73.3 ± 15.2	89.5 ± 13.9∗∗	0.631
Medical knowledge subscore	75.9 ± 10.1	90.1 ± 16.3∗∗	0.521
Practice-based learning and improvement subscore	72.3 ± 12.6	81.4 ± 20.2∗	0.564
Interpersonal and communication skills subscore	83.2 ± 21.5	84.0 ± 18.3	0.477
Professionalism subscore	69.4 ± 10.9	72.9 ± 21.2∗∗	0.499
Systems-based practice subscore	70.9 ± 8.2	82.1 ± 12.7∗	0.621

Data were expressed as mean ± SD; Score = (obtained points/maximum possible points) × 100%.

p < 0.05 versus pre-course.

p < 0.01 (paired t test); (n = 70).

Discussion

In this study, a composite global performance score was determined for each core competency from merging separate items. This is in contrast to unclassified global assessment OSCE scores that may not provide an accurate assessment of the PGY1 resident performance.14, 15 Meanwhile, our items were developed by experienced medical education leaders and were based largely on the conceptual framework contained in the model of the clinical practice of internal medicine. It had been reported that using structured scoring guidelines can appropriately assess whether PGY1 residents have acquired core competency during training.

In Fig. 1B and Table 4, higher scores of core-competency-based OSCE than baseline were found after the training program. This result supported the discriminatory reliability of the six core competency evaluation process of PGY1 OSCE in our study.

PCA is a form of factor analysis. In our study, factor-analytic techniques were used to detect structure in the relationship between variables. With PCA, a “line” (an eigen vector) is defined in the scatter plot that illustrates maximal variance. This is also known as the “factor proportion” or the “factor load”. The square of the factor proportion gives the percentage of the variability that an element contributes to a system. After this first factor has been extracted, the next line (second eigen vector) that maximizes variances is defined. In this way, one is able to determine the contribution of each consecutive factor to the overall variability of the results.11, 12

Reliability on the aspects of professionalism and system-based practice were lower (Cronbach’s alpha = 0.31 and 0.419, respectively) than the other four aspects of core competency (Cronbach’s alpha: patient care = 0.577; medical knowledge = 0.74; interpersonal and communication skills = 0.576; problem-based learning and improvement = 0.596). Certainly, the generally low inter-rater reliability of core-competency-based OSCE indicated that raters were still not familiar with the scoring system in our study. In other words, program directors should further unify the consensus between raters about the core-competency scoring system. Actually, it has been suggested that OSCE is not a good tool to assess medical knowledge. Thus, it is essential for program directors to add chart stimulation recall, oral and multiple choice question written examinations to assess medical knowledge in the future.14, 15, 16

In our study, PGY1 residents should complete each station within 15 minutes, and a 5-minute changeover time was allowed between stations; total testing time was 2 hours. It had been established that limited testing time is another possible factor for the low inter-rater reliability. Indeed, it was difficult for raters to judge 12 items on the checklist during every 15-minute station in our study. The literature is clear that longer periods improve the reliability of OSCE. Accordingly, longer test period is mandatory for more widely focused complex clinical problem in our core-competency-based OSCE.

The significant improvement in performance on patient care was accompanied with an acceptable reliability (Cronbach’s alpha = 0.631) after finishing the internal medicine training program. Nonetheless, the insignificant improvement of professionalism was associated with relatively good reliability (Cronbach’s alpha = 0.499). The inconsistent findings in this study suggested that the professionalism competency in the checklist should be adjusted in the future. However, our study still had the following limitations. First, scores in each group have a major impact on the mean scores and standard deviations at each year level. Despite this limitation, relatively good test/retest reliability and a significant increase in scores with time spent in the training program support the reliability of core-competency-based OSCE in our study.

Second, only six examination stations were included in our OSCE; previous studies had reported that a greater number of stations would increase the reliability. If this examination were to be used for high-stakes purposes, such as certification, a reliability of at least 0.8 would be desirable. One strategy for increasing the reliability over the present level of about 0.65 is to increase the number of stations. Using the Spearman-Brown formula, it is estimated that two additional stations would be necessary using the bench model with global scoring to achieve a reliability of 0.8. In the present study, each OSCE station measured two aspects of ACGME core competency and assessed two to four specific skills. Thus, another solution to increase reliability of ACGME competency-based OSCE might be to increase the number of OSCE stations and evaluate only one specific skill for each aspect of ACGME competency in the future.

Third, variation of PGY1 residents’ OSCEs because of rater bias is a possibility. Faculty raters may have previous experience with a PGY1 resident which could introduce positive or negative bias in scoring.

Fourth, several of the competencies as defined by ACGME were found in more than one of the six major categories. For example, the concept of working with other health care professionals is incorporated in patient care, interpersonal and communication skills, and systems-based practice. In real practice, a single question for this was listed under system-based practice: “works effectively as a member or leader of the team to provide patient-focused care: understands how his or her actions affect others”. It could arguably be placed in either of the other two competencies. Actually, each of the items to be rated was intended to reflect a specific type of observable skill by the resident, not a personality trait.

Taken together, our study was characterized by a longitudinal experience with the core competency-based OSCE to evaluate internal medicine training program of PGY1 residents over 3 years. In line with previous studies, our results indicate a number of useful findings for program directors.18, 19 Notably, the overall internal reliability of our core competency-based OSCE progressively improved within 3 years (from 0.72 to 0.87). Furthermore, the internal reliability between the six core competencies was evenly distributed between 0.642 and 0.709. In contrast, the inter-rater reliability between different aspects of core competency was quite variable.