Impact of clinical performance examination on incoming interns' clinical competency in differential diagnosis of headache.

BACKGROUND: In Korea, clinical performance examination (CPX) has been included in license examination for medical doctors since 2009 in order to improve clinical performance of medical students. This study aimed to evaluate the contribution of CPX to medical education.
METHODS: Clinical competency in the differential diagnosis of secondary headache was compared between the incoming interns in 2009 unexposed to CPX and the incoming interns in 2010 exposed to CPX, using the data of patients who visited the emergency department due to headache (181 patients seen by 60 CPX non-exposed interns and 150 patients seen by 50 CPX-exposed interns). We obtained the data by reviewing electronic medical records and nominal lists of doctors. Clinical competency was assessed by sensitivity and specificity between the diagnostic impression by interns and the final diagnosis. The association between CPX exposure and clinical competency in secondary headache diagnosis was evaluated using multiple logistic regression analysis.
RESULTS: When we assessed clinical competency on the basis of all listed diagnostic impressions, sensitivity and specificity were 67.9% and 80.0%, respectively, for headaches seen by CPX-exposed interns, and 51.7%, and 71.7%, respectively, for headaches seen by CPX non-exposed interns. Multivariable adjusted logistic regression analysis showed exposure to CPX was not associated with increased competency for identifying secondary headache.
CONCLUSION: Exposure to CPX as a part of the medical license examination was not effective for the improvement of clinical competency of interns in identifying secondary headache.

INTRODUCTION

Headache is one of the most common symptoms with great disease burden worldwide and thus, primary care physicians frequently encounter patients with headache in daily practice.1,2) Headache develops from various causes, some of which pertain to fatal secondary headache.3) Therefore, it is very important for physicians to accurately identify secondary headache,3,4,5) which raises the importance of training medical students to make an accurate differential diagnosis of headache.6)

In Korea, the clinical performance examination (CPX) was included in the license examination for qualified medical doctors in 2009 for the first time in order to improve clinical competence of medical students.7) Accordingly, incoming interns in 2010 should have passed CPX as an essential step to acquire a medical license.7,8) CPX is a test that requires great change in the medical education system.9) Therefore, it seems necessary to evaluate whether the introduction of CPX to the Korean medical education system has contributed to the improvement of clinicians' quality of performance as expected.

There have been studies evaluating the impact of CPX on the improvement of interviewing skills of medical students. Han et al.10) reported a significant correlation between CPX score and patient-doctor relationship (r = 0.707). Jang et al.11) also showed a positive correlation between CPX score and the attitude of interviewing with patients (r = 0.191). However, no study has evaluated the impact of CPX on the clinician's diagnostic accuracy of diseases.

In this regard, we aimed to assess the impact of CPX on incoming interns' clinical competency in differential diagnosis of headache. Headache has been included in the list of essential items of CPX in license examinations for qualified Korean medical doctors from the beginning. We hypothesized that interns who have been exposed to CPX (CPX-exposed interns) would show higher competency compared with interns who have not been exposed (CPX non-exposed interns).

METHODS

1. Study Subjects

In order to evaluate the impact of CPX, we compared the clinical performance of identifying secondary headache using the data of patients with headache who visited emergency department (ED) of a tertiary hospital located in Seoul, Korea between March 1st, 2009 and January 31st, 2010 (seen by 69 CPX non-exposed interns), and between March 1st, 2010 and January 31st, 2011 (seen by 60 CPX-exposed interns). All medical records were reviewed by one of the authors.

Initially, we reviewed the electronic medical records of 542 headache patients (290 patients seen by CPX non-exposed interns and 252 patients seen by CPX-exposed interns), respectively. Among the 542 patients, a total of 211 were excluded from the study for the following reasons: headache associated with trauma (18 patients), having discharged from the ED on his/her own (1 patient), medical records lacking information on intern's differential diagnosis (57 patients), unavailable information about final diagnosis due to not showing at recommended follow-up visit (135 patients). Thus, 331 headache cases (181 enrolled in 2009 and 150 enrolled in 2010) seen by 110 interns (60 in 2009 and 50 in 2010) were finally included in our study. This study was approved by the institutional review board of Samsung Medical Center.

2. Study Variables

We collected data on the demographic and clinical characteristics of patients through the review of electronic medical records: age, sex, time of ED visit, characteristics of headache, presence of red flag signs, past medical history, name of duty intern, list of diagnostic impressions for headache given by intern, and final diagnosis for headache. In addition, we collected the working information of duty doctor for each patient: occurrence of patient hand-off, duration of clinical experience of duty intern, and on-duty time of duty intern (more than 16 hours or not).

We collected demographic characteristics of interns by reviewing electronic medical records and the nominal lists of doctors enrolled in the residency program, including sex, exposure to CPX, and enrollment into residency program after completing internship training.

We identified the presence of red flag signs of headache according to the Scottish Intercollegiate Guidelines Network guidelines on diagnosis and management of headache in adults as follows:12) onset of new headache after age ≥50, change in headache frequency or characteristics, headache worsened by postural change or Valsalva's maneuver, neck stiffness, focal neurologic deficit, associated systemic symptoms, previous history of secondary headache or cancer, having risk factors for cerebral venous sinus thrombosis, jaw claudication, and being infected by human immunodeficiency virus. Focal neurologic deficit was defined as the presence of the following features (≥1); mental status change, disorientation, abnormal findings in cranial nerve examination, abnormal findings in motor power or sensory examination, increase (>grade 2) or decrease (

We categorized the final diagnosis of headache into two groups (primary or secondary) according to International Headache Society (IHS) classification.13) Secondary headache was defined as a headache related with infection, neoplasm, vascular disease, cranial nerve pathology, psychiatric cause, musculoskeletal cause, iatrogenic, and other ill-defined causes. Remaining types of headache were categorized as a primary headache.

We also categorized diagnostic impression by interns into two groups (primary headache or secondary headache) according to IHS classification, not only on the basis of primary diagnostic impression but also on the basis of all listed diagnostic impressions. On the basis of primary diagnostic impression, we classified the diagnostic impression as secondary headache when the diagnostic impression in advance of others was ascertained to be on the lists of secondary headaches of IHS classification. On the basis of all listed diagnostic impressions, we classified the diagnostic impression as secondary headache when any of the listed diagnostic impressions was ascertained to be on the lists of secondary headaches of IHS classification.

3. Statistical Analysis

We compared characteristics between the patients seen by CPX non-exposed interns and the patients seen by CPX-exposed interns by chi-square test and t-test. We also compared the characteristics between the CPX non-exposed interns and the CPX-exposed interns.

In order to assess clinical competency, we calculated sensitivity, specificity, positive predictive value, and negative predictive value for identifying secondary headache. We repeatedly assessed the statistics based on both the primary diagnostic impression and all listed diagnostic impressions.

We defined sensitivity as the proportion of suspected secondary headache cases over finally-diagnosed secondary headache cases, and specificity as a proportion of suspected primary headache cases over finally-diagnosed primary headache cases.14) Positive predictive value was defined as a proportion of finally-diagnosed secondary headache cases over suspected secondary headache cases.14) Negative predictive value was defined as a proportion of finally-diagnosed primary headache cases over suspected primary headache cases.14)

In order to evaluate the association between CPX exposure and clinical competency in secondary headache diagnosis, we conducted multiple logistic regression analysis with an adjustment for provable covariates: sex of intern,15) patient handoff,16) enrollment into residency program after completing internship training, duration of clinical experience of duty intern, duration of working at ED, sex of patient, age of patient, patient encounter by intern who was on duty for more than 16 hours,17) presence of red flag sign (≥1), first-onset headache at the age of 50 or older, abrupt-onset headache, presence of focal neurologic deficits, headache aggravated by postural change or by Valsalva' s maneuver, presence of neck stiffness, presence of systemic symptoms (fever, dizziness, and general weakness), and history of previous secondary headache. Covariates related with clinical characteristics of patient were selected because they have been commonly considered to be significant clinical factors suggesting secondary headache.12)

We conducted all the analyses under the assumption that all measurements were independent of each other. The P-value of significance level was set at 0.05 for all statistical assessment. All the analyses were performed using SAS ver. 9.3 (SAS Institute Inc., Cary, NC, USA).

RESULTS

Table 1 shows general characteristics of interns. The mean number ± SD of headache patients per one intern was 3.02 ± 1.97 for CPX non-exposed interns and 3.00 ± 2.14 for CPX-exposed interns. The proportion of male interns and the interns who were enrolled in a residency program after completing internship training did not differ between the two groups.

Table 1

General characteristics of interns

Values are presented as number (%) or mean ± SD.

CPX: clinical performance examination.

*Obtained by t-test for means and chi-square test for proportions.

In Table 2, demographic and clinical characteristics of patients and related working information of the duty doctor were presented. We compared those characteristics between patients seen by CPX non-exposed interns and CPX-exposed interns. Males occupied a higher proportion among patients seen by CPX-exposed interns (46.0%) than among patients seen by CPX nonexposed interns (35.3%) with borderline statistical significance. The proportion of patients experiencing hand-off of duty intern was higher among cases (10.7%) seen by CPX-exposed interns compared to that among cases (5%) seen by CPX non-exposed interns (P = 0.05). The proportion of patients seen by duty interns with longer clinical experience in internship training (≥7 months) was higher among cases seen by CPX-exposed interns (58.0%) compared to that in patients seen by CPX non-exposed interns (44.8%) (P = 0.02). For other characteristics, there was no difference between the patients seen by CPX non-exposed interns and patients seen by CPX-exposed interns.

Table 2

General and clinical characteristics of headache cases

Values are presented as mean ± SD or number (%).

CPX: clinical performance examination, ED: emergency department.

*Obtained by t-test, chi-square test, or Fisher's exact test.

When we repeated the comparison in patients who were finally excluded from the present study, no significant difference was found between the patients seen by CPX-exposed interns and patients seen by CPX non-exposed interns for all selected characteristics.

Table 3 shows a list of ascertained final diagnoses for headaches among patients. In both groups, more than half of cases were ascertained as primary headache, and migraine was the most prevalent diagnosis. The proportion of patients with each ascertained diagnosis did not show a significant difference between both groups (P > 0.05 by chi-square test).

Table 3

Ascertained final diagnosis for headache cases

Values are presented as number (%).

CPX: clinical performance examination, CNS: central nervous system.

*P-value assessed by chi-square test for the distribution of the final diagnosis between two groups of interns > 0.05.

Table 4 shows the estimated accuracy of differential diagnosis for headache on the basis of primary diagnostic impression. All the statistics assessing diagnostic accuracy tended to be slightly higher for the headache cases seen by CPX non-exposed interns than that for the headache cases seen by CPX-exposed interns.

Table 4

Estimated accuracy* of differential diagnosis for headache on the basis of primary diagnostic impression

Values are presented as number (%) or %.

CPX: clinical performance examinations.

*Estimated by sensitivity, specificity, positive predictive value, and negative predictive value.

When we repeated the same analysis for estimating the diagnostic accuracy for headaches on the basis of all listed diagnostic impressions, we observed similar findings (Table 5).

Table 5

Estimated accuracy* of differential diagnosis for headache on the basis of all listed diagnostic impressions

Values are presented as number (%) or %.

CPX: clinical performance examinations.

*Estimated by sensitivity, specificity, positive predictive value, and negative predictive value.

Table 6 shows the multivariable-adjusted associations between CPX exposure and accurate identification of secondary headache with an adjustment for various characteristics of patients and interns. Exposure to CPX was not associated with an improvement in correct identification of secondary headache regardless of whether the estimation was conducted on the basis of primary diagnostic impression or all listed diagnostic impressions.

Table 6

Factors associated with diagnostic values of duty doctors

CI: confidence interval, CPX: clinical performance examination.

*Estimated by multiple logistic regression model in which all the variables in the above table were put.

DISCUSSION

The present study is the first Korean study to assess the impact of CPX on incoming interns' clinical competency by comparing the diagnostic accuracy of identifying secondary headache between CPX-exposed interns and CPX non-exposed interns. Although we hypothesized that CPX-exposed interns would show higher competency than CPX non-exposed interns, findings of our study did not support our hypothesis.

Regarding the null effect of CPX on the improvement of clinical competency as seen in the present study, several possible reasons exist. Firstly, the interns who were selected for the present study had taken CPX for the first time in Korea with very scarce experience with CPX. Thus, their clinical competency might have been barely influenced by CPX. Secondly, the evaluation of clinical competency of interns on the basis of headache diagnosis might have resulted in underestimation of the impact of CPX. A wide range of clinical problems should have been evaluated in CPX8) in order to assess clinical competency more accurately, given that clinical performance consists not only of diagnostic accuracy but also procedure skills, patient education, and communication skills.7) Thirdly, the impact of CPX might have been masked by other unmeasured factors that could have greater influence on clinical competency of individual interns.16,17,18,19,20,21)

Examinations testing clinical competency such as objective structured clinical examination (OSCE) or CPX have been adopted not only to the Korean medical license examination (KMLE) but also to board certifying examinations of plastic surgeons, neurologists, and family physicians. There were studies that provided evidence for positive aspects of the examinations testing clinical competency. In board examinations of plastic surgeons, there was a weak association between written test scores and OSCE scores (r = 0.32).22) In board examinations of neurologists, similar findings were revealed.23) A study using a questionnaire survey reported that levels of satisfaction regarding adopting of CPX into the KMLE were high for physician faculties, nurses, patients, and examinees.9)

Meanwhile, a criticism against the United States Medical Licensing Examination step 2 Clinical Skills (USMLE step 2CS), one type of CPX, has been raised24) such as inappropriately high rate of success (up to 97%),25) lack of adequate supervision or feedback,24) and low cost-effectiveness.24) A study by Clauser et al.26) showed that significant inconsistency may exist regarding the assessment of examinees in the USMLE step 2 CS. In addition, the USMLE step 2 score reflected residency directors' rating for interns less accurately than undergraduate grades did.27) Therefore, further studies evaluating the effect of CPX on clinical competency of medical students and doctors need to be conducted.

Several factors have been suggested to have an association with clinical skill examination scores or clinical performance.16,17,18,19,20,21) Ogunyemi and Taylor-Harris18) reported that undergraduate written test scores, evaluation grade by faculty, and age of examinee were significantly correlated with scores of the USMLE step 2, while sex and race were not. In another study in Canada, clinical experience, case volume, and socioeconomic factors were related with physicians' ability to solve acute health problems.19) Reduced workload has also been found to contribute to completeness of residents' discharge summaries,20) which provided a basis for the Institute of Medicine's recommendation that restricts duty hours of a resident to less than 16 hours.17) However, we could not find factors that were consistently associated with clinical competency in identifying secondary headache.

The present study has several limitations. Firstly, we collected data by reviewing the medical record, which may have led to biased results. Secondly, one of the authors who knew the hypothesis underlying this study performed data collection. This un-blinded method of data collection may have resulted in overestimation of the beneficial impact of CPX. However, this seems unlikely to have occurred, given that rather lower diagnostic accuracy was observed for CPX-exposed interns. Thirdly, selection bias may have occurred because too many cases were excluded from the final analysis and a higher proportion of patients with primary headache who need no further evaluation or follow up might have been excluded. However, we do not believe this distorted our findings, because no significant difference was found between the patients seen by CPX-exposed interns and the patients seen by CPX non-exposed interns for all selected characteristics among the excluded cases. Fourthly, it may not be easy to generalize the findings from the present study because our study focused on one specific clinical entity, headache. Fifthly, although we made an adjustment for the enrollment of studied individual interns into residency program after completing internship training, duration of clinical experience of duty intern, and duration of working at ED, we could not take account for other aspects of individual professional competency of interns including KMLE score, performance level in medical school, or other unmeasured individual clinical abilities due to a lack of information. In addition, as we could not control the number of patients seen by each intern equally, the competency level of individual interns might also have influence the study results. These limitations may have resulted in under- or over-estimation of the impact of the Clinical Performance Examination on incoming interns' clinical competency in differential diagnosis of headache.

In conclusion, this study on incoming interns found that exposure to CPX as a part of the KMLE was not effective for the improvement of diagnostic competency for secondary headache. However, given the limitations of our study, further evaluations of the long-term effect of CPX over more diverse clinical entities are needed.