Relationships between high-stakes clinical skills exam scores and program director global competency ratings of first-year pediatric residents.

BACKGROUND: Responding to mandates from the Accreditation Council for Graduate Medical Education (ACGME) and American Osteopathic Association (AOA), residency programs have developed competency-based assessment tools. One such tool is the American College of Osteopathic Pediatricians (ACOP) program directors' annual report. High-stakes clinical skills licensing examinations, such as the Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation (COMLEX-USA Level 2-PE), also assess competency in several clinical domains.
OBJECTIVE: The purpose of this study is to investigate the relationships between program director competency ratings of first-year osteopathic residents in pediatrics and COMLEX-USA Level 2-PE scores from 2005 to 2009.
METHODS: The sample included all 94 pediatric first-year residents who took COMLEX-USA Level 2-PE and whose training was reviewed by the ACOP for approval of training between 2005 and 2009. Program director competency ratings and COMLEX-USA Level 2-PE scores (domain and component) were merged and analyzed for relationships.
RESULTS: Biomedical/biomechanical domain scores were positively correlated with overall program director competency ratings. Humanistic domain scores were not significantly correlated with overall program director competency ratings, but did show moderate correlation with ratings for interpersonal and communication skills. The six ACGME or seven AOA competencies assessed empirically by the ACOP program directors' annual report could not be recovered by principal component analysis; instead, three factors were identified, accounting for 86% of the variance between competency ratings. DISCUSSION: A few significant correlations were noted between COMLEX-USA Level 2-PE scores and program director competency ratings. Exploring relationships between different clinical skills assessments is inherently difficult because of the heterogeneity of tools used and overlap of constructs within the AOA and ACGME core competencies.

Like residency training programs accredited by the Accreditation Council for Graduate Medical Education (ACGME), American Osteopathic Association (AOA) accredited residency programs are required to assess each of the seven core osteopathic competencies (osteopathic philosophy and osteopathic manipulative medicine, medical knowledge, patient care, interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice) for all residents in each year of training (1, 2). These competencies are comparable to the six core competencies of the ACGME (2).

Program director ratings and annual evaluations have been used to evaluate resident performance (3–8). Since 2005 osteopathic pediatric program directors have been rating resident performance using the American College of Osteopathic Pediatricians (ACOP) program directors’ annual report. This competency-based global assessment tool is completed annually and used to evaluate resident progress throughout training. Independent competency ratings are strongly related to each other (9).

The Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation (COMLEX-USA Level 2-PE) is a high-stakes licensure exam based on standardized patients (SPs) and used to assess clinical skills competency of osteopathic medical students entering graduate medical education (10). For this exam, examinees rotate through 12 stations in which they evaluate SPs who have been trained to simulate a variety of typical clinical presentations. The exam specifically measures a variety of clinical skills, including doctor-patient communication, interpersonal skills and professionalism; medical history taking and physical examination skills; osteopathic principles and manipulative treatment; and written communication skills. COMLEX-USA Level 2-PE clinical skill component scores have good internal consistency (10–12). Furthermore, previous study of first-year osteopathic pediatric residents showed overall program director competency ratings were positively correlated with COMLEX-USA cognitive exam scores (Levels 1–3) as well as biomedical/biomechanical domain scores of the clinical skills exam, COMLEX-USA Level 2-PE (9).

COMLEX-USA Level 2-PE measures fundamental clinical skill competence of medical students in the context of SP encounters. The ACOP program directors’ annual report measures each of the seven competencies of pediatric residents in the context of overall performance throughout each year of residency training. Both assessment tools seem to measure common clinical skills, but in different settings, formats and times during medical training. Do they truly measure similar skills? For instance, both tools measure an examinee's ability to communicate with patients; therefore, one would expect communication skills ratings of the ACOP program directors’ annual report and COMLEX-USA Level 2-PE to be highly correlated. The purpose of this study is to investigate relationships between COMLEX-USA Level 2-PE clinical skill scores and competency-based global assessments of pediatric residents as reported by osteopathic residency program directors. In particular, we investigate the extent to which domain and individual component scores of COMLEX-USA Level 2-PE are related to the individual program director competency ratings of medical knowledge, patient care, interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice of first-year pediatric residents.

Methods

Measures

COMLEX-USA Level 2-PE clinical skill scores

COMLEX-USA Level 2-PE is a 12-station SP-based high-stakes clinical skills examination used for licensing osteopathic physicians, where examinees interview and examine SPs who are trained to simulate different medical complaints (10). Each station provides 14 minutes for an examinee to interview the SP, followed by a nine-minute post-encounter period of time to complete the written patient note (subjective/objective/assessment/plan – SOAP note) summarizing the encounter. Cases vary according to presenting symptoms (respiratory, cardiovascular, neuromusculoskeletal, gastrointestinal and other symptoms), age, gender and race/ethnicity. Cases also vary in clinical complaints that could be acute, chronic or provide opportunities for health promotion or disease prevention.

Examinees, usually medical students toward the end of their third year or beginning of their fourth year of training, must pass two domains (humanistic and biomedical/biomechanical) to pass the examination. Pass/fail decisions based on the standard of minimal competence for entry into graduate medical education (10) are operationalized by each domain's cut score. Examinee performance is scored using several clinical skill measures within each domain (Table 1).

Table 1

COMLEX-USA Level 2-PE domains and components

Domain	Component
Humanistic	Global patient assessment tool (including assessment of professionalism, respect, eliciting information, giving information, empathy and listening)
Biomedical/ biomechanical	Data gathering (history and physical)	Written patient note (SOAP note)	OMTa

Although osteopathic principles are assessed throughout the examination, OMT is specifically scored in 25–40% of all encounters.

The humanistic domain comprises scores to assess the examinee's listening skills, respectfulness, empathy, professionalism, ability to elicit information and ability to provide information. Each of these components is scored by the SP using the global patient assessment tool, a holistic Likert-type scoring instrument (10). As a measure of reliability, the generalizability coefficients for humanistic domain scores are comparable to those of other high-stakes clinical skills examinations used for licensure (10, 11).

The biomedical/biomechanical domain comprises three weighted component scores: data gathering (ability to obtain a medical history and perform a physical exam); written patient note (ability to complete a written communication task and synthesize information, develop a differential diagnosis and formulate a diagnostic and treatment plan); and osteopathic manipulative treatment (OMT: ability to integrate osteopathic principles and use OMT) (10). Data gathering scores are measured on a percentage metric based on the number of history questions and physical examination maneuvers correctly performed during the clinical encounter. Written patient notes are scored by trained physician examiners using a Likert-type holistic instrument. OMT skills, specifically scored in 25–40% of encounters, are scored by trained physician examiners using a different Likert-type holistic tool. Generalizability coefficients for biomedical/biomechanical domain scores are comparable to those of other high-stakes clinical skills examinations (10).

Component scores for patient encounters are averaged across all scored stations to compute average performance during the testing day. After scores are equated for rater leniency or stringency, a pass/fail decision is rendered for each candidate. Pass/fail determinations are based on standards applied to the cut scores of each domain using an examinee-centered standard setting method (10). Only pass/fail scores, not component scores, are reported to examinees, school deans and residency program directors.

ACOP program directors’ annual report: competency ratings of residents

Osteopathic pediatric program directors complete the ACOP program directors’ annual report for each resident at the end of each year of training. This global assessment is used to address performance related to resident scholarly and research activity, continuity patient panel size, overall progress and skill assessment (3). For each resident, the program director assigns a rating to each skill, classified by competency, using a nine-point rating scale ranging from 1 (lowest) to 9 (highest). Program director ratings are recorded by one director from each residency program, thus inter-rater reliability measures are not available. The specific skill assessments for each of the seven osteopathic medical competencies are presented in Table 2. The reports are then forwarded to the ACOP Graduate Medical Education Committee for review and certification of training. ACOP program directors’ annual report ratings have been shown to be reliable, and independent competency ratings are strongly related to each other (9).

Table 2

Competency-based skill assessment of residents as reported using ACOP program directors’ annual report

Competency	Skill assessment
Osteopathic philosophy and OMT	1. Demonstrates competency in the understanding and application of OMT where appropriate.
	2. Integrates osteopathic concepts into the medical care provided to patients as appropriate.
Medical knowledge	1. Demonstrates competency in the application of clinical medicine to patient care.
	2. Knows and applies the foundations of clinical and behavioral medicine.
	3. Demonstrates knowledge of accepted standards of care.
	4. Remains current with new developments in medicine.
	5. Participates in lifelong learning activities.
Patient care	1. Gathers accurate, essential information from all sources.
	2. Demonstrates competency in the performance of diagnostic and treatment procedures.
	3. Provides healthcare services that include preventive medicine and health promotion.
Interpersonal and communication skills	1. Demonstrates effectiveness in developing appropriate doctor-patient relationships.
	2. Exhibits effective listening, written and oral communication skills.
Professionalism	1. Demonstrates respect for patients/families and acts as their advocate.
	2. Adheres to ethical principles in the practice of medicine.
	3. Is sensitive to cultural diversity, i.e. religion, age, gender, sexual orientation and disabilities.
	4. Is cognizant of his/her own physical and mental health in order to care effectively for patients.
Practice-based learning and improvement	1. Treats patients in a manner consistent with current and evidence-based information.
	2. Performs self-evaluations of clinical practice.
	3. Understands research methods, medical informatics and the application of technology.
Systems-based practice	1. Understands national and local health care delivery systems.
	2. Advocates for quality health care on behalf of patients.
	3. Practices cost effective medicine.

Design

Data collected from all pediatric osteopathic residents reviewed by the ACOP Graduate Medical Education Committee from 2005 through 2009 were compiled from the ACOP program directors’ annual reports. Data include competency-based skill assessment ratings for each resident. Program director ratings are reviewed as a global measure of resident performance by the ACOP Graduate Medical Education Committee. Therefore, skill assessment ratings within each competency were summed together into a total competency score for each of the seven core competencies, labeled as program director competency ratings for this study.

A linking variable unknown to the researchers was created to match to the National Board of Osteopathic Medical Examiners (NBOME) database to extract all COMLEX-USA domain and component scores. Results of this study are reported in aggregate form to maintain confidentiality. Institutional Review Board approval was received to analyze these data for this study.

Sample

The initial sample consists of 134 residents, representing all first-year pediatric residents who were reviewed between 2005 and 2009. The sample includes residents from 19 pediatric training institutions across the United States.

Of the total sample, 94 first-year residents (69%) took the COMLEX-USA Level 2-PE examination, as a minority of residents entered a training program prior to taking this exam. Most of the 94 first-year residents completed COMLEX-USA Level 2-PE toward the beginning of their fourth year in medical school.

Analysis

For each subject (N=94), correlations between COMLEX-USA Level 2-PE domain and component scores and program director scores for each competency were analyzed for statistical comparison using SAS version 9.1. The extent to which scores on COMLEX-USA Level 2-PE were related to the program director competency ratings of medical knowledge, patient care, interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice were measured in terms of linear dependency or Pearson correlations (r). Overall domain scores (biomedical/biomechanical and humanistic) and individual component scores (OMT, SOAP note, data gathering and the six skills assessed within the humanistic domain) were analyzed for relatedness in conjunction with the program director competency ratings. In addition, because the program directors’ global assessment tool has not been evaluated for internal consistency, a principal component analysis was performed. This analysis was used to account for the variability of the individual competency rating scores, and then used as a predictor or criterion variable for the subsequent analysis.

Results

COMLEX-USA Level 2-PE biomedical/biomechanical and humanistic domain clinical skill component scores were correlated with program director competency ratings to investigate their relationships (Table 3 and 4). A significant correlation was shown between overall biomedical/biomechanical scores and program director competency ratings (r=0.29, p<.01). In addition, within the biomedical/biomechanical domain, SOAP note scores showed significant correlations with all competency ratings (r=0.22–0.29, p<.05), and data gathering scores showed significant correlations with program director competency ratings for patient care (r=0.23, p<.05) and interpersonal and communication skills (r=0.23, p<.05).

Table 3

Pearson correlation (r) matrix of COMLEX-USA Level 2-PE biomedical/biomechanical domain scores and program director competency ratings of first-year pediatric residents (N=94)

	Program director competency ratings
	Overall	OMT	MK	PC	IC	P	PBL	SBP
Biomedical/ biomechanical domain
Overall	.29b	.22a	.25a	.28b	.34b	.30b	.22a	.22a
OMT	.14	.08	.13	.11	.19	.15	.10	.12
SOAP note	.29b	.28b	.23a	.29b	.27b	.27b	.28b	.22a
Data gathering	.18	.14	.14	.23a	.23a	.19	.12	.12

Note: MK – medical knowledge; PC – patient care; IC – interpersonal and communication skills; P – professionalism; PBLI – practice-based learning and improvement; SBP – systems-based practice.

Significant at p<0.05.

Significant at p<0.01.

Table 4

Pearson correlation (r) matrix of COMLEX-USA Level 2-PE humanistic domain scores and program director competency ratings of first-year pediatric residents (N=94)

	Program director competency ratings
	Overall	OMT	MK	PC	IC	P	PBLI	SBP
Humanistic domain
Overall	.13	.10	.07	.16	.24a	.19	.09	.05
EI	.19	.20	.19	.18	.18	.16	.19	.08
LI	.04	.05	.01	.05	.11	.06	.05	-.01
GI	.12	.12	.07	.11	.16	.15	.14	.08
Resp	.12	.07	.11	.14	.20	.15	.05	.06
EM	.17	.12	.08	.19	.27b	.20	.16	.11
PR	.23a	.18	.18	.22a	.26a	.22a	.21a	.18

Note: EI – eliciting information; LI – listening; GI – giving information; Resp, respect; EM – empathy; PR – professionalism; MK – medical knowledge; PC – patient care; IC – interpersonal and communication skills; P – professionalism; PBLI – practice-based learning and improvement; SBP – systems-based practice.

Significant at p<0.05.

Significant at p<0.01.

COMLEX-USA Level 2-PE humanistic domain scores significantly correlated with program director competency ratings of interpersonal and communication skills (r=0.24, p<.05). Within the humanistic domain, professionalism showed significant correlations with competency ratings for patient care, interpersonal and communication skills, professionalism, practice-based learning and improvement, and overall competency (r=0.21–0.26, p<.01). Empathy also showed significant correlations with program director competency ratings of interpersonal and communication skills (r=0.27, p<.01).

Due to the few significant correlations (between individual component scores within the humanistic domain and program director competency ratings), a principal component analysis was performed on the competency rating data. Principal component analysis assumptions were tested to determine if the 22 items on the ACOP program directors’ annual report could be optimally combined into one or more smaller composites. The assumption that competency items are uncorrelated was rejected by Bartlett's test of sphericity, χ2(231)=4729.83, p<.001. A Kaiser-Meyer-Olkin measure of sampling adequacy, 0.95, was sufficiently large to proceed with the analysis. Cronbach's coefficient alpha showed sufficient internal consistency among competency items (0.982). Three principal components explained 85.9% of the variance between the 22 competency items, with explained variance ranging from 4.7% to 73.6%. The principal component analysis was followed by varimax rotation of the factor matrix. The pattern of item regression on factors lacked simplicity for several items, indicating the three factors underlying competency items are correlated.

Principal component analysis was also completed for each of the competency categories individually (medical knowledge, OMT, patient care, interpersonal/communication, professionalism, practice-based learning and systems-based practice), which yielded a one-component solution for each competency, accounting for 90–99% of total variance explained. Of the 22 skill assessments of the ACOP program directors’ annual report, analysis identified three underlying factors, which we named as existing competency classifications: osteopathic principles and treatment and medical knowledge; systems-based practice and practice-based learning and improvement; and patient care, interpersonal and communication skills, and professionalism. The six ACGME or seven AOA general competencies assessed empirically by program director ratings could not be recovered with a principal component analysis. Instead, the three factors were identified to explain 86% of the variance between competency ratings.

Discussion

Residency program directors routinely evaluate residents of their training programs. Investigating how these ratings correspond to other measures of clinical competence helps to justify resident evaluation by the program director. Overall program director competency ratings have been shown to be positively correlated with the biomedical/biomechanical domain scores of the clinical skills exam, COMLEX-USA Level 2-PE (9). We demonstrate in this study that to some extent domain and individual component scores of COMLEX-USA Level 2-PE are related to the individual program director competency ratings of medical knowledge, patient care, interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice of first-year pediatric residents.

Clinical skills examination performance, as measured by the biomedical/biomechanical domain scores, correlated positively with overall program director competency ratings, exhibiting a significant amount of convergent validity. The constructs measured within this domain of the COMLEX-USA Level 2-PE and the ACOP program director reports are similar, and appear to be measuring similar concepts. When completing global assessment ratings, such as the ACOP program director reports, assessors likely elicit input from many sources and faculty. This information, albeit less reliable than first-hand accounts, is likely to focus on residents’ cognitive abilities and medical knowledge, consistent with the clinical skills targeted in the biomedical/biomechanical domain of COMLEX-USA Level 2-PE.

Humanistic domain scores also correlated positively with program director ratings for interpersonal and communication skills. Program directors are often in a unique position of directly observing morning reports and resident presentations, whereby medical knowledge and communication skills are directly evaluated. Because these skills are directly observed by program directors, this may account for the positive correlations between their competency ratings (particularly professionalism and communication skills) and COMLEX-USA Level 2-PE scores.

Although significant correlations were found between biomedical/biomechanical domain scores and program director competency ratings, few significant correlations between component scores within the humanistic domain and competency ratings were identified. The constructs measured within the humanistic domain may be dissimilar to those measured within the ACOP program director reports, as suggested by the low number of significant correlations. Alternatively, perhaps the tools are inherently different and measuring something unique (divergent validity). Previous studies recognize the complexity of interpreting correlations between medical student and resident performance given the heterogeneity of assessment tools being used at different levels of training (13, 14), and there needs to be a more consistent approach to the use of tools for assessing external validity. Perhaps, as another explanation, both tools are global assessments, and neither is measuring discrete or unique clinical skills. Previous studies demonstrate that current measurement tools poorly discriminate competencies from one another, and competencies cannot be independently assessed (15). In addition, the tool used to measure the humanistic domain in COMLEX-USA Level 2-PE is generalizable and more consistent with a global assessment of humanistic clinical skills (11). Our data suggest that the ACOP program directors’ annual reports are measuring overall resident competence, and one should use caution when interpreting scores of individual skill components.

In a previous study, factor analysis indicated that program directors from a variety of subspecialties assess competence in two domains: medical knowledge and interpersonal skills (8). Another study identified four competency clusters: professionalism, patient care, medical knowledge and communication (16). Using principal component analysis, we identified three factors, rather than the six ACGME or seven AOA general competencies. This may confound our results and help to explain why more correlations were not found between program director competency ratings and COMLEX-USA Level 2-PE component scores.

This study has a few limitations. First, although correlations found in the study were statistically significant, many were weak and may lack practical significance. Further research, such as addressing construct validity of the tools and focus group discussions, may help to elucidate the practical implications of the significant findings. Second, the sample in this study represents all first-year pediatric residents reviewed by the ACOP Graduate Medical Education Committee between 2005 and 2009 who completed the COMLEX-USA Level 2-PE; therefore, results are limited to this sample and may not be applicable to residents of other specialties. The study excluded many osteopathic interns who train in traditionally allopathic programs, where program directors for this group of residents did not complete the ACOP program directors’ annual report. A total of 691 osteopathic physicians were enrolled in ACGME-approved pediatric residency programs in 2009–2010, representing 8.5% of all pediatric residents in allopathic training programs (17), and 190 allopathic pediatric residency positions were filled through the National Resident Matching Program in 2009 (18). This suggests that a large number of osteopathic pediatric interns, whose training is certified by the ACGME and not by the AOA or ACOP, were not available for this study. Future study could involve assessing all osteopathic pediatric residents by creating a common assessment tool used by both AOA- and ACGME-accredited residency program directors.

All residency program directors likely utilize some form of global resident assessment, but these vary from institution to institution. The ACOP program directors’ annual report is a common global assessment tool used by all osteopathic pediatric program directors across all AOA-accredited residency programs. Although findings of this study apply to a limited sample of residents, results suggest that global competency-based resident assessments do correlate with high-stakes SP-based clinical skills assessments to some extent. This information may be helpful for developing future research protocols for assessing clinical skills of residents of ACGME-accredited programs and residents of disciplines other than pediatrics.

Conclusions

Results of this study support the view that program director global assessments, like those used by most AOA and ACGME-accredited residency training programs, correlate with high-stakes clinical skills examination performance. Results specifically supported three major outcomes. First, biomedical/biomechanical domain and SOAP note component scores correlated positively with overall program director competency ratings. Second, humanistic domain scores correlated with program director competency ratings for interpersonal and communication skills. Third, three competency assessment factors were identified using principal component analysis, rather than the six ACGME or seven AOA general competencies. Exploring relationships between different clinical skills assessments, such as COMLEX-USA Level 2-PE and program director competency ratings, is inherently difficult because of the heterogeneity of tools used. Future work in utilizing comparable assessment tools would help to elucidate some of the findings of this study.