Clinical skills assessment of procedural and advanced communication skills: performance expectations of residency program directors.

BACKGROUND: High stakes medical licensing programs are planning to augment and adapt current examinations to be relevant for a two-decision point model for licensure: entry into supervised practice and entry into unsupervised practice. Therefore, identifying which skills should be assessed at each decision point is critical for informing examination development, and gathering input from residency program directors is important.
METHODS: Using data from previously developed surveys and expert panels, a web-delivered survey was distributed to 3,443 residency program directors. For each of the 28 procedural and 18 advanced communication skills, program directors were asked which clinical skills should be assessed, by whom, when, and how. Descriptive statistics were collected, and Intraclass Correlations (ICC) were conducted to determine consistency across different specialties.
RESULTS: Among 347 respondents, program directors reported that all advanced communication and some procedural tasks are important to assess. The following procedures were considered 'important' or 'extremely important' to assess: sterile technique (93.8%), advanced cardiovascular life support (ACLS) (91.1%), basic life support (BLS) (90.0%), interpretation of electrocardiogram (89.4%) and blood gas (88.7%). Program directors reported that most clinical skills should be assessed at the end of the first year of residency (or later) and not before graduation from medical school. A minority were considered important to assess prior to the start of residency training: demonstration of respectfulness (64%), sterile technique (67.2%), BLS (68.9%), ACLS (65.9%) and phlebotomy (63.5%). DISCUSSION: Results from this study support that assessing procedural skills such as cardiac resuscitation, sterile technique, and phlebotomy would be amenable to assessment at the end of medical school, but most procedural and advanced communications skills would be amenable to assessment at the end of the first year of residency training or later.
CONCLUSIONS: Gathering data from residency program directors provides support for developing new assessment tools in high-stakes licensing examinations.

Over the last decade, the Accreditation Council on Graduate Medical Education (ACGME) and American Osteopathic Association (AOA) competencies have been integrated into graduate medical education (1, 2). As a consequence, greater attention has been given to defining competence and identifying specific skills required of residents (3). As an example, the ACGME and American Board of Pediatrics (ABP) have established the Pediatric Milestones Working Group to define specific competencies required of residents at different levels of training (4, 5). Not only are program directors attempting to clarify which skills should be assessed at which level of training, but national licensing boards are also working toward identifying clear assessment objectives (6). The two examinations used to license physicians in the United States, United States Medical Licensing Exam (USMLE) and Comprehensive Osteopathic Medical Licensing Examination – USA (COMLEX-USA) (7, 8), are phasing in a two-decision point model for licensure: entry into supervised practice and entry into unsupervised practice (9, 10). Therefore, identifying which skills should be assessed at each of these decision points becomes critical.

Preparing medical students for residency training has been the focus of medical schools in the United States (11, 12). As part of the Medical School Objectives Project (MSOP), the Association of American Medical Colleges (AAMC) has identified specific skills required of medical students prior to graduation (11), such as patient care, communication and procedural skills. Specific examples include ability to communicate effectively with patients, families and colleagues and ability to perform routine procedures such as venipuncture, lumbar puncture, laceration repair, and thoracentesis (11). In a recent study by the National Board of Medical Examiners (NBME), residents reported performing a number of procedures and communication tasks during their first few months of training (13). The authors conclude that perhaps these clinical skills should be taught and assessed prior to completion of medical school training.

Numerous studies have identified educational gaps between graduating medical students and residents, with regard to clinical skills such as advanced communication and procedural skills (14–21). For instance, Wagner and Lypson described objective standardized clinical examinations (OSCEs) administered to residents at the start of their training (14). New resident performance in communication assessments were consistently high, but patient care scores varied widely with particularly low scores in the areas of hand hygiene and aseptic technique. In another study of new residents, Lypson reported lowest OSCE scores in the areas of informed consent and identification of critical values (15). Both studies not only identified variability among entering residents, but also exposed gaps between residency program faculty expectations and actual performance reflective of undergraduate medical education training. Identifying disparate expectations of medical students and residents has challenged medical educators to identify which specific clinical skills should be taught and assessed. Previous investigators report conflicting opinions regarding which clinical skills should be taught or assessed during medical school (17–24) and residency training (18, 25, 26).

While many studies have attempted to identify the clinical skills required of trainees, most have been limited to a particular institution or discipline. Identifying these clinical skills is particularly important for developing and enhancing assessments for licensing examinations. Two high-stakes clinical skills examinations are used in the United States to assess clinical skills performance of medical students: USMLE Step 2 Clinical Skills (USMLE Step 2-CS) and COMLEX-USA Level 2-Performance Evaluation (COMLEX-USA Level 2-PE) (7, 8). However, neither clinical skills examination currently assesses advanced communication (triadic encounters, death and dying, etc), procedural, or clinical skills which may be unique to a particular specialty. To inform test development and exam enhancement, the goal of this study is to investigate residency program directors’ expectations of assessment of their residents’ procedural and advanced communication skills. In particular, the objective is to survey residency program directors and identify which clinical skills are important to assess, by whom, when, and how (formative versus summative).

Methods

Instrument (survey to residency program directors)

Using content from a variety of questionnaires used in previous studies (17–20, 22–33) and recommendations from the AAMC (11, 34, 35) and ACGME (3), physician staff from the National Board of Osteopathic Medical Examiners (NBOME) compiled a list of specific procedural skills and advanced communication skills.

Recommendations from NBOME's Clinical Skills Testing Advisory Committee (8 members) and strategic planning committee (16 members) were incorporated into the survey. Members from both committees are considered experts in medical education and assessment; members include representatives from undergraduate medical education (deans, associate deans, faculty), graduate medical education (ACGME and AOA-accredited residency program directors, directors of medical education), clinicians, medical educators, and psychometricians.

After receiving input from these expert panels, items were reviewed, and further enhancements were made by NBOME's Research Advisory Committee (12 members) composed of experts in assessment, education, and research. The final instrument (found in Supplemental content) addressed 28 procedural skills and 18 advanced communication skills, was pretested by physician staff, and distributed using Survey Monkey.

Sample

The web-delivered survey was distributed to 3,443 ACGME and AOA-accredited residency program directors with valid email addresses contained in NBOME's residency program director database. Program directors were randomly divided into two groups; each group received one of two versions of the survey (one with procedural skills presented first and another with advanced communication skills presented first). Program directors from all disciplines were included in the sample.

Analysis

Institutional Review Board approval was granted by the Center for the Advancement of Healthcare Education and Delivery (C-AHEAD) to collect, analyze and report these data for this study. Survey responses were analyzed using descriptive statistics. Intraclass Correlations (ICC), which describe the degree of group agreement, were calculated to examine the disparity in responses of program directors of different specialties.

Results

A total of 347 program directors completed the survey, representing a response rate of 10.1%. Program directors from a wide range of disciplines responded to the survey, and specialty distributions were reflective of national data (Table 1) (36, 37). For instance, 45 surgery and surgical subspecialty program directors were included in the sample (13% of the sample), compared to 946 in the national sample (19.9% of all residencies). Primary care residencies were slightly overrepresented in our sample. For instance, 100 family medicine residency program directors were included in the sample (28.8% of the sample), compared to 636 in the national sample (13.4% of all residencies). Among the 347 respondents, 44 were identified as ‘surrogates,’ program director-selected surrogates (e.g., assistant/associate residency program directors) who completed the survey on behalf of the residency program director. Among those who completed the survey, 293 respondents completed the section on procedural skills and 284 completed the section on advanced communication skills. The attrition in survey completion was attributed to the length of the survey.

Table 1

Residency program director respondents in comparison with national sample data, by specialty (n=347)

	Sample				National
Specialtya	ACGME-accredited residency programs n=305b		AOA-accredited residency programs n=42		ACGME-accredited residency programs (total=4,128) (37)		AOA-accredited residency programs (total=636) (36)
Family Med	84	27.5%	16	38.1%	452	10.9%	184	28.9%
Internal Med	40	13.1%	11	26.2%	379	9.2%	88	13.8%
Other	27	8.8%	2	4.8%	1168	28.3%	109	17.1%
Surgical Subsp	26	8.5%	4	9.5%	581	14.1%	78	12.3%
Pediatrics	25	8.2%	1	2.4%	196	4.7%	17	2.7%
OB/GYN	22	7.2%	2	4.8%	246	6.0%	31	4.9%
Emergency Med	16	5.2%	10	23.8%	153	3.7%	43	6.8%
Psychiatry	15	4.9%	–	–	182	4.4%	9	1.4%
Anesthesiology	13	4.3%	–	–	132	3.2%	12	1.9%
Radiology	13	4.3%	–	–	188	4.6%	14	2.2%
PM & R	11	3.6%	–	–	79	1.9%	3	0.5%
Surgery	11	3.6%	4	9.5%	246	6.0%	41	6.4%
Neurology	9	2.9%	1	2.4%	126	3.1%	7	1.1%

Abbreviations: ACGME, Accreditation Council on Graduate Medical Education; AOA, American Osteopathic Association; ObGYN, Obstetrics and Gynecology; PM & R, Physical Medicine and Rehabilitation.

Eleven program directors reported to have more than one specialty so that each of the total percentages corresponding to sample subgroups may exceed 1.

Sixty-four of the ACGME-accredited residency programs are also certified by the AOA.

Procedural skills

Figure 1 presents program directors’ opinions about the importance of assessing 28 procedural skills. Program directors considered a number of procedures to be important to assess (sum of ‘important’ and ‘extremely important’ responses): sterile technique (93.8%), advanced cardiovascular life support (ACLS) (91.1%), basic life support (BLS) (90.0%), interpretation of EKG (89.4%), and interpretation of blood gas (88.7%). Skills such as osteopathic manipulative treatment (35.6%), obtaining a blood culture (37.0%), and PPD placement (38.4%) were considered less important.

Fig. 1

Importance of procedural skills assessment by program directors’ responses. The values reflect responses to the survey question ‘In your opinion, how important is it for each of the following skills to be assessed?’ Values reflect the sum of responses to ‘important’ and ‘extremely important’. The skills are sorted in the descending order of importance rated by all program directors (n=293).

With regard to the importance of assessing procedures, agreement varied among program directors of different specialties. ICC is an index representing proportion of the total variance explained by group effects with higher ICC values indicating larger group variation (lower agreement between specialty groups). ICC values ranged from 0.04 to 0.51. Signifying disagreement between program directors from different specialties, high levels of disparity between specialty groups (ICC values >0.30) were found for central line access, lumbar puncture, incision and drainage, splinting/casting, child birth (vaginal) and pelvic exam. Signifying agreement between program directors from different specialties, low levels of disparity (ICC values <0.10) were found for nasogastric tube placement, obtaining blood culture, cardiac resuscitation (BLS), phlebotomy, sterile technique, and injection (IM/SC). Some procedural skills, such as sterile technique and cardiac resuscitation (BLS), displayed both low group disparity and high importance ratings.

Presented in Fig. 2 are program directors’ opinions regarding how the 28 procedural skills should be assessed. Program directors overwhelmingly reported that each of the procedures should be assessed in a formative fashion, followed by a combination of both formative and summative assessment. Compared to other procedures, ACLS (24.2%), BLS (23.6%), Neonatal Advanced Life Support (NALS) (21.3%), Pediatric Advanced Life Support (PALS) (20.7%), phlebotomy (15.2%), sterile technique (14.9%), injection (14.7%) and intravenous placement (14.7%) were considered to be procedures amenable to summative assessment.

Fig. 2

Type of assessment for procedural skills by program directors responses. The values reflect responses to the survey question ‘In your opinion, please mark whether the assessments should be summative (e.g., used for advancement purposes), formative (e.g., used for feedback and teaching purposes), both or neither.’ Procedural skills are presented in the descending order of responses to ‘Summative Assessment’ (%) by program directors (n=293).

The majority of program directors reported residency program faculty to be the most appropriate for assessing procedural skills (Table 2). Only for the phlebotomy skill, medical school faculty were regarded more appropriate than residency program faculty. A small number of program directors reported that resuscitation (ACLS, BLS, PALS and NALS) could be evaluated in a high-stakes testing environment (28.7, 27.6, 27.6 and 27.0%, respectively).

Table 2

Program directors’ perception of who should be evaluating procedural skillsa

Procedural skills	Medical School Faculty (%)	Residency Program Faculty (%)	Residency Program Director (%)	Director of Graduate Medical Education (%)	High Stakes Testing/Licensing Exam Agency (%)
Sterile technique	49.5	69.3	28.0	5.5	6.5
Cardiac resuscitation (ACLS)	30.4	52.6	28.0	8.2	28.7
Cardiac resuscitation (BLS)	32.8	49.5	27.0	7.2	27.6
Interpretation of EKG	42.7	71.7	28.7	3.4	7.9
Interpretation of arterial blood gas	44.0	70.3	28.7	4.1	7.2
Suturing	41.6	76.8	27.3	3.1	1.7
Pelvic exam	47.1	69.3	26.3	4.4	4.1
Incision and drainage (wound/abscess)	23.9	80.9	27.0	4.1	1.7
Injection (IM/SC)	42.0	63.1	22.9	3.1	2.4
Lumbar puncture	25.6	81.6	28.3	4.1	1.4
Cardiac resuscitation (PALS)	27.6	50.9	26.3	5.8	27.6
Endotracheal tube insertion	21.5	77.8	29.7	4.1	3.1
Cardiac resuscitation (NALS)	26.6	49.5	25.6	5.5	27.0
Urinary catheter placement	42.3	64.5	21.8	3.1	2.0
Nasogastric tube placement	37.2	70.0	23.5	3.4	1.4
Intravenous catheter placement	46.1	59.7	22.9	3.4	3.1
Central line access	19.1	77.1	26.6	3.8	3.1
Phlebotomy	54.6	53.9	21.8	2.7	3.4
Splinting/casting	33.8	73.0	26.6	4.1	2.4
Central line placement	18.4	80.2	28.7	4.1	2.7
Child birth (vaginal)	28.7	72.7	23.9	4.4	1.7
Spirometry	36.9	67.9	24.9	2.7	3.8
Arterial puncture	37.5	68.9	25.6	3.4	2.4
Thoracentesis	22.2	78.2	25.9	3.1	1.0
Paracentesis	22.5	77.8	26.3	3.4	1.4
PPD placement	48.1	54.6	20.8	2.4	3.1
Blood culture	45.1	55.3	23.2	3.1	2.4
Osteopathic Manipulative Treatment (OMT)	45.7	59.4	24.2%	6.5	6.8

The values reflect responses to the survey question ‘In your opinion, WHO would be the most appropriate to make such judgments?’ The skills are sorted in the descending order of importance rated by all program directors (n=293) (Fig. 1).

As for the most appropriate time to assess the procedural skills (Table 3), program directors reported that assessment of most procedures should be completed at the end of the first year of residency or later. Of the responses for ‘end of the first year of residency,’ the largest rates were reported for suturing (62.8%), lumbar puncture (61.1%), and incision and drainage (60.8%). A small number of skills were considered important to assess prior to the start of residency: BLS (68.9%), sterile technique (67.2%), ACLS (65.9%), and phlebotomy (63.5%).

Table 3

Program directors’ perception of when procedural skills should be assesseda

Procedural skills	Prior to start of residency (%)	End of first year of residency (%)	End of second year of residency (%)	Toward the end of residency (%)	After completion of residency (%)
Sterile technique	67.2	42.3	9.2	10.2	5.1
Cardiac resuscitation (ACLS)	65.9	34.8	12.3	11.3	5.5
Cardiac resuscitation (BLS)	68.9	30.4	10.6	10.9	5.1
Interpretation of EKG	47.4	56.7	17.7	12.3	3.4
Interpretation of arterial blood gas	49.5	55.3	12.6	8.9	3.1
Suturing	42.0	62.8	15.4	11.9	4.8
Pelvic exam	56.3	50.5	11.3	7.5	3.1
Incision and drainage (wound/abscess)	20.1	60.8	21.8	13.3	3.8
Injection (IM/SC)	50.5	42.3	9.6	8.5	3.1
Lumbar puncture	18.4	61.1	25.6	14.3	2.7
Cardiac resuscitation (PALS)	49.1	40.3	10.6	11.9	5.5
Endotracheal tube insertion	19.1	57.7	25.6	14.7	3.1
Cardiac resuscitation (NALS)	48.8	39.9	10.2	11.9	5.1
Urinary catheter placement	50.2	48.8	7.5	6.1	2.7
Nasogastric tube placement	38.2	56.0	9.2	7.9	2.0
Intravenous catheter placement	51.5	45.7	7.2	7.2	2.0
Central line access	13.7	58.0	24.9	14.7	3.8
Phlebotomy	63.5	35.8	4.8	6.1	2.0
Splinting/casting	28.7	50.5	23.5	17.1	4.1
Central line placement	11.9	56.3	27.3	16.0	4.1
Child birth (vaginal)	22.2	49.8	20.8	18.1	4.8
Spirometry	36.9	46.1	17.7	14.3	3.4
Arterial puncture	39.2	51.9	10.2	8.5	1.7
Thoracentesis	14.7	46.8	30.7	19.1	5.5
Paracentesis	17.1	43.7	30.4	20.5	4.8
PPD placement	57.3	36.5	5.5	5.8	2.0
Blood culture	53.2	39.9	4.8	5.8	1.7
Osteopathic Manipulative Treatment (OMT)	50.2	31.1	20.1	16.7	8.2

The values reflect responses to the survey question ‘In your opinion, WHEN would be the most appropriate to make such judgments?’ The skills are sorted in the descending order of importance rated by all program directors (n=293) (Fig. 1).

Advanced communication and interpersonal skills

Figure 3 displays program directors’ opinions about the importance of assessing 18 advanced communication skills. Program directors considered most communication skills important to assess (sum of ‘important’ and ‘extremely important’ responses). Responses were the highest for demonstrating professionalism (99.6%), respectfulness (98.9%), good listening skills (98.6%), communication with nursing/ancillary staff (98.6%), and empathy (97.9%). The remaining skills each received ratings of importance higher than 78%. The ICC coefficients examining group agreement between program directors of different specialties ranged from near 0 to 0.13. No significant between-group variation was found.

Fig. 3

Importance of advanced communication skills assessment by program directors’ responses. The values reflect responses to the survey question ‘In your opinion, how important is it for each of the following skills to be assessed?’ Values reflect the sum of responses to ‘important’ and ‘extremely important.’ The skills are sorted in the descending order of importance rated by all program directors (n=284).

Regarding the 18 communication skills, program directors overwhelmingly reported that integrative evaluations using both summative and formative assessment should be utilized (Fig. 4). Exclusive summative assessment was not considered a suitable format by most program directors. The majority of the program directors reported residency program faculty to be the most appropriate to assess advanced communication skills (Table 4).

Fig. 4

Type of assessment for advanced communication skills by program directors’ responses. The values reflect responses to the survey question ‘In your opinion, please mark whether the assessments should be summative (e.g., used for advancement purposes), formative (e.g., used for feedback and teaching purposes), both or neither.’ Advanced communication skills are presented in the descending order of responses to ‘Summative Assessment’ (%) by program directors (n=284).

Table 4

Program directors’ perception of who should be evaluating advanced communication skillsa

Communication skills	Medical School Faculty (%)	Residency Program Faculty (%)	Residency Program Director (%)	Director of Graduate Medical Education (%)	High Stakes Testing/Licensing Exam Agency (%)
Demonstrating professionalism	47.5	77.1	54.9	15.8	8.8
Demonstrating respectfulness	51.4	77.8	47.5	11.3	6.0
Demonstrating good listening skills	52.5	77.5	44.0	5.6	6.0
Communication with nursing/ancillary staff (oral)	39.4	80.6	42.6	7.0	2.1
Demonstrating empathy	49.6	76.8	41.5	5.6	3.9
Eliciting information	52.5	79.9	39.4	6.3	7.7
Giving information	48.2	81.3	40.8	6.7	6.7
Acknowledgment of medical error/mistake	41.2	77.1	47.5	10.2	5.3
Hand offs (e.g., sign out rounds, transfer of care)	34.2	82.4	45.1	8.5	4.2
Delivering bad news	37.3	80.6	39.1	5.6	4.2
Obtaining informed consent	34.5	85.2	40.5	5.3	6.3
Communication with nursing/ancillary staff (written)	38.0	80.6	40.8	7.0	1.8
Referral to consultants (oral)	32.0	85.9	37.7	4.9	2.1
Referral to consultants (written)	32.4	84.9	37.7	4.6	1.8
Demonstrating cultural competence	47.5	72.5	43.0	10.6	6.3
End of life (e.g., advance directives)	36.3	79.6	38.7	7.0	4.6
Triadic encounters (e.g., communication with parent and child)	39.4	81.7	35.6	3.9	2.8
Dictation of medical record	31.7	79.9	41.9	8.5	2.1

The values reflect responses to the survey question ‘In your opinion, WHO would be the most appropriate to make such judgments?’ The skills are sorted in the descending order of importance rated by all program directors (n=284) (Fig. 3).

For all communication skills, except ‘demonstrating respectfulness’, program directors reported the end of first year of residency to be the most appropriate time for evaluation (Table 5). Of the responses for ‘end of the first year of residency,’ the largest rates were for handoffs (83.1%), referral to consultants-oral (76.4%) and dictation of medical record (77.5%).

Table 5

Program directors’ perception of when advanced communication skills should be assesseda

Communication skills	Prior to start of residency (%)	End of first year of residency (%)	End of second year of residency (%)	Toward the end of residency (%)	After completion of residency (%)
Demonstrating professionalism	60.2	64.1	35.9	33.5	20.1
Demonstrating respectfulness	64.1	60.9	32.0	31.3	15.5
Demonstrating good listening skills	60.6	63.7	32.0	27.1	11.6
Communication with nursing/ancillary staff (oral)	43.3	73.6	27.8	26.1	10.9
Demonstrating empathy	57.0	62.7	32.4	30.3	12.3
Eliciting information	53.2	66.5	30.3	25.4	10.2
Giving information	42.3	69.0	37.0	29.2	11.6
Acknowledgment of medical error/mistake	39.8	65.1	34.2	29.2	13.4
Hand offs (e.g., sign out rounds, transfer of care)	27.8	83.1	35.2	25.7	9.5
Delivering bad news	25.7	62.0	41.9	28.5	10.6
Obtaining informed consent	23.2	77.1	39.8	26.8	11.3
Communication with nursing/ancillary staff (written)	37.7	75.0	28.5	24.6	10.6
Referral to consultants (oral)	16.5	76.4	37.3	27.5	9.2
Referral to consultants (written)	16.5	75.7	37.0	28.5	9.5
Demonstrating cultural competence	48.2	57.7	32.7	28.2	9.9
End of life (e.g., advance directives)	23.6	62.0	40.8	33.5	11.6
Triadic encounters (e.g., communication with parent and child)	25.0	69.7	34.5	27.5	7.7
Dictation of medical record	20.1	77.5	29.6	26.1	8.8

The values reflect responses to the survey question ‘In your opinion, WHEN would be the most appropriate to make such judgments?’ The skills are sorted in the descending order of importance rated by all program directors (n=284) (Fig. 3).

Discussion

Program directors reported that all advanced communication tasks and some procedural tasks are important to assess during medical training. Although their responses were consistent across disciplines when considering communication tasks, there was variability among groups when asked about procedures. High levels of agreement between program directors of different specialties were seen for nasogastric tube placement, obtaining a blood culture, cardiac resuscitation, phlebotomy, sterile technique and injections. Strong agreement is likely explained by the fact that these procedures are common to all physicians, not just those of a particular discipline. Identifying consistency among program directors of different disciplines is important, given the recent growth of specialization in graduate medical education (38–40). However, of these procedures with high levels of agreement, only cardiac resuscitation, sterile technique and injection were considered important to assess. In a similar survey of program directors, 89.7% expected competency three months into residency with regard to BLS, and 74.4% with regard to ACLS (18).

For both advanced communication and procedural skills, program directors reported that assessments should include a combination of formative and summative evaluation. This was particularly true for advanced communication tasks, demonstrated by a small number of program directors advocating for exclusive summative assessment. Compared to the other procedural skills, cardiac resuscitation, phlebotomy, sterile technique, injection and intravenous placement were considered amenable to summative assessment.

Program directors reported that most clinical skills should be assessed at the end of the first year of residency (or later) and not before graduation from medical school. Exceptions to this include demonstration of respectfulness, sterile technique, cardiac resuscitation, and phlebotomy; these were considered important to assess before the start of residency. This is a departure from the recommended procedures specified in AAMC's MSOP report, which advocates that students demonstrate the ability to complete the following eight procedures: venipuncture, inserting an intravenous catheter, arterial puncture, thoracentesis, lumbar puncture, inserting a nasogastric tube, inserting a foley catheter, and suturing lacerations (11). Among this list from the MSOP, only venipuncture (or phlebotomy) was considered important to assess at the end of medical school in our study, and the remainder were considered important to assess during the first year of residency or later. Consistent with Raymond's findings that few residents report performing specific procedures early in residency (13), our study of program directors supports that most clinical procedures should be assessed at the end of first year of residency (or later). Similarly, many of the clinical skills tasks assessed by the Medical Council of Canada Qualifying Examination Part II (MCCQE Part II) necessitate clinical experience during residency, and therefore examinees are required to complete a minimum of 12 months of postgraduate training before taking the clinical skills exam (7).

This study has a few notable limitations. First, although the sample of program directors includes the largest sample of physicians from different institutions and disciplines than any other study we could locate addressing communication and procedural skills (347 program directors), the survey response rate was 10.1%, and a higher response rate may provide additional information. Second, the program directors’ rationale for their responses was not elicited, and future study could be improved by complementing the survey with focus group discussion. Third, we did not solicit responses from medical school faculty (e.g., clerkship directors). Perspectives of medical school faculty is important to incorporate in future study, particularly since significant differences of opinion have been reported regarding which skills should be taught in medical school (17–24) and residency training (18, 25, 26). Fourth, the study did not include a formal resident task analysis with verification of completion of procedures; obtaining primary verification (such as a review of credentialing logs) would provide valuable information.

Conclusions

Performing clinical skills in a competent fashion is important for patient care. Ideally, assessments used for licensure should measure clinical skills considered important to assess among residency program directors across all disciplines and amenable to summative high-stakes assessment. As USMLE and COMLEX-USA examination programs begin to augment and adapt current examinations to comply with a two decision point model for licensure, clarifying which skills should be assessed at specific levels of training (entry into supervised practice and entry into unsupervised practice) becomes particularly important. Results from this study support that assessing procedural skills such as cardiac resuscitation, sterile technique, and phlebotomy would be important to assess at the end of medical school (entry into supervised practice), but that the assessment of most procedural and advanced communications skills would be more suited at the end of the first year of residency training or later (entry into unsupervised practice). Gathering data from residency program directors provides support for examination development as new assessment tools are considered for high-stakes licensing examinations.