Assessment and Evaluation in Social Determinants of Health Education: a National Survey of US Medical Schools and Physician Assistant Programs.

BACKGROUND: Social determinants of health (SDOH) curricular content in medical schools and physician assistant programs are increasing. However, there is little understanding of current practice in SDOH learner assessment and program evaluation, or what the best practices are.
OBJECTIVE: Our study aim was to describe the current landscape of assessment and evaluation at US medical schools and physician assistant programs as a first step in developing best practices in SDOH education.
DESIGN: We conducted a national survey of SDOH educators from July to December 2020. The 55-item online survey covered learner assessment methods, program evaluation, faculty training, and barriers to effective assessment and evaluation. Results were analyzed using descriptive statistics. PARTICIPANTS: One hundred six SDOH educators representing 26% of medical schools and 23% of PA programs in the USA completed the survey. KEY
RESULTS: Most programs reported using a variety of SDOH learner assessment methods. Faculty and self were the most common assessors of learners' SDOH knowledge, attitudes, and skills. Common barriers to effective learner assessment were lack of agreement on "SDOH competency" and lack of faculty training in assessment. Programs reported using evaluation results to refine curricular content, identify the need for new content, and improve assessment strategies.
CONCLUSIONS: We identified a heterogeneity of SDOH assessment and evaluation practices among programs, as well as gaps and barriers in their educational practices. Specific guidance from accrediting bodies and professional organizations and agreement on SDOH competency as well as providing faculty with time, resources, and training will improve assessment and evaluation practice and ensure SDOH education is effective for students, patients, and communities.

Introduction

In recent years, there has been increasing recognition of the importance of social determinants of health (SDOH) and how they affect a wide range of exposures, risks, and outcomes related to health, functioning, and quality of life.[1,2] Understanding how to assess and respond to SDOH constructs in healthcare practice is critical to the health and well-being of communities, and this requires effective and well-designed SDOH-related educational opportunities.[3,4]

The published literature regarding educational practices and curricular design considerations regarding SDOH curricula has been growing over the past two decades, with marked acceleration in the last few years.[5-9] However, a major gap persists in the area of effective SDOH learner assessment and program evaluation. Consequently, there is limited information, lack of standardization, and little dissemination of either curricular materials or assessment tools. These deficiencies may be in part due to the vague guidance from oversight bodies for specific SDOH-related curricula or assessment.[10,11]

To avoid reinventing the curricular wheel and advance the field of SDOH education, we surveyed a national sample of educators about their current SDOH assessment and evaluation practices and the barriers that exist. We sought to describe how SDOH curricula are implemented and evaluated at medical schools and PA programs, and how learners’ acquisition of this content was assessed. Data from these critical first steps will be instrumental in establishing best practices in SDOH education assessment and evaluation with the ultimate goal of improving patients’ health and well-being.

Methods

We used a multistep process to develop and conduct our online survey of SDOH educators who included curricular or assessment deans, SDOH course directors, and faculty involved in SDOH curriculum development, teaching, assessment, and evaluation at US medical schools and PA programs.

Survey development

We conducted an extensive review of the published literature on SDOH education in order to identify current gaps in educational practice around SDOH assessment and evaluation. Previously published expert consensus on SDOH curricular content and teaching methods provided the framework for our survey.[12] We then garnered input from two experts in curriculum development and assessment to clarify survey questions and ensure survey content was appropriate and comprehensive.

Survey sections included survey respondent information, program characteristics, SDOH curricular content, learner characteristics, learner assessment methods and frameworks, sources of current SDOH assessment tools/instruments, and barriers to effective assessment and SDOH program evaluation methods.

Participant recruitment and survey dissemination

We identified 155 allopathic medical schools (MD), 37 osteopathic medical schools (DO), and 267 accredited physician assistant programs in the USA using data from the websites of the Association of American Medical Colleges (AAMC), the American Osteopathic Association, and the Accreditation Review Commission on Education for the Physician Assistant (ARC-PA) respectively.[13-15] For schools/programs with more than one campus, we included each campus/site as a distinct school/program as the specific SDOH assessment practice may vary at each site and this resulted in a total of 161 MD programs and 38 DO programs. For PA programs, we removed accredited programs that had not yet matriculated students using accreditation data from the Physician Assistant Education Association (PAEA) Program Directory website.[16]

From June to December 2020, we sent surveys to faculty at 161 MD programs, 38 DO programs, and 237 PA programs using both automated survey response and direct email methodology. We included a digital consent form which was completed before beginning the survey. Surveys were sent in five successive rounds to program leadership in order to capture results from institutions that had not previously responded. The initial two rounds were sent through Qualtrics, while for the subsequent rounds we sent personalized, direct email invitations to each educator to increase the response rate. For programs that had not responded after the first two rounds, we identified and contacted other faculty within the program who had demonstrated expertise in SDOH education as evidenced by disseminated scholarship in academic journals, conference presentations, and webinars. These faculty were often the resident expert SDOH educators at their respective institutions. Respondents could decline and suggest another faculty member to complete the survey for the program and a direct email with survey link was sent to the new faculty.[17] In addition, respondents were encouraged to upload SDOH assessment tools they were currently using into our online repository for review. All participants were offered the opportunity to enter a raffle for $500 or $250 gift cards.

Results

Institution characteristics

We sent surveys to a total of 436 programs (161 allopathic schools, 38 osteopathic schools, and 237 PA programs). One hundred six completed surveys were collected (24% response rate): 62 (58%) from PA programs and 44 (42%) from allopathic or osteopathic medical schools. There was no multi-campus program that produced more than one response. Responses from 40 allopathic schools and 4 osteopathic schools were combined in this analysis. Four responses with less than 50% of items completed were removed from the analysis. Survey respondents included educators from a variety of academic ranks and roles within their programs, including deans, PA Program Directors, SDOH course directors, course faculty, and assessment and evaluation faculty/staff. Medical school respondents were from 23 states and PA programs respondents represented 29 states. These responses represent 26% of all accredited US medical schools and 23% of all accredited PA programs. Overall, 10% of programs identified as rural, 34% as suburban, and 54% urban.

SDOH curriculum characteristics

There is marked variation in where and how programs embed SDOH content within the larger curriculum in both preclinical and clinical phases (Table 1). Of note, 6% (3/44) of MD/DO and 10% (6/62) of PA programs reported they do not teach SDOH content in the clinical phase. In addition, programs described a wide variety of pedagogies in their SDOH curricula. Large group lecture was the most common method reported by both medical schools and PA programs. Cooperative and/or collaborative learning was the second most commonly cited methodology (Table 2).

Table 1

SDOH curriculum teaching sites

	Preclinical		Clinical
	MD/DO	PA	MD/DO	PA
Classroom only	13	37	2	0
Clinical/field experience only	0	1	4	27
Both	31	24	35	29
Not covered	-	-	3	6
Total	44	62	44	62

Table 2

SDOH teaching methods (multi-select item, total >106)

Teaching method	Total # of programs(MD, DO, PA)	MD/DO schools	PA programs
Large group lectures	101	42	59
Direct clinical care	82	34	48
Cooperative/collaborative learning	81	33	48
Case-based learning	79	38	41
Service learning	68	31	37
Self-directed learning	43	23	20
Online modules	35	16	19
Other*	6	5	1

SDOH content and learner assessment

In both PA programs and medical schools, SDOH-related knowledge, skills, and attitudes were reportedly assessed across a wide variety of topics. A handful of programs reported that they did not assess SDOH knowledge (2%, 2/106), skills (6%, 6/106), or attitudes (7%, 7/106). Only 41% (18/44) of medical school respondents and 40% (25/62) of PA programs reported assessing observed SDOH behaviors among their learners.

Who is assessing learners in SDOH education?

For SDOH knowledge and attitudes, the most common assessors were (in order) faculty, self, and peer at both medical schools and PA programs. SDOH skills were most commonly assessed by faculty, followed by self. A small number of programs reported involving the patient/family and social worker in assessing SDOH skills. Rarely were nurses involved in learner assessment of SDOH knowledge, skills, or attitudes (5%, 5/106).

Learner assessment strategies

The assessment strategies used varied across domains (Table 3). In the domain of knowledge, both medical school and PA programs relied heavily on written assessment (59%, 26/44 and 76%, 47/62 respectively). In the skills domain, medical schools most commonly used direct observation (68%, 30/44) and OSCE (52%, 23/44), while PA programs favored OSCE (65%, 40/62) and simulation (47%, 29/62). For assessment of attitudes, medical schools most commonly used direct observation (59%, 26/44), while PA programs used written assessment (48%, 30/62). Overall, respondents ranked simulation-based assessment, group projects/presentation, and oral exam/presentation as having the highest value in their SDOH educational practice, and ranked multiple-choice questions as the lowest in value.

Table 3

Top learner assessment strategies in SDOH Education (multi-select items, colum totals >106)

MD/DO Programs
Knowledge	Skills	Attitudes
• Written assessment (59%, 26/44) • Direct observation (48%, 21/44) • MCQ tests (43%, 19/44)	• Direct observation (68%, 30/44) • OSCE (52%, 23/44) • Simulation (34%, 15/44)	• Direct observation (59%, 26/44) • Written assessment (41%, 18/44) • OSCE (34%, 15/44)
PA Programs
Knowledge	Skills	Attitudes
• Written assessment (76%, 47/62) • OSCE (47%, 29/62) • Oral exam/presentation (45%, 28/62)	• OSCE (65%, 40/62) • Simulation (47%, 29/62) • Written assessment (45%, 28/62)	• Written assessment (48%, 30/62) • OSCE (47%, 29/62) • Simulation (45%, 28/62) • Direct observation (45%, 28/62)

Frequency of learner assessment

The most commonly reported assessment frequency was at intervals across the curriculum. A minority of programs only assessed learners at the end of specific modules/rotations.

EPAs and core competencies in SDOH learner assessment

Respondents noted that SDOH learner assessment data mapped to many of the published competencies for their disciplines. For medical schools, the top three competencies were (in order) interpersonal and communication skills, patient care, and systems-based practice. For PA programs, the top three competencies were (in order) society and population health, health literacy and communication, and patient-centered practice knowledge.

In general, entrustable professional activities (EPAs) were used by 45% (20/44) of medical schools and 24% (15/62) of PA programs. Of the programs that used EPAs, 35% (7/20) of medical schools and 87% (13/15) of PA programs incorporated EPAs into SDOH assessment, and of this group, more than 89% of all programs indicated that EPAs were used for SDOH assessment in both inpatient and outpatient settings. Respondents cited lack of expertise and lack of faculty buy-in as barriers to incorporating EPAs in SDOH learner assessment.

Sources of SDOH learner assessment tools

The most common sources for medical schools were locally developed tools which had not yet been disseminated and tools adapted from other developers. For PA programs, the most common sources were tools adapted from other developers and tools that faculty themselves had developed and disseminated.

How are SDOH assessment results used?

The majority of respondents described using assessment data for both formative and summative purposes. A few programs reported using these results for promotion to the next level (26%, 28/106) or inclusion in the Medical School Performance Evaluation/“Dean’s letter” (13%, 14/106), while some others used it for graduation decisions (23%, 24/106).

Forty-eight percent (21/44) of medical schools and 60% (37/62) of PA programs reported that a student may be required to complete remediation based on their SDOH assessments; the remainder either did not require any remediation for SDOH deficiencies or were unsure if this was possible. Nonetheless, almost all respondents (96%, 102/106) thought that learners should be held to an SDOH curricular standard and 95% (101/106) believe their school would support requiring learners to meet the SDOH standard.

Barriers and facilitators to effective SDOH learner assessment

Respondents reported the lack of agreement on the definition of “SDOH competency” and lack of training in assessment related to SDOH domains as major barriers to effective assessment (Table 4). Facilitators of SDOH learner assessment included faculty and student input into assessment tools and strategies for MD/DO programs, and faculty input into assessments for PA programs (Table 5).

Table 4

Barriers to effective SDOH learner assessment (multi-select item, total >106)

Barriers	Total # of programs(MD, DO, & PA)	MD/DO	PA
No agreement on “SDOH competency”	55	24	31
Not trained/don’t feel competent to effectively assess learners in this area	52	25	27
Lack of time to assess learners	41	19	22
Lack of understanding about assessment tools	39	10	29
Lack of assessment tools	29	11	18
Lack of time to document assessment results	14	7	7
Other	14	7	7

Table 5

Facilitators of effective SDOH learner assessment (multi-select item, total >106)

Facilitators	Total # of programs(MD, DO, & PA)	MD/DO	PA
Having faculty input into SDOH assessment tools and strategies	69	23	46
Having student input into SDOH assessment tools and strategies	39	22	17
Having trained assessment and evaluation staff at my institution	35	17	18
Having community members’ into SDOH assessment tools and strategies	34	15	19
Other	16	8	8

Faculty training in SDOH learner assessment

Only 33% (35/106) of respondents reported that their faculty were trained to assess learners on SDOH content—41% (18/44) of medical schools and 27% (17/62) of PA programs. Fourteen percent (15/106) of respondents did not know if there was training available at their institution. The most common types of training faculty had received were faculty development courses or an advanced degree in education.

Program evaluation

Respondents reported the two most common cycles for SDOH curricular revision were on a rolling basis and annually. Sixty-one percent (65/106) of respondents reported that program evaluation occurred annually or multiple times within a year. Most institutions reported conducting evaluation of their coursework on both the course and curricular levels. However, 14% (6/44) of medical schools and 35% (22/62) of PA programs reported they did not conduct evaluations of their SDOH course/curriculum, and 6% (6/106) of respondents were not aware of the level at which evaluation occurred. For MD/DO programs, the assessment and evaluation staff were the group most commonly responsible for conducting course/program evaluation, while for PA programs, the SDOH course director was most commonly tasked with this responsibility.

Programs reported using a variety of methods for program evaluation including surveys, focus groups, mid-level assessment data, group-level assessment data, and the Physician Assistant Education Association (PAEA) end-of-program survey. The results were most frequently reviewed by program leadership and the course/curriculum directors for both medical schools and PA programs. Learners reviewed these results in 45% (20/44) of medical schools and 11% (7/62) of PA programs. The top three uses of program evaluation results at both medical schools and PA programs were (i) to refine existing curricular content; (ii) to identify the need for new curricular content, and (iii) to improve assessment strategies.

Medical schools and PA programs both reported that the most common measures used to evaluate program effectiveness were learner reaction/satisfaction, learner knowledge, and learner behavior (Table 6). Respondents from both medical schools and PA programs cited lack of time as the most common barrier for evaluating their SDOH courses or programs.

Table 6

Measures of SDOH program effectiveness (multi-select item, total >106)

Measures	Total # of programs(MD, DO, & PA)	MD/DO	PA
Learner reaction/satisfaction	86	38	48
Learner knowledge	81	33	48
Learner behavior	63	24	39
Interprofessional teamwork	41	14	27
Institutional culture around SDOH	27	16	11
Faculty satisfaction with teaching experience	26	9	17
Faculty attitudes towards SDOH	21	7	14
Learner career choice	14	4	10
Patient-centered outcomes/quality of care	14	3	11
Population health measures	9	4	5
None of the above	4	2	2
Other	2	2	0

Discussion

With the burgeoning portfolio of SDOH curricula in medical schools and PA programs come questions regarding the best methods of assessing our learners and evaluating SDOH programs. One important finding from our study is the modest cohesion among respondents. This perhaps reflects the low priority and lack of required SDOH content within the context of biomedical sciences.[18] Without curricular standards or validated educational methodology[19], SDOH content is easily relegated to the non-core, elective part of the curriculum. Recognizing the value of SDOH content in medical education is an essential first step that will be bolstered by assessment and evaluation.[20]

There is significant heterogeneity in assessment and evaluation practices across programs and this is an important gap for accreditation bodies to address. While the mandate to create curriculum and assess learners is a step in the right direction,[10,11] it is imperative that accreditation bodies provide educators guidance and specific standards on what SDOH learner assessment and program evaluation should include, similar to the Canadian Medical Education Directives for Specialists (CanMEDS) framework (see supplementary materials).[21,22]

The survey results also show that faculty and self are the most common assessors of SDOH knowledge, skills, and attitudes in both medical schools and PA programs. Programs need to be more inclusive and interprofessional in SDOH curriculum development, assessment, and evaluation. Input from nurses, social workers, and the patient/family would be invaluable to the training and development of the learner and the program at large. In addition, input from the patient/family in SDOH learner assessment and from community members in SDOH program evaluation would provide a more comprehensive and rigorous understanding of the program’s impact as described by Kirkpatrick’s model of program evaluation.[23] Without input from patients and community members, SDOH educators are at best guessing about the effectiveness of their curriculum for the most important stakeholders.

While the majority of the programs assess learners’ knowledge, skills, and attitudes in SDOH education, less than half of medical schools and PA programs assess SDOH behaviors in their learners. Miller’s pyramid of clinical competence describes the intended progression of the learner from mere knowledge to actual behaviors.[24,25] If our programs are not assessing SDOH behaviors, we are missing out on a key aspect of clinical competence for our learners.

Regarding specific assessment frameworks, while medical schools and PA programs have adapted core competencies for use in their assessment of learners,[26-28] EPAs have not been as widely adopted. They were reportedly used in learner assessment in general by only a third of our respondents—specifically in SDOH education, by only 18%. We hypothesize that the lack of agreement on SDOH behaviors and skills is a reason for this low adoption of EPAs.

The two most frequently cited barriers to effective assessment are lack of training in assessment in this area and no agreement on “SDOH competency” for learners. Only 35% of respondents reported either they or their faculty had been trained to assess learners on the SDOH curriculum. These findings highlight an excellent opportunity for medical schools and PA programs to support their faculty in creating and participating in SDOH-related faculty development courses.

Based on our survey results, we recommend the following changes in SDOH assessment and evaluation:

The assessment framework chosen by the institution should effectively assess SDOH skills and behaviors of learners.

There is a wide variety of assessment frameworks currently in use (core competencies, EPAs, Miller’s pyramid), and it is important that the learners’ SDOH skills and behaviors are assessed regardless of the framework chosen, and not only knowledge.

Institutions need to provide faculty with the time and resources for training in effective learner assessment and program evaluation, specifically in SDOH education.

Faculty development workshops on SDOH education at their home institutions or at professional conferences such as the Society of General Internal Medicine or the American College of Physicians are examples of training opportunities that institutions can support. The Physician Assistant Education Association (PAEA) routinely offers faculty development workshops on a variety of topics, and could consider offering one focused on SDOH education

Program evaluation in SDOH education should be completed at least annually and should involve key stakeholders including learners and patients/community members.

It is imperative that institutional leadership provide time for faculty to conduct program evaluation at least annually, with the involvement of key stakeholders including learners, patients, and/or community members.

We need a multidisciplinary approach to define the standard of SDOH competency in health professions education.

Given the importance of interprofessional education and practice, health professions education organizations (AAMC, Physician Assistant Education Association), professional societies (Society for General Internal Medicine, American Medical Association, American Osteopathic Association, American Association of Physician Assistants), and accrediting bodies of these health professions (Liaison Committee on Medical Education, Commission on Osteopathic College Accreditation, Accreditation Review Commission on Education for the Physician Assistant) must work together to define the standard of SDOH competency. The lack of agreement on SDOH competency is a major barrier for educators and can be frustrating to learners who sense some level of subjectivity in their assessment.

Competency in SDOH should be a graduation requirement in health professions education

Knowing the impact that social determinants of health have on our patients’ health, it is imperative that all graduates of medical schools and PA programs meet a set standard of SDOH competencies. This will ensure that learners are not only prepared for national board examinations, but more importantly are able to address the needs of their patients and their communities.[29]

Limitations

We were limited by a slow uptake in survey response due to the COVID-19 pandemic and had to send out multiple rounds of invitations to faculty, with a lower than expected response rate of 24%. We recognize the potential for selection bias and that our results might actually underestimate the number of programs that do not address SDOH assessment and evaluation in a meaningful manner. We invited respondents to post their SDOH curricula and assessment tools which would have given us the opportunity to review learning objectives. Unfortunately, no respondents posted materials. Future studies will benefit from a thematic review of SDOH learning objectives. We solicited information broadly about learner assessment; however, to develop best practices, we will need granular information about how educators assess attitudes, behaviors, and skills. In addition, the timing of the survey during the pandemic when SDOH and social justice issues were center stage in the nation likely resulted in some measure of social desirability in the survey responses. Nonetheless, the survey findings provide valuable insight into current practice in SDOH learner assessment and program evaluation which hitherto had been unavailable.

Conclusion

This is the first study to provide a description of current practices around SDOH learner assessment and program evaluation in two health professions education disciplines. The heterogeneity in assessment and evaluation practice in SDOH education is not surprising given the lack of agreement on the definition of SDOH competency. Our study findings reinforce the call for more specific guidance from accreditation bodies in health professions education regarding SDOH standards, assessment, and evaluation. With continued investment towards faculty’s time and training, development of standards for SDOH competencies, and engagement of patients and communities in SDOH education, we will be closer to attaining the goals of reducing health disparities and meeting the healthcare needs of vulnerable patients and communities.

(PDF 372 kb)

(PDF 20.1 kb)