Using Mixed Methods With Multiple Stakeholders to Inform Development of a Breast Cancer Screening Decision Aid for Women With Limited Health Literacy.

Background. When stakeholders offer divergent input, it can be unclear how to prioritize information for decision aids (DAs) on mammography screening. Objectives. This analysis triangulates perspectives (breast cancer screening experts, primary care providers [PCPs], and patients with limited health literacy [LHL]) to understand areas of divergent and convergent input across stakeholder groups in developing a breast cancer screening DA for younger women with LHL. Design. A modified online Delphi panel of 8 experts rated 57 statements for inclusion in a breast cancer screening DA over three rounds. Individual interviews with 25 patients with LHL and 20 PCPs from a large safety net hospital explored informational needs about mammography decision making. Codes from the qualitative interviews and open-ended responses from the Delphi process were mapped across stakeholders to ascertain areas where stakeholder preferences converged or diverged. Results. Four themes regarding informational needs were identified regarding 1) the benefits and harms of screening, 2) different screening modalities, 3) the experience of mammography, and 4) communication about breast cancer risk. Patients viewed pain as the primary harm, while PCPs and experts emphasized the harm of false positives. Patients, but not PCPs or experts, felt that information about the process of getting a mammogram was important. PCPs believed that mammography was the only evidence-based screening modality, while patients believed breast self-exam was also important for screening. All stakeholders described incorporating personal risk information as important. Limitations. As participants came from one hospital, perceptions may reflect local practices. The Delphi sample size was small. Conclusions. Patients, experts, and PCPs had divergent views on the most important information needed for screening decisions. More evidence is needed to guide integration of multiple stakeholder perspectives into the content of DAs.

Introduction

Patient decision aids (DAs) aim to support patient involvement in making decisions about health screening and/or treatment by presenting available options and likely outcomes. To promote rigor in DA development and evaluation, the International Patient Decision Aid Standards (IPDAS) were developed.[2,3] The IPDAS guidance delineates a systematic process for designing a DA, which includes establishing a steering committee with relevant stakeholder representation (including patients/end users), conducting needs assessments, and implementing alpha and beta testing to iteratively refine DA content. Over time, the IPDAS collaboration guidance on DA design evolved in response to rigorous reviews of DAs that pointed to areas for improvement, including considerations for health literacy. Current IPDAS guidance suggests considering health literacy in DA design by using plain language and including patient stakeholders in all phases of development and testing. However, few DAs have incorporated end users with limited health literacy (LHL) throughout the development and testing of these tools.

IPDAS guidance is less clear on the extent to which and how input from multiple stakeholders should be incorporated. A recent series of systematic reviews sought to characterize patient inclusion in DA development, seeking to identify practices that promoted user-centered designs. Of the 283 reviewed DA projects, 15% included patients/end users in a formal needs assessment, and 35% included patients in at least one step for refining a prototype. Only 16% included groups that have been marginalized, and 94% did not report the health literacy of the user group. While it is clear that patients are sometimes included in DA development, there remains a need to understand how the perspectives of stakeholder groups that have been marginalized in DA development activities align or diverge with experts and to learn how to best incorporate differing stakeholder perspectives within DA design.

Therefore, we aimed to develop a comprehensive understanding of multiple stakeholder needs in developing and delivering a breast cancer screening DA for women with LHL. This analysis focuses specifically on triangulating perspectives to understand areas of divergence and convergence across stakeholder groups (experts, primary care providers [PCPs], and patients with LHL) that require synthesis in developing DAs for breast cancer screening among limited health literacy populations who have been historically underrepresented in the design process.

Methods

We explored breast cancer screening decision making among three separate sets of stakeholders. Patients and PCPs participated in a qualitative interview study. The study’s primary results are reported elsewhere[8,9] and informed the design of an online Delphi panel conducted among breast cancer screening experts. Here we combine secondary analysis of the patient and provider qualitative interviews with primary data from the expert Delphi panel. The Boston University Medical Center Institutional Review Board approved all activities. Prior to enrollment, all participants were informed of study purpose and potential risks and benefits, and provided consent.

Data Sources 1 and 2: Qualitative Interviews With Patients and Primary Care Providers

In 2018–2019, we completed qualitative interviews with 25 women with LHL (patients) and 20 PCPs at six outpatient primary care practices located at a safety net hospital to explore mammography counseling experiences, including providers’ goals for counseling and patients’ desired information. Methodological details, including the conceptual foundations for the study, content covered, and prior analytic techniques, are available in prior publications.[8,9] Briefly, we invited primary care patients with upcoming visits, ages 40 to 54, with no mammogram in the last 9 months and no breast cancer history. If interested, women completed the Health Literacy Skills Instrument–1010 (HLSI-10) and those who scored <7, indicating limited health literacy, were enrolled. The HLSI-10 measures oral literacy, reading ability, navigation/information seeking, and numeracy—all skills relevant to decision making. Patients with and without prior mammograms were sought for inclusion, and while 13 were mammogram-naïve, 12 had undergone at least one mammogram in the past. Hour-long patient interviews used a flexible interview guide, pilot tested with five women with LHL, that systematically covered knowledge of and experiences with mammography counseling, breast cancer risk and screening potential benefits/harms; various screening modalities; information delivery preferences; and feedback on sample DA information. Patients were compensated $40. PCPs and nurse practitioners in family medicine or general internal medicine clinics were invited to participate in a 30-minute interview via email or practice-wide meetings. PCP interview guides, pilot tested with clinical team members, systematically covered mammography counseling practices, including discussion of breast cancer risk and screening potential benefits and harms; the use of risk assessment tools (including DAs); and shared decision-making (SDM) experiences. Both PCP and patient interview guides were informed by IPDAS guidelines to align with Delphi panel content domains. PCPs were entered into a raffle to receive $200 after study completion. NVivo 12.6.0 qualitative data management software was used to manage qualitative interview data and facilitate analysis.

Data Source 3: Expert Delphi Panel

From May to July 2020, a three-round modified online Delphi panel was conducted among a planned convenience sample of 7 to 10 breast cancer screening and decision science experts to determine content for the breast cancer screening DA. We used the Delphi process to elicit feedback from content experts in successive rounds to build consensus on this complex topic.[12-14] Experts known to the study team from across the United States were invited to participate via email. Delphi panel sample sizes vary from <10 to 100+ depending on the research topic breadth and panelist expertise, heterogeneity, and availability. Our narrow focus and multiple participant inclusion criteria restricted the population of potential participants. Subsequently, 15 experts who met the following criteria were invited: had an advanced degree (i.e., MD, PhD); at least 3 years of either professional, clinical, or research experience; and at least one publication within 5 years related to breast cancer screening, health literacy, decision making, and/or DA design. The principal investigator was blinded to participants, and participants were blinded to one another to minimize social desirability bias. Online Delphi surveys were administered in Qualtrics Management Software and sent via secure email. In Round 1, experts were asked to rate the importance of including 55 statements in a DA designed to be inclusive of women with LHL. The statements were developed based on the IPDAS Checklist. Statements covered topics including screening options, potential benefits and harms of mammography, preferences and values, breast cancer risk factor information, and communication strategies (see Supplement Table 1 for a list of statements). Experts rated each statement on a Likert-type scale from 1 (not important to include) to 9 (very important to include). Experts were encouraged to provide open-ended rationales for their ratings. In Round 1, experts were invited to nominate new statements for review to ensure salient topics were not overlooked. Two statements were added during this process.

Prior to Rounds 2 and 3, experts received a summary of consensus determinations from the prior round; the distribution of ratings among experts with a notation of how the expert rated the statement; and a summary of qualitative comments. Expert panelists were encouraged to review the summary prior to completing the next round. Participants received a $100 debit card after completing all three rounds.

Following the RAND Appropriateness methodology, we calculated a disagreement index (DI; equal to the [70th – 30th interpercentile range]/[interpercentile range adjusted for symmetry]). If the DI was >1, indicating disagreement, the item was moved to the next round. If the disagreement index was <1, and the median was between 4 and 7, the item was categorized as “uncertain” and moved to the next round. Items with medians x≤ 4 or x≥ 7 were categorized as “negative consensus” or “positive consensus,” respectively, and not considered in subsequent rounds. This higher threshold for determining consensus provided opportunities for experts to discuss inconclusive statements in greater detail. Qualitative comments were organized by topic and by round and systematically coded during the data synthesis phase in preparation for triangulation with other data sources.

Synthesis

This article reports on the synthesis of the described data sources. The Delphi quantitative results were used to identify areas of agreement and disagreement among experts. The qualitative, open-ended comments from experts were coded into the major topic areas of the Delphi, and then subcoded using emergent theme labels that described the expert’s perspective. Based on emergent themes from the Delphi qualitative data, we completed a secondary analysis of the patient and provider interview data. Researchers identified corresponding codes in the patient and provider codebooks and further subcoded salient interview data, as well as reviewed transcripts for emergent themes not captured in the original codebooks that aligned with emergent Delphi codes. As described by Boeije, codes were first developed and compared at the level of the individual participant, followed by comparisons within stakeholder groups, and finally comparisons were examined across stakeholder groups. To facilitate multi-stakeholder analysis, researchers created grids that organized potential crosscutting themes and exemplary quotes by each stakeholder group. Study team members reviewed and refined these grids, identifying themes for which all stakeholder groups presented robust data. Below we present the results of the Delphi panel, followed by four salient multi-stakeholder themes relevant to DA development.

Results

The sample demographics for all three data sources are provided in Table 1. Eight of the 15 invited expert panelists (53%) completed Round 1 of the Delphi, with seven of eight (88%) retained through Round 3. Within the eight-person Delphi sample, five panelists were medical doctors or nurses in internal medicine and three were researchers without clinical degrees; five panelists self-identified expertise in DA development; four in each of breast cancer screening, health literacy, and decision-making; and two in risk assessment. Delphi results are detailed in the Supplement. In brief, there were five items that unanimously were rated as important, with a median score of 9: stating that women can choose not to get a mammogram or get one at a future time; describing the benefits of screening; describing the harms of screening; and describing false positives and overdiagnosis as possible screening harms. The inclusion of breast self-exam as a screening option was unanimously rated as unimportant (median = 1.5). After three rounds, the panel did not reach consensus about including statements relating to what women need to do to prepare for the mammogram, and the process of getting a mammogram.

Table 1

Sample Demographics by Data Collection Method

	Women with Limited Health Literacy (N = 25)	Primary Care Provider Interviews (N = 20)	Delphi Panelists (N = 8)
Age, in years, mean (SD)	46.3 (4.75)	Not collected	53.2 (7.2)
Race, n (%)
Black/African American	18 (72)	Not collected	0
White	2 (8)		6 (75)
Asian	0		2 (25)
Not available	5 (20)		0
Ethnicity, n (%)
Hispanic/Latina	3 (12)	Not collected	0
Non-Hispanic/Latina	22 (88)		8 (100)
Gender
Female, n (%)	25 (100)	15 (75)	8 (100)
Male, n (%)	0	5 (25)	0
Number of years in practice (clinicians only), n (%)			N = 5
≤5 years	Not applicable	8 (40)	0 (0)
6–10 years		6 (30)	0 (0)
11—20 years		2 (10)	2 (40)
>20 years		4 (20)	3 (60)

Multi-Stakeholder Synthesis

The expert, PCP, and patient analysis resulted in the development of four themes relevant to DA design. Each is described below, and individual aspects of convergence and divergence by stakeholder group are represented in Figure 1.

Figure 1

Summary of Thematic Synthesis Across Stakeholder Groups

Benefits and Harms of Screening

Our multi-stakeholder comparison revealed diverse perspectives on what is considered a harm of mammography and in groups’ willingness to accept particular harms. Patients in our sample had the least awareness about harms. Patients named pain as the primary downside or harm of mammography that should be communicated, despite it not being considered a harm by other stakeholders. Patients who talked about harms also often made statements about how clinicians or the medical system should work to avoid such harms. This was true both of pain avoidance, and false positives, which by some were perceived to be “mistakes”:

The pain. The way people talk about that pain. . . . I think that’s the only downside to it is the pain. . . . Put us to sleep or something. (Patient 18)

So sometimes some people get false positives. So if they’ve been doing it for this long, at this point, I was like, “Well how do you still make that kind of mistake?” (Patient 20)

Frustration with the harms of mammography and the feeling that these should be mitigated were represented only in the patient perspective.

PCPs, similar to patients, acknowledged the importance of preparing women for possible pain during the mammogram, but emphasized false positives as the primary harm associated with mammography, which was aligned with expert framing of harms. PCPs expressed the various ways they discussed mammography harms with their patients, albeit only one reported ever providing detailed estimates of false positive and true positive rates with patients. Few PCPs viewed the presentation of false positives as essential, while other PCPs perceived the topic to be too complex for patients. Furthermore, discussion of harms appeared to relate to the PCP’s overall preference for age of screening initiation and appeared to vary based on PCPs’ preferred clinical guidelines, which provide different age-based recommendations. Harms were less valued by PCPs who believed that mammography was life-saving and who sought for all patients to receive a mammogram:

They need to know what to expect and that nothing is perfect. But spending a lot of time going into all the negatives of a mammogram, I don’t feel like is helpful. Because ultimately we want to promote everybody get it. (PCP 11)

I’m kind of a snail when it comes to cancer screening. I want there to be strong evidence around it. And I do counsel a lot about the downsides of screening. Like finding things that may not progress to cancer, finding things that need biopsies, the anxiety that can be generated around repeat screenings. (PCP 18)

In sum, PCPs as a group described the importance of emphasizing mammography benefits with patients. For those who described the need for harm information, the nature of harm descriptions was driven by a PCP’s clinical judgment and preferred clinical guidelines about the timing of a patient’s mammography initiation and frequency.

While sharing the benefits and harms was universally endorsed as important by experts, qualitative comments highlighted challenges related to available evidence as well as communicating complex risk estimates. As Expert 2 stated,

There is clear evidence for overdiagnosis and overtreatment as breast cancer screening harms; however, it is possible that we only have a conservative estimate of the rates. The degree of both and the level of harm from the patients’ perspective may exceed our expectations. Best to be humble and cautious here, not downplaying the importance of overtreatment, until we can say with more certainty how badly this hurts women.

Experts noted the challenges in communicating testing features such as false negative and true positive rates: “The issue with presenting true positives is that a proportion will be overdiagnosis, but it still seems important to present women with the expected true positive and false positive changes” (Expert 5). Other experts stated concerns about the complexity of presenting this information: “These are challenging concepts in general and I am concerned about the ability to accurately convey the meaning of these chances in a way that is easily understandable to those with limited literacy (which often may include limited numeracy)” (Expert 3). So while the importance of delineating the nature and scope of harms was a priority for this expert panel, the task of doing so in a way that resonated with women with limited health literacy was acknowledged as a practical challenge. The sampled experts recognized that health literacy, and more important, numeracy skills of the user, are critical in conveying risks and benefits.

Screening Modalities

Decision aid standards set by IPDAS require that all evidence-based options be provided to users to facilitate informed decision making. Experts in this panel had some disagreement about the evidence surrounding various screening modalities beyond mammography, namely, magnetic resonance imaging (MRI) and ultrasound. Furthermore, the distinction between modalities, especially digital mammography and tomosynthesis, was perceived to be of marginal value across all three stakeholder groups. Patients, by and large, were unfamiliar with most modes of screening and primarily named mammography and breast self-exam as effective screening modalities. As noted above, the Delphi panel unanimously rated breast self-exam as of low importance to address (median score 1.5). Experts cited limited evidence for self-exams limited and suggested they be omitted from a DA or described as available, but insufficient: “Given the poor evidence regarding clinical and self-breast exams, if that information is presented it should be in the context that those aren’t sufficient” (Expert 7). Yet patients called on breast self-exam’s utility, and it had widespread support as a viable screening tool:

The self-examinations that I do in the shower and coming to my own physician and them doing the finger examination as well, that one I tend to have more confidence in as far as touching. (Patient 5)

So while patients articulated that they found reassurance in breast self-exam, PCPs and experts indicated that they did not support providing information on breast self-exam as a screening modality, consonant with national guidelines.[18,19]

Experts questioned the utility of providing information on modalities that were potentially unavailable to the screening population. As Expert 5 described, “Persons may not have a choice between digital mammography or tomosynthesis and the evidence for efficacy primarily reflects both modalities. Many health care systems may have primarily one or the other technology.” While many experts felt that “evidence supporting the 20% [risk threshold for MRI] recommendation by [the American Cancer Society] is weak” (Expert 1), they also acknowledged that MRI is relatively common in the general population and DAs may benefit from mentioning that MRI may benefit some women. This was congruent with PCPs’ perception that some minimal information about MRI should be addressed as patients often inquire about other modalities that their social contacts have had: “‘My mom’s getting an MRI, do I need an MRI?’ . . . Well, we are like, here we’re sitting and talking about you. What you read in the letter, what you read here, and there is about everybody. This meeting is about you” (PCP 11). Modalities that were not routine were thus felt to be extraneous to core DA content.

Mammography Preparation and Process

The inclusion of mammography preparation and process information within a DA was a source of disagreement among Delphi panelists. Thematic analysis of the open-ended responses revealed that the source of disagreement stemmed from whether including this information was a fundamental component of decision making, or not. Expert 5 noted, “This seems distinct from the decision to have a mammo[gram], and thus seems lower priority to be a part of the decision-aid itself.” Over subsequent rounds, most agreed with the fact that it might not be part of the decision, but noted, “Understanding what to expect aids in satisfaction. Importantly, knowing how/when results are provided is key to alleviating anxiety” (Expert 4). Largely, PCPs echoed expert opinion that process and preparation information was helpful but nonessential. However, their reasoning here underscored a belief that women knew what to expect already. As PCP 3 stated, “I would say 100% literally, regardless of the language that they speak or their education level, understand what a mammogram is.” Many PCPs acknowledged they counseled less on the procedure than other tests: “I don’t spend as much time on describing what it is or what it’s for, as much as I do for colon cancer screening” (PCP 16).

Women in our study, however, shared that 1) they were not as ubiquitously knowledgeable about mammography as PCPs predicted and 2) they perceived process information about mammography to be fundamental to their own decision-making, in a contrast to both experts and PCPs:

I don’t know what mammogram means because I don’t know what you do when you’re having a mammogram. (Patient 22)

So yeah the information ahead of time is important to me. That way I can make my mind up and say, “Okay, this is what I want to do,” or “I don’t want to do this” . . . Yeah, it is important. (Patient 6)

In sum, our sample of women with LHL expressed a strong desire for process information as a critical component of their decision making, while experts and PCPs perceived this information to be less fundamental.

Breast Cancer Risk Factors

Experts agreed that at a minimum, it is important to tell women what the baseline risk of developing breast cancer is for the average women. Expert 8 stated that relaying a baseline rate of cancer “is important because it anchors women’s baseline risk. Baseline risk is a key element to understand the beneficial effect of screening compared to the harms.” Experts further agreed that using some sort of risk model makes a DA much more desirable and useful: “If women in the target pop[ulation] are capable of reporting their risk factors, then why wouldn’t you want them to use personalized risk info[rmation], rather than population risk?” (Expert 6). Experts brought up concerns about how to accurately and effectively provide individual risk scores that addressed the needs of women with LHL. They suggested that complicated information delivery formats, including complex numbers or data visualizations, could hamper patients’ understanding and ability to make choices consistent with their true preferences. While experts were largely in favor of including risk calculations in a DA, few of the PCPs indicated that they used risk assessment tools, or felt comfortable doing so:

In the last year, maybe I’ve used it once or three times . . . the Gail Model. . . . But it took me so long to figure out how to use it. . . . I think one of the things that it’s hard with the calculator, is that they show you how to use the calculator to get a number, but having those algorithms of what to do with this number, or with this risk, and then that will also help too. (PCP 1)

For experts, personal risk calculations were perceived as essential, but seldom used by PCPs in practice. Patients in our study did seek additional information on what might increase their risk of cancer. Patient 6 commented, “What would put me at risk? Like what are some of the reasons why women get breast cancer and what could be done about it?” Women valued such knowledge, but most did not receive such information in the course of breast cancer screening counseling and decision making.

All stakeholder groups identified breast density as an important yet challenging concept to address, especially for women with limited health literacy. Experts recommended that breast density be included in risk discussions, although both experts and PCPs described limited guidance for addressing breast density in screening:

We lack evidence based guidelines for how to modify screening procedures based on breast density but given the high level of inclusion of density on mammogram reports, it’s important to incorporate this information in order for women to interpret report findings. (Expert 5)

Experts and PCPs commented that breast density information requires careful wording to ensure understanding among patients with LHL. Such concerns were borne out in patient interviews, with many women suggesting that breast density was synonymous with cancer and articulating worry about breast density.

Discussion

The synthesis of three stakeholder groups describes experiences with and desire for breast cancer screening decision-making information in a DA for women with LHL. This analysis suggests that multi-stakeholder research may help identify and respond to varied information and design preferences among patients with LHL, providers, and experts, ultimately enhancing the use of DAs to facilitate SDM.

Our work identified a heterogeneity of opinions among stakeholder groups. This heterogeneity presents challenges in creating a tool responsive to the differing needs of all stakeholders. While experts and PCPs prioritized providing evidence-based options only, emphasized false positives as the primary mammography harm, and downplayed the relevance of delivering mammography process information, patients had a broader scope of harms (e.g., pain), wanted more information on breast self-exam, which evidence does not support,[18,19] and viewed process information as fundamental to their decision making. This is consistent with research that indicates patients may conceptualize cancer screening, including associated harms and benefits, differently than experts or providers with whom they are engaged in SDM processes. Likewise, there is potential for tension between stakeholder groups in the purpose of using DAs to achieve shared decisions that warrants further exploration. For example, in our study, PCPs preferred sharing information on multiple screening modalities, while some experts suggested that evidence-based modalities unavailable in patients’ health care contexts could be omitted from DAs. Such disparate perspectives highlight how perceived actionability of information may inform comprehensive SDM in the short- and long-term. Prioritizing information that is actionable within patients’ immediate health care environments over information that is not actionable currently, but may be in the future, may result in SDM that is responsive to the moment, but undercuts patients’ broader knowledge development and agency in future SDM. Ultimately, these discrepancies pose a challenge in making decisions about what to include in a DA when incorporating multiple stakeholder perspectives.

When developing DAs, the context in which they are delivered remains critical to their implementation, as previously described.[22,23] Our findings about the importance of process information to women with LHL both with and without prior mammogram experience, provides one example where it appears expert and PCP expectations and practices are misaligned with patient priorities. Women’s desire for mammography process information as part of decision making has been previously reported, reinforcing that patient centered tools might consider its inclusion. This analysis underscores that consideration of where this information is best delivered should take into account current practices and ensure the coverage of essential information across venues of information exchange (i.e., PCP visits v. DAs).

Though all stakeholders in our sample valued risk information, PCPs did not routinely provide it to patients with LHL and experts raised concerns about the ability of patients with LHL to digest complex risk information. This is consistent with work indicating that providers modify their risk discussions with patients with limited numeracy skills. However, patients in our sample, the majority of whom scored incorrectly on numeracy items, identified risk information as a high priority for breast cancer SDM. Proven strategies may help overcome these expert and PCP-perceived numeracy-related barriers to sharing risk: Studies have shown precise risk estimates using decimals may be recalled less and perceived as less credible relative to an integer-based risk estimate. But, personalizing information through self-assessment and feedback can enhance the efficacy of DAs, and be particularly beneficial to those with lower education. Furthermore, studies highlight the importance of design in effectively conveying complex information to LHL groups, supporting experts’ recommendations to share complex information such as testing features (i.e., false negative rates) with patients, despite challenges in doing so. The use of varied and/or multiple mediums (e.g., text plus pictures) and including an interpretation of numerical values can improve accuracy of perceived risk and comprehension, and lower thresholds to engagement in SDM.[29,30] However, formatting elements including directionality and color may be interpreted differently among patients and should be tested with LHL groups before use. Finally, the need to address provider discomfort with risk assessment tools and build skills around risk communication can enhance SDM. Ultimately, without a comprehensive understanding of what PCPs do/do not cover and what patients with LHL prefer in DAs, the potential for unmet information needs is high, as we have documented through this analysis.

DAs are designed to be evidence based and include a presentation of all evidence-based options. In our findings, we identified disagreement about what is considered an evidence-based option (breast self-exam) or harm (pain) among stakeholders. For teams developing DAs, the role of experts in defining evidence warrants scrutiny. Others have recommended that patients should be presented with information they deem relevant to their decision rather than what scientists would prefer. Our findings demonstrate that work may be required to establish agreement about the scope of information discussed during breast cancer screening decision making, either through DAs or in patient-provider conversations. Providing patient-centered information could be empowering, even if it falls outside the scope of guideline-determined evidence-based content. There is some risk, however, associated with presenting non-evidence-based screening options, even if only to say that evidence does not support them. Recent studies have pointed to “backfire” effects in attempts to address vaccine misinformation, whereby misinformation, even if presented as a myth, can be reinforced. If misinformation, including common misperceptions, is presented, it should be labelled as such and paired with corrective statements clearly and concisely. Thus, the decision to include information about low value or non-evidence-based options, such as self-exams, should be weighed carefully, with considerations of the patient population; DA design, specifically sequencing of text; context in which a DA will be implemented; and with patient input.

While most DAs include patients in refining content, the lack of inclusion of patients as advisors (9%) or partners (8%) in the design and maintenance of DAs offers a possible reason for how a disconnect in priorities may arise, as we observed in the differing values stakeholders placed on including mammography process information. At a higher level, the composition of the IPDAS steering committee and other expert groups appears to lack formal patient representation despite patient involvement being a stated priority. Patient involvement in guideline development has been shown to affect the inclusion of patient-relevant topics, outcome selection, and recommendations for implementation and dissemination. Centering patient priorities within IPDAS and others seeking to certify DAs creates the potential for creating more inclusive, patient-centered DA standards that explicitly elicit and address patient priorities for decision making. Finally, community-engaged research demonstrates how patient involvement can develop more efficacious measurement tools and user-friendly education materials for patients with lower socioeconomic status and LHL engaged in breast cancer care decision making.[36,37] The National Cancer Institute’s commitment to using community-engaged research to reduce cancer disparities offers a promising foundation and resource infrastructure for future health literacy-focused efforts.

Limitations

Caution in interpreting the priorities of each stakeholder group should be undertaken. Patient and PCP samples were drawn from a single institution, and thus ultimately reflect local practices. The sample size for the Delphi panel was small, rendering consensus determinations potentially less stable than if a larger panel size had been generated. This is likely in part related to low recruitment due challenges posted to the COVID-19 pandemic, which was cited as a reason for nonparticipation. Patients included in the study all had LHL and most identified as Black. The study sampled solely on health literacy status, and the overrepresentation of Black women is, in part, a function of the patient population served by the healthcare system in which the research was conducted. Though racial homogeneity was unanticipated, higher than expected inclusion of participants from historically marginalized populations may be advantageous, ensuring that their perspectives will be included in our future DA content. Including members of this population further provides opportunities to address potential gaps in information provision arising from the emphasis of professional stakeholder involvement over the patient voice.

Conclusion

This multi-stakeholder synthesis highlights the need for DA developers to garner multiple perspectives in designing tools. Evidence and guidance on best practices in integrating diverse and potentially divergent input across groups is needed. Finally, while the field of decision science is a leader in promoting collaborative development of patient-centered DAs, greater integration of the patient voice, especially of those who have LHL, requires additional efforts to promote truly inclusive decisional tools. Our findings will contribute to both the development of a DA responsive to the preferences of women with limited health literacy (e.g., inclusion of mammography process information) and the broader goal of building tools that enhance health equity.

Click here for additional data file.

Supplemental material, sj-pdf-1-mpp-10.1177_23814683211033249 for Using Mixed Methods With Multiple Stakeholders to Inform Development of a Breast Cancer Screening Decision Aid for Women With Limited Health Literacy by Christine M. Gunn, Ariel Maschke, Michael K. Paasche-Orlow, Ashley J. Housten, Nancy R. Kressin, Mara A. Schonberg and Tracy A. Battaglia in MDM Policy & Practice

Click here for additional data file.

Supplemental material, sj-pdf-2-mpp-10.1177_23814683211033249 for Using Mixed Methods With Multiple Stakeholders to Inform Development of a Breast Cancer Screening Decision Aid for Women With Limited Health Literacy by Christine M. Gunn, Ariel Maschke, Michael K. Paasche-Orlow, Ashley J. Housten, Nancy R. Kressin, Mara A. Schonberg and Tracy A. Battaglia in MDM Policy & Practice

Click here for additional data file.

Supplemental material, sj-pdf-3-mpp-10.1177_23814683211033249 for Using Mixed Methods With Multiple Stakeholders to Inform Development of a Breast Cancer Screening Decision Aid for Women With Limited Health Literacy by Christine M. Gunn, Ariel Maschke, Michael K. Paasche-Orlow, Ashley J. Housten, Nancy R. Kressin, Mara A. Schonberg and Tracy A. Battaglia in MDM Policy & Practice

Click here for additional data file.

Supplemental material, sj-pdf-4-mpp-10.1177_23814683211033249 for Using Mixed Methods With Multiple Stakeholders to Inform Development of a Breast Cancer Screening Decision Aid for Women With Limited Health Literacy by Christine M. Gunn, Ariel Maschke, Michael K. Paasche-Orlow, Ashley J. Housten, Nancy R. Kressin, Mara A. Schonberg and Tracy A. Battaglia in MDM Policy & Practice