Consensus-Based Development of an Assessment Tool: A Methodology for Patient Engagement in Primary Care and CPD Research.

ABSTRACT: With cardiovascular disease (CVD) posing a significant disease burden in Canada and more broadly, preventative efforts which incorporate best evidence, patient preference, and physician expertise must continue to take place. Primary care providers play a pivotal role in this effort, and a greater understanding of patient perspectives is needed to guide management and inform training. We used a validated consensus method, the nominal group technique (NGT), to identify patient-reported experience measures (PREM) related to CVD prevention deemed most important by both patients and providers. The NGT was used by using structured discussions between patients and providers to bring ideas about PREM CVD outcomes to a consensus. Four patient partners and four primary care providers were selected to participate in an NGT session. Each participant wrote down items/questions they believed important in CVD preventative care. After discussions, all items underwent anonymous ranking on a 5-point scale. Items were included/excluded based on 75% agreement a priori. The panel produced 10 items from a total of 26 after 2 rounds of ranking. The top two items were as follows: "Is your treatment plan tailored to you" and "Was your physician good at giving information about your risk factors?" These results are significantly different compared with existing quality measures because they highlight aspects of patient experience and therapeutic relationship. A questionnaire consisting of prioritized PREM items is valuable in quality improvement and continuous professional development (CPD).

Consensus methods are used in medical education and health services research to synthesize opinions and prioritize responses for decision-making, education, and assessment, among others.[1,2] In the context of primary care, techniques such as the nominal group technique (NGT), which aims to extract opinions and consensus on a nominal question, have been relied on heavily to synthesize expert opinions to develop consensus care guidelines.[2,3] For example, the NGT has been used to evaluate exercise adherence measures for patients with musculoskeletal disorders.[4] In continuing professional development (CPD), NGT has been used for small group learning and needs assessment.[5-7] However, there is a paucity of evidence and standardization in using consensus methods for generating patient-reported experience measures' (PREM) items for quality improvement and medical education more broadly.[8,9] For more effective NGT implementation in primary care and CPD, there should be an emphasis on composing a more diverse panel, beyond physician experts alone who compose existing CPD programming and care guidelines, to encompass varied patient–physician interactions and adapt training to trends in healthcare delivery goals.[10,11]

One such trend is the emphasis on prevention as seen in cardiovascular care. Understanding patients' experiences living with a health condition and their preferences for primary care services represent important parts of evaluating and improving care delivery to improve patient outcomes. These preferences should be incorporated into existing CPD programming and care guidelines.[11] Thus, using cardiovascular disease (CVD) risk factor reduction in primary care as an example, we describe a methodological study on the use of the NGT by both patients and physicians to codetermine important PREM items. In doing so, we hope to aid in the development of effective assessment tools that capture patient experience, guide CPD, evaluate practice interventions, and ultimately improve practice.[9,12] Moreover, readers can follow the steps in our example for consideration in their own projects.

METHODS

Study Design

This study described the involvement of both patients and physicians using NGT to build a consensus on cardiovascular health outcomes to be used as patient-negotiated assessment items for primary care and CPD. NGT was selected as it is a well-validated method for reaching consensus, especially based on input from experts.[1,13] In brief, NGT involves structured and iterative discussions between 5 and 12 participants with the goal of bringing individually and privately generated ideas to consensus among the larger group. Through this process of discussion, a large number of heterogenous inputs are created, each with the possibility of combination with others. The small number of 5–12 participants allows for robust and dynamic discussion during idea generation and is also more manageable for the facilitator should participants need facilitation or assistance.[14] Compared with other consensus methods such as Delphi, it is more commonly used among lay people and its facilitated nature ensures that all members are involved in idea generation and voting.[9] We followed a recommended and reproducible consensus group method protocol.[15]

Participation/Recruitment

Ethics approval was obtained at the local Research Ethics Board at the Montfort Hospital and Bruyère Hospital. Five patient members of the Montfort Hospital patient partners program with CVD treatment experience were invited to participate in the study. Notices for recruitment of physician panel members were sent to eight total eligible physician participants within the Ottawa Hospital and Montfort Hospital, two large academic teaching hospitals affiliated with the University of Ottawa. As part of the Canadian Institute of Health Research Strategy for Patient-Oriented Research initiative, which is a national initiative that encourages research to focus on patient-identified priorities, we recruited four patient partners from the Montfort Hospital to participate in the nominal group session. Four primary care providers (PCPs) from across the Ottawa region also participated in the session. These physicians were recruited through connections of team members seeking a purposive sample with diverse ages, gender, and practice models. All participants provided informed written consent to participate in the research study.

Procedure, Data Collection, and Analysis

A skilled moderator and expert in NGT led the session in June 2019, which was conducted in English. The project principal investigator first introduced the group to the objectives and purpose of the project, as well as the Canadian Primary Care Information Network (CPIN). This is an automated patient communication and survey system through which the PREM items developed by the group would be trialed as short patient surveys delivered by email or text message. Furthermore, the group was presented with a sample list of measures used to evaluate CVD preventative care and shown examples of high-quality patient education materials. This served to familiarize participants with current literature and help guide their ideas.

Initial items generation: All participants were asked to privately write down any items that they felt would be important in evaluating the quality of CVD preventative care delivered in primary care practices. Each participant was given 10 minutes to write down items as either questions or topics without restrictions to the content or form, in accordance with established protocols from other studies.[16,17] A total of 26 items (Table 1) were recorded on the flipchart paper by an investigator.

TABLE 1.

Codetermination of PREM Items During NGT

Item #	Initial items generated (n=26)	Discussion and combination of items (n=22)	Round 1 (n=9)	Round 2 (n=4)	Combination of items (n=1)	Top 10 ranked (n=10)
16	Is your treatment plan tailored to you?			I (Pt)		1
7	How good was I at giving information about risk factors?			I (P)		2
23	Did I involve you in decision-making?		I (P)			3
12	How comfortable are you talking about your risk factors?		I (Pt)			4
13	To what extent do you feel your physician has your best interest in mind?		I (Pt)			5
10	Were all your questions about CVD medications answered?		I (Pt)			6
15	Did your physician give you good advice for exercise and healthy habits?		I (P)			7
24	Did I explain your risk level well enough?		I (Pt)			T-8a
25	Do you know how to get help in case of a CVD emergency?		I (Pt)			T-8b
20	Did I advise you on ways to reduce your blood pressure?		I (P)			10
1	Is CVD inherited?	E
2	Do you know your blood pressure in the last year?	C		E
3	Did I inform you about your blood pressure results?
4	Are there factors besides blood pressure that contribute to heart attack/stroke risk?			E
5	Do you know the signs/symptoms of heart disease/stroke?			E
6	Do you want more information about risk factors?			E
8	Would it help if I provided you with websites for more info?	E
9	Could you tell your friends/family about your CVD situation?			E
11	Were you given a follow-up for new medication?			I	R	E
14	Do you need more info about community resources such as exercise facilities, smoking cessation, and support groups?			E
17	Did you have your blood pressure measure in the last year?	C		I	R	E
18	How many visits did you have your blood pressure measured?
19	Did I ask you to monitor your blood pressure and report back if elevated?			E
21	Did I explain CVD properly?			E
21	Did I explain the medications properly (side effects)?		I		R	E
26	Did I demonstrate awareness of continuity of care (up to date)?			E

C, combined; CVD, cardiovascular disease; E, excluded; I, included; NGT, nominal group technique; P, primary care physician-centered; Pt, patient-centered; R, recombined; T, tie.

Discussion and combination of items: Subsequent group discussions to combine similar items brought the total number to 22before private ranking.

Rounds: In a group setting, each member of the panel was asked to share their own ideas in a round robin format and moderated discussion subsequently took place for each item. Emphasis was placed on achieving collective understanding of each item. New ideas were generated, irrelevant items were excluded, and similar items were combined. This took place until all items from each panel member were discussed, and no new ideas were forthcoming. Throughout this process, the moderator ensured that there was equal participation among panel members but limited direction to avoid interference with group process; no individual dominated a discussion.

All panel members were then asked to anonymously evaluate each of the items generated based on their importance and relevance in being included in a patient questionnaire evaluating the quality of CVD preventative care delivered in primary care practices. Items were ranked on a five-point scale, with five being highest priority and one being lowest priority.[18] It was determined a priori that questions receiving 75% agreement of ranking 4 to 5 were included, whereas those receiving 75% agreement of ranking 1 to 3 were excluded. Scores were then tallied by the moderator. Any items not achieving consensus were subject to subsequent group discussion and further rounds of reranking. A total of two rounds of reranking were performed, as typically seen in NGT, to produce the top 10 items. Ten items was considered a priori as a reasonable number for patients to complete on a survey. The entire process took about 3 hours to complete.

RESULTS

A total of eight participants joined in the in-person NGT session. Among patient partners (n = 4), all represented both English-speaking and French-speaking populations within the Ottawa area. Among PCPs (n = 4), two joined remotely through telephone, and there were two servicing three socially diverse neighborhoods in the community.

Figure 1 outlines the NGT indicator generation process. Items that were included and excluded during successive rounds of discussion and ranking are listed in Table 1. In brief, 26 items were generated from initial independent brainstorming by panel members. After discussion to eliminate or combine similar items, 22 possible indicators entered the first round of rankings.

FIGURE 1.

Item generation process during NGT. NGT, nominal group technique.

Two iterations of group discussions and independent ranking were performed to identify the top 10 PREM items (Table 1, thick border) for assessing the quality of CVD prevention. After the first round of ranking, nine items met a priori criteria for inclusion. No items achieved a priori criteria for exclusion. Thirteen remaining items continued to a second round of discussions and independent ranking. Of these, four items met criteria for inclusion, whereas nine items met criteria for exclusion. After the second round of discussions, the final list of items was revisited to combine similar items based on discussion and consensus among the group members. Two items were combined, resulting in 10 total items generated. Four of the top 10 items were physician-centered, and 6 items were patient-centered assessment items.

DISCUSSION

Here, 10 highly ranked, patient and physician codetermined PREM items for evaluating primary care of CVD prevention are identified. Central themes of communication, patient–physician relationship, shared decision-making, and communication of risk and risk reduction methods underlie shared interests between patients and physicians. By bringing together physicians and patients with lived experience, we replicated the crosstalk, dynamics, and exchanges that occur during clinical encounters that are critical to influencing outcomes. This study suggests that measuring quality improvement or performance with typical indicators or disease targets alone may not completely capture the goals of care. Patients and physicians may have different views on specific factors that contribute to high-quality CVD care, but they may work together to reach a consensus.[19] This allows for the most accurate representation of the diverse views, values, and preferences of panel members while expanding on existing quality indicators that focus on evidence-based processes, disease management, or prevention targets.[11,19,20] In doing so, this study identifies aspects of cardiovascular care that may not be identified by physicians alone in generating existing quality indicators.[19]

Our results emphasize that, when considering patient perspectives in addition to physician priorities in the evaluation of cardiovascular primary care quality, items emphasizing the therapeutic relationship are prioritized over more conventional measures currently used. These are in line with the Quadruple Aim for Health Systems, which emphasizes improving patient experience as a key to delivering effective health care.[21] These results also mirror the priorities for general primary care performance measurement and reporting identified by in-depth engagement with patients, as well as work conducted in priorities of dialysis outcomes.[22-24] Only item 7 and 10 allude to traditional modifiable risk factors described in major Canadian, American, and United Kingdom guidelines.[25-27] Items one and three are a direct reflection of the long recognized importance of shared decision-making in the context of management of CVD and medication initiation.[19,28] Although the 2019 American College of Cardiology/American Heart Association (ACC/AHA) Guidelines recommend that health care providers engage patients in discussions about cardiovascular risk estimates and their implications in perceived benefits of preventative care, there is significantly less clarity about how to structure these types of discussions. This study suggests that risk factor communication may be a defining characteristic of high-quality cardiovascular primary care; it is a meaningful and clinically useful common ground between patients and physicians when engaging in shared decision-making (items 2, 4, 8a, and 10). For patients, their ability to understand risk depends on factors including their perceived level of control over their own health (item 4) as well as the concordance in communication styles between patients and physicians (items 2 and 8a). For physicians, risk communication involves first knowing the guidelines and novel research applicable to the clinical situation, understanding how this evidence applies to the individual patient, and presenting the risks in a way that is easy to understand (items 2 and 8a). Evidence shows that patients generally have a poor understanding and appreciation of CVD risk, and the way in which risk is communicated can influence patient comprehension, patient satisfaction, and may encourage adherence.[29] Strategies for improving risk communication have been reviewed elsewhere.[30]

Our findings contrast most available CVD outcome measures both inside and outside the primary care setting, which are developed by physician panels and medical expert discussion alone.[31-37] Studies have shown that physicians often mistakenly believe they know what patients' preference are about the initiation of medications for prevention of CVD, resulting in suboptimal shared decision-making.[19] They concluded that patient values are often unpredictable and unlikely to be captured by guidelines as they are currently developed. Similar results have been demonstrated among patients starting antihypertensive medications.[28] Furthermore, both the CANHEART Performance Indicators for Cardiovascular Ambulatory Care and ACC/AHA indicators use clinical outcomes as the primary measures for quality of care delivered,[35-37] whereas it is clear that patient experience does not correlate well with hard outcomes. If improving CVD care requires attention to and even measuring of patients' experience and trust in their health care provider, additional approaches to evaluating care to ensure routine engagement of patients in assessing their experience of care and communication may be required alongside typical chart audit and administrative data analyses.[38]

Further refinement (through criterion referenced validity and member checking) and pilot testing of the codesigned PREM items will be taking place in four Ottawa clinics using an automated patient survey system (CPIN). The reporting of these results may contribute to CPD by enabling comparisons in patient experience, sharing of patient ratings of outreach educational material, and sharing of the different types of information and surveys peers are using. As CPIN expands, this may lead to sharing information across borders. The creation of targets based on survey responses may be useful in improving individual practices by giving individual physicians feedback and comparisons on the subjective experiences of patients they are treating. This would build on objective feedback that physicians receive regarding their care, as delivered through patient report cards.[12] Further efforts to design measurement approaches which incorporate shared priorities, or acknowledge distinct priorities in performance goals, may offer a better understanding of our health system performance and more broadly.

Strengths and Limitations

Our process for identifying priority PREM items is unique in that patients, and providers were equally represented, and group facilitation ensured equal participation during the NGT process. In addition, no restrictions were placed on whether measures involved access, process, or continuity of care, outcomes, or patient–physician interaction. This is in contrast to the typical method used for developing performance measures for clinical care.[31-37] There are a number of limitations to this study example. First, it is a single center study involving participants from Ottawa, Canada, and CPIN is for now a regional and national level program. Therefore, it represents the views, priorities, and preferences of a group operating under a universal, single payer, and publicly funded health care system. The culture of patient and physician dialog may also vary between health care systems based on time, funding, and societal factors. As well, the selection of patient partners willing to participate in the NGT may skew results toward those most willing to be involved in their own care. This underlies the importance of patient–physician communication in all types of encounters to establish expectations and individual preferences.[39] Finally, although the involvement of eight total participants optimizes communication between panel members, it inevitably reduces the number of opinions that are considered. Expert moderation of NGT was used to ensure that all opinions, where present, were heard and discussed.

CONCLUSIONS

Health services' research is increasingly valuing the perspectives of patients when evaluating performance through the use of patient-reported outcomes, which can be identified using NGT.[11,21] In this example, we have generated PREMs which include highly ranked items on CVD prevention, which may be valuable in guiding PCPs in their practice improvement efforts. These indicators are different compared with existing measures as they are the product of interactions between patients and physicians, thereby presenting a unique perspective on patient experience that is not adequately captured by existing quality indicators. Further work to explore the feasibility, utility, and impact of these and similar PREM items will advance the field of performance measurement in health care.

Lessons for Practice

▪ The NGT can be used as a strategy in involving patients in the CPD planning process.

▪ Patient-negotiated assessment items generated through NGT can guide the evaluation of CPD programs.

▪ The patient as a member of the health care team should be empowered to be involved in the development of CPD programs.