Survey of physician attitudes to using multisource feedback for competence assessment in Alberta.

BACKGROUND: The use of multisource feedback (MSF) for assessing physician performance is widespread and rapidly growing. Findings from early very small research studies using highly selected participants suggest high levels of satisfaction and support. However, after nearly two decades of experience using MSF to evaluate all physicians in Alberta, we are sceptical of this.
OBJECTIVES: To determine physicians' actual opinions of MSF using the entire physician population of Alberta, Canada
DESIGN: Online survey.
SETTING: Alberta, Canada. PARTICIPANTS: All physicians with a full licence to practice in Alberta in 2015.
INTERVENTIONS: All participants were asked to grade how well they thought MSF was at assessing various aspects of physician performance using a 10-point Likert-type scale. There was also a text response field for written comments. OUTCOMES: Mean responses to quantitative questions. Qualitative content and thematic analysis of open-ended text responses.We analysed the data using SPSS V.23 and NVivo V.11 and built a multivariate model highlighting the predictors of high and low opinions of MSF.
RESULTS: Survey response rate was high for physicians with 2215 responses (25%). The mean rating for how successful MSF was at assessing a variety of dimensions, varied from a low of 5.03/10 for medical knowledge to a high of 6.38/10 for professionalism and communication. Canadian-trained MDs rated MSF significantly lower on every dimension by approximately 20% compared with non-Canadian-trained MDs.
CONCLUSIONS: Alberta physicians have much lower opinions about the ability of MSF to measure any dimension of their performance than what has been suggested in the literature. Canadian-trained MDs have a particularly low opinion of MSF for reasons that remain unclear. The results of this survey offer a serious challenge to the effectiveness of a programme that is designed to promote self-reflection and performance improvement. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Multisource feedback is of considerable interest to physicians Worldwide given it is increasingly being used as part of Quality improvement (QI) and Quality assurance (QA) assessments.

This is the largest study of its kind to date.

We used a mixed-methods analysis that allowed for open-ended text responses to give an in-depth insight into the quantitative responses.

We used multivariate statistical methods to analyse the quantitative data.

Survey was anonymous but some physicians may still have been uncomfortable giving honest answers to the medical regulator.

Introduction

Multisource feedback (MSF) also known as 360-degree or multisource or multirater feedback has been used as a tool for quality improvement and quality assurance in healthcare for over 20 years and in the business world for even longer.1 2 One advantage of MSF is that it can provide a physician with a much broader range of personal feedback (eg, medical expert, communication, administration and professionalism) from many different sources (patient, coworker, peer, self and leader) than might be obtained traditionally.1 3 In addition, because feedback from many patients and multiple peers and coworkers is typically obtained and aggregated, this should average out any more extreme outlier views that assessors might express, hopefully giving a more balanced overall view.4 5 The overall psychometric properties, in terms of the reliability, validity and feasibility of the tool have been reported to be good.3 6 7

However, despite the widespread and rapidly growing use of MSF for both formative and summative assessments of physician competence and performance, the actual views of physicians regarding MSF have not been well explored. Previous studies of physician attitudes to MSF in Canada and Worldwide have shown very high rates of satisfaction with MSF, for example, in one paper from Alberta 100% of 24 physicians felt that ‘(MSF) was a helpful educational exercise…’.3 Another study of 308 physician volunteers found two-thirds of those who responded to a poststudy satisfaction questionnaire (numbers not given) said they were contemplating or had made changes to their practice.1 Other studies have typically surveyed similar small numbers of highly self-selected physicians.8–10 A survey of 249 junior doctor attitudes in the UK found positive attitudes to MSF but a low perceived opinion about the effectiveness.11 This is surprising, given that a key assumption of MSF is that the ‘weak’ areas or opportunities for practice improvement identified will encourage physicians to make changes and improve in those areas. If the physician being assessed is not supportive of the process, then one can hardly claim that the individual will be likely to make changes. A qualitative study of consultant physicians in the Netherlands identified ‘lack of openness and constructive feedback’ as major barriers for the success of an MSF programme.12 Similarly, a qualitative study of opinions about MSF among allied health staff and paediatric residents in Canada showed strong interest in the concept of MSF but significant potential barriers to success, such as poorly defined roles and responsibilities, perceptions of expertise, hostile hospital culture and negative interprofessionalism and power dynamics.13 Only 1 of 16 studies looking at the effectiveness of MSF in physicians identified any significant positive change in actual behaviour and in that study, the treatment group also received ‘…a tailored coaching session to assist in identifying their strengths and weaknesses and in setting specific behavioural goals’.14–16 Furthermore, it is well established in the business literature that MSF programmes can cause significant negative effects with a decreased performance in up to one-third of all MSF programmes due in part to an ‘exacerbation [of] bureaucracy, heightening [of] political tensions and consumption [of] enormous numbers of hours’.17–20 Also giving good feedback is not easy, ideally the source should be credible and the information should be ‘SMART’: specific, measurable, attainable, relevant and time-bound.21 Additionally, significant anxiety and distress can be imposed on individual physicians who are told they are in the ‘bottom 10%’.22 Several small research projects of selected volunteer physicians who were used as participants to test new MSF programmes or components of programmes found high rates of satisfaction of around 70%.3 9 23–25 However, these results can hardly be said to represent the views of the majority of physicians. Such contradictory findings in the literature have created an urgent need to better understand physicians’ attitudes and opinions regarding MSF.

Since 1999, all physicians in Alberta were mandated to undergo a competence/performance assessment at least once every 5 years, in accordance with Alberta’s Health Professions Act.26 The assessment, also known as the Physician Achievement Review (PAR) programme, comprised an MSF assessment involving questionnaires from 8 peer physicians (colleagues), 8 coworkers, (eg, nurses, receptionists and physiotherapists), 25 patients and a self-evaluation.27 The physician’s aggregated scores were compared with reference data and presented in a report that ranked the physician in comparison with physicians with a similar practice on different aspects of performance (eg, similar to CanMEDS dimensions of medical expert, communication and professionalism).28 The terms ‘multisource feedback (MSF)’ and ‘physician achievement review (PAR)’ can essentially be used interchangeably, but for simplicity, we use MSF exclusively in this paper.

We report on the findings of a survey of the entire physician membership in Alberta in 2015. The objectives of this survey were: (1) to discover physicians’ actual opinions regarding MSF; (2) to see if different types of physicians (eg, male vs female individuals; older vs younger) varied in their opinion of MSF.

Methods

This was an electronic online survey sent via email to all physicians fully licenced to practice medicine in Alberta, Canada in 2015 (n=approximately 9,000). The research team initially developed a series of questions based on the existing literature and from questionnaires and surveys commissioned by the CPSA previously. These questions were then pretested on most of the physicians who worked at the CPSA (a general surgeon, an occupational medicine physician and five family medicine physicians). The draft survey was then tested and reviewed by a special advisory committee (the ‘pilot’ committee) set up to help with piloting, running and interpreting the survey results. The pilot committee consisted of representatives from various physician organisations (Alberta Medical Association, Canadian Medical Association, Alberta Health Services, University of Alberta and University of Calgary), Alberta family physicians (primary care network), a public representative, CPSA Council members, specialist and generalist physicians, CPSA staff and previous leaders of the PAR programme. The penultimate version of the survey was sent out to three family physician offices (approximately 30 family physicians) to additionally test the process for logging on and completing the online survey. Finally, we sent out an initial request and link via email to the final survey to all physicians in the province (wave 1) followed by a reminder request and link 2 months later (wave 2). We deliberately collected two separate ‘waves’ or responders to determine whether the demographics of the potential slow or non-responders differed significantly from each other or the whole population of Alberta physicians. Uniquely in surveys of populations, we already knew the exact expected demographic makeup (because the CPSA holds data on all physicians practising in Alberta) that can help to identify any potential skewing of the responses.

The survey consisted of 12 questions asking the physician how successful MSF was at assessing different areas of physician competence (medical knowledge, clinical skills, communication skills, professionalism, administration, practice management and team functioning) and how inspiring, motivating and reflective MSF was in their opinion. There was also a text response field for physicians to provide written, open-ended feedback and comments at the end of the questionnaire. Finally, because the survey was anonymous, we asked respondents to provide information on their gender, year of graduation, whether they were a family medicine physician/general practitioner (FM/GP), if they practised solo (compared with practising in a group) and whether they obtained their medical degree in Canada (Canadian MD). We used a 10-point modified Likert response scale to answer with three anchors (1= ‘not at all’, 5 ‘fairly’ and 10= ‘extremely’).

Quantitative responses were analysed using Statistical Package for the Social Sciences (SPSS) V.23 for Mac. Descriptive, bivariate and multivariate general linear model analyses were performed. The multivariate general linear model was created using the 12 survey questions as the dependent variables, and years in practice, Canadian MD, solo practice, family physician and gender as the independent variables. Missing data were excluded from the analysis. Any potential interaction terms were tested for and included in the final model. The internal reliability of the survey was measured using Cronbach’s alpha.

Qualitative (open-ended) text responses were examined using a thematic inductive qualitative content analysis by a member of the research team with experience conducting qualitative and mixed-methods research and analyses.29 NVivo V.11 Pro for Windows qualitative data analysis software was used to assist in grouping the open-ended text data into themes for analysis.

Patient and public involvement

A public/patient representative was a member of the survey pilot committee that helped to design the questions and the survey pilot initially. The same committee also reviewed the results and helped to interpret the findings. The results of the survey have already been disseminated to all physicians in Alberta via the monthly newsletter sent out via email to every physician in the province.

Results

A total of 2215 physicians responded out of 9021 fully registered Alberta physicians (response rate of 25%). Cronbach’s alpha for the survey questions was 0.964 indicating very high internal consistency for the survey items.

Wave 1 consisted of 1387 physicians and wave 2 consisted of 828 physicians (table 1). The waves were significantly different from each other in terms of age and proportion of Canadian MDs with wave 1 being slightly longer in practice by a mean of 4 years and having approximately 6% more Canadian MDs. Compared with the entire physician population, respondents (waves 1 and 2) were slightly longer in practice, more female, more FM, more Canadian MDs and substantially less solo practising physicians (table 1).

Table 1

Demographics of responders and non-responders

	Wave 1 (n=1387)	Wave 2 (n=828)	Wave1 vs wave 2 significance	All (n=9021)	Wave 1+2 vs all significance
Years in practice (SD)	26.0 (11.7)	22.9 (11.8)	<0.001	22.0 (12.2)	<0.001
Gender (% female)	38.8	41.4	NS (0.37)	37.2	0.02
Family physician (%)	51.8	57.1	NS (0.07)	48.5	0.001
Solo practice (%)	19.8	18.8	NS (0.67)	30.5	<0.001
Canadian MD (%)	73.8	67.3	0.02	66.6	<0.001

The mean rating for how successful MSF was at assessing a variety of dimensions, varied from a low of 5.03/10 for medical knowledge to a high of 6.38/10 for professionalism and communication (table 2). The overall mean and 95% CI for the 12 MSF survey responses was 5.7 (5.58–5.82) indicating the ‘true’ mean response is likely to be within about 2% of our value 95 times out of 100.

Table 2

Mean rating for question ‘Please rate how successful the existing [multisource feedback] programme is in assessing the following dimensions (1–10 where 1=not at all, 5=fairly and 10=extremely)’

Dimension	Mean score	SE
Medical knowledge	5.03	0.06
Clinical skills	5.17	0.06
Communication skills	6.38	0.06
Practice administration	5.36	0.06
Patient management	5.68	0.06
Adherence to standards of practice	5.34	0.06
Professionalism	6.38	0.06
Team functioning	6.15	0.06
Easy to participate	6.20	0.07
Inspires reflection in practice	5.69	0.07
Motivates clinical practice improve	5.58	0.07
Provide a learning opportunity	5.47	0.07
Overall	5.70	0.06

The multivariate General linear model (GLM) showed that only Canadian MDs had significantly different opinions compared with non-Canadian-trained MDs (table 3). All the other variables were not significant in the model and there were no significant interaction terms.

Table 3

Mean rating for question ‘Please rate how successful the existing [multisource feedback] programme is in assessing the following dimensions (1–10 where 1=not at all, 5=fairly and 10=extremely)’—stratified by Canadian or non-Canadian-trained MD

Dimension	Non-Canadian-trained MD	Canadian-trained MD	Multivariate analysis (p value)
Medical knowledge	5.71	4.74	<0.0001
Clinical skills	5.72	4.92	<0.0001
Communication skills	6.61	6.31	NS
Practice administration	6.04	5.08	<0.0001
Patient management	6.34	5.41	<0.0001
Adherence to standards of practice	6.11	4.98	<0.0001
Professionalism	6.73	6.30	NS
Team functioning	6.52	6.02	NS
Easy to participate	6.79	6.03	<0.0001
Inspires reflection in practice	6.35	5.43	<0.0001
Motivates clinical practice improve	6.29	5.31	<0.0001
Provide a learning opportunity	6.1	5.26	<0.0001
Overall	6.19	5.48

NS, not significant at 0.05 level.

Qualitative thematic content analysis of the 303 responses to the open-ended text box asking for ‘Additional Comments and Ideas’ regarding MSF, revealed that respondents overwhelmingly regarded MSF as negative. Overarching themes included the opinions that MSF is:

To me the [MSF] is an incredible waste of time and money. I do not know one MD who actually pays it much attention, either filling it out or implementing changes.

I believe that [MSF] is really useless as it currently operates. Major changes required.

The responses are biased to some or a large degree as the patients and peers that assess are chosen by the person being reviewed. If the person being reviewed is in an influential position, I'm not sure how accurate the review really is. I try to be as honest as I can when reviewing others, but wonder if that is always the case.

[I]find it [MSF] an extremely stressful exercise that I do every 5 years to maintain my license.

Discussion

The CPSA research team was genuinely surprised to find such a low/mediocre rating for MSF (on average, 5–6/10), given the previous much more optimistic data from published research studies and non-independent small surveys.1 3 8–10 23 Results from this research indicate that substantial efforts are required to investigate and potentially improve the utility of MSF in Alberta physicians. Successful MSF programmes in business tend to involve considerable investment in training reviewers beforehand, educating participants about the potential benefits of MSF and providing feedback directly to the participant to help with understanding and facilitate actionable change.30 These strategies were not used in Alberta with the PAR programme, and perhaps those who are considering applying MSF-type programmes to physicians should incorporate the valuable successes and variable lessons learnt from the business milieu.

Furthermore, the substantially lower ratings by Canadian-trained MDs on almost every dimension of MSF were an even greater surprise to the research team. Again, considerable thought and investigation are required to assist in explaining why there is such a marked difference in opinion between these two groups. One could surmise that Canadian-trained MDs are more familiar with performance assessment (and MSF in particular) and can be much more critical in their responses. Conversely, non-Canadian-trained MDs might feel more grateful or appreciative for the opportunity of receiving feedback, particularly if they had not been exposed to similar forms of feedback in their country of medical training. Additionally, non-Canadian-trained MDs might feel more vulnerable or less comfortable criticising the CPSA even though the study was anonymous and without consequence. Our survey was generic for all types of physicians and surgeons and it may be that a discipline-specific questionnaire might be more valid to allow for different working environments that various physicians experience. Also, the qualitative piece of our study was limited in its scope and not detailed enough to provide more than a tantalising glimpse into the reasons for these results. Future more in-depth qualitative research, such as individual interviews with physicians or focus groups with respondents, is needed to investigate the themes identified in this work further. Some of this work has already commenced in a separate but related project that explored the experiential knowledge of physician-assessors in identifying potential risk and support factors to physician performance.31

Survey participation is notoriously difficult in physician groups who are often bombarded with multiple different survey requests on an almost daily basis.32 33 Previous surveys from CPSA have met with response rates of about 10% and this may not be unusual even for physician professional organisations (CPSA. CPSA Survey 2014, Unpublished, 2014 May, pp 1–228). Our response rate of 25% was more than double the response rate of any previous survey by the CPSA, indicating that this is clearly an important topic for Alberta physicians. There is growing evidence that high-quality, rich and reliable information can be obtained from surveys with lower response rates (20%–25%) and in fact, they may be superior to higher response rate surveys.34–37 First potential reason being that initial ‘non-responders’ who are ‘forced’ to answer do not do so in a benevolent way, which can seriously undermine the validity of the findings.38 Second, there seems to be little if any evidence to support that the typical minimum response rate should be 60%–65%.39 40 Holbrook et al analysed results from 81 national surveys (with response rates from 5% to 54%) and found the extra expense and energy invested in trying to boost response rates may not result in worthwhile improvements in reliability or validity.41

Statistically, there was a difference between the different waves of respondents and between the entire population of Alberta physicians in terms of demographics, but given the large sample sizes, it may be very easy to find statistical differences even when the actual difference is small. Given that we know the composition of the responders and all Alberta physicians, we can predict the potential effects of any differences. Looking at the mean differences, wave 1 and 2 participants were very similar in composition between the entire population and the two waves, with one notable exception: there were appreciably fewer physicians in solo practice who answered the survey (20% vs 30%). Given that the responses from solo physicians were identical to 'non-solo’ physicians (with the exception that they rated ‘group feedback’ measures less importantly for obvious reasons), the validity of the overall results is established. Compared with the entire population of Alberta physicians, respondents were more likely to be female individuals, more likely to be family physicians and more likely to have practised longer, all of which would not skew the survey responses. There were slightly more Canadian-trained MDs, which might skew all opinions on MSF to be slightly worse than if the entire physician population had responded.

Given the way respondents rated MSF in Alberta, it is difficult to see how MSF could inspire the majority of physicians to make any significant changes to their practice, which is surmised to encourage the quality improvement of their performance. The mean score for ‘inspires reflection in practice’ was only fair at 5.69/10. About 30% of physicians did rate the MSF 70% or higher, causing us to question how these physicians might differ from those who rated MSF lower. The statistical analysis found that these higher-MSF-rating physicians are more likely to be non-Canadian-trained MDs. However, the business literature would suggest that these ‘high rating’ respondents may have unique personality variables that are receptive to MSF, such as high levels of emotional stability, extroversion and conscientiousness.42 Unfortunately, because the survey was anonymous, the personalities of the ‘high-rating’ respondents cannot be further explored. Our findings complement research already completed in Alberta, where 72% of a cohort of surgeons taking part in the PAR programme indicated initially that they were ‘contemplating or [had] initiated change on the basis of multisource feedback…’.10 However, after following-up with the cohort 3 months later, the same authors concluded that ‘surgeons made few changes in practice in response to feedback data’; postulating that perhaps the surgeons did not value the information they received from MSF because it was not based on surgical outcomes, ‘the cornerstone of surgical practice’.43

Physicians in Alberta have much lower opinions regarding the ability of MSF to measure the dimensions of performance compared with previous anecdotal reports and published literature to date on this subject. Canadian-trained physicians have a particularly low opinion of MSF for reasons that remain unclear. Further investigation into the conclusions from this research will allow for a richer understanding of these opinions and offer an opportunity for exploration into the appropriateness of the application of MSF for medical doctors. Individual interviews and/or focus groups with physicians to further explore the themes identified in this inaugural study may lead to an increased understanding of participants’ differing opinions of, and experiences with, multisource feedback as it pertains to performance. The results of this survey offer a serious challenge to the effectiveness of a programme that is intended to promote self-reflection and performance improvement in physicians.