Can the results of a randomized controlled trial change the treatment preferences of orthopaedic surgeons?

AIMS: The aim of this study was to investigate surgeons' reported change of treatment preference in response to the results and conclusion from a randomized contolled trial (RCT) and to study patterns of change between subspecialties and nationalities.
METHODS: Two questionnaires were developed through the Delphi process for this cross-sectional survey of surgical preference. The first questionnaire was sent out before the publication of a RCT and the second questionnaire was sent out after publication. The RCT investigated repair or non-repair of the pronator quadratus (PQ) muscle during volar locked plating of distal radial fractures (DRFs). Overall, 380 orthopaedic surgeons were invited to participate in the first questionnaire, of whom 115 replied. One hundred surgeons were invited to participate in the second questionnaire. The primary outcome was the proportion of surgeons for whom a treatment change was warranted, who then reported a change of treatment preference following the RCT. Secondary outcomes included the reasons for repair or non-repair, reasons for and against following the RCT results, and difference of preferred treatment of the PQ muscle between surgeons of different nationalities, qualifications, years of training, and number of procedures performed per year.
RESULTS: Of the 100 surgeons invited for the second questionnaire, 74 replied. For the primary outcome, six of 32 surgeons (19%), who usually repaired the PQ muscle and therefore a change of treatment preference was warranted, reported a change of treatment preference based on the RCT publication. Of the secondary outcomes, restoring anatomy was the most common response for repairing the PQ muscle.
CONCLUSION: The majority of the orthopaedic surgeons, where a change of treatment preference was warranted based on the results and conclusion of a RCT, did not report willingness to change their treatment preference.Cite this article: Bone Joint Open 2020;1-9:549-555.

Introduction

Published orthopaedic research is increasing.[1] Despite this increase, the proportion of Level I and Level II studies, even in some of the highest indexed orthopaedic journals, is still only 10% to 20%.[2,3] Furthermore, the majority of the published randomized controlled trials (RCT) have methodological deficits.[4] Several studies have also shown a delay in implementation, a lack of acceptance of new evidence, and subsequently an evidence-practice gap.[5-10] The implementation of evidence-based practice in orthopaedic surgery is poorly understood. The purpose of evidence-based medicine (EBM) is “doing the right things”.[11] For surgeons to change practice, outcome studies must have external validity, and the surgeons have to be convinced that in their hands this change will benefit their patients.[12] Research on factors of published articles that influenced decision making among orthopaedic surgeons identified four main features: study design (RCT rated highest); sample size (more than 100); reputation of investigators; and quality of the journal.[13] Several theories and models to aid implementation of EBM have been published.[14-16]

We sought to investigate a part of the implementation process in EBM: surgeons’ acceptance of the results and conclusion from an outcome study (RCT) to their reported change of treatment preference and to study different patterns of change between subspecialties and nationalities.

Part of the author group (JS, SB) performed a RCT on repair or non-repair of the pronator quadratus (PQ) muscle during volar locked plating of distal radial fractures (DRFs). The study was published online in November 2019.[17] Previous studies on the functional outcome between repair and non-repair of the PQ muscle were either retrospective cohorts or smaller prospective series with limitations to the methodology.[18-20] We saw this as an opportunity to study the impact of the results of a RCT on orthopaedic surgeons’ preferences, by surveying surgeons before and after the RCT was published. The results of the RCT were not known before planning this survey study and it has later shown no difference in functional outcome between the PQ muscle repair and non-repair group. The conclusion of the RCT publication was that there was no difference and thus the recommendation was not to repair the PQ muscle during volar locked plating of DRFs.

The null hypothesis was that the results and conclusion from a RCT would not change the treatment preference of the repair of the PQ muscle by surgeons who used volar locked plating for DRFs.

Methods

Study design

The study design was a cross-sectional survey of surgical preference. The study was approved by the South Western Sydney Local Health District Human Research EthicsCommittee in Australia on 19.06.2017 (HREC Reference: LNR/17/LPOOL/246, SSAReference LNRSSA/17/LPOOL/247). The study did not require ethical approval in Denmark.

Outcome measurements

The primary outcome was the proportion of surgeons for whom a treatment change was warranted, who then reported a change of treatment preference following the RCT. Secondary outcomes included the reasons for repair or non-repair, reasons for and against following the RCT results, and difference of preferred treatment of the PQ muscle between surgeons of different nationalities, qualifications, years of training, and number of procedures performed per year.

Study population

Due to the qualitative nature of the study, we aimed to gather as many replies as possible. Minimal clinically important difference (MCID) was defined from a previous study that estimated a MCID in change of treatment preferences in orthopaedic surgery at 15%.[21]

The invited population were Australian or Danish orthopaedic surgeons or registrars in orthopaedic surgery. The Australian surgeons were identified via the authors’ network (IH, JS) and the Australian Orthopaedic Association (AOA) webpage through the search function ‘find a surgeon’ with subspecialty ‘trauma’ (n = 140) or ‘Hand’ (n = 96). The search was performed in April 2017. A total of 236 surgeons were identified after removing duplicates. Email addresses were obtained through official webpages or telephone contact. In all, 22 of the 236 surgeons were either not interested, unavailable, or had incorrect contact details. From the authors’ network, 57 surgeons were identified and a total of 271 email addresses of Australian orthopaedic surgeons were included for possible participation. Due to EU regulations on data protection, the Danish orthopaedic surgeons could not be identified through webpages. The possible participants were therefore established through the authors’ network (SB, JS) and 109 surgeons were identified. The total number of surgeons from both Australia and Denmark was 380. Inclusion criterion for participation in the survey was medical doctor working within orthopaedic surgery in either Australia or Denmark. Exclusion criteria were: 1) not performing volar locked plating of DRFs; and 2) performing volar locked plating of DRFs through a ‘minimally-invasive’ approach. Figure 1 is a flowchart of the participants.

Fig. 1

Flowchart of participants.

Questionnaire design

Two questionnaires were developed for this study. The first questionnaire was developed through a Delphi process with 13 orthopaedic surgeons and trainees from Nordsjaellands Hospital. The second questionnaire was developed through consensus between the authors. The questionnaires are described in the Appendix section.

The first questionnaire was sent out before the RCT was published. This questionnaire had two sections:

Baseline data on participants: Years of experience, qualifications, performing surgery with volar locked plating of DRFs, numbers of procedures a year, performing volar locked plating of DRFs through a ‘minimally-invasive’ approach.

Preferred treatment of the PQ muscle during volar locked plating of DRFs: Repair or non-repair, if the treatment of the PQ muscle was dependent on different factors, reasons for the preferred treatment of the PQ muscle.

The second questionnaire was sent out after publication of the RCT. This questionnaire also had two sections:

Baseline data: Identification email.

Treatment preference changes based on the RCT publication: Establishing if the abstract or the article was read, preferred treatment of the PQ muscle during volar locked plating of DRFs (repair or non-repair), if the conclusion of the RCT warrants the surgeon changing treatment, if the surgeon will change preferred treatment of the PQ muscle, reasons for changing or not changing.

Questionnaire administration

The questionnaires were administered via email through the web-based survey programme SurveyMonkey (San Mateo, California, USA, www.surveymonkey.com). The first questionnaire was open from 9 October 2019 to 9 November 2019. The second questionnaire was open from 8 December 2019 to 9 January 2020. The RCT was published online on 30 November 2019. A reminder was sent out up to three times if there was no reply. The abstract of the RCT was attached and a link to the article included in the email for the second questionnaire.

Statistical analysis

Responses were imported from the survey programme to a survey-specific database using Excel 2019 (Microsoft, Redmond, Washington, USA). Data were analyzed descriptively. Incomplete answers were discarded. Categorical variables were reported as counts and, where relevant, also percentages.

Results

Overall, 74 out of 100 surgeons (74%) responded to the second questionnaire. This was 19% (74 of 380) of the surgeons invited to the first questionnaire. In all, 32 of the 74 (43%) routinely repaired the PQ muscle and 42 of the 74 (57%) did not repair. Baseline characteristics of the questionnaire participants are shown in Table I.

Table I.

Survey responder characteristics.

Characteristics of survey responders	n (%)
Surgical experience
< 10 years	80 (70)
> 10 years	35 (30)
Qualifications
Hand surgeons	32 (28)
General orthopaedic surgeons	27 (23)
Other subspecialty or not yet specialized	56 (49)
Number of operations per year
0	7 (6)
1 to 20	53 (46)
< 20	55 (48)
Performing minimally-invasive surgery
Yes	8 (7)
No	100 (87)
Not performing DRF surgery	7 (6)
Routinely repair PQ muscle
Yes	53 (46)
No	47 (41)
Not performing DRF surgery or do minimally-invasive surgery	15 (13)

DRF, distal radial fracture; PQ, pronator quadratus.

For the primary outcome, six of 32 surgeons (19%) (who usually repaired the PQ muscle and therefore a change of treatment preference was warranted) reported a change of treatment preference based on the RCT publication.

Secondary outcomes are shown in Tables II–IV.

In the cohort of 74 surgeons who did not usually repair the PQ muscle (and therefore would not be expected to change practice), no surgeons chose to change their practice. Further subgroup descriptions of practice preferences are shown in Table IV.

Table IV.

Difference between proportions of change of treatment preferences according to qualifications/years of training/number of procedures per year (volar plating DRFs).

Surgeons who do repair PQ routinely (n = 32)
Change of treatment preference		Will change and not repair (n = 6)	Will not change and continue repairing (n = 26)
Years in practice	0 to 10 years	3	4
	> 10 years	3	22
Number of procedures per year	1 to 20 years	1	12
	> 20 years	5	14
Qualifications	Hand surgeon	3	11
	General orthopaedic surgeon	1	8
	Surgeon with other specialty/not yet specialized	2	7

DRF, distal radial fractures; PQ, pronator quadratus.

Responses from the second questionnaire showed that routine repair of the PQ muscle was conducted by 25 of 41 Australian surgeons (61%) and seven of 33 Danish surgeons (21%). In all, 12 of 74 surgeons (16%) replied that they had read the full paper, and 54 of 74 surgeons (73%) had read the abstract only. Eight of 74 surgeons (11%) read neither the abstract nor the full paper. Three of 25 Australian surgeons (12%) and three of seven Danish surgeons (43%) who routinely repaired the PQ muscle replied that they would change treatment preference based on the result of the RCT publication.

Discussion

In this cross-sectional survey, we investigated orthopaedic surgeons’ treatment preferences of the PQ muscle before and after publication of a RCT. To our knowledge, this study is the first to survey orthopaedic surgeons before and after a RCT to assess a potential treatment change. The primary outcome of proportions of surgeons who routinely repaired the PQ muscle and would change practice was six out of 32. The MCID for practice change was defined at 15%. We observed that 19% of surgeons described a change of treatment preference and therefore an acceptable result. However, the vast majority of surgeons were not willing to change treatment preference.

Limited change of treatment preferences in light of new evidence or delays in implementation have been identified in orthopaedic surgery. For example, the non-superiority of knee arthroscopies with partial meniscectomies in patients over 40 years of age with degenerative changes, compared with physiotherapy or sham surgery, is known.[22-24] Despite this evidence, change of practice has been limited.[9,25] Some surgeons in our study postulated that when there was no difference in outcomes, there was no reason for them to change their practice. The protection of flexor tendons and restoring former anatomy were the most commonly used reasons to repair the PQ muscle. Interestingly, a large retrospective study on DRFs and flexor tendon ruptures did not find repair of the PQ muscle to be protective against tendon ruptures.[26]

This study, based on limited data, has shown that implementation of new treatment preferences in orthopaedic surgeons is challenging. The low number of surgeons who elected to change treatment preferences despite RCT evidence is discouraging. Some explanations may be the smaller size of the RCT (less than 100 participants), limited comparable studies, and some statistical concerns with the RCT itself. However, the results of this survey showed that only 16% of respondents had read the full papers and 11% had read neither the abstract nor the full paper.

When looking at the effort, time, and money required to complete good RCTs successfully, is there a more impactful method of research that could be considered? If a RCT is not able to change practice, what other avenues are there to implement change? Even orthopaedic conferences have difficulty changing practice for optimal care.[7] Orthopaedic surgeons are known to have significant autonomy over treatment preferences and therein lies the challenge to change.[27-29] The Australian Orthopaedic Association, as part of its strategic plan for 2019 to 2021, prioritizes a practising community of musculoskeletal EBM.[30]

In the second questionnaire, 25 of 41 Australian surgeons (61%) and seven of 33 Danish surgeons (21%) routinely repaired the PQ muscle. This is a large difference between nationalities. The Danish surgeons were from the same network as the authors that performed the RCT. There is a risk of bias in that some surgeons may have already known the outcomes of the RCT prior to publication and thus, their treatment preferences may have already changed, affecting the results and conclusion of this study. The Australian surgeons, on the other hand, would not have had access to the data. They did not demonstrate a willingness to change treatment preference despite the evidence. Further research is required to ascertain if this willingness is culture-dependent. Also of note, there is no financial incentive as risk of bias in either country to repair or not repair the PQ muscle to influence preferred practice.

In the group that routinely repaired the PQ muscle, the surgeons with less than ten years of practice were more inclined to omit repairing after the RCT publication was released. To our knowledge, there is no published research to investigate years of practice and EBM implementation but it seems reasonable to believe that a younger surgeon is more inclined to change treatment preferences due to having less established practices.

The results of the RCT showing no difference in outcomes in repair and non-repair of the PQ was not known prior to the design of the questionnaire. Showing no difference in outcome could be used to maintain current treatment preference, however, only three surgeons replied that they were unwilling to change due to the reason of “does no harm” (Table III). Future orthopaedic research that does show a clear benefit of one treatment arm over another may change the clinical decision making of orthopaedic surgeons. This study shows the process of conducting an analysis of a target audience before and after the publication of a RCT which may be a worthwhile endeavour for future researchers to undertake.

There is an inherent conservatism in orthopaedics for accepting new knowledge. In order to protect patients against treatment using new ideas without due process, a resistance exists until this new knowledge is tested and proven. This inherent evidence-practice gap takes time to overcome and often requires multiple studies. It is beyond the scope of this paper to discuss this dilemma further. This study investigated if orthopaedic surgeons are ready to change treatment preferences based on new evidence. We recognized that a change of practice is complicated likely due to conservatism, confirmation bias, and herd mentality. Surgeons can be an EBM supporter without necessarily acting upon the evidence. By investigating a specific portion of the decision-making process, we sought to further understand parts of the pathway in implementing EBM.

There are some limitations to this study. First, the response rate was low with the first survey at 30%. However, this is comparable with other studies that have had similar outcomes (11% to 17% response rate) when surveying orthopaedic surgeons.[10,31] This low response rate creates a risk of selection bias. In the first survey, 53% of respondents routinely repaired the PQ muscle, while in the second this was reduced to 43%. This would lean towards a risk of selection bias. We attempted to reduce selection bias by prompting the non-responders three times prior to closing the questionnaire. Secondly, due to the low total number of respondents, it is difficult to draw conclusions from the subgroup analysis. Conversely, the number of respondents for the primary outcome was enough to indicate that the acceptance of EBM is still challenging. Thirdly, the nature of this survey is based on preferences rather than actual treatment change. We cannot know if the veracity of the responses will be upheld in daily practice.

Future directions from our study would include repeating the survey with the same group of surgeons in one to two years to ascertain if they did change treatment preferences as stated. Further research is required to find the best ways of changing practice in response to new evidence. A qualitative study that explores the reasons for surgeons to incorporate study findings into practice would further add to our understanding of the impact of EBM.

In conclusion, the majority of the orthopaedic surgeons who reported routinely repairing the PQ muscle during volar locked plating of DRFs were not influenced by a published RCT showing PQ muscle repair to be unnecessary.

Take home message

- The main objective of this study was to investigate surgeons’ reported change of treatment preference in response to the results and conclusion from a randomized clinical trial.

- The majority of orthopaedic surgeons report not willingly changing treatment preferences based on the results of a randomized clinical trial.

Table II.

Ranking reasons of treatment to either repair or not repair the PQ muscle before publication of the results of the RCT. (n = 100, with 15 excluded as either not operating on distal radial fractures or performing minimally-invasive surgery).

Surgeons who repair PQ routinely (n = 53)
Surgeon responses for factors influencing repair of PQ	Number
Thickness and fascia of muscle	29
Grade of injury to muscle	27
Age of patient	11
Other	11
Surgeon responses for reasons to repair the PQ	Number
Prevent tendon irritation	42
Restore anatomy	15
Preserve soft tissue	8
Other	1

Surgeons who do not repair PQ routinely (n = 47)
Surgeon responses for factors influencing repair of PQ	Number
Thickness and fascia of muscle	17
Never repair	15
Grade of injury to muscle	8
Age of patient	7
Other	2
Surgeon responses for reasons to not repair the PQ	Number
Does not improve outcomes	30
Not necessary	24
Other	7
Increased operative time	2
Risk of compartment syndrome	1

PQ, pronator quadratus; RCT, randomized controlled trial.

Table III.

Ranking reasons of treatment to either repair or not repair the PQ muscle after publication of the results of the RCT.

Surgeons who do repair PQ routinely and will not change (n = 26)
Surgeon responses for factor for not changing	Number (surgeons allowed multiple reasons)
Restore previous anatomy and cover hardware	11
Minimal surgical time required	5
Nil reasons provided	7
Follow-up too short to determine long-term sequelae	3
Does no harm	3
Evidence not strong enough in RCT publication	3
Repair not adequate in RCT publication	1
It is what I was taught by my mentor	0
It is what others are doing	0
My results are good in my hands and therefore I do not wish to change	0
More studies are required	0

Surgeons who do repair PQ routinely and will change (n = 6)
Surgeon responses for factors for change	Number (surgeons allowed multiplereasons)
The results of the RCT publication are evidence enough to change treatment	6
Increased length of operative time	2

PQ, pronator quadratus; RCT, randomized controlled trial.