An international survey to identify the intrinsic and extrinsic factors of research studies most likely to change orthopaedic practice.

OBJECTIVES: Evidence -based medicine (EBM) is designed to inform clinical decision-making within all medical specialties, including orthopaedic surgery. We recently published a pilot survey of the Canadian Orthopaedic Association (COA) membership and demonstrated that the adoption of EBM principles is variable among Canadian orthopaedic surgeons. The objective of this study was to conduct a broader international survey of orthopaedic surgeons to identify characteristics of research studies perceived as being most influential in informing clinical decision-making.
MATERIALS AND METHODS: A 29-question electronic survey was distributed to the readership of an established orthopaedic journal with international readership. The survey aimed to analyse the influence of both extrinsic (journal quality, investigator profiles, etc.) and intrinsic characteristics (study design, sample size, etc.) of research studies in relation to their influence on practice patterns.
RESULTS: A total of 353 surgeons completed the survey. Surgeons achieved consensus on the 'importance' of three key designs on their practices: randomised controlled trials (94%), meta-analyses (75%) and systematic reviews (66%). The vast majority of respondents support the use of current evidence over historical clinical training; however subjective factors such as journal reputation (72%) and investigator profile (68%) continue to influence clinical decision-making strongly.
CONCLUSION: Although intrinsic factors such as study design and sample size have some influence on clinical decision-making, surgeon respondents are equally influenced by extrinsic factors such as investigator reputation and perceived journal quality.Cite this article: Dr M. Ghert. An international survey to identify the intrinsic and extrinsic factors of research studies most likely to change orthopaedic practice. Bone Joint Res 2016;5:130-136. DOI: 10.1302/2046-3758.54.2000578.

The intrinsic and extrinsic factors of research studies perceived as being most influential in informing clinical decision-making amongst orthopaedic surgeons.

Randomised controlled trials, meta-analysis and systematic reviews are the most valued study types among orthopaedic surgeons.

Orthopaedic surgeons value perceived reputation of highly prestigious journals while additionally valuing research published by highly skilled subspecialist surgeons.

Orthopaedic surgeons were most likely to apply literature evidence to their practice if sample sizes ranged from 101 to 500 subjects and if p-values, with associated 95% confidence intervals are reported.

Low survey response rate of 3.9%.

Relatively equal proportion of academic practice and community/private practice respondents helps to support ‘generalisability’ of our results.

Methodological rigour of the survey development process with two-stage piloting process ensured our results represent, to our knowledge, the largest known survey of intrinsic/extrinsic elements of a study necessary to inform a change of practice for orthopaedic surgeons.

Introduction

While evidence-based medicine (EBM) has gained widespread support across healthcare specialties, its implementation among surgeons has lagged behind best evidence by several years.[1-4] Despite favourable attitudes towards EBM amongst surgeons, a lack of understanding of how to assess, interpret and apply current evidence leaves significant room for incorporation of evidence into a surgeon’s practice.[5]

Currently the literature in orthopaedic surgery across the 100 most viewed orthopaedic journals is expanding at a rate of 4000 articles every month.[6] Such a wealth of literature saturates already busy orthopaedic surgeons and would require an orthopaedic surgeon to critically review a minimum of 17 studies daily in order to remain up-to-date with this ever-evolving field.[6] In addition, remaining in touch with the best evidence is particularly challenging, given that the vast majority of orthopaedic surgery publications constitute Level IV (lowest quality) evidence.[7,8] High-quality evidence (randomised-controlled trials (RCTs), systematic reviews and meta-analyses of RCTs) is recognised as a requirement for orthopaedic surgeons to consider a change of practice.[9,10] Mattila et al[11] have shown that high-quality RCTs can in fact initiate evidence-based practice changes in orthopaedic surgery. Yet, only 3% of the orthopaedic literature represents RCTs.[12]

When evidence is less clear on a management strategy within orthopaedics, surgeon preference becomes a significant factor in clinical decision-making.[13,14] We have previously shown in our pilot survey of the Canadian Orthopaedic Association (COA) membership that three key study designs (RCTs, meta-analysis and systematic reviews) are most likely to influence a change in practice in Canada.[9] Furthermore, studies with sample sizes greater than 100 participants and studies published by high-profile investigators in journals of perceived high quality are significant factors when Canadian orthopaedic surgeons interpret evidence to guide practice.[9]

Further high-level research will ultimately drive future developments in evidence-based orthopaedics.[15] In order to guide the design and conduct of future studies, it is important to understand and assess what factors among research studies potentially play the most significant role in impacting orthopaedic surgery practices globally. The objective of this study was to conduct an international survey of orthopaedic surgeons to identify characteristics of research studies perceived as being most influential in informing clinical decision-making.

Materials and Methods

Survey development

A multi-stage development strategy was used to create our survey. An initial detailed review of the available literature was performed to formulate a template questionnaire in keeping with previously established guidelines for academic survey development.[16,17] Following template questionnaire formulation, a four-person focus group, which included a senior orthopaedic surgeon with clinical research experience, two senior orthopaedic residents and a biostatistics graduate student, was established. The focus group provided assistance with increasing survey readability and usability, while emphasising high-priority content and minimising survey bias.

We formulated an eight-question demographic section and 21-question respondent preference section (garnering opinions on both extrinsic and intrinsic features of research studies likely to influence a change in practice). To maximise usability of the survey and minimise respondent burden, we opted to use a web-based survey interface, administering the survey via SurveyMonkey (www.surveymonkey.com, Palo Alto, California). A cover letter introducing the study to potential survey respondents was created by the study team to highlight the study objective, explain passive consent via survey completion, emphasise the confidential nature of the questionnaire, and approximate the time expected to be required to complete the survey in its entirety.

Next, a two-stage piloting process was employed. The questionnaire was first piloted via the online interface to eight independent orthopaedic surgeons (four surgeons with academic practices, four surgeons in community practice). Pilot respondents were asked to provide feedback on survey usability, respondent burden as well as utility of the survey to effectively assess respondent opinions on our topic of inquiry. The identification of any technical challenges experienced with the online survey interface was requested.[16,17] From this feedback, the question stems were further revised and a status bar added to the online user interface to increase potential responses.[16] A large pilot trial was then administered to active members of the COA (841 members). From this pilot study, we received a response total of 95 completed questionnaires.[9] Feedback from this pilot trial was incorporated to decrease the timeline for accepting survey respondents and adjust the rollout period of the survey to avoid coinciding with major conferences.[9] The proposed survey was reviewed with the communications department of the Journal of Bone and Joint Surgery (JBJS) (Am) and further revisions were made to adjust question stem wording to a more global sample population with an unbiased tone. Additionally, country of practice was added to the demographic section of the questionnaire.

The final survey was divided into an eight-question demographics section and a 21-question respondent preference section. The full survey is available online in supplementary material. Questions within the demographics section assessed surgeon age, gender, country of practice and formal academic research training. The respondent preference section sought to assess extrinsic factors of research studies, such as the role of residency and fellowship training on informing practice change as well as the importance of industry and patient requests in guiding clinical practice. Those factors pertaining to study methodology and conduct (intrinsic factors) included study design, sample size and methods used to present study outcomes.

Questionnaire administration

Our population of interest included a broad range of globally practicing orthopaedic surgeons from a convenience-sampling frame of the readership of JBJS (Am). Requests for participation were sent via two mass email communications from JBJS (Am)’s headquarters. The first email sent to members of JBJS (Am) included the study cover letter on departmental letterhead as well as a secure link to our SurveyMonkey hosted survey, which invited participants to complete the questionnaire. Passive consent was outlined in the cover letter and was implied as participants independently volunteered to complete the survey and all responses were completely anonymous. No monetary incentives or pre-notification of the study were provided before initial email contact from JBJS (Am). A reminder email was sent four weeks later. Data collection concluded eight weeks post-launch. This study and the questionnaire were approved by the Hamilton Integrated Research Ethics Board (Project Number: 14–169).

Statistical analysis

Completed responses were entered into a study-specific database and the data was analysed descriptively. Incomplete responses were discarded. Categorical variables are reported as counts and percentages, and continuous variables are summarised with means and standard deviations (sd). Composite combination of positive Likert scale responses (i.e. ‘strongly agree’ and ‘agree’) was performed to demonstrate trends among like-opinioned respondents. Response rate was defined as the number of individuals who completed the survey across those who received the email invitation. The analyses were performed using IBM SPSS version 21 (IBM, Chicago, Illinois) and Microsoft Excel (Microsoft, Santa Rosa, California).

Results

The JBJS (Am) communications department initially sent the survey to the entirety of their readership, a total of 9071 registered members. A total of 28 emails failed to be delivered to the members’ listed preferred address and thus a total of 9043 potential respondents received the invitation to complete the questionnaire. A second email inviting participation in the study was sent to 8795 surgeons (who had not yet completed the survey) four weeks after the initial initiation to participate. Of the 9043 individuals who received the survey, a total of 3490 individuals (38.6%) opened the survey, of which, 353 completed the survey in its entirety. Thus, our overall response rate was 353 out of a possible 9043 respondents (3.9%).

Demographics

Respondent characteristics are shown in Table I. In total 92% of surgeons (325/353) are male, while 218/353 (62%) of respondents are > 50 years of age, with a mean respondent age of 52 years (sd 10.1). Hip and knee arthroplasty, 68/353 (19%) and sports medicine 67/353 (19%) composed the bulk of fellowship-trained respondents, with 103/353 (29%) of respondents not listing a subspecialty (fellowship-trained) practice focus area. Approximately one-third, 114/353 (32%) of respondents are in academic practice and 189/353 (54%) in private practice. Just over half, 197/353 (56%), responded that they do not supervise residents in training and 319/353 (90%) responded that they do not have a graduate degree in clinical research. Of 348 respondents who disclosed current continent of practice, 294/348 (84%) practice in North America, with the fewest respondents 3/348 (0.9%) practicing in Africa.

Table I.

Respondent demographics

Total responses: 353	Number (%) of respondents
Age (yrs)
30 to 34	14 (4)
35 to 39	42 (12)
40 to 44	41 (12)
45 to 49	38 (11)
50 to 54	59 (17)
55 to 59	68 (19)
60 to 64	46 (13)
> 65	45 (12)
Gender
Male	325 (92)
Female	28 (8)
> 10 yrs in practice
Yes	169 (48)
No	184 (52)
Current practice type (select all that apply):
Academic	114 (32)
Community	94 (27)
Private	189 (54)
Public	41 (12)
I am currently supervising residents
Yes	156 (44)
No	197 (56)
I obtained subspecialty fellowship training in the following discipline (select all that apply):
None	103 (29)
Hip and knee reconstruction/total joint reconstruction	68 (19)
Sports medicine	67 (19)
Upper extremity	51 (14)
Trauma	46 (13)
Spine	39 (11)
Foot and ankle	25 (7)
Pediatrics	23 (7)
Oncology	9 (3)
I hold a graduate degree in clinical research
Yes	34 (10)
No	319 (90)
Continent of practice (348 responses)
Africa	3 (0.9)
Asia	8 (2.3)
Australia	7 (2)
Europe	22 (6.3)
North America	294 (84)
South America	14 (4)

Extrinsic factors of research studies affecting clinical decision-making

88% of respondents (297/337) ‘strongly agree’ or ‘agree’ that the judicious integration of best-available research with patient values and clinical expertise is an important part of their clinical decision-making (Table II). Responses demonstrated that 244/337 (72%) of respondents ‘strongly agree’ or ‘agree’ that research published in a highly prestigious journal is likely to influence their clinical decision-making, while 227/335 (68%) ‘strongly agree’ or ‘agree’ that research published by highly skilled subspecialist surgeons is likely to influence their clinical decision-making. Only 21/336 (6.3%) of respondents ‘agreed’ or ‘strongly agreed’ that they would continue to practice based on concepts learned in training, independent of the current literature. Similarly, only 14/334 (4.2%) of respondents indicated they would implement a proposed intervention, which appeared to cause no harm to patients, independent of the literature showing benefit.

Table II.

Extrinsic factors of research studies affecting clinical decision-making

Item asked	Number (%) of respondents
Journal name/impact factor of the journal (325 responses)
Very important	79 (24)
Important	137 (42)
Somewhat important	88 (27)
Not at all important	21 (6.5)
Profile of the study investigators (322 responses)
Very important	62 (19)
Important	139 (43)
Somewhat important	89 (28)
Not at all important	32 (10)
Study type (325 responses)
Very important	147 (45)
Important	103 (32)
Somewhat important	70 (22)
Not at all important	2 (0.6)
Study sample size (323 responses)
Very important	168 (52)
Important	86 (27)
Somewhat important	67 (21)
Not at all important	2 (0.6)

Intrinsic factors of research studies affecting clinical decision-making

A total of 77% of respondents (250/325) reported that the study design was an ‘important’ or ‘very important’ factor likely to influence their clinical decision-making (Table III). Respondents indicated that the top three study designs with the greatest potential to influence a change in current clinical practice were: randomised controlled trials 307/327 (94%), meta-analysis 244/327 (75%) and systematic review 215/327 (66%) (Table III). Randomised controlled trials 270/327 (83%) and meta-analyses 192/327 (59%) were the study designs felt by respondents to have had the most significant impact on their practice in the last five years, with case reports 16/327 (4.9%) and narrative reviews 17/327 (5.2%) having the lowest impact on practice over the same interval. Sample size was reported as ‘important’ or ‘very important’ for 254/323 (79%) of respondents, with the greatest proportion of respondents, 190/327 (58%) stating that a minimum sample size of 101 to 500 participants would be required from a study in order for a surgeon to consider such a study to influence their practice. In terms of outcomes reporting, 235/327 (72%) and 221/327 (68%) of respondents agreed that a study should report both p-values and 95% confidence intervals, respectively.

Table III.

Intrinsic factors of research studies affecting clinical decision-making

Item asked	Number (%) of 327 respondents
The results of the following study designs have the potential to influence a change in my clinical practice (select all that apply to you)
Meta-analysis	244 (75)
Systematic review	215 (66)
Randomised controlled trial	307 (94)
Cohort study	149 (46)
Case-control study	155 (47)
Cross-sectional study	71 (22)
Case series	105 (32)
Case report	54 (17)
Narrative review	46 (14)
Editorial	66 (20)
For a study to influence my clinical practice it would require a sample size of (select all that apply to you)
1 to 10	25 (7.7)
11 to 50	52 (16)
51 to 100	142 (43)
101 to 500	190 (58)
501 to 1000	138 (42)
> 1001	118 (36)
For a study to influence my clinical practice the results should report (select all that apply to you)
p-value	235 (72)
95% confidence interval	221 (68)
Relative risk reduction	107 (33)
Absolute risk reduction	79 (24)
Odds ratio	86 (26)
Number needed to treat	120 (37)
Mean difference	42 (13)
Minimally important difference	75 (23)
Sensitivity analysis	83 (25)
Adjusted analysis	41 (13)
None of the above	41 (13)
The study designs that have had the most profound impact on my practice in the last five years are (select all that apply to you)
Meta-analysis	192 (59)
Systematic review	166 (51)
Randomised controlled trial	270 (83)
Cohort study	58 (18)
Case-control study	78 (24)
Cross-sectional study	24 (7.3)
Case series	46 (14)
Case report	16 (4.9)
Narrative review	17 (5.2)
Editorial	21 (6.4)

Discussion

Key findings

High-quality health care implies practice that is consistent with best evidence.[18] In keeping with the advancing implementation of evidence-based practice in orthopaedics, 84% of survey respondents agreed that judicious integration of best-available research with patient values and clinical expertise is vital to their clinical decision-making. Concurrently, journal reputation remains central to individual surgeon influence, as surgeons continue to place emphasis on research published in highly prestigious journals in addition to valuing research published by highly skilled subspecialist surgeons. Orthopaedic surgeon respondents are more likely to apply evidence to their practice if a study design is amongst the highest quality (i.e. meta-analyses, systematic reviews, randomised controlled trials), if sample sizes range from 101 to 500 subjects, and if p-values with associated 95% confidence intervals are reported.

Our response rate of 3.9% is the greatest limitation of this survey and is in keeping with the declining response rates seen among clinician surveys.[19-22] While clinicians, in particular specialists, are less likely to respond to electronic-based surveys, our primary goal in targeting our convenience sample was minimising respondent burden by using an accessible format.[20,22] Our low survey response rate may introduce selection bias and impact ‘generalisability’. However, Table I demonstrates that the respondents have a sufficient breadth in terms of surgeon age, length of practice, and fellowship training. Among respondents, a relatively equal proportion identified themselves as practicing in academic and community/private practice, thus supporting some generalisability of our results.[24] Importantly, a common trend has been shown from the responses received and while these responses may not be representative of the entire sample population, strong methodology in the survey development process in spite of a low response rate does help reduce the effect of non-responder bias in our study.[24]

One strategy that may have improved response rates would have been to develop and administer the survey in the five most commonly spoken first languages of the JBJS (Am) readership. We are unable to quantify the number of potential respondents who may have participated had multiple survey languages been available to participants. Additionally, the authorship was not made privy to the number of potential participants who have office policies, which prohibit participation in online surveys, which likely further reduced our response rate.[23] Distribution of our survey through the main communication department of JBJS (Am) as opposed to personalised invitations to participate in the survey likely depleted our response rate. It is probable that potential participants may not have viewed the invitation as directly applicable to them and thus ignored the two invitations to participate in the survey, without reading the invitation.[21]

Another limitation, one common to all surveys, is the possibility that responses to our questionnaire may differ from one’s true practice. Efforts were made to emphasise the confidentiality and anonymity of our survey and therefore, to minimise response bias. While efforts were employed to determine the importance of individual study factors on informing a surgeon’s practice, not all extrinsic and intrinsic factors surveyed are of equal importance.[9] The bulk of respondents were from North America and Europe. As such, it is possible that factors identified to be important for informing change of practice may not be applicable outside of these jurisdictions.

One consideration not addressed by our survey is the impact of cost and financial reimbursement on a surgeon’s clinical decision-making process. However, Hageman et al[13] have previously shown that reimbursement is relatively unimportant for orthopaedic surgeons in informing clinical decisions in both Europe and North America, when evidence is inconclusive. However, it is possible that reimbursement could have a subconscious effect on decision-making not accounted for by our survey.[25]

A strength of this study is the methodological rigour employed in the survey development. We developed a user-friendly survey interface with emphasis on reducing judgment and bias in the question stems, with increased usability of the survey via our online platform. A two-stage piloting strategy aided in establishing face and content validity, while strengthening the confidence in our findings. In keeping with previously established guidelines to maximise the number of respondents, we were able to ensure respondent anonymity, secure data collection, and provide appropriate time estimates for completion.[16] In keeping with the recommendations for academic survey design by Burns et al,[16] we designed our questionnaire to be applicable to all orthopaedic surgeons, incorporating questions relevant to general orthopaedics as well as the subspecialty population. To our knowledge, this is the largest sample size of orthopaedic surgeon respondents questioned in regards to features of research study designs important for informing practice changes.

Emerging trends from our survey

The results of our pilot Canadian survey and our current global survey indicate that orthopaedic surgeons recognise and respect the role of high-quality study types in practicing evidence-based orthopaedics.[9] RCTs were the most important study types for potentially changing practice among our respondent population, consistent with previously reported surveys of orthopaedic surgeons.[9,10] Most published literature in orthopaedics is of a low level of evidence (retrospective non-comparative studies).[7] In fact, Chaudhry et al[12] demonstrated that RCTs comprise only 3% of the orthopaedic literature, with only 11% of published orthopaedic literature classified as Level I evidence. Bhandari et al[26] further demonstrated there is a need for improved reporting of orthopaedic RCTs and emphasised that findings from RCTs cannot be taken at face value without critical appraisal of the literature. Poolman et al[27] have also shown that studies labeled as Level I evidence should not be automatically assumed to have high reporting quality.[26,27] The role of observational studies, particularly in surgery when many trials would be unethical or unfeasible, must not be overlooked. Given that observational studies currently represent the vast majority of the current literature, a surprisingly low number of respondents indicated that observational studies have had a significant impact on their practice in the last five years.[28]

Recent high-quality RCT evidence has demonstrated an impact on clinical practice related to indications for knee arthroscopy.[11] However, in most instances, best evidence in the literature has seen a significant lag in clinical application.[1] Morris, Wooding and Grant[29] reviewed 23 papers that examined the lag of translational research to clinical practice and found that the mean delay from research evidence to widespread clinical practice change was 17 years. Similarly, Fitzpatrick[30] indicated that it could take anywhere from ten to 30 years for research to be implemented into clinical practice.

Only 10% of respondents hold a graduate degree in clinical research. Atesok et al[15] recently stated that surgeon scientists will drive future developments of evidence-based orthopaedics. The incorporation of research methodology into the training of orthopaedic residents has the potential to increase scientific inquiry and improve musculoskeletal care.[31,32] It has been shown that greater pursuit of academic careers among orthopaedic surgeons can be promoted by encouraging research at the resident level.[33-35] Furthermore, a focus on research training during residency correlates with a greater likelihood of publication in peer-reviewed journals and higher impact journals.[33,35] Increasingly, active forms of continuing medical education will be crucial to advance evidence-based practice in orthopaedics and in medicine in general.[36,37]

In conclusion, our results demonstrate that the vast majority of orthopaedic surgeon respondents recognise the role of high-level evidence in guiding their clinical practices. High-quality studies in the form of RCTs, meta-analysis and systematic reviews are most valued among orthopaedic surgeons for informing clinical practice changes. However, it is clear that some elements of literature appraisal among surgeons require further development. The majority of respondents continue to emphasise the reputation of the authors and the journal of publication when considering clinical practice changes. Continued educational emphasis on the generation of large, methodologically sound clinical trials and the tenets of evidence-based orthopaedics remain paramount to translating high-quality research findings into clinical practice.