APPLICABILITY OF RANDOMIZED TRIALS IN HAND SURGERY: SURVEY STUDY.

OBJECTIVE: To assess the applicability of randomized clinical trials and whether certain factors (surgeon experience/journal impact factor) influence their applicability.
METHODS: In this survey study we used the Pubmed/Medline database to select 32 consecutive randomized clinical trials published between 2013 and 2015, involving hand surgery (high/low impact). These studies were independently assessed by 20 hand surgeons (with more or less than 10 years of practice) who answered 4 questions regarding their applicability. Agreement was assessed using Cohen's kappa and comparison of proportions via chi-square statistics. P-value <5% constituted statistical significance.
RESULTS: A total of 640 evaluations were produced, generating 2560 responses. A weak correlation was observed between less and more experienced respondents (kappa <0.2; range 0.119-0.179). Applicability between the least and most experienced respondents was similar (p = 0.424 and p = 0.70). Stratification by journal impact factor showed no greater propensity of applicability (p = 0.29) for any of the groups.
CONCLUSIONS: Low agreement was found between the respondents for the applicability of the randomized studies. Surgeon experience and journal impact do not seem to influence this decision. Level of Evidence II, Prospective comparative study.

INTRODUCTION

The number of published clinical studies has been increasing dramatically, with a challenging volume of information to be evaluated and summarized. , Scientific journals are largely responsible for disseminating such information, but there is some doubt about the connection between what is published in high-impact journals and its relevance in everyday practice. ,

In an effort to achieve visibility for their findings, researchers suffer from the high methodological requirements necessary for publication in high-impact journals. These transform good clinical questions into studies that do not reflect real practice scenarios, sometimes controlled artificially by the particularities of controlled randomized clinical studies. This is the current criticism of the purism of evidence-based medicine.

Furthermore, a confounding factor can be found in the potential conflict of interest between high-impact journals and external funding by industry or various research sources. ,

Even so, the consensus is that comparative clinical studies controlled by randomization are the best and most reliable for everyday application. There is great doubt, however, correlating high levels of methodological excellence and practical applicability; studies involving hand surgery and orthopedics report that approximately half of randomized clinical studies change treatment paradigms in a north American orthopedists. -

This study starts from the hypothesis that applying research to daily clinical practice (external validity) is not directly related to the impact or importance of the journal in which the study was published and also not associated with experience gained over time.

The objective of this current study is to verify whether characteristics of the study/periodical (journal impact) are related to the propensity of applicability (feasibility/relevance) of these results in daily clinical practice, using the opinion of physicians (hand surgeons and their experience) as the parameter.

MATERIALS AND METHODS

This study was approved by the institutional review board (CAAE: 55969916.5.0000.5505), and participants signed an informed consent form after explanation of the study.

Strategy for identifying studies eligible for the survey

Randomized clinical trials on treatment involving the topic of hand surgery. Consecutive articles were selected by a survey strategy involving MeSH terms (hand surgery OR hand therapy OR wrist surgery) associated with the filter for randomized clinical trials and limited to the past 4 years (2016–2012).

18 studies from high-impact journals - , ,23, , , , - , - (defined as Qualis/CAPES A1, A2, B1, AND/OR impact factor above 1.5);

14 studies from low-impact journals , , , , , , , , , - (defined as Qualis/CAPES B2 and lower and/or impact factor below 1.5).

The structured abstracts of these studies were evaluated by: 20 (twenty) physicians:

10 hand surgeons with more than 10 years of training;

10 hand surgeons/orthopedists with less than 10 years of training.

Verification of applicability/relevance/feasibility

For each study, questions related to applicability/feasibility/clinical relevance were developed, and participants responded during a single session under the supervision of the researcher, who did not influence the responses:

Do you consider the topic (clinical research question) relevant/important?

Under ideal conditions, would you apply these results in your patients?

In everyday clinical practice, is application of these results feasible?

Do you consider this methodology appropriate to answer the research question?

These questions were answered categorically (Yes/No). We considered studies which earned more than 75% “yes” responses applicable, study by study.

Analysis of subgroups

The responses were evaluated and categorized according to:

Journal impact: low vs. high impact;

Surgeon’s experience: more or less than 10 years of experience.

Statistical analysis

The numerical data were presented as mean and standard deviations or percentages, and confidence intervals of 95%.

Categorical data were assessed in accordance with 2x2 contingency tables and subsequent inferential analysis by means of the chi-square test. Assessment of interobserver concordance consisted of the Kappa statistic, with the level of concordance standardized according to Cohen. P values lower than 5% were considered significant.

RESULTS

Thirty-two studies were included, and evaluated by 20 hand surgeons, totaling 640 evaluations and 2560 responses.

Applicability/Feasibility/Relevance: high versus low-impact publications

When the study-by-study responses were stratified according to “great chance of applicability” and “low chance of applicability,” and “low and high scientific impact publications,” no significant difference was seen between the propensity of applicability. (Table 1)

Table 1

Summary of studies with low and high chance of applicability, stratified by journal impact factor.

Journal impact	High chance of applicability	Low chance of applicability
High	187	173
Low	157	123

Chi-square; 1.079; p=0.29.

Concordance: more vs. less experienced

In general, the responses regarding the studies demonstrate low reproducibility and low significance when more experienced surgeons were compared with less experienced surgeons. Of the 32 studies, only four demonstrated statistical significance. (Tables 2 and 3)

Table 2

Interobserver agreement stratified by study and potential applicability: more vs. less experienced.

Concordance	Kappa	P-value
Vanni et al 13	-0.032	0.629
Van Heest et al 14	0.179	0.002*
White et al. 15	-0.009	0.852
Zaino et al. 16	-0.021	0.757
Rubin et al. 17	0.110	0.103
Dundar et al. 18	-0.114	0.087*
Orlandi et al. 19	0.060	0.355
Roh et al. 20	0.012	0.755
Kolbenschlag et al. 21	-0.027	0.677
Mickelson et al 22	0.069	0.308
Ekrol et al. 23	0.000	0.998
McMillan et al. 24	0.174	0.010*
Costa et al. 25	0.119	0.046*
Prosser et al. 26	0.043	0.525
Vermeulen et al. 27	0.018	0.721
Vermeulen et al. 28	0.012	0.815
Walenkamp et al. 29	0.059	0.376
Buijze et al. 30	0.096	0.066
Paschos et al. 31	0.023	0.729
Bentohami et al. 32	0.036	0.593
Gradl et al. 33	0.000	1.000
Rocchi et al. 34	0.018	0.752
Yamazaki et al. 35	0.108	0.107
Gautam et al. 36	0.062	0.353
Zhang et al. 37	0.110	0.092
Lindan et al. 38	0.035	0.597
Østerås et al. 39	-0.031	0.633
Nam et al. 40	0.052	0.262
Karlsson et al. 41	0.019	0.775
Drac et al. 42	0.150	0.026*
Geetha et al. 43	0.116	0.073
Koman et al. 44	0.064	0.321

Table 3

Interobserver agreement stratified by study and potential applicability: more vs. less experienced - significant results.

Concordance	Kappa	P-value
Van Heest et al. 14	0.179	0.002
McMillan et al. 24	0.174	0.010
Costa et al. 25	0.119	0.046
Buijze et al. 30	0.150	0.026

External validity: more vs. less experienced

Among the studies, the difference between the proportions of “yes” answers (number of “yes” answers per study), stratified by experience, showed a difference for eight studies (25%). Among these studies, the proportion of “yes” responses was greater among less experienced respondents in five studies. Among the more experienced physicians, the proportion of “yes” responses was higher in three studies, with no difference between the proportions (Fisher’s exact test, p=0.70).

When “yes” answers (>75%) were categorized by study, there was no difference between the less and more experienced professionals (less experienced: 37.5% versus more experienced: 28%, chi-square = 0.637; p=0.424). (Table 4)

Table 4

Number and percentage of “yes” answers, stratified by experience.

“Yes” answer	More Experienced		Less Experienced		P-value
	N	%	N	%
Vanni et al. 13	32	80.0%	34	85.0%	0.556
Van Heest et al. 14	25	62.5%	31	77.5%	0.143
White et al. 15	18	45.0%	32	80.0%	0.001*
Zaino et al. 16	21	52.5%	18	45.0%	0.502
Rubin et al. 17	28	70.0%	32	80.0%	0.302
Dundar et al. 18	27	67.5%	31	77.5%	0.317
Orlandi et al. 19	20	50.0%	21	52.5%	0.823
Roh et al. 20	33	82.5%	39	97.5%	0.025
Kolbenschlag et al. 21	17	42.5%	15	37.5%	0.648
Mickelson et al 22	27	67.5%	25	62.5%	0.639
Ekrol et al. 23	20	50.0%	26	65.0%	0.175
McMillan et al. 24	21	52.5%	16	40.0%	0.262
Costa et al. 25	34	85.0%	24	60.0%	0.012*
Prosser et al. 26	17	42.5%	16	40.0%	0.820
Vermeulen et al. 27	14	35.0%	26	65.0%	0.007*
Vermeulen et al. 28	33	82.5%	37	92.5%	0.176
Walenkamp et al. 29	35	87.5%	33	82.5%	0.531
Buijze et al. 30	20	50.0%	32	80.0%	0.005*
Paschos et al. 31	31	77.5%	26	65.0%	0.217
Bentohami et al. 32	20	50.0%	23	57.5%	0.501
Gradl et al. 33	22	55.0%	11	27.5%	0.012*
Rocchi et al. 34	32	80.0%	22	55.0%	0.017*
Yamazaki et al. 35	26	65.0%	21	52.5%	0.256
Gautam et al. 36	23	57.5%	21	53.8%	0.744
Zhang et al. 37	31	77.5%	35	87.5%	0.239
Lindan et al. 38	22	59.5%	28	70.0%	0.333
Østerås et al. 39	32	80.0%	28	70.0%	0.302
Nam et al. 40	28	70.0%	33	91.7%	0.018*
Karlsson et al. 41	11	27.5%	16	40.0%	0.237
Drac et al. 42	18	45.0%	20	50.0%	0.654
Geetha et al. 43	27	67.5%	31	77.5%	0.317
Koman et al. 44	14	35.0%	16	40.0%	0.644

DISCUSSION

The objective of this study was to verify the factors leading to applicability of high-quality clinical research (level I, randomized clinical studies) in the spectrum of hand surgery. It is plausible that in clinical practice, surgeon experience and journal impact are relevant factors for decision-making. More experienced surgeons are expected to be more skeptical about applying new evidence compared to less experienced professionals. The same is also expected in terms of journal impact factor: higher-impact journals are expected to publish studies with greater clinical relevance and external validity. However, the results of this study refute common sense, demonstrating that these factors are not relevant.

With the recent advent of open-access journals that charge for publication, there is a tendency for some studies to migrate to these journals since they offer impartial theoretical assessment and swifter publication. This scenario permits publication of studies with high methodological quality in “open” journals with lower impact and visibility in the area. Our results demonstrate a greater tendency to applicability, when stratified by journal impact or importance.

The applicability of the evidence, as disseminated by the Canadian school, should follow the cycle of evidence, which consists of five stages: 1. formulate the clinical question; 2. look for the best evidence; 3. critical analysis, verify the effects and their applicability; 4. integrate the evidence with the experience, in the clinical setting; 5. evaluate effectiveness and efficiency. This present study systematically created a controlled environment in which we can evaluate stages 2, 3 and 4 of evidence-based medicine in an integrated manner. ,

The results of our study are unprecedented and there is no standard for comparison in the area literature, making our results relevant and groundbreaking. Future studies could consider larger samples and investigate other influencing factors, such as external funding, costs, and specific regional characteristics (related to assistance and economic aspects).

CONCLUSION

In summary, in this study a low concordance was observed between a group of specialized surgeons in terms of applicability of randomized clinical trials. Potentially influencing factors, such as surgeon experience and journal impact factor were not seen to be relevant for this sample.