Publication characteristics of studies published in The Spine Journal from 2005 to 2015.

There is a growing demand for evidence- based practices and informed clinical decision making supported by reliable, high-quality research. The aim of the study is to analyze trends in the level of evidence of publications and to evaluate the publication characteristics that influence the quality of research in The Spine Journal (TSJ). This is a comprehensive publication assessment that reviews and analyses all studies published in TSJ from the years 2005, 2007, 2009, 2011, 2013, and 2015. Level of evidence, study type, funding source, author country, author department, number of citations were considered as the outcome measures. Multivariable logistic regression, multivariable linear regression analyses, and chi square tests were used to analyze the trends of published studies level of evidence, study type, the specialties of authors, author countries, number of citations, and funding sources. A total of 1456 articles were evaluated. There was a decrease in the percentage of high-level evidence (level 1 and 2) studies from 73.6% in 2005 to 49.8% in 2015 (P=0.0045). There was a significant increase in the percentage studies with reporting funding support (P<0.0001). Funded studies were more likely to have a higher level of evidence (P<0.0001). The percentage of studies from international authors increased from 17.8% in 2005 to 69.1% in 2015 (P<0.0001). The percentage of studies with orthopedic authors decreased from 67% in 2005 to 44.9% in 2015 with a corresponding increase in the percentage of studies with neurosurgeon authors from 14.4% in 2005 to 23.2% by 2015, as well as an increase in the percentage of studies with a collaboration of authors from both specialties from 5.1% in 2005 to 8.7% in 2015 (P=0.0007). Orthopedic and neurosurgery collaboration in authorship did not affect the level of evidence of the studies nor the number of citations of the studies (P=0.7583). Earlier studies had a higher Scopus citation number but were not affected by the level of evidence (P=0.2515) nor the department of the author(s) (P=0.9107). We can conclude that the publication characteristics of articles in TSJ have evolved between 2005 and 2015 with a 3.9-fold increase in international authorship and a 32% decrease in the proportion of Level I and Level II studies. Inter-departmental collaboration, funding source, and country of origin may affect level of evidence and number of citations. Continued efforts to increase level of evidence should be considered. ©Copyright: the Author(s), 2019.

Introduction

There is a growing demand for evidence- based medicine-practices and informed clinical decision making that is supported by reliable, high quality research. In 2003, The Journal of Bone and Joint Surgery adopted and tailored the levels of evidence ranking system to the field of orthopedic surgery.[1] This system gave readers an efficient method for gauging the quality of a study, and it created a standard in scientific research that constantly encourages improvement and higher quality methodology and study design.[2]

Several studies have examined the characteristics of research publications and level of evidence in other orthopedic subspecialties. In 2005, Obremskey et al. analyzed all studies published from January to June 2003 from nine orthopedic journals.[3] This analysis demonstrated that the majority of studies published were therapeutic studies (270 out of 382; 70.7%) and were predominantly Level IV evidence.[3] Wupperman et al. analyzed publications in Spine from January to June 2003 and found similar results (43.8% therapeutic; 53.6% Level IV evidence).[4] These studies examine the characteristics and quality of orthopedic research within a single year. These crosssectional analyses are not able to evaluate the trends in evidence over time. Given the push for high quality research over the last decade, an analysis of publication characteristics over a longer time period better demonstrates the impact of this agenda. For instance, Cvetanovich et al. investigated trends in The American Journal of Sports Medicine by analyzing studies published in 1996, 2001, 2006, and 2011.[5] That investigation demonstrated an overall increase in higher level (I and II) studies (P=0.007) and a significant increase in therapeutic studies (P=0.004). The authors also reported an increase in private funding, financial conflicts of interest, and number of countries represented by authorship. They suggest that increased private funding makes expensive higher level randomized controlled trials possible.[5] The primary objective of this investigation was to analyze trends in the level of evidence of publications in the The Spine Journal (TSJ).

Materials and Methods

All studies published in The Spine Journal from the years 2005, 2007, 2009, 2011, 2013, and 2015 were reviewed and analyzed. These studies were identified using the PubMed database: www.ncbi.nlm.nih.gov/pubmed (National Center for Biotechnology Information, U.S. Library of Medicine, Bethesda, MD). Variables recorded for each article included title, date of publication, level of evidence, study type, the specialties of authors, author countries, number of citations, and funding sources.

Using the JBJS Level of Evidence grading system, each study was categorized by type of study (diagnostic, therapeutic, prognostic, economic/decision) and level of evidence was ranked level 1 (highest level) to level 5 (lowest level) based on study methodology (Table 1).Nonclinical studies such as Presidential addresses, case reports, case reviews, letters to the editor, and commentaries were not ranked using the LOE guidelines but were included in the analysis. Number of citations, which is a measure of impact factor, was measured and collected using Google Scholar.

Table 1.

JBJS Level of Evidence grading system.

Level	Therapeutic Studies	Prognostic Studies	Diagnostic Studies	Economic and Design Analyses
I	Randomized controlled trials. Systemic review of Level I randomized controlled trials.	Prospective study. Systemic review of Level I studies.	Testing of already established diagnostic criteria. Systemic review of Level I studies.	Clinically sensible costs and alternatives; values obtained from many studies; multiway sensitivity analyses. Systemic review of Level I studies.
II	Prospective cohort study. Poor-quality randomized controlled trial. Systemic review of Level II studies or non-homogenous Level I studies.	Retrospective study. Study of untreated controls from a previous randomized controlled trial. Systemic review of Level II studies.	Development of diagnostic criteria on basis of consecutive patients. Systemic review of Level II studies.	Clinically sensible costs and alternatives; values obtained from limited studies; multiway sensitivity analyses. Systemic review of Level II studies.
III	Case-control study. Retrospective cohort study. Systemic review of Level II studies.		Study of nonconsecutive patients. Systemic review of Level III studies.	Limited alternatives and costs; poor estimates. Systemic review of Level III studies.
IV	Case series	Case series	Case-control study. Poor reference standard.	No sensitivity analyses.
V	Expert opinion	Expert opinion	Expert opinion	Expert opinion

Statistical analysis

Studies with level of evidence of 1 or 2 were designated as high-level studies. Studies with levels of evidence of 3-5 were designated as low-level studies. Chi-square test was used to determine whether level of evidence, funding source, author country, or author department varied over time. Chisquare test was also used to determine whether level of evidence varied by funding source. Multivariable logistic regression controlling for level of evidence, author department, and year of publication were used to determine study characteristics associated with greater number of citations. Multivariable linear regression was also used to determine whether interdepartmental collaboration between orthopedics and neurosurgery produced studies with greater number of citations, controlling for year of publication and level of evidence. Statistical analysis was performed using SAS 9.4 (SAS Institute Inc., Cary, NC). Statistical significance was set at P<0.05.

Results

Level of evidence

A total of 1456 published studies were analyzed. There has been an overall decrease in high level of evidence studies (level of evidence I or II) from 2005 to 2015 (P=0.0045) (Table 2). In 2005, 73.6% of published studies were level of evidence I or II. By 2015, this percentage decreased to 49.8% (Table 2). The number of high-level studies has progressively increased from 39 studies in 2005 to 122 in 2015 while low level studies increased from 14 in 2005 to 123 in 2015.

Table 2.

Percentage of high level of evidence studies decreased over time.

Level of evidence	2005	2007	2009	2011	2013	2015	Total
% studies, 2005-15
High (1,2)	73.6	63.2	55.3	56.0	57.5	49.8	55.7
Low (3,4,5)	26.4	36.8	44.7	44.0	42.5	50.2	44.3
No. studies, 2005-15
High (1,2)	39	12	47	56	103	122	379
Low (3,4,5)	14	7	38	44	76	123	302
Total	53	19	85	100	179	245	681

Logistic regression shows that the percentage of higher-level studies (level 1&2) decreased over time (P=0.0045).

Sources of funding

In terms of sources of funding, the overall number of studies with funding has also increased significantly over time (P<0.0001) (Table 3). From 2005 to 2015, studies with public funding alone increased from 6.7% to 35.1% while studies with both private and public funding increased from 4.8% to 10%. The percentage of studies with private funding alone, however, decreased from 21% to 15.4% (Table 3). Additionally, funded studies were found to impact the level of evidence of studies than studies without funding (P<0.0001) (Table 4). A larger percentage of high-level studies were found to be funded from private and/or public sources when compared with funding sources of lower level studies (Table 4). When comparing the odds ratio of the level of studies based on sources of funding, it was found that public-funded studies were 4.34 times more likely to be high level studies when compared to studies without funding. Privately funded studies were 2.76 more likely to be high level studies when compared to studies without funding. Studies with both public and private funding were 1.67 times more likely to be high level studies when compared to studies without funding (Table 4).

Table 3.

Percentage of Spine Journal studies with funding is increasing over time.

	2005	2007	2009	2011	2013	2015	Overall
Percentage of studies
None	67.6	72.7	28.6	7.6	8.6	39.4	34.1
Public*	6.7	9.1	15.3	34.2	50.4	35.1	29.9
Private**	21.0	15.2	53.1	53.2	28.8	15.4	28.2
Both***	4.8	3.0	3.1	5.1	12.2	10.0	7.9
Number of studies
None	71	24	28	6	12	102	243
Public*	7	3	15	27	70	91	213
Private**	22	5	52	42	40	40	201
Both***	5	1	3	4	17	26	56

Chi-Square test shows that the percentage of studies receiving funding increased significantly over time (P<0.0001).

*Public: Estimate (-0.7193), SE (0.1733), Chi-Square (17.2321), Pr>ChiSq (<0.0001).

**Private: Estimate (-0.2651), SE (0.1671), Chi-Square (2.5176), Pr>ChiSq (0.1126).

***Both: Estimate (0.2356), SE (0.2507), Chi-Square (0.8827), Pr>ChiSq (0.3475).

Table 4.

Studies with funding were more likely to be higher level studies.

	None	Public	Private	Both	Overall
Percentage of studies
High	39.8	74.2	64.6	52.5	60.7
Low	60.2	25.8	35.4	47.5	39.3
Number of studies
High	43	112	93	21	269
Low	65	39	51	19	174
Total	108	151	144	40	443

Frequency Missing: 1013. Chi Square test shows that funding source impacted level of evidence (p<0.0001). OR estimates are as follows. Public vs No Funding: estimate (4.341), 95%CI (2.554, 7.378). Private vs No Funding: estimate (2.756), 95%CI (1.647, 4.613). Both vs No Funding: estimate (1.671), 95%CI (0.805, 3.468). Logistics regression shows that publicly funded studies were over 4 times more likely to be high level studies (Odds ratio of 4.34/1) than non-funded studies. Privately funded studies were 2.76 times more likely to be high level studies. Studies with both public and private funding were 1.67 times more likely to be high level.

Country of origin

When analyzing trends in the country of origin of published studies, it was found that the percentage of international authors increased significantly from 2005 to 2015 (P<0.0001) (Table 5). In 2005, 17.8% of studies were international and 82.2% of studies were studies from the United States. Over the next 10 years, the percentage of international studies increased to 69.1% while the percentage of studies from the United States decreased to 31.9%. The number of both international and U.S. studies have both increased since 2005. The number of international studies increased from 21 studies to 378 studies while the number of U.S. studies increased from 97 studies to 177. Even so, Spine Journal has been publishing a greater percentage of international studies.

Table 5.

The percentage of studies from international authors increased over time.

	2005	2007	2009	2011	2013	2015	Overall
Percentage of studies
International	17.8	40.6	33.5	43.7	50.5	68.1	51.0
USA	82.2	59.4	66.5	56.3	49.5	31.9	49.0
Number of studies
International	21	13	66	97	166	378	741
USA	97	19	131	125	163	177	712
Total	118	32	197	222	329	555	1453

Frequency Missing=3. Chi-Square statistics shows that the percentage of international authors increased significantly over time (P<0.0001).

Authorship of studies by specialties

Trends in the authorship of studies based on specialties showed an overall decrease in the percentage of studies with only orthopedic surgeon authorship and an increase in the percentage of neurosurgery and collaborative research. Studies with sole orthopedic authorship have gradually decreased, trending from 67% of studies in 2005 to 44.9% in 2015. This corresponds to the increase in studies published with neurosurgery authorship as well as studies with a combined orthopedic and neurosurgery authorship. Studies with sole neurosurgery authorship showed a general increase from 14.4% in 2005 to 23.2% in 2015. Studies with a collaborative authorship increased from 5.1% in 2005 and as low as 3.0% in 2007 to as high as 10.1% in 2009 and, more recently, 8.7% in 2015 (P=0.0007) (Table 6). In addition, 55.6% of studies performed without collaboration of authors from both specialties were found to be high level studies while 47.5% of studies with a collaboration of authors were found be high level studies. Studies with collaboration of both orthopedic and neurosurgeon authors did not increase level of evidence or number of citations (P=0.1789) (Table 7).

Table 6.

Percentage of orthopedic surgeon authors decreased significantly over time.

Years	2005	2007	2009	2011	2013	2015	Overall
Percentage of studies
Neither	13.6	12.1	20.6	16.7	14.0%	23.2%	18.8%
Ortho	67.0	63.6	51.3	59.0%	56.2%	44.9%	52.7%
Neurosurg	14.4	21.2	18.1	16.2%	19.8%	23.2%	19.9%
Both	5.1	3.0	10.1	8.1%	10.0%	8.7%	8.7%
Number of studies
Neither	16	4	41	37	46	129	273
Ortho	79	21	102	131	185	249	767
Neurosurg	17	7	36	36	65	129	290
Both	6	1	20	18	33	48	126
Total	118	33	199	222	329	555	1456

Chi-Square statistics shows that the percentage of orthopedic surgeon and neurosurgeon authors changed significantly by year (P=0.0007).

Table 7.

Ortho/neuro collaboration did not increase LOE or number of citations.

Parameter	Estimate	Standard Error	t Value	Pr > | t |	95% Confidence Limits
Multivariable Linear regression
LOE	1.00457	1.0242089	0.98	0.3271	-1.00705, 3.01620
Year	-6.16736	0.3661724	-16.84	<0.0001	-6.88655, -5.44817
Collaboration	0.99983	3.2406552	0.31	0.7578	-5.36507, 7.36472
Increasing in collaboration	1.164	3.782	0.31	0.7583*	-6.256, 8.585

*Earlier year increased citations; LOE and collaboration did not. Univariable Linear regression: P=0.7583. Level of evidence as follows: High (1,2): No Collaboration 280 (55.6%) + Ortho/Neuro Collaboration 38 (47.5%) = 318. Low (3,4,5): No Collaboration 224 (44.4%) + Ortho/Neuro Collaboration 42 (52.5%) = 266. Total: No Collaboration 504 (100%) + Ortho/Neuro Collaboration 80 (100%) = 584.

The citation number of studies did show a relationship based on the year published and the level of evidence. Earlier years showed an increase in citation number of 6.2 citations per year (P<0.0001). Citation number increased by 1.0 per increased level of evidence. However, citation number did not change significantly with author departments or higher level of evidence (P=0.9107) (Table 8).

Table 8.

Earlier year increases SCOPUS citation number; LOE and department do not.

Variable	Category	N	Mean	Lower 95%CL	Upper 96%CL	DF	F-value	Pr > F*	Estimate	t Value	Pr > | t |	95% CL
Evidence Level	1	257	20.6	16.3	24.8	1	1.32	0.2515	1.04	1.15	0.2515	-0.74, 2.83
	2	119	20.6	14.4	26.9
	3	200	19.1	15.2	23.1
	4	101	16.4	10.9	21.9
	5	1	119.0	-	-
	Missing	775	12.1	8.7	15.5
Year	2005	117	51.9	42.4	61.5	1	349.75	<.0001	-6.22	-18.7	<.0001	-6.88, -5.57
	2007	33	29.2	19.1	39.4
	2009	198	26.5	20.9	32.1
	2011	222	26.9	16.1	37.6
	2013	329	9.7	8.3	11.0
	2015	554	2.4	2.1	2.7
Department	Orthopedics	766	18.3	15.0	21.6	3	0.18	0.9107	3.20	-0.38	0.7039	-7.49, 5.06
	Neurosurgery	289	9.8	7.2	12.5				3.97	-0.05	0.9585	-7.97, 7.56
	Neither	272	13.8	8.1	19.4				0.73089	0.19	0.85	-7.03, 8.49
	Both	126	17.1	12.2	22.1				3.20
		1453	15.7	13.5	17.8

Multivariable regression shows that earlier year was associated with increased citation number (P<0.0001). There was an estimated increase of 6.2 citations per year, and 1.0 citations per increased evidence level. Author department (P=0.9107) and higher level of evidence (P=0.2515) did not significantly affect number of citations.

Discussion

The purpose of this investigation was to examine publication characteristics in Spine Journal from 2005 to 2015, and found a 3.9-fold increase in International authorship and a 32% decrease in the proportion of Level I and Level II studies over the study period. In contrast to our hypothesis that Level I and Level II studies would become more common, the proportion decreased although the number of high-level studies did increase. Over the 10 year span, the annual number of higher level publication increased from 39 to 122 while the number of lower level publications increased from 14 to 123 (Table 2). As we hypothesized, there was a greater increase in the percentage of international studies. In terms of collaborative authorship, however, studies with an orthopedic/neurosurgery collaborative authorship did not yield higher level of evidence or an increased number of citations.

Amiri et al. compared the level of evidence and impact factor of studies published in The Spine Journal in 2010 with those of Spine, European Spine Journal (ESJ), Journal of Neurosurgery: Spine (JNS), and Journal of Spinal Disorders and Techniques (JSDT). They reported a predominance of lower level studies (level III and IV) in The Spine Journal (62.1%) with 37.9% of higher-level studies (level I and II). In contrast, we found a higher number of levels I and II studies around that time - 55.3% in 2009 and 56% in 2011 (Table 2). The disparity of these findings may be, in part, due to the use of different classification systems used to assign level of evidence. Amiri et al. used the Oxford Centre for Evidence-based Medicine criteria whereas our study used the JBJS Level of Evidence grading system.[6]

The predominance of lower level studies in the five journals analyzed by Amiri et al. and the decreasing trend of higher-level studies demonstrated in our study reflects the difficulties of conducting higher level studies in spine surgery. As Cvetanovich et al. discussed, the nature of orthopedic cases unfortunately makes randomized control or prospective studies impractical or impossible. Designing a randomized control trial requires a study treatment comparable to the already established treatment option.[5] Therefore, with the challenges of balancing efficacy, ethics and practicality, randomized control trials in testing surgical interventions are, in most cases, the least feasible study design.[7] Additionally, the substantial costs and time required for randomized control trials that further limit the ability of researchers and clinicians to use this study design.[5] As such, there has been discussion as to whether a better alternative method may be necessary for categorizing non-RCT evidence supporting therapies.[8]

Solomon et al. further investigated the factors that influence a clinician’s decision regarding study designs and any potential obstacles that may impede carrying out a randomized control trial in surgical specialties. Interestingly, the top three factors that dissuaded higher level study designs were patient preferences, uncommon conditions, and lack of community equipoise, rather than methodological barriers. Without the promise of equivocal therapeutic efficacy, as discussed earlier, clinicians were hesitant to initiate an RCT where the patients’ preference for the established treatment option would discourage recruitment. Even so, Solomon et al. discusses potential clinician bias that plays a role in the supposition that patients would not be willing to participate, especially given that therapeutic RCT’s, though few in numbers, have been completed. Additionally, uncommon conditions prove difficult to study due to poor recruitment rate. Solomon et al. posits that RCT’s may be more realistic when investigating complications in already established therapeutic options with strong evidence of efficacy and lack of community equipoise.[9] Still, there remains a paucity of data surrounding this multifaceted issue that warrants further studies to better understand the trends in publishing higher level studies.

Nevertheless, level III and IV studies are invaluable in spine research where chronic disease processes and long-term outcomes need to be constantly assessed in an efficient and cost-effective manner. Retrospective cohort studies and case-control studies allow researchers to accomplish this ethically and practically in ways that randomized controlled trials or prospective cohort studies cannot guarantee. It is also important to understand that while the level of evidence ranking system is an important tool that gauges reliability of a research study, it is not an all-inclusive tool. There are numerous factors that contribute to the quality of a study such as the study’s effort to account for bias. The weight of a study’s impact and clinical significance requires greater scrutiny of the study design, methodology, and data.[10] Our data demonstrated an overall increase in the number of funded investigations from 2005-2015. The number of privately funded studies has remained fairly consistent- ranging from 40 to 52 studies per year. The greatest contributor to the increase in funding comes from the number of publicly funded studies. From 2009 to 2015, the number of publicly funded studies increased significantly- from 15 to 91 publications (Table 3). Interestingly, we found an increase in publically funded spine research publication even though there has been a decrease in the amount of available public funding over the last decade.[11] The National Institutes of Health (NIH), the largest source of biomedical research funding, reports that it has lost 22% of its capacity to fund research due to budget cuts, sequestration, and inflationary losses.[12] The NIH appropriations (in constant dollars) have decreased from $20,590,000 in 2005 to $16,332,000 by 2015 with the subsequent number of grants decreasing from 37,270 to 33,199 for the respective years.[13] Even with the overall decrease in public funding, the increase in public funding in orthopedic research suggests continued federal interest in orthopedic spine research and provides some reassurance regarding the future of orthopedic studies. There are potential limitations of our investigation. Similar to the study design utilized by Cvetanovich et al., our study reviewed publications from the following years: 2005, 2007, 2009, 2011, 2013, and 2015 as a representation for the span of 10 years. A full evaluation of every year between and including 2005 and 2015 may have given a more accurate depiction of various trends over time but it was hypothesized that evaluating 2 year intervals would be sufficient.[5] In addition, our study reviews publications only from The Spine Journal and may not be representative of all spine surgery research. The Spine Journal was chosen because of its high regard in spine surgery. Furthermore, because levels of evidence are not published in The Spine Journal, analyzing The Spine Journal would yield information not as readily accessible to clinicians and readers. Therefore, it is important note that while our findings provide insight into the level of evidence of publication in The Spine Journal over the last decade, the publications we reviewed represent only a small piece in the larger framework of spine research. A subsequent study of interest would be to broaden the breadth of our findings by incorporating this study design to compare trends between various orthopedic journals. The disparities in trends between different journals may provide greater insight in understanding the variables most essential in encouraging higher level research and higher quality evidence-based medicine. Our methodology was another source of limitation as it involved a single reviewer to rate publications using the JBJS Level of Evidence grading system. A larger number of reviewers would decrease potential variability and error. Finally, the number of citations per publication over time was used in this study as a marker for high impact studies.[14] There has been debate whether the number of citations qualifies as an accurate surrogate for the quality or significance of research. Some believe certain specialties with greater turnover rates in publications influence the number of citations. [15] For the purposes of this review, including the number of citations as a possible standard in evaluating overall trends and correlations was important, even if it was not considered the deciding factor in evaluating research. The results of this investigation provide an impetus for analyzing the trends in The Spine Journal as well as a comparison in trends between other orthopedic journals. Further investigation could also entail including and analyzing even numbered years from 2005 to 2016 in The Spine Journal using multiple reviewers to provide a more accurate depiction of trends over the years. It would be beneficial to incorporate a risk of bias assessment in future studies to further ensure quality of data.

Conclusions

The publication characteristics of articles in TSJ have evolved between 2005 and 2015 with a 3.9-fold increase in International authorship and a 32% decrease in the proportion of Level I and Level II studies. Inter-departmental collaboration, funding source, and country of origin may affect level of evidence and number of citations. Continued efforts to increase level of evidence should be considered.