Impact of academic collaboration and quality of clinical orthopaedic research conducted in low- and middle-income countries.

BACKGROUND: Little is known about the quality of orthopaedic investigations conducted in low- and middle-income countries (LMICs). Academic collaboration is one model to build research capacity and improve research quality. Our study aimed to determine (1) the quality of clinical orthopaedic research conducted in LMICs, (2) the World Bank Regions and LMICs that publish the highest quality studies, (3) the pattern of collaboration among investigators and (4) whether academic collaboration between LMIC and non-LMIC investigators is associated with studies that have higher levels of evidence.
METHODS: Orthopaedic studies from 2004 to 2014 conducted in LMICs were extracted from multiple electronic databases. The World Bank Region, level of evidence and author country-affiliation were recorded. Collaboration was defined as a study that included an LMIC with non-LMIC investigator.
RESULTS: There were 958 studies that met inclusion criteria of 22,714 searched. Ninety-seven (10.1%) of included studies achieved Level 1 or 2 evidence, but case series (52.3%) were the most common. Collaboration occurred in 14.4% of studies and the vast majority of these (88.4%) were among academic institutions. Collaborative studies were more likely to be Level 1 or 2 (20.3% vs. 8.4%, p < 0.01), prospective (34.8% vs. 22.9% p = 0.04) and controlled (29.7% vs. 14.4%, p < 0.01) compared to non-collaborative studies.
CONCLUSIONS: Although orthopaedic studies in LMICs rarely reach Level 1 or 2 evidence, studies published through academic collaboration between LMIC and non-LMIC investigators are associated with higher levels of evidence and more prospective, controlled designs.

Introduction

There is increasing recognition of the importance of musculoskeletal disease as a global health issue, particularly as it relates to injury. An estimated 1.2 million deaths and 50 million nonfatal injuries occur each year due to road traffic accidents alone [1], with traumatic injuries contributing to more global disability than human immunodeficiency virus (HIV), tuberculosis and malaria combined [2, 3]. The burden of injury is greatest in low- and middle-income countries (LMICs), where resource-limited orthopaedic surgeons face a daunting volume of musculoskeletal disease [2, 4]. Efforts to address this disparity in care have manifested through surgical missions, donated implants and educational programmes [4, 5]. Less, however, is known about the impact of clinical research [4, 5]. Recent literature suggests that clinical orthopaedic research conducted in LMICs may help to answer relevant clinical questions and shape public health policy [4]. Although a growing body of research exists to inform treatment for orthopaedic disease in high-income countries (HICs), generalizing results to LMICs is difficult due to delays in treatment, differences in training and availability of equipment [6, 7]. Thus, it would be ideal for orthopaedic surgeons practicing in LMICs to base their treatment on high-quality research conducted in a similar setting [4].

To date, the quality of clinical research conducted in LMICs has not been assessed. Orthopaedic surgeons weigh a study’s level of evidence (LOE) before determining how the results apply clinical practice [8-10], as Level 1 or 2 studies can potentially reveal important therapeutic, diagnostic, prognostic and economic outcomes [11, 12]. Given the importance of study quality in shaping treatment protocol, the state of orthopaedic research in developing countries requires comprehensive review. Some have hypothesized that collaboration between investigators from LMICs and non-LMICs (e.g. upper-middle and high-income countries) can improve study quality [4, 13]. While there is growing evidence that investigators from LMICs strive to answer clinically relevant questions through research, they may lack resources, training and protected time away from clinical responsibilities [14]. However, collaboration with academically affiliated non-LMIC authors may help local investigators overcome these barriers through funding support, research education and division of labour [4]. Thus, investigations are needed to evaluate how international academic partnership can improve study quality.

Our study aimed to determine (1) the quality of clinical orthopaedic research conducted in LMICs, (2) the World Bank Regions and LMICs that publish the highest quality studies, (3) the pattern of collaboration among investigators and (4) whether academic collaboration between LMIC and non-LMIC investigators is associated with studies that have higher levels of evidence.

Material and methods

We conducted a scoping review using the Arksey and O’Malley framework with modifications from Levac et al. and Daudt et al. [15-17]. Five authors sought and assessed orthopaedic studies conducted in LMICs that were published between June 2004 and June 2014. The LMICs were defined as any country listed as a “Low-income” or “Lower-middle-income” economy according to the 2014 World Bank classification [18]. Eighty-two countries met our study’s definition of LMIC (Table 1). The studies were identified by comprehensive textword and MeSH-based electronic searches of PubMed/MEDLINE, EMBASE and Cochrane library that was developed with assistance from a research librarian. Our search strategy combined terms for orthopaedic surgery, LMICs, musculoskeletal injury, musculoskeletal anatomy and human studies to be as inclusive as possible (Appendix).

Table 1.

Countries with low-income or lower-middle-income economies as defined by The World Bank. Table adapted from http://data.worldbank.org/about/country-and-lending-groups.

Low-income economies ($1045 or less) [n = 31]
Afghanistan	Gambia, The	Niger
Benin	Guinea	Rwanda
Burkina Faso	Guinea-Bisau	Sierra Leone
Burundi	Haiti	Somalia
Cambodia	Korea, Dem Rep.	South Sudan
Central African Republic	Liberia	Tanzania
Chad	Madagascar	Togo (Sub-Saharan)
Comoros	Malawi	Uganda
Congo, Dem. Rep	Mali	Zimbabwe
Eritrea	Mozambique
Ethiopia	Nepal
Lower-middle-income economies ($1046 to $4125) [n = 51]
Armenia	Indonesia	Samoa
Bangladesh	Kenya	São Tomé and Principe
Bhutan	Kiribati	Senegal
Bolivia	Kosovo	Solomon Islands
Cabo Verde	Kyrgyz Republic	Sri Lanka
Cameroon	Lao PDR	Sudan
Congo, Rep.	Lesotho	Swaziland
Côte d’Ivoire	Mauritania	Syrian Arab Republic
Djibouti	Micronesia, Fed. Sts.	Tajikistan
Egypt, Arab Rep.	Mongolia	Timor-Leste
El Salvador	Morocco (North Africa)	Ukraine
Georgia	Myanmar	Uzbekistan (Central Asia)
Ghana	Nicaragua	Vanuatu
Guatemala	Nigeria (West Africa)	Vietnam
Guyana	Pakistan (South Asia)	West Bank and Gaza
Honduras	Papua New Guinea	Yemen, Rep.
India	Philippines	Zambia

Full-text articles were assessed for eligibility in the order of title, abstract and manuscript. Included studies (1) deal primarily with a low- or lower-middle-income country (LMIC), (2) pertain to orthopaedic surgery, (3) enrol humans and (4) were original peer-reviewed publications. Studies that reported three cases or fewer, were non-English and pertained to a high-income country (HIC) at war, animals, biomechanics or laboratory values were excluded.

Each included study was read in its entirety and a REDCap survey for data extraction was created that identified the study location, author’s academic affiliation and study quality. Study location and author affiliations were categorized into eight World Bank Regions: East Asia and Pacific, Europe and Central Asia, Latin America and the Caribbean, Middle East and North Africa, Latin America and the Caribbean, Middle East and North Africa, North America, South Asia (Table 2). Collaboration was determined by the investigator country-affiliation and defined as LMIC-only, Multicentre (LMIC with LMIC investigator), Collaborative (LMIC with non-LMIC investigator) and non-LMIC (e.g. upper-middle or high-income country investigators) only. Academic partnerships were defined as a study where the authors from both LMICs and non-LMICs were affiliated with an academic institution.

Table 2.

Countries by World Bank Regions, adapted from http://data.worldbank.org/about/country-and-lending-groups.

East Asia and Pacific, n = 37
American Samoa	Korea, Rep.	Philippines
Australia	Lao PDR	Samoa
Brunei Darussalam	Macao SAR, China	Singapore
Cambodia	Malaysia	Solomon Islands
China	Marshall Islands	Taiwan, China
Fiji	Micronesia, Fed. Sts.	Thailand
French Polynesia	Mongolia	Timor-Leste
Guam	Myanmar	Tonga
Hong Kong SAR, China	New Caledonia	Tuvalu
Indonesia	New Zealand	Vanuatu
Japan	Northern Mariana Islands	Vietnam
Kiribati	Palau
Korea, Dem. Rep.	Papua New Guinea
Europe and Central Asia, n = 57
Albania	Germany	Netherlands
Andorra	Greece	Norway
Armenia	Greenland	Poland
Austria	Hungary	Portugal
Azerbaijan	Iceland	Romania
Belarus	Ireland	Russian Federation
Belgium	Isle of Man	San Marino
Bosnia and Herzegovina	Italy	Serbia
Bulgaria	Kazakhstan	Slovak Republic
Channel Islands	Kosovo	Slovenia
Croatia	Kyrgyz Republic	Spain
Cyprus	Latvia	Sweden
Czech Republic	Liechtenstein	Switzerland
Denmark	Lithuania	Tajikistan
Estonia	Luxembourg	Turkey
Faeroe Islands	Macedonia, FYR	Turkmenistan
Finland	Moldova	Ukraine
France	Monaco	United Kingdom
Georgia	Montenegro	Uzbekistan
Latin America and the Caribbean, n = 41
Antigua and Barbuda	Dominica	Peru
Argentina	Dominican Republic	Puerto Rico
Aruba	Ecuador	Sint Maarten (Dutch part)
Bahamas, The	El Salvador	St. Kitts and Nevis
Barbados	Grenada	St. Lucia
Belize	Guatemala	St. Martin (French part)
Bolivia	Guyana	St. Vincent and the Grenadines
Brazil	Haiti	Suriname
Cayman Islands	Honduras	Trinidad and Tobago
Chile	Jamaica	Turks and Caicos Islands
Colombia	Mexico	Uruguay
Costa Rica	Nicaragua	Venezuela, RB
Cuba	Panama	Virgin Islands (U.S.)
Curacao	Paraguay
Middle East and North Africa, n = 21
Algeria	Jordan	Qatar
Bahrain	Kuwait	Saudi Arabia
Djibouti	Lebanon	Syrian Arab Republic
Egypt, Arab Rep.	Libya	Tunisia
Iran, Islamic Rep.	Malta	United Arab Emirates
Iraq	Morocco	West Bank and Gaza
Israel	Oman	Yemen, Rep.
North America, n = 3
Bermuda	Canada	United States
South Asia, n = 8
Afghanistan	India	Pakistan
Bangladesh	Maldives	Sri Lanka
Bhutan	Nepal
Sub-Saharan Africa, n = 48
Angola	Gabon	Nigeria
Benin	Gambia, The	Rwanda
Botswana	Ghana	São Tomé and Principe
Burkina Faso	Guinea	Senegal
Burundi	Guinea-Bissau	Seychelles
Cabo Verde	Kenya	Sierra Leone
Cameroon	Lesotho	Somalia
Central African Republic	Liberia	South Africa
Chad	Madagascar	South Sudan
Comoros	Malawi	Sudan
Congo, Dem. Rep.	Mali	Swaziland
Congo, Rep	Mauritania	Tanzania
Côte d’Ivoire	Mauritius	Togo
Equatorial Guinea	Mozambique	Uganda
Eritrea	Namibia	Zambia
Ethiopia	Niger	Zimbabwe

Finally, the study quality was assessed by the levels of evidence (LOE), presence of control group, prospective or retrospective design, type of study and presence of randomization. Due to the heterogeneity of study design in the global orthopaedic literature, which includes qualitative and epidemiologic studies that cannot be categorized with the LOE scale, no other quality assessment tools were utilized. To determine the level of evidence of each study, we used the 2015 scale adopted by the Journal of Bone and Joint Surgery and derived from recommendations given by the Centre for Evidence-Based Medicine in Oxford, United Kingdom [19, 20]. The LOE scale divides studies into four categories: Diagnostic, Prognostic, Therapeutic, and Economic and Decision Analyses. Each category can then be subdivided into Level 1–5 evidence, with each level having its own definition [19]. Level 1 and 2 studies were considered high levels of evidence, while those of Level 3, 4 and 5 were considered low levels of evidence. Studies that had no levels of evidence, such as epidemiologic or qualitative studies, were also noted. LOE was used as the primary indicator of study quality due to its ubiquitous use as a validated measure of study strength [9, 11, 21–23]. In addition, studies that do include control groups, collect data prospectively and randomize interventions have been shown to produce higher quality evidence than studies that do not [8, 10].

Data analysis

Descriptive statistics were used to summarize all data. Discrete variables were summarized as counts or proportions, and skewed continuous variables were reported as medians with interquartile ranges (IQRs). Chi-square analysis was used to determine the association between nominal variables. Significance was set at p < 0.05.

Results

Out of 22,714 unique articles assessed, 958 met all inclusion criteria (Figure 1). Over the past decade, 265 (27.6%) of clinical orthopaedic research studies conducted in LMICs were epidemiologic or qualitative and had no LOE. Of the clinical studies that could be assessed with levels of evidence, only 97 (10.1%) were Level 1 or 2. Studies were most commonly designed as prospective or retrospective case series (501 [52.3%]). In addition, only 158 (16.5%) of all studies had a control group and only 24 (2.5%) underwent randomization. Overall, the majority of data from these studies (75.4%) were collected retrospectively. From 2004 to 2014, the median percentage of global orthopaedic studies published per year with a high LOE (e.g. Level 1 and 2 studies) was 11.2% (IQR: 5.8–11.7). The percentage of high-quality global orthopaedics being conducted in LMICs annually has been stagnant over time (Figure 2), even though the number of LMIC orthopaedic studies has steadily increased over the same time period. A chi-square contingency table shows no statistically significant difference in the percentage of Level 1 and 2 studies published over the past decade (p = 0.4) (Figure 3). Of note, out of 503 therapeutic studies, 374 studies recommended the use of tested treatment or intervention, even though most of the studies 394 (78.3%) were Levels 4 and 5 with respect to evidence.

Figure 1.

Flowchart shows included and excluded studies from search to data extraction.

Figure 2.

Percentage of Level 1 and 2 global orthopaedic studies conducted in low- and middle-income countries per year from June 2004 to June 2014.

Figure 3.

Percentage of total levels of evidence of global orthopaedic studies conducted in low- and middle-income countries per year from June 2004 June 2014.

Most studies took place in South Asia (48.4%) and Sub-Saharan Africa (32.6%). Of the 97 studies that were Level 1 or 2 evidence, 55 (56.7%) took place in South Asia and 30 (30.9%) took place in Sub-Saharan Africa (Figure 4). Pakistan (25), India (19), Nigeria (14) and Nepal (9) were the LMICs with the most Level 1 and 2 studies published in the past decade (Table 3). Out of the 82 countries classified as LMICs by the World Bank, only 23 (28.0%) had published a Level 1 or 2 orthopaedic study in the past decade. LMICs from the Middle East and North Africa (3.1%), Latin America and Caribbean (4%) and East Asia and Pacific (5.2%) produced the lowest percentage of Level 1 and 2 studies since 2004.

Figure 4.

Percentage of high-quality studies (e.g. Levels 1 and 2) published by World Bank Region from 2004 to 2014.

Table 3.

Study setting region, Level 1 and 2 studies by region and countries that have published Level 1 and 2 studies.

Variables	Study characteristics (n = 958)
Study setting region
Sub-Saharan Africa (n [%])	312 [32.6%]
Level 1 or 2 studies (n [%])	30 [30.9%]
Number of Level 1 or 2 orthopaedic studies published by country since 2004	Nigeria (14), Malawi (5), Sudan (3), Cameroon (2), Kenya (2) Ethiopia (1), Sierra Leone (1), Tanzania (1), Uganda (1)
East Asia and Pacific (n [%])	53 [5.5%]
Level 1 or 2 studies (n [%])	5 [5.2%]
Number of Level 1 or 2 orthopaedic studies published by country since 2004	Cambodia (1), Mongolia (1), Myanmar (1), Philippines (1), Vietnam (1)
Europe and Central Asia (n [%])	17 [1.8%]
Level 1 or 2 studies (n [%])	0 [0%]
Latin America and Caribbean (n [%])	30 [3.1%]
Level 1 or 2 studies (n [%])	4 [4.1%]
Number of Level 1 or 2 orthopaedic studies published by country since 2004	Haiti (2), Guatemala (1), Guyana (1)
Middle East and North Africa (n [%])	82 [8.6%]
Level 1 or 2 studies (n [%])	3 [3.1]
Number of Level 1 or 2 orthopaedic studies published by country since 2004	Egypt (3)
South Asia (n [%])	464 [48.4%]
Level 1 or 2 studies (n [%])	55 [56.7%]
Number of Level 1 or 2 orthopaedic studies published by country since 2004	Pakistan (25), India (19), Nepal (9), Bangladesh (2), Afghanistan (1)

Over the past decade, 138 (14.4%) of global orthopaedic studies have been Collaborative, 30 (3.1%) have been Multicentre, 85 (8.9%) have only included authors from non-LMICs and 705 (73.6%) have only included authors from LMICs. The majority (75.8%) of them took place in academically affiliated institutions. In addition, the majority of non-LMIC authors practised either in Europe and Central Asia (44%) or in North America (42.8%) (Figure 5).

Figure 5.

Percentage of non-LMIC collaborative authors by World Bank Region from 2004 to 2014.

Out of the 138 studies where LMIC and non-LMIC authors worked together, 122 (88.4%) of them represented academic partnerships. Collaborative studies were more likely to be Level 1 or 2 (20.3% vs. 8.4%, p < 0.01), prospective (34.8% vs. 22.9% p = 0.04) and controlled (29.7% vs. 14.4%, p < 0.01) compared to non-collaborative studies (Table 4). There was no difference in randomization between Collaborative and non-collaborative studies (p = 0.53).

Table 4.

Characteristics of collaborative global orthopaedic studies from 2004 to 2014.

Variables	Study characteristics (n = 138)
Region of collaborating site by WHO classification
Sub-Saharan Africa (n [%])	4 [2.9%]
North America (n [%])	59 [42.8%]
East Asia and Pacific (n [%])	10 [7.2%]
Europe and Central Asia (n [%])	60 [43.5%]
Middle East and North Africa (n [%])	6 [4.3%]
Latin America and Caribbean (n [%])	15 [10.9%]
South Asia (n [%])	0 [0%]
Quality of collaborative studies
Level 1 and 2 studies (n [%])	28 (20.3%)
Level 3, 4, 5 studies (n [%])	57 (41.3%)
No Level of Evidence (n [%])	53 (38.4%)
Controls (n [%])	41 [29.7%]
Prospective (n [%])	48 (34.8%)
Randomized (n [%])	1 (0.7%)

Discussion

While the importance of evidence-based clinical practice is increasingly recognized in high-income countries, there is an enormous gap in the volume and quality of musculoskeletal research relevant to clinicians and patients in LMICs. Our study shows that the majority of clinical orthopaedic researches published in LMICs are designed as case series, and these studies rarely attain high levels of evidence, a phenomenon that has persisted over the past decade. Notably, there appears to be no significant difference between the proportion of Level 1 and 2 studies published in 2004 and the years leading up to 2014. Studies with low levels of evidence cannot accurately determine the treatment outcome and may be harmful if misinterpreted [11]. Our data shows that while the majority of therapeutic studies conducted in LMICs recommended the use of the tested intervention, they were mostly designed as case series. Given the importance of research in the effort to treat global orthopaedic disease, efforts to enable local researchers to produce high-quality studies are warranted.

One factor associated with higher quality research is academic collaboration. Our data suggest that while studies produced by collaboration between LMIC and non-LMIC investigators account for less than 15% of total research, they are associated with higher levels of evidence and more prospective, controlled designs.

Our finding that orthopaedic research conducted in LMICs was unlikely to attain high levels of evidence is similar to the literature published in major Western journals. For instance, Reich et al. evaluated seven years’ worth of publications from The Journal of Bone and Joint Surgery (American) and found that only one of the Level 1 and 2 studies published was conducted in an LMIC [12]. In addition, the proportion of high level studies published in LMICs does not compare favorably with that of non-LMIC studies. An analysis of three American journals found that 23% of studies published in the last decade achieved Level 1 or 2 evidence [9], more than double the proportion of Level 1 and 2 LMIC-conducted studies during the same time frame. Notably, our data also suggests that the overall quality of LMIC publications has not improved over the past decade, which lies in stark contrast to studies of high impact journals that suggest the number of high-quality studies has increased significantly over time [9, 12]. Future research into why a significant gap in research quality exists between high-resource and low-resource environments is warranted.

Even though the quality of research from LMICs is lower overall, studies conducted in South Asia and Sub-Saharan Africa were more likely to attain high levels of evidence. Notably, Pakistan, India, Nigeria and Nepal produced over two-thirds of Level 1 and 2 studies in the past decade. In contrast, 59 of the world’s 82 LMICs did not produce a single Level 1 or 2 study over the same time frame. This finding is similar to recent literature that indirectly suggests LMICs from South Asia produce the highest quality studies in Western journals compared to other LMIC regions [12]. However, it is still unknown why this disparity by region exists, which warrants further investigation.

Collaboration between LMIC and non-LMIC investigators is associated with significantly higher quality of published clinical research with respect to levels of evidence, prospective design and presence of control groups. Notably, over 88% of these studies represented academic partnerships, in which both the LMIC and non-LMIC investigators were affiliated with an academic institution. Although, it is unclear why Collaborative studies are associated with higher research quality, recent literature has hypothesized that investigators from high-income countries can enable their counterparts in LMICs to conduct research through longitudinal partnership [4, 13, 24]. For example, Morshed et al. outline a potential model of partnership in which investigators from an academic HIC institution helped LMIC-based surgeons develop a prospective cohort study through assistance in protocol development, research training, funding and resource procurement [4]. Academic affiliation of LMIC partners may also be an important ingredient for successful partnerships, since academic advancement and research requirement for trainees are often cited by LMIC authors as incentives for research participation [4, 14].

Our study had several limitations. First, given the heterogeneity of included articles, we were unable to apply additional standardized quality assessment tools, which may have affected our ability to gauge the study quality. However, LOE has been shown to be a validated measure of study quality and have adequate interobserver reliability [9, 21]. Second, our review only included studies since June 2004 and could have been strengthened by including more publication years. However, our study provides a snapshot of study quality over the last decade and is by far the most comprehensive existing review of the global orthopaedics literature. Third, as a literature review, our study design cannot prove causal inference exists between academic collaboration and study quality. Our data does, however, show that non-LMIC and LMIC collaboration is associated with higher study quality, which warrants further investigation. Finally, our study did not include primarily non-English publications, which may have excluded high-quality studies published in other languages. However, recent literature suggests that language restrictions of reviews may not produce meaningful bias [25], although future studies with broader language requirements are needed.

Conclusion

Recent literature has shown that many investigators who practice in LMICs have a strong interest in participating in research studies [14]. However, surgeons in LMICs contend with a lack of formalized research training, resources and protected time as significant barriers to improving study quality [4, 14]. These barriers may explain why clinical orthopaedic research conducted in LMICs rarely attains high levels of evidence compared to those conducted in non-LMICs, a phenomenon that has not improved in the past decade. Thus, more research is needed to determine how LMIC investigators can best overcome impediments to high-quality research in resource-poor environments. Academic collaboration is one of the most promising solutions, as it is associated with higher quality studies published over the last decade. Thus, greater investment of resources into developing and investigating the impact of academic collaboration is warranted.

Funding

No funding to disclose.

Conflict of interest

Dr. Coughlin reports AAOS-International Committee Institute for Global Orthopaedics and Traumatology/UCSF: Board or committee member.

Dr. Morshed reports personal fees from Microbion, grants from Philips outside the submitted work; Orthopaedic Research Society: Board or committee member; Orthopaedic Trauma Association: Board or committee member.

Dr. Miclau reports personal fees from Acelity, personal fees from Amgen Co, grants from Baxter, grants from Synthes outside the submitted work; Inman Abbott Society: Board or committee member; Orthopaedic Research Society: Board or committee member; Orthopaedic Trauma Association: Board or committee member; Osteosynthesis and Trauma Care Foundation: Board or committee member.

The other co-authors of this study declare no conflict of interest for this project.