Potential benefits, unintended consequences, and future roles of artificial intelligence in orthopaedic surgery research : a call to emphasize data quality and indications.

Artificial intelligence and machine-learning analytics have gained extensive popularity in recent years due to their clinically relevant applications. A wide range of proof-of-concept studies have demonstrated the ability of these analyses to personalize risk prediction, detect implant specifics from imaging, and monitor and assess patient movement and recovery. Though these applications are exciting and could potentially influence practice, it is imperative to understand when these analyses are indicated and where the data are derived from, prior to investing resources and confidence into the results and conclusions. In this article, we review the current benefits and potential limitations of machine-learning for the orthopaedic surgeon with a specific emphasis on data quality.

The race to be first

The application of artificial intelligence (AI) and machine-learning in orthopaedics is a concept that has elicited an aura of complexity, mystery, and excitement in recent years among researchers and clinicians. However, machine-learning and AI are rather well-established methodological techniques. Indeed, these have formed an extensively applied and well-accepted set of methodological processes among many industries that depend on data science for their success, and AI has been applied in other realms of medicine. AI is a heterogeneous term, but may be defined as technologies or machines capable of performing tasks like problem-solving and learning, language interpretation, pattern recognition, and planning, similar to human cognitive function. Machine-learning is a subset of AI that uses experiential learning through incremental adjustments in internal parameters to objectively strengthen or weaken select features (inputs) through mathematical functions to optimize model prediction accuracy. Orthopaedic research had simply not caught wind of the potential applications of AI and machine-learning, though recent years have seen it quick to adopt and laud these methods proposed to be superior. However, the adoption of these techniques and their potential for clinical impact should not proceed without understanding the limitations of these methods, and their potential to misinform readers and healthcare professionals. Therefore, it is imperative to understand the importance of data quality, appropriate reporting, and proper application of these models as it pertains to machine-learning in orthopaedic research.

Who has the data has the power

In 2021, Tim O’Reilly’s maxim, “who has the data has the power” assumes a new meaning, and comes down to man versus the machine. Medical data were estimated to double every 73 days in 2020, and with the costs and resources used to collect so much patient data, it is imperative to use them in meaningful ways to benefit healthcare centres and patients. For example, by applying AI to image recognition, it may be possible to save time and resources by allowing for the development of preoperative risk assessment or surveillance tools. It is undeniable that big data will transform medicine, and though high-quality data is imperative, ultimately all data must be appropriately analyzed, interpreted appropriately, and allow for change in the way we practice medicine. To this end, there currently exists much “hype” as to the potential of machine-learning; however, this currently remains disproportionate to the translation of such models in clinical practice.

What are the benefits?

Machine-learning models are critical to the advancement of research in orthopaedic surgery, given their ability to analyze and interpret vast numbers of predictor variables in non-linear and highly complex manners, “learn,” and make accurate predictions. Furthermore, incremental inputs can improve the predictive ability of these models as they continue to learn from new data. The manner in which a model makes these predictions was appropriately named after our own brains: artificial neural networks. Similarly to how we as humans unconsciously recognize patterns and use these to inform decision-making, while sometimes not being able to articulate how we came to our conclusions (trusting our ‘gut instinct’), computer algorithms exist in a “black box.” While we can literally open this black box, view the code, and reapply the model to more data in a more consistent application than a human is capable of, the mapping of input to output is complex, and involves numerous connections between components, thus rendering it indescribable. While publications can cite how accurately their models predict outcomes in their dataset, these models often exist only in the paper and are not extended across institutions or implemented in practice.

However, machine-learning may serve as a solution to handling the plethora of medical data now available. In the 1950s, the amount of medical data was doubling at a rate of every 50 years, which increased to every seven years by 1980. By 2010, the amount of medical data was doubling every 3.5 years, and is currently estimated to double every 73 days. This trend necessitates rigorous and efficient data analytic methods, and machine-learning is already transforming medicine by advancing prognostication, interpreting digitized images, and improving diagnostic accuracy. Interestingly, early machine-learning analyses have already demonstrated these capabilities within orthopaedic surgery specifically. A recent systematic review of 11 studies performed by Kunze et al investigated the ability of AI to identify anterior cruciate ligament (ACL) tears and meniscus lesions on imaging. The authors found that the accuracy of detecting ACL tears ranged between 90% to 98%, meniscus lesions 85% to 91%, and that the addition of AI models significantly increased the diagnostic performance of radiologists compared to their efforts without these models. Within total joint replacement (TJR), several recent studies have demonstrated clinically relevant capabilities of AI by using them to personalize risk prediction, monitor rehabilitation, and identify implant types (Table I). Furthermore, these algorithms hold the ability to autonomously retrieve data from electronic medical charts, which may expedite registry creation and chart review. The future of AI in TJR and orthopaedics will likely leverage several recently established applications to automate data extraction and analysis, where a patient’s likelihood of experiencing a clinically meaningful outcome or complication, or the exact manufacturer and sizing of their hip arthroplasty, will be automatically calculated and presented to the clinician.

Table I.

Recent applications and rise of machine-learning in arthroplasty literature since 2015.

Year	Studies, n	Study topic(s)
2015	1	Lower limb muscle activation patterns after TKA
2016	1	Gait analysis after TKA/UKA
2017	3	Classification of revision TKA cause, effect of femoral stem morphology on stress shielding, prediction of opioid use after THA
2018	5	Cost use after THA and TKA, patient activity monitoring after TKA, image-based rating of corrosion severity for THA implants, readmissions after TJR
2019	32	Clinical outcome prediction (adverse events and patient-reported outcomes), resource use and cost of episodes of care, patient activity monitoring (wearable sensors, gait analysis), automatic chart review using natural language processing, implant identification*
2020	56	Clinical outcome prediction (adverse events and patient-reported outcomes), resource use and cost of episodes of care, patient activity monitoring (wearable sensors, gait analysis), automatic chart review using natural language processing, implant identification*
2021 to date	45	Clinical outcome prediction (adverse events and patient-reported outcomes), resource use and cost of episodes of care, patient activity monitoring (wearable sensors, gait analysis), automatic chart review using natural language processing, implant identification*

Applications categorized into four major domains for brevity.

THA, total hip arthroplasty; TJR, total joint replacement; TKA, total knee arthroplasty; UKA, unicompartmental knee arthroplasty.

Unintended consequences

Despite numerous theoretical and realized benefits, we must use machine-learning appropriately and avoid the temptation of the “gold rush” to publish as doing so may lead to misleading conclusions. Furthermore, several limitations of the current use of machine-learning in orthopaedic surgery research must be recognized. Though we must trust algorithms to explore and learn from data, we cannot simply assume they are accurate and valid. Algorithms frequently overfit predictions, responding to patterns specific to one dataset that do not accurately reflect the trends of an entire population, especially when multicollinear correlated predictors are present. This may lead to inflated accuracy and inaccurate conclusions about the potential benefits in practice, highlighting why it is essential to externally validate all algorithms on independent populations.

Data quality must also be considered. The pattern and degree of missing data, such as when not missing at random, or variables exceeding 40% missing data, may cause statistical performance bias and portray inaccurate relationships within the model. Furthermore, data must not be biased, and included variables should be representative of the sample of patients. For example, though a feature on a preoperative CT scan of the knee might be an excellent predictor of long-term aseptic loosening after total knee arthroplasty (TKA), it is possible that preoperative CT scans are only obtained in a small and non-representative sample of TKA patients, and therefore limit the generalizability of the model. For these reasons, we recommend abiding by predictive modelling guidelines and expert consensus, such as the Transparent Reporting of a Multivariate Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guidelines, and the Guidelines for Developing and Reporting Machine Learning Models in Biomedical Research, to help ensure that such analyses are appropriately conducted (Table II). It is imperative that researchers and clinicians look for such adherence when evaluating machine-learning research. Furthermore, we recommend against adopting the use of predictive tools presented in these research articles until rigorous external validation has been performed to confirm their efficacy and reliability.

Table II.

Transparent reporting of a multivariate prediction model for individual prognosis or diagnosis guideline checklist.

Topic	Question number	Checklist item
Methods
Source of data	4a	Describe the study design or source of data (e.g. randomized trial, cohort, or registry data), separately for the development and validation data set, if applicable
	4b	Specify the key study dates, including start of accrual; end of accrual; and, if applicable, end of follow-up
Participants	5a	Specify key elements of the study setting (e.g. primary care, secondary care, general population) including number and location of centres
	5b	Describe eligibility criteria for participants
	5c	Give details of treatments received, if relevant
Outcome	6a	Clearly define the outcome that is predicted by the prediction model, including how and when assessed
	6b	Report any actions to blind assessment of the outcome to be predicted
Predictors	7a	Clearly define all predictors used in developing the machine learning model, including how and when they were measured
	7b	Report any actions to blind assessment of predictors for the outcome and other predictors
Missing data	9	Describe how missing data were handled (e.g. complete-case analysis, single imputation, multiple imputation) with details of any imputation method
Statistical analysis methods	10a	Describe how predictors were handled in the analyses
	10b*	Specify type of model, all model-building procedures (including any predictor selection or hyperparameter selection if applicable), and method for internal validation
	10d	Specify all measures used to assess model performance and, if relevant, to compare multiple models
Risk groups	11	Provide details on how risk groups were created, if done
Results
Participants	13a	Describe the flow of participants through the study, including the number of participants with and without the outcome and, if applicable, a summary of the follow-up time. A diagram may be helpful
	13b	Describe the characteristics of the participants (basic demographics, clinical features, available predictors), including the number of participants with missing data for predictors and outcome
Model development	14a	Specify the number of participants and outcome events in each analysis
	14b	If done, report the unadjusted association between each candidate predictor and outcome
Model specification	15a	Present the full prediction model to allow predictions for individuals (i.e. links to the final model online, code, and final parameters/coefficients), with the architecture described in full in the article
	15b	Explain how to use the prediction model
Model performance	16	Report performance measures (with CIs) for the prediction model

CI, confidence interval.

Where we are, and where we need to go

There remain many challenges as we continue to navigate the use of machine-learning, such as better explaining “black box” phenomenon, inadequate data, and model regulation, and integrating such models into clinical workflow. Publishing methods will be essential as more original research continues to be published, likely with differing results. Not only should open source code be published, but explanations as to how and why decisions were made in algorithmic development processes should also be shared among institutions. This increased clarity in reporting will allow researchers and clinicians to be more confident in the results and conclusions presented, and will be a step towards integration of automated models into electronic medical records for real-time use.

Additionally, we must ensure that we are investing in and using high-quality data, as machine-learning models are only as good as the data that are fed into them. High-quality data are unbiased, and it will be essential that electronic medical records and patient data are collected in a standardized and comprehensive manner to avoid erroneous errors and bias. In the private sector, companies spend substantial amounts of money and resources to ensure that algorithms are developed using high-quality and unbiased data. Researchers should not take conclusions at face value without a comprehensive understanding of where the data were derived from and how it was handled.

If we are to continue using machine-learning, we must thoroughly understand the benefits and limitations of these processes. We must be responsible and informed clinicians and surgeons, and practice integrity with our research; this requires concerted efforts to improve the standardization, integration, and availability of relevant high-quality data. Orthopaedic research is now at a crossroads in terms of man versus machine; however, it must be recognized that we are still in the era of needing man to coexist with machine, especially when defining “meaning.” Though machine-learning is great at working quickly and efficiently in defined roles, we still require humans to define these parameters, as the machine cannot learn what is meaningful and does not functional autonomously. Therefore, physician input is another exceedingly important component in creating these models.

AI and machine-learning can find patterns in any large sets of data, but the larger question becomes do they mean anything and can we trust the data? We should continue to attempt to develop clinically meaningful applications, and embrace collaboration in order to ensure the creation of models trained on diverse sets of patients with the potential to be used in real-world settings. However, until there is increased reporting of methodological processes and data by authors, in addition to efforts by clinicians and researchers to become familiar with machine-learning methods, these studies will likely remain limited to interesting analyses criticized by scepticism. Machine-learning has great potential to benefit patients, and as it continues to be applied in orthopaedic research, it will be up to us to be ahead of or behind the curve.

Take home message

- Artificial intelligence and machine-learning research have seen a substantial increase in recent years given several clinically relevant applications.

- Despite widespread optimism and excitement about the capabilities of artificial intelligence and machine-learning, researchers and clinicians must be critical as to whether these methods are indicated and the quality of data used.

- Increased methodological and data reporting clarity, and increased reader familiarity with machine-learning methods, will be two major steps towards transforming machine-learning studies from interesting studies into clinically meaningful and trustworthy applications.