Follow-up Interactive Long-Term Expert Ranking (FILTER): a crowdsourcing platform to adjudicate risk for survivorship care.

OBJECTIVES: To develop an online crowdsourcing platform where oncologists and other survivorship experts can adjudicate risk for complications in follow-up.
MATERIALS AND METHODS: This platform, called Follow-up Interactive Long-Term Expert Ranking (FILTER), prompts participants to adjudicate risk between each of a series of pairs of synthetic cases. The Elo ranking algorithm is used to assign relative risk to each synthetic case.
RESULTS: The FILTER application is currently live and implemented as a web application deployed on the cloud. DISCUSSION: While guidelines for following cancer survivors exist, refinement of survivorship care based on risk for complications after active treatment could improve both allocation of resources and individual outcomes in long-term follow-up.
CONCLUSION: FILTER provides a means for a large number of experts to adjudicate risk for survivorship complications with a low barrier of entry.

BACKGROUND AND SIGNIFICANCE

An estimated 5% of the US population (16.9 million people) are cancer survivors. For these individuals, care should focus on disease surveillance and health promotion to prevent or ameliorate chronic health issues and subsequent malignancies. Adverse health outcomes are well documented among cancer survivors, especially in underserved populations due to barriers to obtaining care such as financial toxicity, low income, transportation, and insurance inadequacy. Services must therefore address and minimize adverse cancer treatment sequele and decrease risk for recurrent or subsequent malignancies.

For well over a decade, national best practice guidelines have recommended that cancer survivors receive survivorship care. Definitions of levels of survivorship care vary widely, but generally determine the frequency that patients are seen by an oncology team vs a primary care provider., A “one-size-fits-all” model is neither feasible nor sustainable. As cancer care is becoming more precise, so too should survivorship care. Care plans should be tailored to the level of services required and based on each patient’s unique set of risk factors. This concept of stratified survivorship care exists in the United Kingdom through the National Cancer Survivorship Initiative and has been appropriately proposed in the United States.

As defined by Oeffinger and McCabe, an individual’s survivorship risk is their chance of premature mortality, serious morbidity, or adverse health status. Currently, risk stratification models for survivorship primarily consider intensity of the cancer treatment and likelihood of adverse health conditions, based on limited expert opinion. Development of a data-driven survivorship risk model would require rigorously collected and sufficiently comprehensive long-term follow-up data—a resource currently lacking in the electronic health record (EHR) or cancer registries. Expert opinion remains the best available resource to assign a survivor into a low, medium, or high-risk group. However, access to these experts is limited in the healthcare settings where they are needed most, such as in community-based rural clinics.

OBJECTIVES

To address this gap, we are seeking to develop a survivorship risk model that calculates a patient’s required level of follow-up care based on their disease, treatment, genetic, socioeconomic, and demographic factors using clinical knowledge from a large group of experts. We have developed a risk stratification crowdsourcing platform called follow-up interactive long-term expert ranking (FILTER), which invites oncologists and survivorship care experts to judge survivorship follow-up complexity.

MATERIALS AND METHODS

Design considerations for clinical expert crowdsourcing

Our goal was to leverage the expertise of many oncologists and survivorship experts to create a risk-informed algorithm for survivorship care. Crowdsourcing as a means of codifying clinical knowledge is a relatively new concept in oncology and clinical research. Nevertheless, expert data curation through tools custom-designed for crowdsourcing has been essential for generating datasets for machine learning and artificial intelligence research in healthcare.,

For success, such tools must address barriers inherent to development and implementation of a crowdsourcing platform. Table  1 summarizes some of these challenges and how our innovative design overcomes each barrier. Because we used a synthetic dataset instead of actual cancer cases, our institutional review board (IRB) determined the adjudication process to be nonhuman subjects research. We were able to avoid many of the regulatory hurdles common in crowdsourced medical research such as waivers of informed consent, privacy controls, and handling of sensitive data. Instead of developing extensive training materials to ensure experts of different backgrounds applied a uniform approach to rating, we developed our interface with simple instructions. Experts can sign up for an account and start adjudicating cases within minutes. Furthermore, each expert may adjudicate any number of cases with no minimum requirement. Even with just a few cases, each expert still provides information for the ranking algorithm to determine a synthetic case’s relative risk. This low barrier of entry facilitates knowledge capture from busy experts across a diverse range of expertise.

Table 1.

Summary of crowdsourcing platform barriers and how FILTER addresses these barriers

Crowdsourcing platform barrier	FILTER solution
Experts are limited to a single institution’s staff and/or external experts under a data use agreement (DUA). IRB approval and oversight can delay data collection.	Use of synthetic cases instead of real patient information allows a large number of adjudicators from many institutions without IRB approvals or DUAs.
Due to experts of different types and levels of expertise, training participants on uniform criteria and approach for adjudication is difficult and can lead to low inter-rater agreement.	Use of a self-explanatory interface with simple instructions for adjudication obviates the need for training.
Each expert must adjudicate a minimum number of cases to ensure sufficient overlap for assessment of inter-rater reliability. This can delay the project as experts tend to be busy.	Experts can adjudicate as many or as few cases as they wish. Even with just a few cases, each expert provides information useful to determine risk.
Due to the complexity of clinical cases, accurate assessment of risk on an absolute scale may present challenges.	Requiring only one-to-one comparison of relative risk is easier to understand and judge.

FILTER: follow-up interactive long-term expert ranking.

RESULTS

Ranking interface and algorithm

The FILTER application has been deployed on the cloud and is available for experts to create an account and adjudicate cases. Figure  1 is a screenshot of the FILTER interface with an example matchup that an expert might adjudicate. For each adjudication, the expert is presented with the question, “Which of the following scenarios requires a higher level of survivorship follow up?”

Figure 1.

Expert adjudicator FILTER interface.

The expert has the option to choose the case to the right, the case to the left, or rank as equal. After each judgment, the scores for the two cases are adjusted using the Elo rating algorithm.

R is the current score and RBi is the score of opponent.

This algorithm, originally developed to rate chess players, sets an expectation that cases with higher scores will likely “win” against cases with lower scores. With each selection, the “winning” case is increased in point value, and the point value of the “losing” case is decreased. The magnitude of point-value change is dependent on how far apart the scores started, with larger changes in the event of an “upset”. We chose the Elo ranking algorithm over others primarily because Elo does not require us to predefine the number of cases that we want to adjudicate, which allows us to add new cases as more matchups are adjudicated. Additionally, the resulting Elo score is parametric, which allows us to consider the magnitude of differences in scores rather than just their order.

New case creation and matchup algorithm

Because neither the number of experts nor the number of adjudications per expert is prespecified, we designed FILTER to dynamically generate new synthetic cases whenever a sufficient number of matchups have occurred. Each new case is randomly assigned risk factors from each domain in Table  2. This list was generated by authors (TO, DF, and TP) who are experts in oncology survivorship and genetic risk factors in cancer. Our matchup algorithm ensures new cases matched to enough existing cases to establish a starting rank. Existing cases are also periodically rematched against one another to reconfirm their place in the ranking.

Table 2.

Survivorship risk factors by domain

Surgery	Radiation
Breast resection	Radiation to the breast
Lung resection	Radiation to the lung
Kidney resection	Radiation to the kidney
Colon resection	Radiation to the colon
Small intestine resection	Radiation to the small intestine
Extremity resection	Radiation to the extremity
Pancreas resection	Radiation to the pancreas
Liver resection	Radiation to the liver
Brain resection	Radiation to the brain
Larynx resection	Radiation to the larynx
Esophagus resection	Radiation to the esophagus
Lymph node resection	Radiation to the lymph node
Testicle resection	Radiation to the testicle
Ovary resection	Radiation to the ovary
Uterus resection	Radiation to the uterus
Bladder resection	Radiation to the bladder
Prostate resection	Radiation to the prostate
Breast removal	Radiation to the neck
Lung removal	Radiation to the stomach
Kidney removal
Colon removal	Systemic drug
Small intestine removal	Anthracyline (like adriamycin)
Extremity removal	Vinca alkaloid (like vincristine)
Pancreas removal	Tumor antibiotic (like bleomycin)
Liver removal	Alkylating agent (like cyclophosphamide)
Larynx removal	Cisplatin
Esophagus removal	Carboplatin
Lymph node removal	Oxaliplatin
Testicle removal	Microtubule inhibitor (like paclitaxel)
Ovary removal	Immunotherapy (like pembroluzimab)
Uterus removal	Monocloncal antibody (like blinatumomab)
Bladder removal	Tetrahydrofolate reductase inhibitor (like pemetrexed)
Prostate removal	Corticosteroids
Stomach removal	Antimetabolites (like mercaptopurine or cytarabine)
Thyroid removal	Topoisomerase I inhibitor (like topotecan)
	Topoisomerase II inhibitor (like etoposide)
Immune modulation
Allogeneic transplant (CyTBI conditioning)	Genetic risks
Allogeneic transplant (BuCy conditioning)	Multiple close family members with cancer
Allogeneic transplant (BuFlu conditioning)	Inherited cancer gene mutation (eg, BRCA, Lynch) identified
CAR-T cell therapy	Increased risk of treatment toxicity due to inherited gene mutation
	Multiple primary cancers of paired organs or different organs
Comorbidity
Active autoimmune disease	Age (years)
Traumatic brain injury	0–10
Congestive heart failure (CHF)	11–20
COPD or obstructive airway disease	21–30
Renal failure	31–40
Obesity	40–65
Tobacco use	65+
Substance abuse
Developmental delay	Socioeconomic status
Hepatic impairment	Low
Hypertension	Medium
Psychiatric illness	High
Neuropathy
Stroke

Since risk factors for each synthetic case are selected randomly, it is possible that clinically unlikely combinations may occur. We considered creating a list of such combinations and eliminating them from possible synthetic cases. However, for almost every unlikely combination, we were able to come up with an edge case where that combination might occur. Our solution was to instruct experts to make their best determination of risk based on the synthetic case even if the combination of treatments would be impossible. We included a disclaimer whenever an expert logged in that stated that synthetic cases were generated randomly, and that many combinations would not be realistic. We also informed experts that the goal of the process was to determine the contribution of each factor to overall risk independently.

Figure  2 illustrates the sequence of matchups starting from new case creation. In Phase 1, a new case is matched against roundup [log2(n)] existing cases selected at random, where n is the total number of existing cases. In Phase 2, each case matched to the new case is matched to roundup [log2(n)] other existing cases, selected at random. In each matchup both cases involved are score-adjusted immediately, according to the Elo formula. After all Phase 2 matchups have occurred, the newly added case is considered an existing case. Another new case is added, and the process repeats. We designed this algorithm so that matchups would be well distributed among cases. Additionally, we hold one case the same in a series of matchups (the new case in Phase 1 and subsequently matched cases in Phase 2) so that both cases do not change from matchup to matchup. We believe this will assist the expert cognitively to improve the ease and speed of adjudicating cases.

Figure 2.

Matchup algorithm following the addition of a new synthetic case “9” with eight existing cases.

Incentives

Participation of experts is incentivized with gift cards for the three experts who adjudicate the most cases. When logged into FILTER, each expert can see his/her own case count, and the counts for the top adjudicators, on the leaderboard. Each expert is required to affiliate with an institution at the time of account creation. This information is used to display an institution leaderboard to encourage friendly competition among groups of experts.

Initial testing

We have done initial testing with a group of 13 Vanderbilt-Ingram Cancer Center oncologists. These oncologists have adjudicated 1174 matchups for 64 cases. In the next phase of FILTER implementation, we plan to invite members of the National Comprehensive Cancer Network (NCCN) Survivorship Guidelines Panel to participate as experts. These individuals will be authenticated through their institutions’ email addresses.

DISCUSSION

We have created an application with a low barrier of entry to obtain expert adjudication of risk. Although we have designed FILTER for risk of clinical complications in cancer survivorship, the platform is generalizable to other medical use cases that require risk or severity scores generated through crowdsourcing. As a crowdsourcing platform, FILTER is unique and powerful because it does not use real patient data, it does not require much instruction for experts to use, and it does not prescribe a minimum input for the contribution of each expert to be considered complete.

FILTER has several limitations. Its ranking algorithm is limited to adjudicating a single ordinal or continuous scale. This precludes FILTER’s use to identify individual, disease, or treatment phenotypes, a common use case for crowdsourcing in cancer research. As experts start to use FILTER, other limitations may emerge. Given our invitation to a wide range of physician participants, inter-rater disagreement may arise due to differences in opinion based on training or background. In addition, “bad actors” may enter purposefully wrong or random information. We believe that, as with other crowdsourcing platforms, these limitations can be overcome by having a large number of adjudicators so that the wisdom of the crowd is captured.

After adjudication of a sufficient number of cases, our next step is to use the risk scores as outcomes in a regression model that will ascertain the contribution of each factor to survivorship risk. The end result will be an online tool that calculates survivorship risk based on the risk factors in Table  2. One limitation to determining how many experts we must engage is that there is no prior data to determine the extent to which experts will disagree on levels of risk. Part of what we will assess in pilot testing with the NCCN survivorship guidelines panel is expert agreement and risk score variability. We estimate that there will need to be at least 870 synthetic cases to obtain a reliable regression model. Assuming FILTER’s matchup algorithm effectively rates those 870 cases, we would need 77 266 matchups adjudicated. Therefore, we anticipate that there must be 772 experts adjudicating an average of 100 matchups each to get a reliable model.

CONCLUSION

The FILTER crowdsourcing platform addresses a critical need for capturing clinical knowledge from experts when real-world data are scarce. Results from data obtained using FILTER will allow oncologists to better assess patient need for cancer survivorship follow-up care, thus allowing healthcare systems to allocate resources and services according to need.

FUNDING

This work was supported by the National Cancer Institute grant numbers R01 CA240093-02S1 and P30 CA068485.

AUTHOR CONTRIBUTIONS

TO conceived of the crowdsourcing platform and use of the Elo rating algorithm. TO, TP, and DF compiled the list of risk factors. LW, AC, and YL developed the platform. TK created the risk model and LB built the risk calculator. All authors contributed to the final manuscript.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

No new data were generated or analyzed in support of this research.