Comparison of Artificial Intelligence with a Conventional Search in Dermatology: A Case Study of Systematic Review of Apremilast in Hidradenitis Suppurativa Performed by Both Methods.

Dear Editor,

Systematic reviews and meta-analyses play an invaluable role in the practice of evidence-based medicine.[1] Unfortunately, the process is time-consuming, on average requiring 67 weeks to sift through all available literature, collate relevant data, and analyze results to form conclusions.[2] However, recent advances in natural language processing (NLP) and machine learning have enabled “artificial intelligence” (AI) to “learn” through algorithms and assist with text classification and data extraction.[3] Semi-automation, with “human-in-the-loop” systems, can potentially assist with several labor-intensive steps of the systematic review process and make it faster.[13] Nevertheless, skepticism as to the accuracy of automated tools exists which presents a barrier to their widespread acceptance.[13]

Two independent investigators conducted a systematic database search of PubMed and ClinicalTrials.gov. SK conducted the search manually and SS performed the search using an AI. All the tools so used were developed in-house using hypertext preprocessor (PHP) language. The different steps so used are shown in Table 1. The difference between the manual workflow and NLP-assisted workflow is shown in Table 2. The time taken for the search and data extraction was recorded. The machines used a mix of NLP and automation. By automation, the AI screened articles and put extracts of relevant articles in their database in a convenient format, for later use. NLP then used “bags-of-words” technique to extract the relevant lines that captured our curated keywords (statistical/genomic/metabolomics). The extracted data were then entered into Microsoft Excel (2010) after which SS filtered the relevant papers. A similar technique using NLP helped analyze the full-text papers.

Table 1

Development of tools for this systematic review

Development of tools for this systematic review
1. Data from PubMed were extracted using their public API and we built hypertext preprocessor (PHP) language-based web codes to extract the data and store in relational database (API)
2. Further, bag-of-words expression was stored in a separate table
3. Further, PHP codes were written to extract the relevant lines having these bag of words
4. The ClinicalTrials.gov data were downloaded in XML format and stored in “mysql” database by creating PHP codes for conversion in respective formats
5. Alternatively, codes are written to parse the data from ClinicalTrials.gov API and stored in relational database (mysql) programmatically
6. Text of full-text paper was added in the code to further extract the relevant expressions and their lines in the paper. The extracted lines were stored in “mysql” database
7. The relevant expression dump was further extracted in excel format for final analysis

API: Application programming interface

Table 2

Differences between manual workflow and natural language processing (NLP)-assisted workflow

Manual workflow	NLP-assisted workflow
Part A: For PubMed, we created the search expression and searched through the abstracts	The machines used a mix of NLP and automation
	By automation, it automatically screened through list of relevant articles and dumped their extracts in the relational database for later use in convenient format
Then, we read through each abstract manually and documented the relevant points/lines separately in Excel. Further, we selected the articles for full-text review
The work was divided in groups and separate Excel sheets so created were finally collated in one	NLP further used bags-of-word expression technique to extract the relevant lines that captured our curated keywords
After selecting the relevant papers, we downloaded and read the full-text articles	The entire dump was taken in Excel over which the team then easily filtered the relevant papers
The relevant lines were again extracted and collated in the Excel	A similar technique around NLP was further used to analyze the full-text papers
Part B: For ClinicalTrials.gov, we again created the search expression and searched through trial data
	For ClinicalTrials data, the dump was extracted in Excel/CSV from the ClinicalTrials website for quick review
We collated data from the ClinicalTrials.gov and collated the findings in Excel
The results were again reviewed
The group then sat to filter the relevant evidence for systematic review

NLP: Natural language processing

We included trials that studied the efficacy of apremilast in hidradenitis suppurativa published in English, from database inception till January 2021. The process of article selection is detailed in Figure 1.

Figure 1

Details of the article selection process and time taken by both semi-automated and manual methods. The PubMed search terms used were (“apremilast”[Supplementary Concept] OR “apremilast”[All Fields]) AND (“hidradenitis suppurativa”[MeSH Terms] OR (“hidradenitis”[All Fields] AND “suppurativa”[All Fields]) OR “hidradenitis suppurativa”[All Fields])”. Abbreviations: AI, artificial intelligence; min, minute (s); n, number; sec, seconds; MS Excel (version 2010)

We found that the papers were selected and conclusions reached were the same by the semi-automated and completely manual methods. The time taken both for the article selection and data extraction was lower for the search conducted with AI assistance [Figure 1]. A little more than half the patients (54.2%; 19/35) treated with 30 mg twice daily of apremilast achieved ≥50% reduction in Hidradenitis Suppurativa Clinical Response (HiSCR50) from baseline at 16 weeks compared with none in the placebo group.[45] [Table 3]

Table 3

The characteristics and summary of included trials

Author, year	Study design	Apremilast group (n)	Placebo group (n)	Apremilast dose	Treatment duration	Achieved HiSCR50 in the treatment group at 16 weeks, n (%)	Achieved HiSCR50 in the placebo group at 16 weeks, n (%)	Follow-up duration
Vossen, 2019	RCT	15	5	30 mg twice daily	16 weeks	8 (53.3)	0 (0)	8 weeks
Kerdel, 2019	CT	20	NA	30 mg twice daily	24 weeks	11 (55)	NA	28 weeks

Abbreviations: Single-arm clinical trial; HiSCR50, ≥50% reduction in Hidradenitis Suppurativa Clinical Response from baseline (a 50% reduction in total abscess and inflammatory nodule count); NA, not applicable; RCT, randomized control trial

Recognition of the potential for AI to simplify and expedite the systematic review process led to the formation of the International Collaboration for Automation of Systematic Reviews.[1] In this review, we found that the use of automation drastically reduced the total time used to process available literature. This will be critical in larger systematic review that retrieves large number of articles for screening. It also eliminates time lost due to unplanned disturbances and fatigue that inevitably creeps in after perusing a large amount of literature. Machine-assisted processing minimizes mundane tasks, such as extracting several sentences manually for review by peers. This leaves us free to work on critical tasks.

Through this preliminary and small-scale systematic review, we assessed the utility of semi-automation and NLP for systematic review. Our study was limited by the fact that we performed this systematic review for a topic which yielded only 15 articles. Other than the advantage of time, we were unable to find any other significant difference between the two methods. Further large-scale comparative systematic reviews are needed to assess machine accuracy and gain more confidence in using machines.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.