An automated system for retrieving herb-drug interaction related articles from MEDLINE.

An automated, user-friendly and accurate system for retrieving herb-drug interaction (HDIs) related articles in MEDLINE can increase the safety of patients, as well as improve the physicians' article retrieving ability regarding speed and experience. Previous studies show that MeSH based queries associated with negative effects of drugs can be customized, resulting in good performance in retrieving relevant information, but no study has focused on the area of herb-drug interactions (HDI). This paper adapted the characteristics of HDI related papers and created a multilayer HDI article searching system. It achieved a sensitivity of 92% at a precision of 93% in a preliminary evaluation. Instead of requiring physicians to conduct PubMed searches directly, this system applies a more user-friendly approach by employing a customized system that enhances PubMed queries, shielding users from having to write queries, dealing with PubMed, or reading many irrelevant articles. The system provides automated processes and outputs target articles based on the input.

Introduction

Herb-drug interactions (HDIs) are one of the causes of adverse drug effects and one of the factors causing reduction of the safety of patients[1]. As a consequence, HDIs are becoming a concern in clinical practice. During the past decade, a variety of articles related to HDIs have been published, including research results, clinical trials, case report, and reviews[2,3]. Most of these articles are collected by Medical Literature Analysis and Retrieval System Online (MEDLINE), which is the largest database for peer-reviewed and evidence-based articles[4]. During clinical practice, physicians can access these articles through PubMed, which is the most popular online search engine providing office-based MEDLINE searching[5]. For building queries, PubMed provides varied tags which represent different search fields related to information associated with the articles, for example, tag “AU” represents author, and “DP” represents publication date[6]. However, studies suggested that searches conducted by physicians can only retrieve between 31-46% of relevant articles[7,8]. This is because the physicians have to write queries using PubMed. When developing queries, physicians often face the problem of limited time and limited information retrieval techniques[9,10,11]. As a result, a user friendly system for searching papers related to HDIs with high performance is needed to translate research reports and clinical findings from academic literature to actionable information at point of care.

Machine learning is one of the methods previously used in related work[12,17]. A previous study focused on building an annotated corpus to support the automated extraction of drug-related adverse effects from medical case reports[12]. This study showed that a maximum entropy classifier could achieve an F1 score of 0.7 on their corpus. Another study applied a kernel based support vector machine to detect potential adverse drug events, and showed that the size of training set can significantly impact machine learning based methods in this area[17]. The lack of training data is one of the current concerns in related work using machine learning methods as the latter study showed the limitation of machine learning methods.

However, previous studies demonstrated that certain filter-based PubMed queries can lead to a sensitivity of 99% at a specificity of 70% in finding papers on glomerular disease[10,11]. A more recent study automatically generated queries to search for papers related to adverse drug effects for specific drugs[12], and resulted in a sensitivity of 90% and a precision of 93%[12] But none of these studies focused on HDIs, which has different challenges. First, unlike drugs or other well recognized medical terms, herbs are often mentioned in more diverse ways in different dialects, different countries, and even in different published articles, which leads to difficulties in identifying the herb terms[13]. And for each herb-drug pair, only a very limited number of articles mention their potential interactions, and the sources of these articles are diverse, including clinical trials, case reports and biological lab studies[14,15,16].

To search for HDI articles accurately from the large MEDLINE database, a system needs to be very sensitive to the target articles while also having to have a high precision in order to avoid retrieving too many unrelated articles. A recent study described techniques involving applying filters and using Medical Subject Headings (MeSH) based search terms[18]. MeSH is a controlled vocabulary used by the MEDLINE database for indexing articles in the life science area, and is an important resource in article retrieving tasks[19]. It contains a wide range of medical related subject headings, such as the terms for drug and herbs[19]. And some of the headings are noted by major notations, which indicate that these heading are one of the main topics discussed in the papers[6]. In addition, the term “herb-drug interactions” in MeSH is especially useful in our project[21,22].

Based on previous studies and resources, this project aims to combine the characteristics of HDI related papers and create an accurate HDI paper searching system for MEDLINE articles. The objectives of the system are to have high sensitivity and precision, to be automated and user-friendly, and to be able to completely shield users from having to generate PubMed queries or to read many irrelevant articles.

Method

Overview

The workflow of our system is shown in Figure 1. The overview of the system has four steps: fetcher, query generator, scorer, and filterer, and the overall process is sequential. With an input, which is an herb-drug pair of terms, our system will display the papers that discuss interactions related to this pair. To accomplish this task, four main components contribute to our system. The term fetcher preprocesses the name of an herb-drug pair and returns a list of identifiers or terms corresponding to the input, including MeSH descriptors, synonyms, supplementary concepts in MeSH, and U.S. Food and Drug Administration, Substance Registration System - Unique Ingredient Identifier (UNII) codes[21]. With the identifiers, a query generator will generate a broad query to retrieve a large number of papers from PubMed. A knowledge based scorer then will score these papers and rank them with the level of relevance to the interaction of the input herb-drug pair. In the final step a filter will set a cut off value for the output to restrict the number of output papers to a certain number.

Figure 1.

Overview of the methodological framework

Term fetcher

The tasks associated with the term fetcher involve finding the MeSH descriptors, a comprehensive list of synonyms and UNII codes for the input herbs and drugs. In the most common scenario, herbs and drugs will have multiple names or identifiers that will prevent simple queries from retrieving all the relevant papers. As a result, building a fetcher to determine all the synonyms and identifiers for herbs and drugs is crucial for the remaining steps. Our term fetcher works in three steps: first by querying the name of the drugs and herbs in the MeSH database to find the unique MeSH descriptors. If they are successfully fetched, which is the most common case, then the descriptors will be considered to be the main terms of the queries; otherwise the main terms will remain to be the original input terms. After obtaining the main terms, we then fetch all the synonyms of the main terms from the MeSH database[23], which will result in a list of all synonyms for each of the specific drug and the specific herb. For example, the corresponding MeSH descriptor for “St. John’s Wort” is “Hypericum”, and the potential synonyms for it as listed in the MeSH database are “Hypericum perforatum”, “St. Johns Wort”, “Wort, St. John’s” and 19 other different representations of this herb. After fetching the relevant descriptors and synonyms, we search the main term in the UNII[21] database, to collect the UNII code as a supplementary identifier to the input terms. For example, the UNII for “St. John’s Wort” is “UFH8805FKA”. The first two steps are achieved by utilizing the E-utilities[5], which is an online application programming interface provided by NCBI, and the third step is accomplished by downloading the UNII database from the UNII website[21] and using automatic searching code created by us. Frequently, MeSH descriptors and synonyms will suffice to identify most of the drugs and herbs. In some scenarios, especially when new drugs have not yet been assigned asa MeSH descriptors, they can be found as MeSH supplementary concept data, and corresponding UNII codes will be used as a supplementary identifiers of the drugs and herbs. For example, currently, the drug “Rofecoxib” can be found as a MeSH supplementary concept and ea corresponding UNII code 0QTW8Z7MCR. Table 1 shows a sample output, which has been shortened, from the term fetcher for Ginseng.

Table 1.

Sample output from term fetcher for Ginseng.

Mesh Descriptor	Panax
Synonyms	Panaxs Renshen Shinseng Ninjin Schinseng Ginseng Panax ginseng .......
UNII code	CUQ3A77YXI

Query generator

After determining the identifiers of the herb-drug pairs, we next built a broad PubMed query to collect paper references in MEDLINE. This query is designed to be the most inclusive to broadly cover many possible papers. The structure of the query is shown in Figure 2.

Figure 2.

Structure of broad query

In Figure 2, Drug_MT is the main term of the specific drug while Herb_MT is the main term of the specific herb, and the synonyms and UNIIs are the terms found in the previous step. “[MH]” is a tag indicating that the phrase before it should be a MeSH Heading part of the paper, “[TIAB]” is the tag for title or abstract where “TI” represents title and “AB” represents abstract, “[RN]” is the tag for registration number such as UNII code and “[LA]” is the tag for language. The query consists of two components and a set of filters where the components are designed to identify all the possible papers related to the input drug and herb respectively, and the filters are designed to restrict the articles so that they are related to humans and are written in English. An example of the query for the drug “Dextromethorphan” and herb “St. John’s wort” is shown in Figure 3.

Figure 3.

Broad query for drug “Dextromethorphan” and herb “St. John’s wort”

By including all the possible representations of the herb and drug, we retrieve all the papers that mention the input terms or their variations. By using the keyword “AND” to connect the two components and the filters, we retrieve only the intersection of the output of the three components, which are all the papers mentioning both the herb and the drug, which are also associated with humans and written in English. This query was manually developed, by querying possible herb-drug pairs on PubMed and iteratively improving the query to improve sensitivity in order to include more potentially relevant papers. Through E-utilities, we apply this query using PubMed and collect all the papers that were retrieved in MEDLINE format. An example of a paper retrieved in MEDLINE format is shown in Figure 4. It is a paper related to an interaction between Echinacea and Midazolam, and “PMID” is the PubMed identifier of the paper. In Figure 4 the paper is not shown in the original version for this article, but the relevant elements are shown in a modified version to provide an easier display and analysis.

Figure 4.

Information in an article related to the interaction between Echinacea and Midazolam

Scorer

After collecting the results of the broad query from the previous step, we then analyze the contents of the papers to determine their level of relevance to the input herb-drug pair’s interaction. Based on the information extracted from MEDLINE, which contains a set of manual annotations of the papers assigned by expert indexing MEDLINE curators, we build a scorer. The scorer has three components: a drug term scorer, an herb term scorer and an interaction term scorer. These three components are designed to capture different types of information of the articles. For example, if a drug name and herb name both appear together in the title of a paper, as in the paper “Interactions of warfarin with garlic, ginger, ginkgo, or ginseng: nature of the evidence”, then it is highly possible that the paper mentions an interaction of the specific herb-drug pair. Similarly, if the “Herb-Drug Interaction” term is one of the Mesh Headings of the paper and with major notation, then the paper most likely concerns a herb-drug interaction. The scores of the three scorers are calculated separately, for each scorer. The system will match the input article to the scoring criteria and output a list of scores. For each scorer, the score of the article will be the maximum score associated with the article. Figure 5 shows the scoring process of the Herb Term scorer. The overall process of the other two scorers is exactly the same as this process except it uses different scoring criteria, as shown in Table 2. The final score of the article is the sum of all three scores. The final scores can range from 2 to 21 points.

Figure 5.

Scoring process for Herb Term scorer.

Table 2.

Scoring criteria for scorers

Scorer type	Drug Term Scorer	Herb Term Scorer	Interaction Term Scorer
Term appear in title	5	8	8
Term appear in abstract	2	2	1
UNII code included	1	1	N.A.
Term appear in Mesh-Heading	2	3	5
Term appear in Mesh-Heading with major notation	2	3	8
Term appear in Mesh-Heading with “/adverse effects” notation	5	5	N.A.
Term appear in Mesh-Heading with “/pharmacokinetics” notation	5	N.A	N.A.
Term appear in Mesh-Heading with “/chemistry” notation	N.A.	4	N.A.
Mesh-Heading “drug interaction”	N.A.	N.A.	3
Mesh-Heading “Plant extractions/pharmacology”	N.A.	N.A.	8

Term referring to Name/Synonyms for drug and herb, to the phrase “herb-drug interaction” for Interaction.

Table 2 shows the scoring criteria for all three scorers. The article with the herb “Echinacea” and the drug “Midazolam” in Figure 4 can be shown as an example of the scoring process. Since “Midazolam” appears in both UNII and MeSH with “/pharmacokinetics”, the drug term scorer will assign it 5 points. The herb “Echinacea” will have a score of 8 points since it appears in the title. For the interaction term scorer, the score will be 8 points since the “Herb-Drug Interactions” term appears in MeSH as a major notation. Therefore, the final score for this article is 21. The scorer’s criteria was developed based on a manual analysis of a subset of herb-drug interaction articles mentioned in a previous review[22]. The subset of training articles were all articles related to ginkgo, which is associated with 26 drugs and found in 34 articles. We first built the initial scoring criteria based on observations after reading the 34 articles. We then iteratively refined the criteria by adding and deleting features as appropriate, and changing the leverage of the scores for each feature after evaluating the scorer’s performance on the same developmental training set. After three iterations, the scoring criteria converged to the final version. As shown in Table 2, for each scorer, different leverage values were assigned to different items, and some of the items were notapplied to all the scorers. For example, the Mesh-Heading “Plant extractions/pharmacology” will be assigned 8 points by the interaction term scorer, but it is not applicable to either the drug or herb term scorer.

Filter

After the score of each paper is calculated, we then apply a filter to the papers, to output only those papers higher than a cut-off score which is determined by the user. As the score goes higher, the precision will likely increase while the sensitivity will likely decrease and vice versa. Users can choose the level of precision and sensitivity by changing the cut-off score. Based on the observation of the best precision and sensitivity in our training articles, the default cut-off score of the articles was set to be 12.

Evaluation method

Evaluation of the method was performed by the author, and the cut-off score was set to be the default. The evaluation set consisted of 5 randomly selected herb-drug pairs in the review article[22] excluding the herb gingko because it formed the training set. For the evaluation of precision, we manually looked at the articles retrieved for each herb-drug pair, and decided whether they were relevant or not based on our knowledge. For the evaluation of sensitivity, since it is impossible to manually review all the articles in the MEDLINE database, we used the review paper to simulate sensitivity. For each herb-drug pair in the review paper, several articles related to the pair’s interaction were provided[22]. Our approach for the sensitivity simulation was to determine whether the articles that were mentioned in the review paper were also retrieved by our system for each herb-drug pair in the test set. We then calculated the sensitivity to be the ratio of correct articles that were retrieved over all those that should have been retrieved. For example, for the herb “Garlic” and drug “Caffeine”, the review mentioned 2 articles, and our system retrieved 5 articles which consisted of the 2 articles in the review; however, only 4 of the articles were determined to be relevant, and therefore the sensitivity for that pair was 100% while the precision was 80%[22]

Results

Table 3 shows the evaluation results for our system. The system retrieved 11 articles out of 12 from the review, where 27 were relevant out of the total of 29 retrieved, resulting in a sensitivity of 92%, a precision of 93%, and an F-1 score of 92%. The results shown in Table 3 demonstrates that this system can achieve high sensitivity and precision for all of the test cases. The majority of time, a 100% sensitivity and prevision was achieved. This indicates the reliability of this system and the potential future use of it.

Table 3.

Evaluation result

Herb	Drug	Sensitivity	Precision
Garlic	Caffeine	2/2, 100%	4/5, 80%
Kava	Midazolam	2/2, 100%	4/4, 100%
Saw palmetto	Alprazolam	1/1, 100%	3/3, 100%
Ginseng	Phenelzine	2/2, 100%	9/10, 90%
St. John’s wort	Dextromethorphan	4/5, 80%	7/7, 100%
Total		11/12, 92%	27/29, 93%

Discussion

In this project, we introduced an automated and user-friendly herb-drug interaction PubMed-based article retrieval system. The method is based on use of MeSH descriptors, synonyms, supplementary concept records, and UNII codes of herbs and drugs as well as the MeSH headings, titles and abstract information associated with the abstracts of the articles. Evaluation was based on a randomly selected set of herb-drug pairs from a previous review article[22]. The precision of the system was 93% and the sensitivity of the system was 92%, demonstrating that this method performed very well. This result conforms to previous results of queries developed for adverse drug reactions and other query generation articles[9,10,18]. Our results also conforms to the finding that using information in MeSH headings can achieve a high precision when performing PubMed searches[20].

Previous work focused on areas where many articles are usually retrieved because there is more information in the literature associated with those types of queries (i.e. queries about adverse drug events), while our work focused on a very small subset among all medical papers[9,10]. This difference introduced difficulties during system building and evaluation. A highly sensitive system is required to retrieve only relevant articles from a large pool of all articles. Because of the sensitivity-precision trade off, the system’s precision was hard to maintain on a relatively high level. To achieve this goal, we developed a multi-step system that is more complicated than a single query, and successfully secured the performance of our system to a high level.

To achieve the aim of article retrieval, prior work focused on building queries and searching PubMed[9,10,18]. Our project focused on creating a more independent domain specific system for each type of information involved in the query, allowing us more freedom in using all the related information, instead of being constrained by the limitations of PubMed querying capability. Since PubMed is designed for a much broader usage, a customized querying system can be a better fit for a specific task[4,5].

Our system completely shields users from having to write a PubMed query. It takes simple medication and herb names as input and directly delivers the relevant articles. This saves the user’s time, and addresses concerns discussed in a previous study, which showed physician’s lack of time and ability to query articles[8]. The filter at the last layer of our system provides users choices toward the precision and recall level they want. With the default cut off, the system can achieve a best balance of precision and recall, but if a user wants the system to be more inclusive, decreasing the cut-off value is very straight forward and an easy way to address that. Thus, we achieve our goal in creating a user-friendly system.

Figure 6 and Figure 7 shows the error cases from our evaluation results, where only the related areas in the MEDLINE format are displayed. Figure 6 shows an error case in sensitivity for the herb “St. John’s wort” and the drug “Dextromethorphan”, and Figure 7 shows an error case in precision for the herb “Garlic” and the drug “Caffeine”.

Figure 6.

Error case for sensitivity in evaluation results

Figure 7.

Error case for precision in evaluation results

From Figure 6 we can see that although both the herb and drug are mentioned in MEDLINE, we could not find evidence of interactions in the MeSH Headings or the term “Herb-Drug Interaction” in the title or abstract. This indicates that one limitation of our system is that the detection of an interaction is heavily based on MeSH Headings, and therefore when the MeSH Heading does not completely list that characteristic of the paper, our system is very likely to fail detecting an interaction. And for the error case in Figure 7, although the paper mentioned both the herb and the drug, the subject of the paper concerns the substance in both the herb and the drug, which influences human health. Correspondingly, the MeSH Heading “Plant Extracts/pharmacology” in Figure 7 indicates the “pharmacology” use of the “plant extracts” instead of the “pharmacology” influence of the “plant extracts”. This error example shows another limitation for our system, which is that it cannot handle ambiguous situations, when a herb and a drug appears together with an ambiguous interaction term when the term actually does not indicate an interaction. In those cases, our system will tend to make the wrong decision.

There are other limitations in our system. First, we have only very limited data. There were around 100 articles mentioning herb drug interactions in the previous review paper, which is not enough for training a very stable system[22]. In future development, a larger training set will be collected but will be challenging to collect. Additionally, due to the limitation of human resources, the evaluation method only randomly picked five herb-drug pairs, which is not large enough for an insightful evaluation. In addition, the developer of the system evaluated precision and could have introduced errors. When evaluating the recall, we applied a simulated recall by calculating the percentage of the articles mentioned in the review that were also retrieved by our system. It would be impractical to review all the articles in MEDLINE to calculate the recall, but a larger pool of articles is needed to have a better simulation of recall. In future work, the evaluation method can be ameliorated by including a larger evaluation set of articles in the evaluation of recall, as well as by including experts in the evaluation of precision. Another limitation of our system is that after the search using the broad query, the system needs to download all the MEDLINE information corresponding to the articles as well as the articles, which reduces the speed if the total number of the articles is large. A better approach for this problem would be to maintain a database, which includes all articles that were collected and were relevant to ensure a faster query.

Conclusion

The system described in the paper provides an easy, accurate way to retrieve herb-drug interaction related articles from the MEDLINE database. It displayed high sensitivity and precision at 92% and 93% respectively in our preliminary evaluation. We believe this system should be further studied and can be used in clinical practice where it can improve the safety of patients, as well as improve the physicians’ article retrieving quality, speed and experiences.