Characterizing California licensure status and tobacco user experience with adverse events using Yelp data.

Various tobacco vendors, including alternative tobacco product sellers, are listed on the popular crowdsourced business listing platform Yelp. Yelp is used to rate and choose tobacco, electronic nicotine delivery systems (ENDS) goods/services and includes self-reporting of user experiences with shops and products. We cross-referenced California Department of Tax and Fee Administration (CDTFA) licensed tobacco, vape, and head shop retail stores with publicly available Yelp business listings to identify licensed and unlicensed stores in California. We extracted metadata associated with store accounts and analyzed user comments and ratings for discussion of tobacco-related complaints and adverse events. We detected a total of 3,717 shops that were categorized as tobacco/vape/head shops on Yelp and by cross-referencing with CDTFA data, licensed businesses accounted for 49.5% (n = 1,841), licensed individual retailers 31.6% (n = 1,174), and suspected unlicensed storefronts 18.9% (n = 702). Businesses and individuals with a state tobacco retail license received a higher average rating from Yelp users (3.86 out of 5) compared to unlicensed shops (3.57) (p < 0.001). Additionally, 4,682 unique comments about licensed businesses, 1,535 unique comments about individual retailers, and 560 unique comments about unlicensed vendors were reviewed, with themes including discussion about defective and counterfeit products and adverse events including coughing, difficulty breathing and reports of hospitalization detected. In contrast, comments about licensed stores predominantly discussed customer service issues. Close to one-fifth of tobacco, vape and/or head shops reviewed on Yelp were not in CDTFA's licensure database. Overall self-reported tobacco user experiences appeared to differ in content and severity based on whether an establishment was licensed. These results have the potential to identify unauthorized stores and adverse events associated with their tobacco and vaping products or services.

Introduction

Electronic Nicotine Delivery Systems (ENDS) are increasingly being used simultaneously with cigarettes (dual use) or as an alternative to traditional cigarettes (Choi and Chen-Sankey, 2020, Pepper and Brewer, 2014). The popularity of ENDS, which first became available in the United States in 2007, has shown overall growth among youth and nonsmokers, even though there have been substantial decreases in reported use of cigarettes in this same population (FDA, n.d, Attfield et al., 2020, CDC, 2020). Hence, despite very recent reports of decreased use of e-cigarettes among youth in 2020, the health risks associated with ENDS may be evolving due to changes in user risk perception of vaping behavior and corresponding expanding retail and product availability (Sun et al., 2021).

Further, the public health impact of ENDS remains unclear, with vitamin E acetate THC-containing ENDS largely attributed to the 2019 e-cigarette, or vaping, product use-associated lung injury (EVALI) outbreak, along with nicotine containing ENDS associated with other adverse health effects (e.g., brain development, lung and cardiovascular health concerns) (Ren and Lotfipour, 2019, Martin and Sayette, 2018, McGrath-Morrow et al., 2020). The long-term impacts of exposure to ENDS remains concerning, particularly in the context of ongoing debate seeking to regulate the sale of these products due to their potential health harms (Attfield et al., 2020, Callahan-Lyon, 2014, Hajek et al., 2014, Kalininskiy et al., 2019).

The ENDS retail environment includes conventional retail shops and retail stores that may also have an online presence, including small and independent businesses. Expansion of ENDS retail access points has coincided with aggressive marketing strategies that target vulnerable populations such as young adults for initiation, and transition of use between combustible and alternative and new nicotine products (Soneji et al., 2019). Many of these retailers now sell different combustible, vaping devices (e.g., open and closed systems), other components (e.g., e-liquids, replacement parts), and also cannabis/cannabidiol vaping products and e-liquids often in response to changes in the tobacco and marijuana regulatory environment (Berg et al., 2020). State and local tobacco retail licensing (TRL) laws can apply to both sellers of traditional combustible tobacco and ENDS retailers (commonly known as “vape shops”). As of September 2021, 37 states and 2 U.S. territories require licenses for retail sales of e-cigarettes (Public Health Law Center, 2021).

Specific to California, every cigarette, tobacco and ENDS product retailer is required to obtain a TRL from the California Department of Tax and Fee Administration (CDTFA) and renew it annually (CDTFA, 2003). On June 9, 2016, California state law expanded the definition of a tobacco product and its regulatory oversight to ENDS products, including requirements for licensing. Importantly, licensed tobacco retailers in California cannot sell cannabis. In addition to a state license, tobacco and ENDS retailers may also be subject to local tobacco retail licensing ordinances depending on the jurisdiction and location of the business. In 2019, 61 California cities and counties adopted and/or updated their tobacco control policies, including 16 that strengthened requirements for tobacco retailers to obtain licenses by requiring retailers to pay an annual fee, renew the license annually, and increase enforcement of violations through fines, penalties or license suspension (American Lung Association, 2020).

Yelp is a popular crowd-sourcing business listing website that had 31 million unique app device users and had 224 million reviews in 2020. The reviews and ratings provide insight into consumer experiences with specific businesses. Reviews specific to a store/business type can serve as a potential data source for identifying consumer safety issues, including those specific to ENDS and other tobacco products. Prior studies have examined Yelp listings of tobacco, vaping and hookah shops to identify store characteristics, marketing strategies, and examined geographical location of retailers (Berg et al., 2020, Cawkwell et al., 2015, Kong et al., 2017, Lee et al., 2018). For example, Sussman et al. used Yelp reviews in California to generate insights into consumers’ perceptions about ENDS, including health concerns, and Lee et al. used Yelp along with other sources to better identify general characteristics of vape shops in North Carolina (Lee et al., 2016, Sussman et al., 2014). However, no existing studies have attempted to use Yelp to identify TRL status or its potential association with user-reported adverse events or product safety issues. Hence, our study builds on prior research by using advanced methods in data mining to characterize the licensure status of California tobacco and ENDS retailer listings on Yelp while also reviewing user reports for topics related to adverse events, underage selling, and presence of counterfeit products.

Methods

Data collection

A list of licensed tobacco and ENDS retailers operating in the state of California from 2017 to 2020, which included categories of (a) licensed businesses; and (b) individual sellers, was obtained from CDTFA in August 2020. The list of state licensed tobacco retailers operating in California provided detailed tobacco licensure information on businesses consisting of corporations and/or partnerships that were wholesalers, distributors, and retailers. Data was also available for individual sellers consisting of sole proprietors and husband and wife co-owners and domestic partners. In this study, licensed businesses and individual sellers will be referred to as “licensed storefronts” only when comparing the broader categories of licensed and unlicensed stores.

We cross-referenced CDTFA licensing data with business listings from Yelp using custom programming scripts in Python, which used data mining to match retailer names and business addresses to the state licensing records (see Supplementary File for additional information). Data mining returned and collected results from the Yelp search function, which included metadata associated with business account results (e.g., “store name”, “store owner”, “categories”, “location”, “rate”, “comments”.) We matched Yelp store metadata to CDTFA licensure data in order classify which stores were licensed (i.e., those business listings that matched to licensure data) versus those suspected of being unlicensed (i.e., those that had no results that matched to licensure data).

After identification of Yelp listed tobacco and vaping store/retailers, we collected all publicly available user-generated messages, including comments, reviews, and ratings associated with these listing. Our review of user-generated comments was limited to posts occurring from 2017 to 2020 (i.e., after ENDS retailers became required to obtain a license), and those with a negative rating. Negative ratings (those with a score of below 2 (out of 5)) were chosen on the basis of likelihood for complaints, reports of product issues, and adverse events in comparison to neutral (rating of 3) or positive reviews (rating of 4–5). After user-generated data was collected, we conducted inductive content coding stratified for tobacco and/or vape store licensure status: (a) licensed tobacco and/or vape businesses that matched with CDTFA business listing; (b) licensed individual retailers that matched with CDTFA individual proprietor listing; and (c) unlicensed storefronts that did not match to CDTFA listings but were located in California. Content analysis results are summarized for two periods, before the EVALI outbreak (prior to April 2019) and after.

Content and statistical analysis

Reviews posted by Yelp users were manually coded for content analysis using an inductive coding scheme to identify themes related to self-reporting of: (a) adverse events associated with tobacco/ENDS products; (b) other product safety and quality concerns; and (c) underage selling (see Supplementary File for code book). Reviews were also separately examined for discussion related to a THC or Vitamin E acetate ENDS product given their association with EVALI, with these results reported separately. First and second authors coded all Yelp reviews independently and achieved a high intercoder reliability (kappa = 0.95) for codes. For inconsistent results, authors reviewed the comments with other authors and conferred on the correct classification. Independent samples t-test were used to compare mean ratings of licensed and unlicensed storefronts. Inductive coding was followed by Fisher’s exact tests to determine significant proportional differences in themes for hypothesis testing to compare licensed and unlicensed stores. All statistical analyses were conducted using RStudio version 3.6.1. A p-value of < 0.05 was considered statistically significant.

Results

A total of 29,249 state licensed CA tobacco businesses and 14,417 individual retailers were provided by CDTFA for 2017–2020. Of these, 3,717 stores were matched on Yelp, categorized as tobacco, vape, and/or head shops. The vast majority of CDTFA license holders that were not matched to tobacco, vape, and/or head shop specific Yelp business listing were determined to be large retail chain outlets (e.g., grocery stores, big-box retailers, gas stations, etc.) that were not specifically categorized as tobacco retail outlets. Businesses accounted for 49.5% (n = 1,841) and individual retailers were 31.6% (n = 1,174) of the matched dataset. Additionally, we detected 18.9% (n = 702) tobacco, vape, or head shop business listings that did not match to CDTFA records, which we categorized as possible unlicensed storefronts (see Table 1 and Supplementary File). After removing all duplicate comments, a total of 40,001 unique user-generated comments were analyzed from all matched Yelp listings identified (see Supplementary File for additional results). Hypothesis testing to compare the proportional differences in retail characteristics and user reported themes of interest for licensed and unlicensed stores were examined (see Table 2). Additionally, 167 stores accounted for multiple (2 or more) reports of adverse events/counterfeit and defective products/underage selling. Among these stores, 13 stores had 5 reports or more, 8 stores had 4 reports, 29 had 3 reports, 117 had 2 reports. THC-related topics were only detected in the general customer service and complaints comments category, and none were related to or referenced the 2019 EVALI outbreak or originated from retailers where an adverse event was reported by a user.

Table 1

Retail characteristics and study themes by licensure status.

Retail Characteristics	Licensed Businesses			Licensed Individual Retailers			Unlicensed Stores			Total
Retailer Count, N (%)	1,841 (49.5%)			1,174 (31.6%)			702 (18.9%)			3,717
Mean Rating on Yelp	3.9			3.8			3.6
% Comments with rating 2 or below	21.8% (4,682 out of 21,501)			9.5% (1,535 out of 16,121)			23.5%(560 out 2,379)
Themes	Licensed Businesses			Licensed Individual Retailers			Unlicensed Stores			Total
	Pre 04/2019 (27 months) Rate per month (count)	Post 04/2019 (16 months) Rate per month (count)	Total count	Pre 04/2019 (27 months) Rate per month (count)	Post 04/2019 (16 months) Rate per month (count)	Total count	Pre 04/2019 (27 months) Rate per month (count)	Post 04/2019 (16 months) Rate per month (count)	Totalcount
Adverse Events	0.04 (1)	0.2 (3)	4	0.1(3)	0.3 (4)	7	0.07 (2)	0.06 (1)	3	14
Product Safety and Quality	7.9 (2 1 3)	14.7 (2 3 5)	448	4.4 (1 1 8)	9.5 (1 5 2)	270	1.7 (45)	2.4 (38)	83	801
Underage Selling	0.04 (1)	0.3 (5)	6	0.2 (5)	0.6 (9)	14	0.04 (1)	0.0 (0)	1	21
Customer Service Complaints	86.8 (2,344)	100.6 (1,610)	3,954	24.9 (6 7 1)	32.8 (5 2 5)	1,196	9.5 (2 5 6)	12.1 (1 9 3)	449	5,599
Non-relevant Themes*	4.7 (1 2 8)	8.9 (1 4 2)	270	1.2 (33)	0.9 (15)	48	0.5 (14)	0.6 (10)	24	342

*Reviews unrelated to tobacco/ENDS products.

Table 2

Fisher’s exact test results for study themes among licensed and unlicensed tobacco and/or vape storefronts.

Themes	Licensed Stores N (%) (n = 3,015)	Unlicensed Stores N (%) (n = 702)	Fisher’s Exact Test p-value
Adverse Events	11 (0.2%)	3 (0.5%)	0.1
Product Safety and Quality	718 (11.6%)	83 (14.8%)	0.03
Customer Service Complaints	5,150 (82.8%)	449 (80.2%)	0.1

Product safety and quality

Thematic topics related to product quality and safety accounted for 11.8% (n = 801) of all low rated comments reviewed. When broken up into different retailer/storefront categories, licensed businesses made up 55.9% (n = 448) of these comments, individual retailers 33.7% (n = 270), and unlicensed stores 10.4% (n = 83). Sub-themes in this category included reporting of counterfeit merchandise, defective products, and expired products (see examples in Table 3). Overall, the proportion of comments reporting product safety and quality issues was significantly higher among unlicensed shops compared to licensed storefronts (p = 0.03). Product safety and quality posts that occurred in the pre-EVALI period totaled 376 (13.9 per month) compared to a higher number of these post in the post-EVALI period totaling 425 (26.6 per month).

Table 3

Examples of defective products and expired product from user comments.

Vendor Type N (%)	Theme	Posts N (%)a	Example Postb
Licensed Businesses n = 448 (55.9%)	Counterfeit Merchandise	185 (59.9%)	“THEIR PUFF BARS ARE FAKE!!!! I bought a box of Lush Ice Puff Bars here and I thought the weight and amount of puffs were off so I checked the authentication code online. Product came back as invalid. I was wondering why they gave me such a great deal. BE CAREFUL!.“
	Defective Product	231 (32.8%)	“I bought a vape from this location in which the worker who sold it to me didn't know shit about the damn mod and sold me a defective item. I've been back three days in a row for various problems . Didn't give me a receipt either so now it's gonna be a pain in the ass to get a refund so thanks a ton.”
	Expired Product	32 (4.0%)	“I was sold expired e-liquid on 8/11/19. I wasn't aware of this until I sampled and it tasted like a desiccated mushroom not “strawberry tutti-frutti.” Then I saw that it was “best by January 2018.” I know some people say that e-liquid doesn't “go bad” but it really does. Didn't ask for a receipt but I will next time at another store. If you are looking for a place on Sunday morning expect it to open 10–15 min late.“
Licensed Individual Retailers n = 270 (33.7%)	Counterfeit Merchandise	93 (30.1%)	“They sell bad products. Bought several puff bars from them and all of them were burnt. Guy working refused to replace them. Shady bad products and even worse customer service. FYI- They sell counterfeit Puff Bars and the owner knows but denies it. Box on the right is real one from different smoke shop and left one is from this store fake product.“
	Defective Product	162 (22.1%)	“Purchased 2 products from this place 2 days ago and both don't work. Called to do an exchange NOT A RETURN!!!! And I was told sorry read the front door next time.”
	Expired Product	15 (1.9%)	“The guy that works the ecig space is never there and anyone that is there never seems to know anything about the cost or merchandise. They barely understood or even spoke English. They're also selling expired ejuice. Not just a couple weeks but more like 3 + years expired. Very dangerous considering the ones there are with at least 18–24 mg nicotine.“
Unlicensed Stores n = 83 (10.4%)	Counterfeit Merchandise	31 (10.0%)	“They sell fake puff bars here for $16!!!!! When a real one will cost you $9.99.. the taste is so harsh you can tell it's fake . And they tell you to take it out of the packaging and try it to make sure it works . A real one would work right off the first hit. They then ask to throw your box away so you can't go on puffbar.com and verify if it's fake.“
	Defective Product	45 (6.5%)	“I buy vape 2 week ago now my vape leak and liquid come out side . I'm never go this shop again.“
	Expired Product	7 (0.9%)	“Dusty. Dirty. Dark. They have Cheap prices because their products are days from expiring. Save your money and go to the liquor store / market 1–3 blocks away.“

Number of posts and the percentage of total signal posts that contained the theme, b Yelp comments that discussed severity of counterfeit merchandise, defective product and expired product.

Counterfeit merchandise

A total of 38.6% (n = 309) of user comments reported suspected counterfeit merchandise sold by licensed businesses, individual retailers, and unlicensed stores. More than half (59.9%, n = 185) of these comments were from licensed business storefronts, with 30.1% (n = 93) from individual retailers, and 10.0% (n = 31) from unlicensed stores (Table 3). There was no significant difference in the proportion of comments reporting counterfeit merchandise among unlicensed shops and licensed storefronts. Discussion of counterfeit merchandise centered around users believing that product was fake after use and then writing negative reviews about products or store. This typically involved reporting malfunctioning and sensations like a burning taste attributed to a suspected counterfeit product. Users also reported verifying or failing to verify authenticity of a suspected product using its barcode or other identifying information on a manufacturers’ official website to confirm status. Reviews mentioned store owners refusing to accept returns for suspected counterfeit products and alleged that employees were aware of counterfeit products being sold but refused to acknowledge fraudulent sales when confronted. Counterfeit merchandise was reported co-occurrent with adverse events in some of the reviews (n = 6).

Defective and expired product

A large volume of user comments included discussion of defective products reported as broken, cracked, and otherwise deficient merchandise. Users reported problems with vaping products bought from specific stores that were discovered immediately after purchase or after a few uses. Licensed businesses had 32.8% (n = 263) of these user complaints, individual retailers had 22.1% (n = 177), and unlicensed stores had 6.5% (n = 52). A second sub-theme in this category included user comments reporting expired product in the form of stale cigars/cigarettes or expired e-liquid/e-juice sold to consumers. This coincided with users reporting that employees did not check inventory expiration frequently, leading to concerns about consumer safety issues. User-generated reports of expired products were roughly evenly distributed across store types (licensed business 4.0%, n = 32; individual retailers 1.9%, n = 15; and unlicensed stores 0.9%, n = 7).

Underage access and sales

A total of 21 comments were associated with underage sales of tobacco and/or vaping products. The majority of comments were complaints about individual proprietors with a state license (14 of 21, 66.7%). In comparison, licensed businesses accounted for 28.5% (n = 6) and unlicensed stores accounted only 4.8% (n = 1) of alleged underage selling activities. Underage access and sales posts totaled 7 (0.3 per month) during the pre-EVALI period and increased to a total of 14 (0.9 per month) during the post-EVALI period. Claims of underage selling included user accusations that mentioned a specific violation and offered a form of first-hand knowledge or alleged proof of violation, in addition to allegations that offered no specific proof but nevertheless claimed underage selling. Specifically, comments with discussion of specific violations were clear about types of activities involved including firsthand observation of product sales to minors, personal experiences and observations of stores not checking IDs, allowing minors in store, targeting youth and minors using social media posts, shops with a known history of selling tobacco products to minors, allowing minors to use product in store, and sale of other drug substances to minors (see Table 4).

Table 4

User-reported underage selling by licensure status.

	Licensed Businesses N (%)	Licensed Individual Retailers N (%)	Unlicensed Sores N (%)
No explicit proof but claim of underage selling	1 (16.7%)	4 (28.6%)	1 (100.0%)
Reporting of Specific Violations	5 (83.3%)	10 (71.4%)	–
Total	6	14	1
Reporting Underage Selling with specific violations
Not Checking ID	2 (40.0%)	1 (7.7%)	–
Allowing Minors in the store	1 (20.0%)	1 (7.7%)	–
Targeting minors in social media post	–	1 (7.7%)	–
Neighborhood stores known for previous underage sales transactions	–	1 (7.7%)	–
Sales in front of minors or minors using product in store	1 (20.0%)	4 (30.8%)	–
Sale of other drug substances to minors	1 (20.0%)	5 (38.5%)	–
Total	5	13	–

Adverse events

A small volume (0.2%, n = 14) of user-generated comments, originating from 14 different store fronts, reported an adverse event, which were then classified into three distinct sub-categories. Adverse event posts that occurred during the pre-EVALI period before the earliest reported EVALI cases totaled 6 posts (0.2 per month) and slightly increased to a total of 8 (0.5 per month) in the post-EVALI period. All sub-categories consisted of either first or secondhand experiences with adverse events. The first group included users discussing hospitalization after using ENDS or its components (e.g., e-juice). A second group included users discussing personal accounts of emergency room admission or doctor visits as a result of ENDS use. A third group of characteristics included users reporting symptoms associated with possible adverse events or lung injury, such as coughing, chest pain, shortness of breath, abdominal pain, nausea, vomiting, diarrhea, fever, chills, and/or weight loss (see examples in Supplementary File Table 1).

Licensed business storefronts generated 4 adverse event comments (1 in pre-EVALI period (0.04 per month) and 3 in post-EVALI period (0.2 per month)), all reporting various symptoms ranging from persistent cough, chest pain, vomiting, being sick for a few days, and self-reporting of hospitalization after using expired or counterfeit merchandise. Seven adverse event comments (0.1 per month in pre-EVALI period and 0.3 per month in post-EVALI period) were associated with individual retailer storefronts and included reports of coughing, nausea, headache, hospitalization, and purported death of a known individual due to products sold. Unlicensed store comments included 3 adverse event posts (0.07 per month in pre-EVALI period and 0.06 per month in post-EVALI period) associated with symptoms which included nausea, brief sickness, sore throat, and severe sickness. There was no significant difference in the proportion of comments reporting adverse events among unlicensed and licensed storefronts.

Discussion

To our knowledge, this is the first study to detect and report on possible state licensure violations of tobacco and vaping storefronts by collecting and cross-referencing state database information and publicly available data from different data sources including social media. Our study found that 96.9% of stores reviewed matched with tobacco retailer license information provided by CDTFA, which is slightly higher than the 2017 CA State Coverage Study reporting 95.8% compliance to licensure requirements by conducting ground canvasing and retail audits per federal Synar reporting that assesses compliance to underage selling requirements (Franklin and Gretchen, n.d.). This study also found that users from different retailer categories and licensure status reported experiences with counterfeit and defective merchandise sales, along with a smaller number of user reports describing adverse events and underage selling activities as have been documented in other studies (Roeseler et al., 2019). Importantly, these results may assist state regulators in identifying retailers that may not be following legal guidelines or licensure requirements and who may pose heightened risk for consumer safety concerns (Sussman et al., 2014).

Publicly available Yelp user-generated reviews are a potential tool for public health surveillance that can aid further investigation and possible enforcement action, particularly in the event of a cluster of safety events by identifying the “who” (e.g., store), “what” (type of product) and “where” (location of store) of these events and experiences. User-generated reviews from crowdsourcing platforms provide important insights into firsthand and secondhand consumer experiences that may not be captured in other public health surveillance approaches (e.g., surveys, focus groups, etc.) and can be analyzed closer to real-time to identify potential safety events as they arise. Reviews reporting counterfeits can also provide information about unauthorized ENDS products circulating in retail settings, and underage selling can inform policymakers about strategies used to violate laws that mandate protection of underage populations.

The aggregated results of this study will be shared with the CDTFA in order to inform state-wide licensure compliance efforts and could also enable characterization of unlicensed stores operating in California but that remain available to consumers via Yelp. The study also identified stores with multiple violation reports that may warrant triage and prioritization in state enforcement. Unlicensed stores detected likely violate the California TRL requirements and guidelines and are subject to California Business and Professions Code Sections 22971-22971.1, 22972-22973.3, 22980.2, 22981, 22990.7. Specifically, penalties for operating without a valid license, suspended or revoked license, for individuals and companies, can include a misdemeanor punishable by a fine of no more than $5,000, imprisonment not exceeding one year in county jail, or both fine and imprisonment. If sales of tobacco products continue after violation, tobacco products can be seized by the board, or by a law enforcement agency and are deemed forfeited. Businesses that are unlicensed also have a financial impact on the state of their residence, by possibly failing to pay proper taxes at the federal, state, and county level.

Reports of counterfeit products could also help with broader federal efforts to ensure appropriate oversight of new ENDS products introduced into the marketplace. In 2016, the U.S. Food and Drug Administration (FDA) “Deeming Rule,” recognized e-cigarettes as tobacco products, which gave the FDA sole authority over ENDS product standards, packaging, and labeling requirements. For products not on the market as of February 2007, the FDA considers them new tobacco products that must be authorized by FDA. Hence, user generated reports of counterfeit or unapproved ENDS products could form the basis for FDA investigation of unauthorized manufacturers, illegal importation, or other illicit sources that may pose unique patient safety risks. Investigation and enforcement against unapproved ENDS would also align with existing regulatory actions taken by FDA, which issued its premarket authorization review policy in January 2020, including information about enforcement actions against the manufacture, distribution, and sale of unauthorized flavored cartridge -based e-cigarettes (FDA, n.d.).

Importantly, previous studies have shown that jurisdictions with stronger local TRL ordinances can lower the odds of ever cigarette initiation and use, that implementation of retail permitting can lead to better underage sales prohibition compliance, and that presence of ENDS licensing requirements/policy can reduce use among adolescents (Astor et al., 2019, Azagba et al., 2020, Coxe et al., 2014). However, enforcement against unlicensed stores in the digital environment where we detected Yelp comments associated with underage selling, and counterfeit merchandise, are likely in isolation not sufficient to constitute an actionable legal violation. Further investigation should confirm violations of applicable state and/or federal laws (e.g., the state Stop Tobacco Access to Kids Enforcement (STAKE) Act which seeks to enforce a statewide enforcement program to decrease tobacco sales to youth) (Landrine et al., 2000). Results from our study support the need for further research to assess the impact of state and local TRLs in the context of potential adverse events, counterfeit product presence, and underage selling that may be reported in the digital environment, while also assessing questions about enforcement and policy implementation.

Limitations

Our categorization of an “unlicensed” store may be subject to certain limitations. This includes that we only included licensure data and Yelp user comments from January 1, 2017, and onward (when ENDS retailers became subject to CA tobacco retailer licensure requirements) and any storefront in operation before 2017 not found on the CDTFA 2017–2020 list was not included in this study. It is also unclear how this change in licensure requirements extended to ENDS retailers may have been adopted by businesses and Yelp-related retailers. Further, the Cigarette and Tobacco Product Licensing Act defines a retailer as someone who sells tobacco products directly to the public from a California retail location but does not clarify if licensure is required for an online-only retailer that sells into California or otherwise seeks to sell directly to Californian consumers. Our study was limited to examining potentially unlicensed storefronts on Yelp that did not match CDTFA listings but reported a business address, though the presence of online-only storefronts reporting a general business address is a possibility. More research is needed to assess the potential presence of unlicensed and unregulated online tobacco and ENDS retailers that operate within and across multiple states and countries. Additionally, a number of CDTFA licensed entities did not match on Yelp when filtering business listing for those specifically categorized as tobacco, vape, or head stores by the platform. As previously mentioned, many of these license holders were non-tobacco specific retailers, though other tobacco/vape-specific shops may have had their licenses expire/lapse or may be classified as retailers of other products or business types on Yelp. Additional limitations are detailed in the Supplementary File.

Conclusion

This study represents an innovative approach to augment state licensure databases with other surveillance to detect unlicensed tobacco and ENDS retailers while enabling enhanced consumer product safety monitoring. Future studies should validate these approaches and work with state public health officials to carry out enforcement of tobacco control and TRL laws, particularly in relation to reporting adverse health events, prohibiting underage selling, and preventing counterfeit and defective product sales. Additionally, new user comments generated on platforms can be used to assess changing attitudes and user experiences that may be associated with business characteristics, TRL status, and the broader tobacco control policy environment. Tobacco regulatory science can be augmented by digital tobacco retail surveillance, to detect public safety issues and also inform data-driven approaches to future tobacco and ENDS control policies.

Funding

This study was funded by the University of California Tobacco-related Disease Research Program award no. T29IP0384 and T31IP1928.

Disclaimer

The opinions expressed are those of the authors alone.

Data availability statement

Data are available on the CDTFA and Yelp business website. Individual licensed data must be requested directly to CDTFA.

CRediT authorship contribution statement

Matthew C. Nali: Conceptualization, Methodology, Writing – original draft. Vidya Purushothaman: Methodology, Formal analysis, Writing – review & editing. Jiawei Li: Data curation, Software. Tim K. Mackey: Conceptualization, Writing – review & editing, Supervision, Funding acquisition.

Declaration of Competing Interest

MN, JL, and TKM are employees of the startup company S-3 Research LLC. S-3 Research is a startup funded and currently supported by the National Institutes of Health – National Institute of Drug Abuse through a Small Business Innovation and Research contract for opioid-related social media research and technology commercialization. The remaining author declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.