Response Rates of Medical Providers to Internet Surveys Regarding Their Adoption of Preexposure Prophylaxis for HIV: Methodological Implications.

In 2016 to 2017, we surveyed primary care providers (PCPs) in upper Manhattan and the South Bronx, New York, on their knowledge, attitudes, and practices surrounding preexposure prophylaxis (PrEP) for HIV. Despite efforts to promote survey response, we were only able to obtain a meager response rate, limiting our ability to interpret results. In this short communication, we examine our survey's methodology, as well as the methods used by other similar studies, in order to suggest how certain strategies appear to influence PCP response to PrEP surveys. Administering the survey in a variety of modes, sampling from a professional organization's listserv, promoting the survey topic's relevance to potential participants, and offering monetary incentives to each survey respondent all appear to be promising strategies for increasing response rates in PrEP provider surveys.

Introduction

Preexposure prophylaxis (PrEP) has been demonstrated to be effective in reducing HIV incidence among at-risk individuals and is now an important addition to the HIV prevention toolbox.[1] As gatekeepers for biomedical interventions, health-care providers play an essential role in the scale-up of PrEP for individuals at risk for HIV infection.[2,3] Understanding how medical providers view and prescribe PrEP is a first step in informing future scale-up efforts. For this reason, many researchers have recently surveyed providers on their knowledge, attitudes, and practices surrounding PrEP (Table 1). Some attempts at surveying providers about PrEP have resulted in low response rates (RRs).[16-18] Because low response yield may result in a sample that does not adequately represent a study’s sampling frame,[20] a low RR can limit the utility of the survey’s findings. In 2016 to 2017, we surveyed primary care providers (PCPs) in upper Manhattan and the South Bronx to assess knowledge, attitudes, and practices regarding PrEP services. We obtained a poor RR, and we were thus unable to interpret the study’s results. In this brief report, we consider our survey methodology in the context of similar, recently published surveys in order to provide guidance to other researchers conducting Internet-based surveys of providers regarding PrEP adoption.

Table 1.

Characteristics of Online Physician Surveys on Pre-exposure Prophylaxis.

Authors	Estimated RR	RR Notes	Sample	Survey Mode	Recruitment	Incentive	Reminders
Excellent RR (>50%)
Seidman et al, 2016[4]	62%a (580/948)	This is authors’ estimate of RR due to e-mail recruitment, denominator = No. of opened e-mails. 24 additional respondents had incomplete forms. There was an average of 1343 web site visits per week (No. of repeat visits unknown). 188 individuals participated in meeting (amount of overlap between recruitment modes unknown)	Family planning providers in the United States, particularly members of Title X Clinics	Online	Web site link, listserv e-mail encouraging providers to forward the survey, flyers distributed at organization’s national meeting	None mentioned	None mentioned
Castel et al, 2015[5]	61.5% (142/231)	Authors’ report: No. of respondents /No. of solicitations.	HIV providers in Miami and DC	Online, paper and pencil	E-mail, mail	All Rs received USD20 incentive	Periodic mail, telephone, and e-mail reminders (intervals not specified)
Smith et al, 2016[6]	MDs = 59.8% (Average) NPs = 53.7% (Average)	Average RR for each group over the 6 years. Authors’ report: No. of completions/No. of eligible participants	National opt-in panel of (mostly) PCPs	Annual online survey 2009-2015	Opt-in panel	None mentioned	None mentioned
Good RR (30%-50%)
Karris et al, 2013[7]	48.8% (573/1175)	Authors’ report: No. of respondents /No. of members on listserv (ie, No. recruited)	Adult ID MDs who are members of EIN	Online	Listserv e-mail	None mentioned	2 at 1-week intervals (mode not specified)
Krakower et al, 2016[8]	48.1% (573/1191)	Authors’ report: No. of respondents/No. of members on listserv (ie, No. recruited). Denominator restricted to active members who had previously participated in at least one EIN survey	Adult ID MDs who are members of EIN	Online	Listserv e-mail and fax	No incentive	2 e-mails at 1-week intervals
Walsh et al, 2017[9]	30% (627/2088)	Authors’ report: No. of individuals who entered the study web site/No. invited. Of these, 525 met eligibility criteria, provided consent, and took the survey (25.1% of those invited). Analytic sample consisted of the 280 PCP respondents (13.4% of those invited)	PCPs from zip codes with high HIV incidence in 10 US cities	Online	Postal mail and some e-mail, from 3 professional organizations	All Rs received USD100	Three reminders (intervals and mode not specified)
Fair RR (10%-29%)
White et al, 2012[10]/Mimiaga et al, 2014[11]	18.4% (115/625)	Same survey (post-iPrEx in White). Authors’ report (Mimiaga) is 18.4% RR and 115 respondents; no mention of denominator	(Mostly) HIV specialists practicing in Massachusetts	Prepost Online	Listserv e-mail and direct e-mail recruitment	All Rs received USD25 gift card	None mentioned
Bacon et al, 2017[12]	14.4% (99/686)	Authors’ report: No. of completions/No. of those invited.	PCPs and HIV specialists in San Francisco	Online	E-mail via San Francisco Bay Area Collaborative Research Network	All Rs received USD30 gift card	Three e-mail reminders (intervals not specified)
Adams et al, 2016[13]	14% (363 respondents)	Authors’ estimate: No. of completions/No. of those e-mailed that were also eligible based upon prescription privileges. Total number recruited was 3484. After further eligibility exclusions, total analytical sample consisted of 260 completions.	Members of the Am Acad HIV Med	Online	E-mail	No incentive	None mentioned
Krakower et al, 2015[14]	11.2% (184/1637)	Authors’ report: No. of completions/No. of those invited. In total, 207 individuals began the survey.	Attendees of NE AETC event	Online	E-mail	All Rs received USD25 gift card	Up to 4 telephone calls (intervals not specified)
Tellalian et al, 2013[15]	10.5% (189/1800)	Our calculation: No. of respondents/No. of surveys distributed	Members of the Am Acad HIV Med	Online	E-mail	None mentioned	None mentioned
Poor RR (<10%)
Edelman et al, 2017[16]/Blackstock et al, 2017[17]	9% (250 responses)/8.6% (266/3093)	Authors’ estimate from 250 responses. The SGIM had approximately 3093 physicians at the time of the study. 363 providers initiated the survey, but 113 were excluded from analysis due to ineligibility or incomplete responses./Authors’ estimate: No. of completed surveys/No. of those invited. A total of 363 surveys were initiated, but 97 were excluded due to ineligibility or incomplete responses.	Members of the SGIM	Online	Materials disseminated during national annual meeting, e-mails sent through online community forum, direct e-mailing/Listserv e-mail, direct email messaging	Raffle for 2 iPads	Five follow-up reminders sent every week through online community forum/E-mailed weekly for 6 weeks
Silverman, personal communication (survey attempted by these authors, 2017)	5.4% (16/294)	Authors’ calculation: No. of completed surveys/No. of invitees. No respondents were ineligible and no responses were incomplete.	Medicaid Providers (MDs & NPs) in HMO serving Upper Manhattan	Online	Mailed letters	Lottery of 3 USD50 gift cards	Two monthly postcards, e-mails (when available), phone calls
Sachdev et al, 2014[18]	9.7% (146/1508)b	Authors’ calculation: No. of completed surveys/No. of eligible invitees who opened at least one e-mail. E-mails were sent to a total of 5672 physicians. Of those who opened at least 1 e-mail, 37 were deemed ineligible.	MDs in 13 metro areas with highest HIV incidence, from AMA MD master file (84% of respondents were PCPs)	Online	E-mail	None mentioned	Three reminders over 4 weeks (mode not specified)
RR not reported
Sharma et al, 2014[19]	Not provided	The sampling strategy did not allow for RR estimations. A total of 104 responses were received, but 18 were excluded from analysis due to missing data for the primary outcome. Authors refer to sample size as “small.”	Canadian family, ID, internal medicine, and public health MDs	Online	E-mail to organization listservs	Lottery for 1 iPad or 5 USD20 gift cards	None noted

Abbreviations: Am Acad HIV Med, American Academy of HIV Medicine; AMA, American Medical Association; EIN, Emerging Infections Network; HMO, Health Maintenance Organization; ID, infectious diseases; MD, medical doctor; NE AETC, Northeast AIDS Education & Training Center; NP, nurse-practitioner; PCP, primary care physician; RR, response rate; Rs, respondents; SGIM, Society of General Internal Medicine.

a Authors note this estimate is likely inflated due to the inability to calculate denominator from the web site link and snowball referrals.

b Authors’ term: “Cooperation rate.”

Our Survey Methods

We designed a brief survey to examine PCPs’ adoption of PrEP into their practices. The survey assessed providers’ knowledge, attitudes, and clinical experience with PrEP and asked them to respond to hypothetical patient scenarios. We pilot tested the survey and found it took approximately 10 minutes to complete.

Our objective was to construct a sampling frame inclusive of providers most likely to encounter individuals who might not be aware of being at substantial risk for HIV but who would benefit from taking PrEP. We therefore approached 294 primary care medical providers affiliated with a Health Maintenance Organization (HMO) that accepts Medicaid and who serve areas of New York City with high HIV incidence, upper Manhattan and the South Bronx. Infectious disease specialists were excluded because they are presumed to have specialized access to training and resources about PrEP.

Because we were unable to obtain e-mail addresses from the online HMO database, recruitment letters were mailed to the providers’ offices. The recruitment letters included information about the survey and detailed measures taken to safeguard respondents’ confidentiality; an easy URL that linked to the survey; and description of incentives, that is, all respondents were entered into a lottery for three USD50 gift cards. Letters were printed using official letterhead and included personalized addresses and salutations, blue-ink signatures, stamped postage, and other methods to improve RRs.[21] Two reminder postcards were sent in follow-up 1 month apart in addition to periodic telephone reminders when phone numbers were available. The Columbia University institutional review board approved the survey (IRB-AAAO0852); e-consent was obtained.

Results and Discussion

Our RR was strikingly low. While few of the recruitment mailings were returned as undeliverable, only 16 providers responded to the survey after reminder mailings and calls, yielding a RR of 5.5%. To understand this low RR, we assessed our survey instrument and recruitment methods, drawing on published literature to inform our assessment.

Our questionnaire appeared to conform with criteria for appropriate and sound measurement tools: none of the respondents dropped out or had missing data, the average survey completion time was 13 minutes, the items elicited good variability in response options, and the skip and display logic worked appropriately. Additionally, no respondents reported difficulties or negative experiences in taking the survey.

Given the apparent soundness of our instrument, we examined other factors that may affect RRs.[20,21] In August 2017, we searched PubMed, Medline, and Google Scholar for recently published articles that report surveying health-care providers via the Internet on their knowledge, attitudes, and/or practices regarding PrEP in order to explore how methods used by these surveys may have affected their RRs. All articles that met these criteria and provided sufficient description of survey methodology were included. Table 1 shows the target sample size, survey administration and recruitment modes, use of incentives, and final RRs of reviewed surveys. In cases where study authors did not report RRs, they were calculated by dividing the number of respondents by the number of providers invited to participate.

While no conclusive best practices can be drawn, some strategies appeared to be more effective in eliciting high RRs than others. In general, using samples assembled from professional organizations’ membership lists were effective, especially if these organizations were smaller in size.[4,7,8,10] The one study that recruited members from a panel of individuals who have actively agreed to participate in surveys obtained an especially high RR (59.8%).[6] Such a strategy, which aims to recruit respondents who have already indicated interest in survey participation, has been suggested to be a more effective recruitment method than those that advertise a survey to a group of individuals whose interests are unknown. However, use of such a methodology may result in a less diverse and possibly less representative sample.[22]

While all surveys were administered online, the approach by Castel et al[5] to also include a paper-and-pencil option appeared to be effective (RR = 61.5%). While most studies recruited participants by e-mailing a professional organization’s listserv, those that mixed recruitment modes appear to be most effective (mean RR = 38.13%).[4,5,8-11,16,17] Additionally, while the 2 studies that included mailed letters in their larger recruitment plans[5,9] achieved strong RRs (61% and 30%), the only study that exclusively utilized a mailed letter recruitment strategy for all potential participants was our survey, which achieved a poor RR (5.4%); we were only able to follow-up by e-mail with about one quarter of our target list of providers. This suggests that mailed letters may help bolster e-mail-based recruitment efforts, but on their own may be an ineffective strategy for achieving an adequate RR. These findings are concordant with reviews suggesting mixed-mode recruitment and administration may be beneficial for increasing RRs.[23-25]

While reminders were not always needed to produce high RRs, 2 to 3 reminders at 1-week intervals appear to be the most effective way to increase the RR (RR = 48.8% and 30%).[7,9] As with distribution and recruitment, the most effective specified reminder strategy involved mixing physical and electronic modes (RR = 61.5%).[5]

Incentives were not always necessary in order to obtain a high RR. Providing universal incentives to all those who complete the survey[5,9,10-12,14] appears to be more effective than lotteries,[16,17,19] although we were not able to conduct a test for statistical significance due to low sample size and limited RR reporting. The studies that offered cash and gift card incentives[5,9-12] appear to be more effective than those that offered lotteries for iPads,[16,17,19] but we had similar limitations in assessing whether this difference is significant. These findings are also consistent with past research indicating that monetary incentives can improve RRs in surveys of physicians.[26-28] Other evidence suggests that unconditional incentives sent with the questionnaire may be best at promoting response even if they are small, as the respondent may feel obliged to reciprocate the gift.[21,29]

Perhaps the most important factor to address when advertising a survey is stressing its relevance to the potential respondents. Whether or not a respondent perceives a survey’s subject matter to be personally relevant is one of the most critical factors determining his or her participation.[30] Indeed, a few of the PCPs who were asked to participate in our study replied that they have never prescribed PrEP and/or that their practice has nothing to do with PrEP; therefore, they declined to be surveyed. Low RRs precipitated by low perceived relevance would be particularly concerning because these surveys aim to measure whether such attitudes and practices exist among providers. Not capturing a lack of adoption could result in an overestimation of the amount of providers offering PrEP and an underestimation of the need for PrEP-related medical training and dissemination efforts.

Interestingly, the low response we received in our survey of PCPs may relate to the “purview paradox” of PrEP[31,32]—HIV specialists are comfortable prescribing antiretroviral medication but are unlikely to see HIV-negative patients; providers who care for people at substantial risk for HIV are unfamiliar with antiretroviral medication and may overestimate the difficulties in prescribing it as PrEP. This paradox may lead PCPs to underestimate their potential role in making PrEP more accessible. The low RR in our study may suggest that this paradox is present in our response catchment area. On the other hand, it may simply be indicative of “survey fatigue” due to the growing number of surveys made possible by the ease of Internet-based survey administration. We found no significant difference between studies that surveyed infectious disease and HIV specialists[5,7,8,10,11,13-15] or providers who serve areas of high HIV incidence[9,18] and those surveying general practitioners[4,6,16,17] in achieving RRs above or below 14% (Fisher exact test P >.05).

It is also likely that publication bias significantly limits the survey efforts we were able to review. Many surveys that do not obtain RRs that are sufficient for representing their catchment area or powering their analyses are not published; therefore, it is difficult to conclusively determine which methods are effective and which are not. Nevertheless, due to the critical need for surveys of the PrEP attitudes of care providers, this review highlights the methods that have tended to produce higher RRs than others for the benefit of this future research.

Conclusion

Our survey experience illustrates how difficult it can be to effectively survey PCPs regarding their adoption of PrEP. Even when the researchers follow evidence-based administration and recruitment strategies, survey nonresponse can be an insurmountable limitation to this type of research. Nevertheless, this type of research is fundamental in assessing how PrEP is being used and how to improve its use, and some of the surveys reviewed achieved adequate response in spite of the challenges of survey research. We hope that this report can lend important information to researchers when planning surveys of providers on PrEP and similar topics. Our survey attempt and our review of other recent physician PrEP surveys suggests that when asking providers about their knowledge, attitudes, and practices surrounding PrEP, it may be prudent to utilize a mixed-mode administration strategy, utilize a professional organization’s listserv if possible, develop strategies to promote relevance, and offer a monetary incentive to all participants in order to achieve high RRs.