Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial.

Background: Postal questionnaires are frequently used in randomised controlled trials to collect outcome data on participants; however, poor response can introduce bias, affect generalisability and validity, and reduce statistical power. The objective of this study was to assess whether a pen and/or social incentive text cover letter sent with a postal follow-up questionnaire increased response rates in a trial. Method: A two-by-two factorial randomised controlled trial was embedded within the OTIS host trial. Participants due their 12-month (final) follow-up questionnaire were randomised to be sent: a pen; a social incentive text cover letter; both; or neither. The primary outcome measure was the proportion of participants in each group who returned the questionnaire. Secondary outcomes were: time to return, completeness of the questionnaire, necessity of a reminder letter, and the cost effectiveness.
Results: The overall 12-month questionnaire response rate was 721 out of 755 (95.5%). Neither the pen nor social incentive cover letter had a statistically significant effect on response rate: pen 95.2% vs. no pen 95.8%, adjusted OR 0.90 (95% CI 0.45 to 1.80; p=0.77); social incentive cover letter 95.2% vs. no social incentive cover letter 95.8%, adjusted OR 0.84 (95% CI 0.42 to 1.69, p=0.63). No statistically significant differences were observed between either of the intervention groups on time to response, need for a reminder or completeness. Therefore, neither intervention was cost-effective. Conclusions: We found no evidence of a difference in response rates associated with the inclusion of a pen and/or social incentive cover letter with the final follow-up postal questionnaire of the host trial. However, when these results are combined with previous SWATs, the meta-analysis evidence remains that including a pen increases response rates. The social incentive cover letter warrants further investigation to determine effectiveness. Trial registration: ISRCTN22202133 (21st June 2020). Copyright:

Introduction

Randomised controlled trials (RCTs) are the gold standard to assess effectiveness of treatment options and to inform care decisions , yet only a few hundred studies exist to assess the effectiveness of different methods to improve retention or recruitment into RCTs .

Trial methodologists and funders have highlighted the need to evaluate participant recruitment and retention strategies in order to provide evidence on which to base decisions around the design and conduct of RCTs .

Postal questionnaires are frequently used in randomised controlled trials to collect outcome data on participants; however, poor response can introduce bias, affect generalisability and validity, and reduce statistical power. Several systematic reviews report on the topic of retention strategies, including improving response rates to questionnaires . However, there remains a lack of definitive evidence regarding some commonly adopted practices such as sending a pen or using a cover letter with a questionnaire to encourage the participant to return it . The results of a study within a trial (SWAT) evaluating these two strategies are reported here.

Methods

Design

A two-by-two factorial RCT was embedded within the OTIS trial of occupational therapist-led home assessment and modification for the prevention of falls ( ISRCTN22202133) . OTIS recruited participants over the age of 65 years who were at risk of falling. Participants were randomised to receive an occupational therapist delivered visit or usual care. They were followed up for 12 months for falls data and were sent postal questionnaires at four, eight and 12 months. This SWAT was embedded at the 12-month time point. Ethical approval for this SWAT was received from the NHS West of Scotland Research Ethics Committee 3 (16/WS/0154) and Health Research Authority and Research Ethics approval in July 2018. Approvals were obtained from the University of York, Department of Health Sciences Research Governance Committee. Participants provided informed consent to be enrolled into the OTIS trial and to be sent study related information by post. Consent for the SWAT was therefore waived by the above-named ethics committee.

Participants

A total of 779 participants due to receive their 12-month questionnaire between 16 th October 2018 and 2 nd August 2019 were randomised into the SWAT in a single tranche in September 2018. Participants who had withdrawn from the OTIS study prior to this were excluded from randomisation.

The allocation sequence was generated by the OTIS statistician, who was not involved with the sending of the questionnaires, using STATA v15 . The identification numbers of OTIS participants to be involved in the SWAT were randomised 1:1:1:1 in a single block. Because there were no descriptive details of the participants attached to the identification numbers this meant the randomisation was concealed.

Interventions

Table 1 details the combination of interventions sent in the post with the 12-month questionnaire. We included an unconditional £5 note with the questionnaire for all participants.

Table 1.

Intervention groups.

Pen York Trials Unit branded pen, standard cover letter (Supplementary File 1) *	Control Group No pen, standard cover letter (Supplementary File 4).
Pen and Social Incentive Cover Letter York Trials Unit branded pen, social incentive cover letter (Supplementary File 3).	Social Incentive cover letter Social incentive cover letter ( Supplementary File 2), no pen.

*Supplementary Files are available as Extended data .

The non-standard cover letter offered a mild level of social incentive, in the form of a personalised table that indicated whether or not a questionnaire had been received from the participant at the earlier (4 and 8-month) time points. The concept of social incentive that underpinned the intervention for this study was that a social incentive is something that persuades people to behave in a certain way by the promise that their actions will be noticed or made public . Therefore, the cover letter was intended to highlight to the participant that their questionnaire responses are noted and valued .

Blinding and quality assurance

Participants were blind to their participation. Research administrators and research team members posting the questionnaire packs were not blind to the intervention; however, administrators who recorded the outcome data were blind to allocation.

Primary objective

To assess whether a pen and/or social incentive text cover letter sent with the 12-month questionnaire increased postal questionnaire response rates for participants in the OTIS trial.

Primary outcome

The primary outcome was response rate, defined as the proportion of participants in each group who returned the 12-month questionnaire.

Time to return 12-month questionnaire

The completeness of the 12-month questionnaire

The requirement for a reminder letter to be sent

Cost effectiveness

Statistical analysis

The data were analysed in SPSS v25 using two-sided tests at the 5% significance level on an intention-to-treat basis. Participants who withdrew or died before the 12-month questionnaire was sent were excluded from the analysis. The primary outcome was compared using a logistic regression model adjusting for age (retention is generally higher in participants ˂75 years and older adults may respond differently to incentives ), gender (to control for potential differences in anticipation of social incentives between males and females ) and host trial treatment allocation. The presence of an interaction between the two interventions was tested by introducing the interaction term into the logisit model. Time to questionnaire return (calculated as days from questionnaire sent to return) was analysed using Cox Proportional Hazards regression, adjusting for the same covariates as in the primary analysis. The proportional hazards assumption was assessed using Schoenfeld residuals . Completeness of response (defined as number of items completed) was analysed by linear regression model and adjusted as for the primary analysis.

Cost effectiveness was calculated for each group using the total cost of the pen/letter/postage/stationary and staff time.

Due to SWATs typically being under-powered to show small effects, it is essential that the results are seen within the context of the wider literature. A fixed effect meta-analysis using the Mantel-Haenszel method was conducted using review manager v5.3 to pool the results of this study for enclosing a pen with the 12-month questionnaire with other RCT evidence. These were located utilising the Cochrane systematic review search strategy (Supplementary file 14) in MEDLINE and EMBASE, along with hand searching of previous systematic reviews references, published retention research reference lists, conference papers and co-author personal knowledge of studies. The results of this study were pooled with four previous SWATs investigating the same intervention, with the same dichotomous outcome of response to the questionnaire or not. Pooled odds ratios and corresponding 95% CIs were calculated. Heterogeneity between trials was assessed using the Chi-squared and I 2 statistics. The meta-analysis was facilitated by the PROMoting THE USE of SWATs (PROMETHEUS) programme, which supports host trial teams to conducted SWATs and for data obtained to be collated and meta-analysed.

A meta-analysis of the results of the social incentive intervention was not undertaken as the only previous study using this was conducted within a cohort study rather than an RCT .

Results

Figure 1 depicts the recruitment and retention of participants in the embedded trial. Table 2 presents summary statistics for the baseline characteristics of the SWAT participants.

Figure 1.

Flow diagram depicting the recruitment and retention of participants in this embedded trial.

Table 2.

Baseline characteristics of the SWAT participants.

	Pen only (n=192)	Pen and social incentive cover letter (n=184)	Social incentive cover letter only (n=189)	Standard 12- month cover letter (control) (n=190)
Age
n	192	184	189	190
Mean (SD)	80 (6.3)	80 (6.1)	79 (6.2)	80 (6.2)
Min, Max	67, 98	66, 98	65, 98	69, 94
Gender
n	192	184	189	190
Male	73 (38.0%)	56 (30.4%)	59 (31.2%)	69 (36.3%)
Female	119 (62.0%)	128 (69.6%)	130 (68.8)	121 (63.7%)
Host trial randomisation
n	192	184	189	190
OT visit (intervention)	61 (31.8%)	49 (26.6%)	59 (31.2%)	65 (34.2%)
GP standard care	131 (68.2%)	135 (73.4%)	130 (68.8%)	125 (65.8%)
Number of falls in 12 months prior to randomisation
n	145	139	149	135
Mean	2.2 (3.0)	1.8 (1.4)	2.0 (1.7)	2.2 (2.1)
Min, Max	1, 21	1, 11	1, 10	1, 15

#= How good or bad your health is today rated from 0 worst, 100 best.

Between randomisation into the SWAT and being sent their 12-month questionnaire, 24 randomised participants either died or withdrew from the host trial and so were not sent the questionnaire. A total of 721/755 (95.5%) returned the 12-month questionnaire. The response rate was identical in the pen only group (184/192, 95.8%), social incentive cover letter only group (181/189, 95.8%) and control group (182/190, 95.8%). However, it was marginally lower in the pen and social incentive cover letter group (174/184, 94.6%).

No evidence of a difference in response rates was found between participants with or without pens (pen: 358/376 [95.2%]; no pen: 363/379 [95.8%]; adjusted OR 0.90, 95% CI 0.45 to 1.80, p=0.77) nor with or without the social incentive cover letter (cover letter: 355/373 [95.2%]; no cover letter: 366/382 [95.8%]; adjusted OR 0.84, 95% CI 0.42 to 1.69, p=0.63) ( Table 3).

Table 3.

Primary outcome results.

Primary outcome	Group	Hazard ratio (HR)/ Odds ratio (OR)/Mean difference (MD)	95% Confidence Interval	p-value	Other
Response rate	Pen received vs. not received	OR = 0.90	0.45, 1.80	0.77	Total of 721/755 (95.5%) returned tde 12-month questionnaire
	Social incentive cover letter received vs. not received	OR = 0.84	0.42, 1.69	0.29
	Host trial allocation (intervention vs. control)	OR = 1.40	0.64, 3.23	0.38
	Age (per year)	OR = 0.96	0.91, 1.01	0.11
	Gender (male vs. female)	OR = 0.71	0.35, 1.44	0.35

The interaction between the interventions was found to be non-significant (interaction effect size estimate OR 0.79 95% CI 0.20, 3.15 p = 0.74).

Secondary outcomes

. Median time to return the questionnaire was nine days, with a mean of 12.2 days. No statistically significant difference between the groups was found ( Table 4).

Table 4.

Secondary outcome results.

Secondary outcome	Group	Hazard ratio (HR)/ Odds ratio (OR)/Mean difference (MD)	95% Confidence Interval	p-value	Other
Time to return	Pen received vs. not received	HR = 1.08	0.93, 1.25	0.30	Mean time for all participants to return questionnaire = 12.2 days. Median time for all participants to return questionnaire = 9 days.
	Social incentive cover letter received vs. not received	HR =1.101	0.87, 1.17	0.92
	Host trial allocation (intervention vs. control)	HR = 0.85	0.73, 1.00	0.05
	Age (per year)	HR = 0.99	0.97, 1.00	0.02
	Gender (male vs. female)	HR = 1.80	0.92, 1.26	0.35
Reminders sent	Pen received vs. not received	OR = 0.89	0.56, 1.42	0.63	83/755 (11.0%) required a reminder p value associated with the Kruskal- Wallis test statistic p=0.190
	Social incentive cover letter received vs. not received	OR = 0.92	0.58, 1.47	0.74
	Host trial allocation (intervention vs. control)	OR = 1.611	1.00, 2.59	0.05
	Age (per year)	OR = 1.04	1.00, 1.08	0.03
	Gender (male vs. female)	OR = 0.87	0.53, 1.42	0.57
Completeness of response	Pen received vs. not received	MD = 0.14	-0.46, 0.74	0.65	Overall average completeness of the questionnaires was 27.8/31 questions (89.6% complete)
	Social incentive cover letter received vs. not received	MD = 0.09	-0.69, 0.51	0.78
	Host trial allocation (intervention vs. control)	MD = -0.10	-0.55, 0.75	0.77
	Age (per year)	MD = -0.10	-0.46, 0.74	0.65
	Gender (male vs. female)	MD = -1.06	-1.69, -0.42	˂0.001

. In total, 83/755 (11.0%) participants required a reminder letter. The pen and social incentive cover letter group required the least reminders (19/184 10.3%) and the control group required the most reminders (24/190 12.6%). No statistically significant evidence was found of a difference of participants requiring a reminder between the groups ( Table 4).

. Overall average completeness of the questionnaires was 27.8/31 questions (89.6% complete) with no evidence of a difference in completeness of the questionnaire between pen received or not ( Table 4).

. Due to the non-statistically significant effect of the interventions on response rates calculating overall associated costs provides evidence of potential cost savings not to send the social incentive cover letter and/or pen ( Extended data: Supplementary File 9 ).

Meta-analysis

A fixed effect meta-analysis of enclosing a pen with a follow-up postal questionnaire on response rate was conducted ( Figure 2). This included five studies (n=13012 participants) and gave a statistically significant pooled OR favouring the intervention (1.21, 95% CI 1.09 to 1.35 p = 0.0004). Negligible heterogeneity was observed (chi-squared = 2.88 I 2= 0%). The risk of bias was low, as indicated by the Cochrane’s risk of bias tool assessment undertaken ( Extended data: Supplementary File 10 ).

Figure 2.

Meta-analysis of enclosing a pen with a questionnaire to increase response rate to a postal questionnaire.

Discussion

This SWAT found no evidence that sending a pen and/or a social incentive cover letter with a postal, trial follow-up questionnaire improved response rate, time to return, requirement for a reminder, or questionnaire completeness.

A limitation was the average age of the participants (79.9 years) giving a narrow age demographic thus restricting generalisability of results. Further investigation of the pen and social incentive cover letter in RCTs are required across more diverse populations.

The OTIS trial hosted three other methodological SWATs; therefore, there was a potential for contamination or interaction. It is preferable to plan all SWATs that will be undertaken in the early design stages , to ensure they are planned accordingly to reduce the potential of this.

The overall response rate of the 12-month postal questionnaire for all SWAT participants was 95.7%, which may have been helped by the inclusion of £5 to all participants as standard. This high response rate is therefore difficult to improve upon, furthermore the incentives may not have been as effective with participants who are very committed to the behaviour . The incentive required for committed participants may be different . A learning point being that future SWATS testing these interventions should avoid doing so in trials with already high response rates.

Conclusion

Whilst neither the pen nor the social incentive cover letter showed an effect on response rate, the meta-analysis evidence remains that including a pen increases response rates. This reinforces that for interventions where small effects are likely, it is important to undertake a number of trials and combine these to be confident of an intervention’s effectiveness. Further investigation of the social incentive cover letter in RCTs is required to determine effectiveness.

Data availability

Underlying data

Open Science Framework: Pen and Social Incentive Cover Letter Retention SWAT, https://doi.org/10.17605/OSF.IO/7TDRB .

Extended data

Open Science Framework: Pen and Social Incentive Cover Letter Retention SWAT, https://doi.org/10.17605/OSF.IO/7TDRB .

This project contains the following extended data:

Full study protocol

Supplementary File 1- Cover letter for the Pen only group.

Supplementary File 2 - Cover letter for the Social incentive cover letter only group.

Supplementary File 3 - Cover letter for the Pen and social incentive cover letter group.

Supplementary File 4 - Cover letter for the control group.

Supplementary File 5 - Results table by intervention group

Supplementary File 6 - Graph Survival curve of pen vs no pen and time taken to return 12-month questionnaire.

Supplementary File 7 - Graph Survival curve of Social incentive cover letter vs no social incentive cover letter and time taken to return 12-month questionnaire.

Supplementary File 8- Survival curve of host trial allocation and time taken to return 12-month questionnaire.

Supplementary File 9 – Costings table

Supplementary File 10 – Cochrane Risk of bias tool assessments for Bell et al., (2016) , Sharp et al., (2006) , Cunningham-Burley et al., (2020) , Mitchell et al., (2020) and James et al., (2020).

Supplementary File 11 – Summary of all SWATs undertaken in the OTIS study

Supplementary File 12 – Copy of the OTIS reminder letter

Supplementary file 13 – Summary of studies included in the meta-analysis

Supplementary File 14 – Copy of the search strategies of Brueton et al., (2014)

Reporting guidelines

Open Science Framework: CONSORT checklist for ‘Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial’, .

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Comments have been addressed.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Health services research, design and evaluation of interventions to change behaviour; the application of behavioural theory to trial recruitment and retention.

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Thank you for thoroughly revising this manuscript. The limitations of the study are now addressed in the discussion, nevertheless diminish the importance of the results. However, the significance of the study was increased by the meta-analysis. I only have one very minor comment:

Table 3, column “Other”: should it be “the…” instead of “tde 12-month questionnaire…”?

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Midwifery, Epidemiology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Thank you very much for giving me the opportunity to review this article. Studies Within A Trial (SWAT) are important and a good opportunity to make optimal use of resources. Despite their disadvantage that the original trial has not been designed for the purpose of the SWAT, they provide important indications for further studies. In this sense, the results of this SWAT are important for further research.

General impression:

This SWAT was planned at the same time as the original trial. The methods are described in detail and are the strongest part of the article. Nevertheless, there are some important concerns. Participants were rewarded with a 5-pound note. This is in itself also an intervention to increase response rates. As the authors themselves note, the response rate is generally very high. Possibly the additional reward and together with the high response rate made it impossible to show an effect of the two tested interventions.

Abstract: Background: Methods: Results: Discussion:

Background: Is it still true that postal questionnaires are used frequently in randomized controlled trials? One could imagine that a swift towards online surveys can be observed.

This very short background is the weakest part of the manuscript.

The rational for the SWAT differs between the abstract and the main text. The general problem of low response rate for the generalizability of results seems important also for the article, not only for the abstract.

This is the strongest part of the article and mostly very clear.

It was a good idea to combine the results of this SWAT with results of other studies in a meta-analysis. However, it could be meaningful to provide further details about the literature research for the articles, which were included. There is not enough information to decide if the research was really systematic or not. Especially the comment that co-author personal knowledge of studies was considered for finding studies raises some questions. Did the included articles test the same interventions?

Results are presented in a clear way with nice tables. Especially Figure 1 which gives very good overviews about the study process.

Are differences between the study groups described in Table 2 significant or not?

The enclosed 5-pound note represents a further limitation and should be mentioned.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Midwifery, Epidemiology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

We thank the reviewer for the helpful comments and points raised. We hope that our responses are satisfactory and the article has been updated in response to the points raised from both reviewers (v2).

General impression:

This SWAT was planned at the same time as the original trial. The methods are described in detail and are the strongest part of the article. Nevertheless, there are some important concerns. Participants were rewarded with a 5-pound note. This is in itself also an intervention to increase response rates. As the authors themselves note, the response rate is generally very high. Possibly the additional reward and together with the high response rate made it impossible to show an effect of the two tested interventions.

Response: Thank you, these are valid points. Brueton et al., (2013) report that the addition of a monetary incentive was more effective than no incentive (RR 1.31; 95% CI 1.11 to 1.55, p=0.002) so the £5 monetary reward was included as standard in the OTIS trial at the 12 month time point.  All participants received the £5 so this was not felt to be a limitation in assessing the effectiveness of the 2 interventions investigated by this SWAT, but it may have added to the high response rate and this is now noted in the discussion. At the time this SWAT was designed and added to the OTIS trial the high response rates were not known. The high response rate is mentioned in the discussion of the paper with a learning point for future SWATs, as we acknowledge that this limits the scope for the interventions to have a large benefit.

Abstract: Response: We believe it is still true, indeed most of the trials we conduct in York Trial Unit still use postal questionnaires; however, we agree that the use of online data collection has increased since this SWAT was designed 4 years ago. With RCTs in this demographic/population and for participants in RCTs who do not have internet provision, in order to ensure inclusivity, there is likely to remain a need for future RCTs to still have a postal paper questionnaire option available and so keeps this relevant.

Background: Is it still true that postal questionnaires are used frequently in randomized controlled trials? One could imagine that a swift towards online surveys can be observed.

Background: Response: Thank you for this, we have updated the paper v2 to include this information from the abstract into the introduction to ensure consistency. We were limited by the word count of the article to be able to provide any further detail in this section.

This very short background is the weakest part of the manuscript.

The rational for the SWAT differs between the abstract and the main text. The general problem of low response rate for the generalizability of results seems important also for the article, not only for the abstract.

Methods: Response: All the studies included in the meta-analysis are now published and peer reviewed with the references provided so that these can be located and a summary of the key information of the studies included in the meta analysis is now provided in Supplementary file 13.

This is the strongest part of the article and mostly very clear.

It was a good idea to combine the results of this SWAT with results of other studies in a meta-analysis. However, it could be meaningful to provide further details about the literature research for the articles, which were included. There is not enough information to decide if the research was really systematic or not. Especially the comment that co-author personal knowledge of studies was considered for finding studies raises some questions. Did the included articles test the same interventions?

[Added to paper v2: The Cochrane systematic review search strategy (supplementary file 14) was used to search MEDLINE and EMBASE. Additionally previous systematic reviews references were hand searched, along with published retention research reference lists, conference papers and co-author personal knowledge of studies.]

Results: Response: Table 2 presents summary statistics for the baseline characteristics of the SWAT participants. According to the CONSORT statement, significance testing of baseline differences in randomized controlled trials should not be performed. Indeed, this practice has been discouraged by numerous authors.

Results are presented in a clear way with nice tables. Especially Figure 1 which gives very good overviews about the study process.

Are differences between the study groups described in Table 2 significant or not?

(Reference for the above information = , ,  ,  , and (2015) Testing for baseline differences in randomized controlled trials: an unhealthy research behavior that is hard to eradicate. Int J Behav Nutr Phys Act. 2015; 12: 4.Published online 2015 Jan 24. doi:  )

Discussion: Now mentioned in the discussion

The enclosed 5-pound note represents a further limitation and should be mentioned.

The present article by James et al. details a study within a trial (SWAT) of including a pen and/or cover letter with a postal questionnaire. A two-by-two factorial randomized control trial was conducted that assessed the effect of including a branded pen, a ‘social incentive’ cover letter, both a pen and ‘social incentive’ cover letter, or a standard cover letter as control. The primary outcome measure was the return of a postal questionnaire with secondary outcomes of time to return the questionnaire, completeness of the questionnaire, requirement for a reminder letter to be sent, and overall cost effectiveness. Overall, there was no effect of any intervention on return of questionnaires and no effect on secondary outcome measures. A meta-analysis was conducted with respect to including a pen with the questionnaire and its effect on response rate, with pooled results supporting the inclusion of pens.

We would firstly like to commend the authors for presenting the methods clearly and providing supplementary material which enhances replicability. Below we indicate points which could be addressed and identify areas of improvement to further enhance the quality of this article.

Potentially confounding incentives Theoretical foundation of cover letter Minor points to address:

The ‘social incentive’ intervention cover letter is described as including a “mild level of social incentive”. In referencing the supporting citation for the cover letter (Cotterill et al., 2017) , the social incentive used is social pressure. Social pressure is described as conveying to participants that “information about their behavior will be noticed (Cotterill et al., 2017) ”. The present paper then describes the cover letter as “intended to highlight to the participant that their questionnaire responses are noted and valued”. It is apparent that reminder letters were also sent to some participants. A reminder letter could be argued to be another “mild level of social incentive” as it is similarly conveying to participants that their behaviour (i.e. not returning the questionnaire) is being noticed. Can the authors comment on the potential for the reminder letters to confound the results? The reminder is not included among the supplementary documents, but we would be interested in the authors’ thoughts on the potential for unintended social pressure cues or behaviour change techniques being delivered through the reminder letter.

In reviewing both the standard and social incentive cover letter, it is noted that they are identical apart from the personalized table indicating the participants’ previous responses. Similarly to the above point regarding unintended social pressure cues, we wonder if the phrases “We would be very grateful if you could return the completed questionnaire to us[…]” and “Your input to this trial is very important to us[…]”can be unintentionally introducing confounding social pressure by highlighting that their questionnaire responses are “noted and valued”. Can the authors reflect on how the addition of these phrases (and thereby the possibility that unintended ‘social incentives’ were applied within the control group) may have potentially affected results?

Under the section, “Interventions”, on page 3, it is mentioned that an unconditional £5 note was included with the questionnaire for all participants. Due to the financial incentive present, we would just like to clarify if the inclusion of the £5 note was controlled for in the analysis.

It is referenced in the “Discussion” section that the host trial included three other methodological SWATs that have a potential for contamination or interaction. If possible, it would be good to offer brief descriptions or links to other SWATS in the supplementary materials to better understand their potential to confound intervention effects.

Applying the Behaviour Change Techniques (BCT) taxonomy definitions to the intervention letter raises the possibility that the intervention being tested is more akin to “feedback on behaviour” (see Michie, 2013) , given that the letter provides informative feedback on the frequency of questionnaire return, rather than the BCT taxonomy definition of ‘social incentive’ (i.e. “Inform that a verbal or nonverbal reward will be delivered if and only if there has been effort and/or progress in performing the behaviour"). It is worth noting that some readers may interpret the meaning of 'social incentives' quite differently than how it is conceptualised here. Can the authors offer some clarification on this for the reader and on the theoretical underpinnings of the intervention? Adding citation(s) to clarify their conceptualisation of 'social incentive' would be helpful.

It is also unclear whether the authors regard the social incentive cover letter as a form of “social reward”: noted on page 4, the authors state that gender was controlled for within the logistic regression model to “control for potential differences in anticipation of social rewards…”. It does not appear that participants are presented with additional verbal or non-verbal reward above the standard letter with just the inclusion of the response table.

It is noted in Cotterill et al. (2017) that social pressure is “unlikely to be effective among those who are very committed to the behavior” which appears to be the case given that response rates across groups are >95%. Given that there are two prior postal response timepoints, was identifying if this group is highly committed to questionnaire response considered?

Regarding Table 2, are BMI, EQ-55D-5L score, and number of falls relevant to include? The table is already quite extensive and this data may detract the reader’s attention from other variables (e.g. age, sex) more pertinent to the study.

The pen is described as York Trials Unit branded. Was there branding specific to the host trial? It might be that participants would be quicker to recognize the trial versus the CTU. How might this affect the social incentive?

Could it be mentioned briefly in the main body of the paper that the meta-analysis was undertaken as part of PROMETHEUS? We notice it is mentioned in the acknowledgments but one of the review team members was confused about the inclusion of the meta-analysis.

This peer review was conducted by Taylor Coffey, Dr Louisa Lawrie and Dr Eilidh Duncan as part of TC and LL’s training. All reviewers contributed to writing the report.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Health services research, design and evaluation of interventions to change behaviour; the application of behavioural theory to trial recruitment and retention.

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.

Potentially confounding incentives Thank you to the reviewers for their valued comments.

The ‘social incentive’ intervention cover letter is described as including a “mild level of social incentive”. In referencing the supporting citation for the cover letter (Cotterill , the social incentive used is social pressure. Social pressure is described as conveying to participants that “information about their behavior will be noticed (Cotterill ”. The present paper then describes the cover letter as “intended to highlight to the participant that their questionnaire responses are noted and valued”. It is apparent that reminder letters were also sent to some participants. A reminder letter could be argued to be another “mild level of social incentive” as it is similarly conveying to participants that their behaviour (i.e. not returning the questionnaire) is being noticed. Can the authors comment on the potential for the reminder letters to confound the results? The reminder is not included among the supplementary documents, but we would be interested in the authors’ thoughts on the potential for unintended social pressure cues or behaviour change techniques being delivered through the reminder letter.

For this paper the term ‘social incentive’ was used rather than ‘social pressure’ on the advice of two of the co-authors of this paper who were involved in the original Cotterill et al., (2017) embedded trial to test the social incentive cover letter intervention within a cohort study. They noted that following this paper the term ‘social pressure’ is not commonly used outside of politics literature.

A copy of the reminder letter that was sent to 83 participants has been added to the OSF as supplementary document 12.

Whilst it could be argued that the reminder letter provided another mild level of social incentive, it was not felt that it would provide additional social incentive as per the definition used for this study (discussed in relation to the next comment).  Furthermore, the number of these sent out was evenly distributed across the randomised and analysed groups: 83 reminder letters sent were sent out of 755 and these were evenly distributed across the 4 groups (pen and social incentive cover letter group 19/184 (10.3%), control group 24/190 (12.6%), social incentive cover letter only group 20/189 (10.5%) and pen only group 20/192 (10.4%). So it would be unlikely that these additional reminder letters would confound the results.

There was no statistically significant evidence of a difference in the likelihood of participants requiring a reminder between the groups (pen vs no pen: adjusted OR = 0.89 [95% CI 0.56, 1.42 p = 0.63]; social incentive cover letter vs no social incentive cover letter: adjusted OR = 0.92 [95% CI 0.58, 1.47 p = 0.74]. The standard text for this cover letter was the exact text used in the cover letters previously sent to participants with their 4 and 8 month questionnaires for the OTIS trial. The text of both the ‘control’ and ‘social incentive’ cover letters were identical the only difference was the addition of the table therefore it was not felt that this would affect the results. The example sentences noted could fall into some definitions for social incentive, but they would not meet ‘persuades people to behave in a certain way by the promise that their actions will be noticed or made public’ (Cotterill et al., 2017). Clarification of this definition has been added to the paper to help to clarify the comments around this.

In reviewing both the standard and social incentive cover letter, it is noted that they are identical apart from the personalized table indicating the participants’ previous responses. Similarly to the above point regarding unintended social pressure cues, we wonder if the phrases “We would be very grateful if you could return the completed questionnaire to us[…]” and “Your input to this trial is very important to us[…]”can be unintentionally introducing confounding social pressure by highlighting that their questionnaire responses are “noted and valued”. Can the authors reflect on how the addition of these phrases (and thereby the possibility that unintended ‘social incentives’ were applied within the control group) may have potentially affected results?

Since all participants received the £5 it was neither necessary nor appropriate to control for this in the analysis.

Under the section, “Interventions”, on page 3, it is mentioned that an unconditional £5 note was included with the questionnaire for all participants. Due to the financial incentive present, we would just like to clarify if the inclusion of the £5 note was controlled for in the analysis.

A supplementary file 11 has been added to OSF outlining the other SWATs that were undertaken within the OTIS Trial.

It is referenced in the “Discussion” section that the host trial included three other methodological SWATs that have a potential for contamination or interaction. If possible, it would be good to offer brief descriptions or links to other SWATS in the supplementary materials to better understand their potential to confound intervention effects.

Theoretical foundation of cover letter Although recent work has pointed towards the importance of using behaviour change theory to support and influence behavioural actions required by participants in RCTs (Gilles et al., 2018), behaviour change theory and associated research is such a vast area that this is a valid point that different concepts can be defined in different ways by different theories/authors. To help to clarify this in the paper sentences have been added to provide further detail and detail how the concept of social incentive was interpreted in relation to this study.

Applying the Behaviour Change Techniques (BCT) taxonomy definitions to the intervention letter raises the possibility that the intervention being tested is more akin to “feedback on behaviour” (see Michie, 2013) , given that the letter provides informative feedback on the frequency of questionnaire return, rather than the BCT taxonomy definition of ‘social incentive’ (i.e. “Inform that a verbal or nonverbal reward will be delivered if and only if there has been effort and/or progress in performing the behaviour"). It is worth noting that some readers may interpret the meaning of 'social incentives' quite differently than how it is conceptualised here. Can the authors offer some clarification on this for the reader and on the theoretical underpinnings of the intervention? Adding citation(s) to clarify their conceptualisation of 'social incentive' would be helpful.

[added to paper v2: ‘The concept of social incentive that underpinned the intervention for this study was that a social incentive is something that persuades people to behave in a certain way by the promise that their actions will be noticed or made public [10].’ Error in terminology of the paper, social reward was used interchangeably with social incentive in error. Social reward removed from this sentence and replaced with social incentive with a further reference added.

It is also unclear whether the authors regard the social incentive cover letter as a form of “social reward”: noted on page 4, the authors state that gender was controlled for within the logistic regression model to “control for potential differences in anticipation of social rewards…”. It does not appear that participants are presented with additional verbal or non-verbal reward above the standard letter with just the inclusion of the response table.

[changed paper v2 to read: ‘gender, to control for potential difference in the effect of social incentives between males and females]

Thank you, this is a valid point.  At the time this SWAT was designed and added to the OTIS trial the high response rates of the previous time points were not known. This is currently noted in the paper as a consideration for future research to consider when using this type of social incentive cover letter.

It is noted in Cotterill that social pressure is “unlikely to be effective among those who are very committed to the behavior” which appears to be the case given that response rates across groups are >95%. Given that there are two prior postal response timepoints, was identifying if this group is highly committed to questionnaire response considered?

These variables have been removed from Table 2 as suggested.

Regarding Table 2, are BMI, EQ-55D-5L score, and number of falls relevant to include? The table is already quite extensive and this data may detract the reader’s attention from other variables (e.g. age, sex) more pertinent to the study.

The pen had the logo of York Trials Unit and was not specific to the OTIS trial. This would be an interesting follow on question for future work, whether the branding of the pen sent with a questionnaire had any effect on response rate. The pen was not considered to provide a social incentive for this SWAT, only the cover letter.

The pen is described as York Trials Unit branded. Was there branding specific to the host trial? It might be that participants would be quicker to recognize the trial versus the CTU. How might this affect the social incentive?

[Added to paper v2: ‘Due to SWATs typically being under-powered to show small effects, it is essential that the results are seen within the context of the wider literature (added to methods section under statistical analysis)

Could it be mentioned briefly in the main body of the paper that the meta-analysis was undertaken as part of PROMETHEUS? We notice it is mentioned in the acknowledgments but one of the review team members was confused about the inclusion of the meta-analysis.

The results of this study were pooled with four previous SWATs [8,9,22,23] investigating the same intervention, with the same dichotomous outcome of response to the questionnaire or not. Pooled odds ratios and corresponding 95% CIs were calculated. Heterogeneity between trials was assessed using the Chi-squared and I 2 statistics. The meta-analysis was facilitated by the PROMoting THE USE of SWATs (PROMETHEUS) programme, which supports host trial teams to conducted SWATs and for data obtained to be collated and meta-analysed.’(Summary details for the studies included in the meta-analysis are provided in Supplementary file 13)]