Concerns about the external validity of the study 'Prevalence of persistent symptoms after treatment for Lyme borreliosis: A prospective observational cohort study'.

Ursinus et al. have provided a very detailed, carefully described and analyzed study on persistent symptoms after treatment for Lyme borreliosis (LB)., It is an impressive body of work with three questionnaires sent to over 7000 participants at five time points (baseline, 3,6,9,12 months). Great care was taken to ascertain case validity, especially by checking the erythema migrans (EM) cases via photographs. The study compared the cohort of LB cases with two different control cohorts.

The STROBE statement-checklist recommends to: “Discuss the generalizability (external validity) of the study results.”

The target population is not explicitly defined by the authors, but for the ensuing discussion it could be defined as consecutive patients treated for LB in the study period in the Netherlands or perhaps Northwestern Europe. An important point of discussion is, whether the study population of LB patients (n=1135) or EM patients (n=1076) were recruited as a representative sample of the target population.

It is described that “patients were included after online self-registration (www.tekenradar.nl) or through participating clinical LB centers”. The study was advertised broadly in the public, via medical clinics and patient societies.

The recruitment strategy was different for the three groups as described in Fig. 1 (in ref. 1) and Table S5 (in ref. 1):

1768 LB patients were screened for eligibility. 1135 LB cases were included and of these 924(81%) completed the 12 month follow up.

28395 individuals were randomly invited from the general population and matched to the included cases. 4000(14%) were included and 1212 (30%) of these completed the 12 month follow up.

5456 individuals from the tick bite cohort were screened for eligibility and 2405 (44%) were included. Of these 1111(46%) completed the 12 month follow up.

All recruited persons registered data via the tekenradar website. There was a large difference in participation as 81% of cases completed the study compared to 30% and 46% of the control groups. Also a large number (85.5%) of the invited population controls did not respond at all. This points to differences in motivation to participate. It should be noted, that the case group was reminded by telephone, but not the control groups. Follow up questionnaires were optional for the group of population controls (footnote Table S5 in ref. 1). Thus, it is not clear how many with missing follow up data would have fulfilled the definition of persistent symptoms.

The recruitment strategy of advertising on the internet and via other channels with voluntary recruitment via the internet may have a high risk of self selection or volunteer bias. Also the selection process of cases included by the clinics is not described. This could selectively amplify the number of included patients with self-reported health complaints or enhance inclusion of clinical cases with a more severe presentation.

The authors assume that the tick bite cohort without LB would be comparable concerning the online methods. This assumption is not backed by the differences in missing follow up. Furthermore, the EM group shows nearly the same level of symptoms at all individual time points. There is no effect of an acute clinical infection with a higher symptom load at baseline and a subsequent decline as is seen with disseminated LB (Fig 3, Fig S2, in ref. 1). This feature is consistent with an assumption of mild and local pathology of European EM with negligible systemic effect and that remaining symptoms are absent. The small increased stable level of subjective complaints found in the study, does not appear associated with the EM episode. It is also not completely clear if the baseline, at the time of acute illness, was included in the definition of persistent symptoms?

A clear description of the selection of the final subset used in the main results is not found.

N=1084, 1942 and 1887 fullfilled “primary analysis criteria” for the LB, Pop control and tick bite group, respectively. These numbers are shown as 1135, 4000 and 2405 “included in the primary analysis” in the flowchart fig. 1 and table S5 (in ref. 1).

Multiple substitution was used and can be of advantage. Missing scores were substituted using the data from available preceding and following scores. A too large proportion of missing data could be >40% as a proposed rule of thumb and in which case multiple imputation is not recommended. The proportion of missing data (complete or incomplete cases) was quite high as shown in table S4B (in ref 1), but it is not quite clear how the rate of missing data was assessed. For example, (1135-600)/1135 (47%) of LB and (2405-701)/2405 (71%) of the tick bite cohort needed substitution. The main outcome definition for “persistent symptoms” is in short reporting reduced health in at least 3 consecutive questionnaires. For the substitution to be valid it is assumed, that all three cohorts have the same random a tendency to report similar results in the missing questionnaires. The high rate of missing questionnaires especially in the cohorts without LNB does not support this assumption. It is difficult to assess, if it was safe to perform multiple imputation. However, the sensitivity analyses does point to some robustness

Another point which could affect the external validity of the study, is the inclusion periods for the various cohorts:

LB: April 2015 – October 2018, Tick cohort: April 2016 – July 2019 and Population: October 2017 – September 2018. These differences are not discussed, or explained and no information is given on when patients were included or started follow-up within these periods. A graph with the start of follow-up for each group (month and year on the x-axis, and counts on the y-axis) could be appreciated. Did all three groups have similar seasonal variation in recruitment?

In conclusion it is a large well performed study with excellent attention to many details, but it is a shortcoming, that the authors do not present an explicit discussion of the external validity of the study, as being a representative sample of consecutive clinical patients with LB. It is unclear if the results of the study may be generalized beyond the study population due to risk of volunteer bias and differences in the rate of missing data.

Contributors

RBD has written the original draft, RBD, KAK and LFO have discussed, reviewed and edited the complete contents.

Declaration of interests

RBD has received support for attending ECCMID in 2019 and 2018, participation in an Advisory Board Meeting Roche diagnostics 2018, Member of the executive committee of ESGBOR (www.escmid.org/esgbor) 2011-2019. LFO has received research grants from Region Värmland, Sweden. KAK has no conflicts of interest to disclose.