The lower COVID-19 related mortality and incidence rates in Eastern European countries are associated with delayed start of community circulation.

BACKGROUND: The purpose of this analysis was to assess the variations in COVID-19 related mortality in relation to the time differences in the commencement of virus circulation and containment measures in the European Region.
METHODS: The data for the current analysis (N = 50 countries) were retrieved from the John Hopkins University dataset on the 7th of May 2020, with countries as study units. A piecewise regression analysis was conducted with mortality and cumulative incidence rates introduced as dependent variables and time interval (days from the 22nd of January to the date when 100 first cases were reported) as the main predictor. The country average life expectancy at birth and outpatient contacts per person per year were statistically adjusted for in the regression model.
RESULTS: Mortality and incidence were strongly and inversely intercorrelated with days from January 22, respectively -0.83 (p<0.001) and -0.73 (p<0.001). Adjusting for average life expectancy and outpatients contacts per person per year, between days 33 to 50 from the 22nd of the January, the average mortality rate decreased by 30.1/million per day (95% CI: 22.7, 37.6, p<0.001). During interval 51 to 73 days, the change in mortality was no longer statistically significant but still showed a decreasing trend. A similar relationship with time interval was found for incidence. Life expectancy and outpatients contacts per person per year were not associated with mortality rate.
CONCLUSION: Countries in Europe that had the earliest COVID-19 circulation suffered the worst consequences in terms of health outcomes, specifically mortality. The drastic social isolation measures, quickly undertaken in response to those initial outbreaks appear effective, especially in Eastern European countries, where community circulation started after March 11th. The study demonstrates that efforts to delay the early spread of the virus may have saved an average 30 deaths daily per one million inhabitants.

Introduction

COVID-19 was declared a pandemic by the World Health Organization (WHO) on March 13, 2020 [1]. The WHO, in its first statement on the 22nd of January 2020, reported that there was evidence of human-to-human transmission of the new coronavirus identified in the Wuhan outbreak [2].

In Europe, the infection spread from China with the first cases reported in second half of January in France, Germany, Italy, Spain and the United Kingdom [3]. Sustained community circulation of SARS-Cov-2 began in late February and early March, and by the end of March, almost all European countries had reported their first 100 confirmed cases [4].

Countries in Europe started to talk about public health containment measures in late January and early February [5,6], but the majority of drastic, countrywide containment measures in Europe started in mid-March. This followed the spike of cases in Lombardy, Italy, which provided strong evidence for the devastating potential of the new virus. The WHO declared the pandemic on 11th of March, which gave official clarity to the scope and urgency of the issue. Containment measures varied from country to country and included such actions as closure of schools, closure of most non essential businesses and services, ban of non-essential travel, and total lockdown of cities. For most European countries, these measures had never been experienced before at such a widespread degree and intensity.

A striking difference can be seen in COVID-19 indicators between countries in Western Europe and those in Eastern Europe, with much lower cumulative incidence and mortality rates in Eastern Europe. Mortality rates range from more than 500 per million inhabitants in Spain, to less than 10 per million in Ukraine [4]. The reasons for these differences are still largely unexplained.

Recently, peer reviewed publications and other reports have explored biological factors responsible for the differences in incidence and mortality. For instance, host angiotensin-converting enzyme (ACE) receptor polymorphism and Bacillus Calmette-Guerin (BCG) vaccination have been cited as possible explanations [7,8]. While such biological factors are of important clinical significance, they are unlikely to explain the wide population differences in incidence and mortality observed across countries and regions in Europe. While there seems to be general consensus among professionals [9,10] about the overall efficacy of containment measures, an active debate remains about the effect of specific interventions such as stay at home orders and closure of all businesses [11,12].

Strong evidence about these topics is important to understand effective responses to the current pandemic and to prepare for future events.

As of the writing of this manuscript, no peer reviewed papers reported on the effect of the timing of containment measures in relation to the spread of COVID-19 across countries in Europe. Thus, a relatively straightforward analysis of the timing the epidemic spread across various European countries and its effects on mortality rates, is warranted and informative.

The study objective was to assess whether differences in COVID-19 related mortality were associated with differences in the timing of documented community circulation of the virus across European countries. We hypothesize that countries where the COVID-19 outbreak started later were in a better position to implement drastic control measures in time to minimize further spread of infection and consequent negative health outcomes in their population.

Methods

Data

All countries of the WHO European region with a population over 100 000 were included in the analyses (n = 50).

Our outcome variables were COVID-19 mortality and cumulative incidence on the 7th of May, 2020.

As an indicator of the initiation of SARS-Cov-2 community circulation—our primary predictor variable—we use the date of reporting the first 100 confirmed cases. The country specific COVID 19 related mortality rates and the dates countries reported their first 100 cases were retrieved from John Hopkins dataset [4]. The data were verified at the European Centre for Disease Control (ECDC) database [13].

We estimate similar epidemiological surveillance capacities among study units. Since the 27th of January, 2020, all WHO European region countries were included in a COVID-19 standardized surveillance system, coordinated by ECDC and the WHO Regional Office for Europe. By the end of January cross-border inter laboratory systems were in place to arrange testing and reporting of cases [3].

Time to the first 100 cases is a synthetic and relatively robust metric for measuring the initiation of a pandemic. This metric, the 100 first cases, has been used in relevant publications to classify early COVID-19 cases [14]. The number seems to represent a critical mass of cases documented during initial community circulation. Some publications have described transmission dynamics in samples of the first 100 COVID-19 cases demonstrating community circulation [15,16].

To compare the time differences between European countries concerning initiation of community circulation, we use the interval between the date of the country reporting its first 100 COVID-19 cases and the 22nd of January. The latter is the date when the WHO stated there was human-to-human transmission of the novel coronavirus. We call this indicator throughout the paper ‘time interval’ or ‘days from 22nd January’. It is the main independent variable in our model.

To control for the influence of different proportions of older adults across European countries, the country average life expectancy at birth was included in the multivariate analyses. These data were retrieved from the World Bank dataset on the 7th of May 2020 [17]. To control for different health-care system utilization patterns, outpatients contacts per person per year was also included in the multivariate analyses. We used data on outpatient contacts from latest year available as retrieved from WHO dataset on the 10th of October 2020 [18]. We ran additional models using US data to see if the findings from Europe could be replicated in another context.

Statistical analysis

Univariate descriptive statistics were used to summarize all variables. Non-parametric methods were applied for bivariate analysis. We calculated Spearman correlations to measure the strength of associations and used scatterplots with a locally weighted smoothing line to examine if there were non-linear relationships between dependent variables (i.e. mortality or incidence), with time interval (number of days between the date when the 100 first COVID cases were reported and the 22nd of January). The scatterplots revealed a change of linear pattern at day 50 of the time interval (which corresponds to 11th of March, as the date when 100 first COVID-19 cases were reported); therefore, summary statistics were calculated for all variables for the time interval from day 31 to 50, and time interval from 51 to 73. One-way ANOVA tests were performed to calculate p-values for differences in means for all variables.

Next, we carried out bivariate and multivariable piecewise linear regression analyses [19] for mortality and incidence with independent variables ‘time interval’ with break point at day 50, ‘life expectancy’ and ‘outpatients contacts per person per year’. Model diagnostics were performed to examine normality and influential data points were used to assess model performance. Model diagnostics showed that Italy, Belgium and Germany were influential countries in mortality analysis whereas Luxemburg and Iceland were influential for incidence analysis using Cook’s D criteria. Residuals for both multiple regression analyses were approximately normal. Log-transformation of the two outcomes improved the normality but the adjusted R2 was smaller for the mortality model. Interpretation of original measures of mortality and incidence were used for all models so that the interpretations were clear and meaningful. Finally, we conducted sensitivity analysis using data from the United States, since the total sample size is similar to the combined European countries. These analyses followed the same procedures as above and were done to assess whether the model explored for Europe could be replicated elsewhere.

Results

Table 1 shows the summary statistics for all data and by time interval (time interval in days before or after day 50), and the Spearman correlations between variables. The mean mortality, incidence and life expectancy were higher in the time interval 31–50 compared to time interval 51–73 (p<0.001). Mortality and incidence were highly intercorrelated (r = 0.84, p<0.001), and positively associated with life expectancy (r = 0.75, p<0.001). The correlation between mortality with time interval was -0.83 (p<0.001) and -0.73 (p<0.001) for incidence. Outpatients contacts per person per year had a weak association with mortality and incidence.

Table 1

Summary statistics for mortality per million, incidence per million, life expectancy, and outpatients contacts per person per year for the full sample and by number of days from the 22nd of January (31–50 days or 51–73 days), and the Spearman correlation between the variables.

Variable (Range)	All Sample (N = 50)	Days 31–50 (N = 16)	Days 51–73 (N = 34)	p-value
	Mean ± SD	Mean ± SD	Mean ± SD
Mortality (0.3–720)	97.4 ± 162.6	234.8 ± 225.2	32.8 ± 52.2	< .0001
Incidence (40–6134)	1504 ± 1526	2634 ± 1476	972.3 ± 1247.1	0.0001
Life expectancy (71–84 years)	78.2 ± 4.1	82.1 ± 1.3	76.3 ± 3.6	< .0001
Outpatients contacts per person per year	6.4 ± 2.5	6.3 ± 2.2	6.4 ± 2.7	0.889
Spearman Correlation and p-value
	Mortality	Incidence	Days from 22 Jan	Life expectancy
Incidence	0.84 (p<0.001)
Days from 22 Jan	-0.83 (p<0.001)	-0.73 (p<0.001)
Life expectancy	0.75 (p<0.001)	0.76 (p<0.001)	-0.81 (p<0.001)
Outpatients contacts per person per year	0.05 (p = 0.718)	0.10 (p = 0.482)	-0.04 (p = 0.810)	-0.14 (p = 0.326)

Fig 1 is the scatterplot of mortality and incidence per million vs. days from the 22nd of January with a locally weighted smoothing line. The figure shows that the slopes for both mortality and incidence before day 50 were steeper than during the time interval between days 51–73.

Fig 1

Scatterplot of mortality and incidence per million versus days from the 22nd of January.

Countries labels as blue dots are influential data points using Cook’s D criteria.

Results from bivariate and multivariable piecewise regression after removing influential data are displayed in Table 2. The parameter estimates for time interval attenuated slightly in the multivariable analysis, but remained statistically significant.

Table 2

Bivariate and multivariable regression analysis of mortality and incidence per million with three independent variables: Days from the 22nd of January with break point at day 50 (time interval 31–50 and 51–73 days), life expectancy, and outpatients contacts per person per year.

	Bivariate Analysis		Multivariable Analysis
	Beta (95% CI)	p-value	Beta (95% CI)	p-value
Mortality per million			Adjusted R2 = 71%
Time interval (31–50 days)	-31.24 (-38.14, -24.34)	<0.001	-30.14 (-37.64, -22.65)	<0.001
Time interval (51–73 days)	-2.33 (-5.70, 1.05)	0.171	-0.05 (-5.60, 5.50)	0.985
Life expectancy (in years)	17.17 (9.58, 24.76)	<0.001	5.26 (-4.31, 14.83)	0.282
Outpatients contacts per person per year	-3.35 (-18.48, 11.77)	0.664	2.67 (-5.74, 11.08)	0.534
Incidence per million			Adjusted R2 = 56%
Time interval (31–50 days)	-153.23 (-221.58, -84.88)	<0.001	-122.67 (-194.61, -50.74)	<0.001
Time interval (51–73 days)	-62.52 (-106.76, -18.29)	0.007	-2.04 (-71.77,67.69)	0.953
Life expectancy (in years)	213.88 (146.44, 281.31)	<0.001	147.29 (22.92, 271.66)	0.020
Outpatients contacts per person per year	-0.33 (-19.68, 19.02)	0.973	53.07 (-49.45, 155.60)	0.310

The multivariable analyses demonstrated that, between days 33 to 50 from the 22nd of January, the average mortality rate decreased by 30.1/million per day (95% CI: 22.7, 37.6, p<0.001). During interval 51 to 73 days, the change in mortality was no longer statistically significant, but still showed a decreasing trend (Beta = -0.05, 95% CI: -5.60, 5.50, p = 0.985). Life expectancy and outpatient contacts per person per year were not associated with mortality rate. A similar relationship with time interval was found for incidence; however, in contrast to the mortality model, higher life expectancy was found to be associated with higher incidence (Beta = 147.3, 95% CI: 22.9, 271.7, p = 0.02). The adjusted R2 (proportion of variation explained by the model) was 75% for mortality and 60% for incidence.

Table 3 and Fig 2 show the results from the sensitivity analyses using data from the United States. These analyses demonstrate very similar findings to those reported from Europe.

Table 3

Regression analysis for USA states: Mortality per million with independent variable days from the 22nd of January with break point at day 59 (time interval 46–59 and 60–70 days).

(a) Mortality per million in the U.S. on May 7th
	Beta (95% CI)	p-value
Time interval (46–59 days)	-35.40 (-56.87, -13.93)	0.001
Time interval (60–70 days)	-12.10 (-34.55, 10.36)	0.284
(b) Mortality per million in the U.S. on Oct 3
	Beta (95% CI)	p-value
Time interval (46–59 days)	-35.61 (-67.03, -4.20)	0.026
Time interval (60–70 days)	-39.21 (-72.06, -6.36)	0.020

Fig 2

Scatterplot for USA data: Mortality per million versus days from the 22nd of January.

Discussion

Timing matters. Results from this study indicate that the time when sustained virus circulation started in a country is associated with the health impact of the pandemic for that country. Overall, later sustained circulation in a country was associated with lower overall mortality and incidence rates.

Results from these analyses of 50 European countries also reveal that the relationship between mortality and incidence rates and time of commencement of community circulation of SARS-Cov-2 does not follow a linear gradient. Mortality and incidence rates decreased significantly for every day of delay until the 50th day from the 22nd of January (i.e. March 11th). After that point, the gradient became less steep and time becomes non significant.

The non linear nature of the relationship may reflect the fact that in the days after the declaration of world pandemic on March 13, 2020, countries were quick to introduce drastic containment measures. The majority of countries, where pandemic started after mid-March, likely benefited similarly from the timely measures to contain viral spread and the differences between them became insignificant. Google mobility reports confirm the quick effects on social distancing of the mid-March measures [20].

The countries in Europe that observed the earliest COVID-19 circulation, suffered the worst consequences in terms of health outcomes, specifically mortality. While only a matter of a few weeks difference, this time interval may have changed dramatically the reservoir of virus circulating in the community and thereby the trajectory of incident disease and resulting deaths.

Reaching the first 100 cases in a country depends also on seeding episodes. Large urban areas of Western Europe have more intensive global connection, more international travel, and a higher potential for more seeding events, which increased the chances of earlier community transmission in these countries [21].

The longer that it took for a country to get to 100 reported cases, the easier it may have been to control the epidemic using public health policy action. The drastic social isolation measures undertaken in European countries where community circulation of the virus started after March 11th seem to have been well-timed. This may explain their significantly lower COVID-19-related incidence and mortality in Eastern European countries compared with the Western European countries.

On March 10th and 11th, the Italian government was the first in Europe to issue decrees [22,23] introducing countrywide lockdown measures, limiting movement out of the home and banning the operation of a number of businesses (bars, retail shops etc). On March 20th the measures were further tightened, banning all non essential open air walking. During a short period, after the 11th of March, most European countries introduced similar measures [24]. Evidence suggests that these measures were effectively enforced and that community mobility was significantly reduced [20]. As the disease spread, drastic measures in Eastern European countries, and in the Western ‘periphery’ of Europe (i.e.Portugal), appear to have been effective in mitigating COVID-19 mortality. In the West, where community circulation had initiated much earlier, the measures taken in mid-March (or even later) were comparatively late and allowed a large mass of COVID-19 cases, building a critical reservoir of infection in the population. Consequently, the efficacy of public health actions was greatly reduced and the most vulnerable members (older adults, those with chronic conditions) of society were deeply affected.

While some governments in Asia (i.e. China) had already taken drastic public health measures to effectively curb the epidemic [25], we assume that after the declaration of a global pandemic,decision-makers in Europe were in a better position to take and enforce such extreme measures, which only weeks before had seemed too draconian. International mass media coverage of the pandemic outcomes on the Italian health system and the high risk of dying in Italian northern regions, also influenced quick decision-making by political leadership by mid-March. In countries with swift responses, including those fortunate to have experienced later community spread and fewer seeding events, the outbreaks were less pervasive and the most vulnerable less affected.

As the pandemic is ongoing, there may be small observed changes in the health outcome differences documented here, with regard to the timing of a critical mass of cases in various regions of Europe. However, it is highly unlikely they will significantly change in the associations documented during the first wave of pandemic. With few exceptions (for example Russia), as of May 7th, in European countries the epidemic curves were flattened, the epidemic peaks were past, and the effective reproductive numbers were around 1 [26].

We also tested the validity of the proposed model with 50 states of United States of America using the same source of data. Our results showed a similar pattern and very similar statistics for the two interval gradients. Using the U.S. data, the cut-off was only about one week later compared to Europe (59 days from 22nd January, or 19th of March). We also applied the model using the 3d of October mortality data and the direction of the trend remained the same; although, the association becomes more linear. The later findings most probably confirm the weaken of the early effect of spring measures.

While life expectancy is lower in the Eastern Europe [17] and a potential confounder in analyses comparing different European countries with varying life expectancies, the multivariate analyses show that timing of the outbreak was a more important factor for mortality. Further, the burden of chronic disease is higher in Eastern compared to Western Europe [27], potentially putting Eastern European populations at higher risk of mortality from COVID-19. Country level results indicate, however, that Eastern European countries have fared better than their Western European counterparts indicating that relative chronic disease burden is an improbable explanatory factor for differences in country level outcomes.

A number of other factors can be discussed to explain differences in COVID-19 outbreak trajectories between Western and Eastern Europe, including urbanization and population density. Currently, these factors are controversial [28,29] and no publications demonstrate consistent evidence to support these as deciding factors.

Compulsory BCG vaccination programs across countries have also been associated with COVID-19 morbidity and mortality in a geographical correlation study [8], but the WHO states that there is insufficient evidence to confirm this [30]. A trial testing the potential effect of BCG vaccines to boost immunity against COVID-19, is underway in Germany and the Netherlands [31]. Geographical variations of ACE2 receptor polymorphism has also been reported as a possible explanation to COVID-19 epidemiological findings [7]. Nonetheless, the two last hypotheses cannot explain the important intra country variations observed in Italy or elsewhere. The delay in infection and timeliness of the measures may better explain the much lower COVID-19 morbidity and mortality rate in Sicily, Sardinia or other regions in ‘periphery’ of Italy, compared to the Northern regions where first clusters where reported.

In conclusion, the results of this study provide secondary evidence about the effect of public health measures taken in Europe after the 11th of March, when pandemic was declared by WHO. They demonstrate that efforts to delay the early spread of the virus might save daily an average 30 deaths per one million inhabitants. The study can help public health professionals to better understand the pattern of pandemic spread and its relation to pandemic mortality. The results will support decision makers during pandemic to take early and swift public health measures in order to assure important health benefits in subsequent months.

Our study is subject to several limitations. Our model may explain some important country differences; for example, the slope in the first regression segment is apparently driven by lower mortality in countries with community circulation reported during 8-11th March (from Austria to Czech Republic). It does not explain all observed differences, such as that between Belgium and Germany or Sweden and Norway. Further research, focused on comparing specific country situations, is needed in the future.

We examine mortality and cumulative incidence as they are reported by countries. The data may have issues which can be only partially validated. While incidence is highly affected by country testing strategies, the reported mortality has been used as a valid health outcome in other studies [7,8] We also use the date of first 100 cases, as they are reported by countries health authorities. Similar sources of data have been considered valid to systematically document community COVID-19 cases [14]. We know that silent community circulation of the virus started before initial detection. Nonetheless, the observation of country epidemic curves seems to generally confirm the ranking of reported initiation dates [32]. Further, genetic sequencing methodology has been used to track early outbreaks in Europe and the USA and it supports the timeline pattern provided by early documented cases [33].

We acknowledge that the ecological study method used in the analyses has limitations and does not allow conclusions on causality. We include only a few variables, and two outcomes. We recommend that future analyses include more potential covariates to explore the complex causal web of COVID-19 health outcomes and their relationships with policy decisions and provide estimates of effects of specific factors on outcomes. Finally, while our hypothesis about higher efficacy to control the epidemic where it started later seems logical, more research is needed on specific public health measures taken by European countries. Such research should find ways how to standardise the interventions and how to make use of relevant non-English literature.

Annex countries data.

(DOC)

Click here for additional data file.

Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

25 Sep 2020

PONE-D-20-15377

The lower COVID-19 related mortality and incidence rates in Eastern European countries are associated with delayed start of community circulation

PLOS ONE

Dear Dr. Ylli,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Your manuscript was reviewed by 2 experts in the field. Both identified significant problems in your submission. Please review the attached comments and provide point-by-point responses.

Please submit your revised manuscript by Nov 09 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Yury E Khudyakov, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

3. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: No

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Review PLOS One: The lower COVID-19 related mortality and incidence rates in Eastern European countries are associated with delayed start of community circulation

Thank you for the opportunity to review this interesting manuscript.

In general, please proofread this manuscript carefully. There are multiple small typos, missing or extra words, and missing or extra punctuation. Given there are no line numbers, it is unwieldly to note them all in the specific comments below.

Abstract – background

This sentence is long and unwieldly and should be simplified or separate into two sentences.

Abstract – conclusion first line

No comma after “circulation”

Introduction first paragraph, second sentence

“statement of on” – delete “of”

Introduction first paragraph, second sentence

Structure of the sentence makes it unclear whether human-to-human transmission or whether the novel coronavirus was first described to the WHO on Dec 31.

Introduction second paragraph, second sentence

Citations needed

Introduction third paragraph, second sentence

Should read something like “These measures varied from country to country and included such actions as” – otherwise, it implies all actions were taken in all places, which is not true.

Introduction third paragraph, last sentence

“size” is an incorrect word here – perhaps “widespread degree”

Page three, final paragraph

Should read “For instance, host angiotensin…” – these are not the only biological differences noted and should not be implied as such.

Page four, first paragraph, final sentence

You mention “measures” – to what are you referring?

Page four, second paragraph, final sentence

This sentence is not grammatical nor is it clear.

Page five, second paragraph

This sentence is convoluted. Please simplify or separate into two sentences.

Page five, fourth paragraph

March 11th was the first 100 cases where?

Page six, first paragraph

Is the interval 51-71 or 51-73? It is mentioned as both in the previous paragraphs. Why was 71/73 chosen as a cut off? It appears that all countries reported 100 cases by this time period, if so, please specify in the body of the manuscript.

Page seven, final sentence

Was higher or lower life expectancy associated with incidence?

Discussion

Please carefully proofread – there is a lot of non-standard English here.

You need citations throughout the discussion, including for “seeding events”, “intensive global connection”.

Your paragraph on life expectancy and chronic disease in Western v Eastern Europe is difficult to understand. Please clarify.

Page ten, final paragraph

Should be “intra country variations observed in Italy” not “inner”

General comments

How do you control for different rates and implementation dates of testing in these fifty countries? You do address incidence/mortality but you do not address how implementation dates of testing may wildly vary your date of first 100 confirmed cases per country. This may significantly impact the outcome of your analyses.

How do you control for different health-care system utilization across these countries?

How do you control for racial differences across these countries?

How do you control for socio-economic differences?

Why do you think there was a statistically significant difference in days 31-50 and not days 51-73?

Given that the virus can have an incubation period of up to 14 days post exposure, was there a significant change in viral transmission 14 days after individual countries imposed their containment measures? Was there a change in trend of time to 100 cases in the 14 days after Mar 11?

Reviewer #2: The manuscript examines the variations in COVID-19 mortality and incidence rates in relation to the delayed start of community virus circulation in Eastern European countries. While there are some potentials in this study, I have concerns about the novelty, social significance, and policy implications of this study, and therefore I would suggest resubmission. Authors need to address the following comments:

--The modeling without validation is not reliable. The authors need to examine whether their model is consistent with independent (test) dataset/recent data or not. With such a small sample size, the associations are not unlikely and not very reliable.

--Page 5 last line: I don’t think model diagnostics are used to improve “model fit”? This is used mostly to check the underlying assumptions. Also, I don’t see any accuracy assessments done for the model.

--The authors need to justify based on what criterion they have selected the first 100 cases as the main predictor. I mean, why not 90,80, 10? It may cause significant differences in the results. What is the scientific reason behind it?

--Country-level age structure, racial diversity, and socio-economy should be considered as confounders.

-What is the significance and broad impacts if the gap is filled? What is the added value of this work to public health? How the results can help public health decision-makers in practice.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

19 Oct 2020

Answers to Reviewer #1

In general, please proofread this manuscript carefully. There are multiple small typos, missing or extra words, and missing or extra punctuation. Given there are no line numbers, it is unwieldly to note them all in the specific comments below.

RESPONSE: We proofread all the manuscript carefully, corrected all errors and improved the English as needed.

Abstract – background

This sentence is long and unwieldly and should be simplified or separate into two sentences.

RESPONSE: Done. We simplified it.

Abstract – conclusion first line

No comma after “circulation”

RESPONSE: Corrected as suggested

Introduction first paragraph, second sentence

“statement of on” – delete “of”

RESPONSE: Corrected as suggested

Introduction first paragraph, second sentence

Structure of the sentence makes it unclear whether human-to-human transmission or whether the novel coronavirus was first described to the WHO on Dec 31.

RESPONSE: We revised the sentence and clarified it is about human-to-human transmission.

Introduction second paragraph, second sentence

Citations needed

RESPONSE: Reference provided

Introduction third paragraph, second sentence

Should read something like “These measures varied from country to country and included such actions as” – otherwise, it implies all actions were taken in all places, which is not true.

RESPONSE: True. We corrected it following the reviewer suggestion.

Introduction third paragraph, last sentence

“size” is an incorrect word here – perhaps “widespread degree”

RESPONSE: Corrected as suggested.

Page three, final paragraph

Should read “For instance, host angiotensin…” – these are not the only biological differences noted and should not be implied as such.

RESPONSE: Corrected as suggested.

Page four, first paragraph, final sentence

You mention “measures” – to what are you referring?

RESPONSE: We specified the measures under debate (stay at home orders and closure of all businesses)

Page four, second paragraph, final sentence

This sentence is not grammatical nor is it clear.

RESPONSE: We corrected it.

Page five, second paragraph

This sentence is convoluted. Please simplify or separate into two sentences.

RESPONSE: We separated the concept into two sentences.

Page five, fourth paragraph

March 11th was the first 100 cases where?

RESPONSE: This describes the distribution of time interval (days from the date the first 100 cases were documented from January 22). The change of the linear pattern was found to be at interval time 50. This interval time corresponds to March 11th.

It is the main independent indicator in our model and its full name is a bit too long and complex. It creates some difficulties in sentences. In the revised manuscript we clarify the terminology. We also standardize its use throughout the text.

Page six, first paragraph

Is the interval 51-71 or 51-73? It is mentioned as both in the previous paragraphs. Why was 71/73 chosen as a cut off? It appears that all countries reported 100 cases by this time period, if so, please specify in the body of the manuscript.

RESPONSE: 71, in previous paragraph was an error. We corrected it to 73. Yes, the range of the time interval is 51-73 days.

Page seven, final sentence

Was higher or lower life expectancy associated with incidence?

RESPONSE: Higher life expectancy. We specified it in the revised text.

Discussion

Please carefully proofread – there is a lot of non-standard English here.

RESPONSE: We carefully proofread the English.

You need citations throughout the discussion, including for “seeding events”, “intensive global connection”.

RESPONSE: Now we provide more citations, including for seeding and global connection.

Your paragraph on life expectancy and chronic disease in Western v Eastern Europe is difficult to understand. Please clarify.

RESPONSE: Corrected

Page ten, final paragraph

Should be “intra country variations observed in Italy” not “inner”

RESPONSE: Corrected as suggested

General comments

How do you control for different rates and implementation dates of testing in these fifty countries? You do address incidence/mortality but you do not address how implementation dates of testing may wildly vary your date of first 100 confirmed cases per country. This may significantly impact the outcome of your analyses.

RESPONSE: This is an important point. Thank you. Since the 27 of January, all WHO European region countries were included in a COVID-19 standardized surveillance, coordinated by the European Centre for Disease Prevention and Control (ECDC) and the WHO Regional Office for Europe.By the end of January, cross-border inter laboratory systems were in place to arrange testing and reporting of cases. (#3 in the references). We now describe this on page 5, second paragraph.

We also know that middle-income countries outside the European region, such as Iran, where the pandemic seems to have started earlier than in most Europe, had demonstrated capacities to identify COVID-19 cases since early in February. Thus, the ability to detect COVID-19 in Europe existed by the end of January.

In the discussion we also mention the observation of country’s epidemiological COVID-19 curves, which confirm the trend of pandemic initiation timeline in specific countries, as expressed by 100 first official cases. Curbs show that outbreaks in France, Spain and Germany are slightly delayed compared to Italy, while the pandemic in UK starts even later. Additionally, while Russia had already confirmed its first 2 COVID-19 cases at the end of January, the community circulation there (first 100 cases), started only in March. The ‘Russian’ curb reached the peak much later than most countries in west.

A recent relevant publication based on viral genetic sequence data backs the main timeline pattern provided by early confirmed cases in Europe and USA. We cite this now in the discussion on page 14, second paragraph.

(https://science.sciencemag.org/content/early/2020/09/11/science.abc8169)

Despite its limitations, time of 100 first cases seem to be a synthetic and relatively robust indicator for measuring the initiation of pandemic. Other recent relevant publications are using it to indicate early cases.

(https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30581-8/fulltext).

In the revised version we elaborate the issue and the metric more extensively in the methods and discussion. We also cite more relevant references

How do you control for different health-care system utilization across these countries?

RESPONSE: We agree this is an important point. To address this question, we looked at outpatient contact per person per year in the European Region. Upon observation, this variable does not appear to be a meaningful confounder. For example, countries of Ex Soviet Union have the highest per capita utilization of health care, and Nordic countries the lowest. Also, in Belarus, Hungary or Lithuania this indicator is much higher than in most European Union. Yet, utilization rates do not map onto COVID-19 mortality or incidence rates.

Nevertheless, because utilization patters could affect study findings, in the revised version of the paper, we have included this new variable in the model. Neither the pattern nor the magnitude of association between time interval and mortality and incidence rates change, when including this variable.

How do you control for racial differences across these countries?

RESPONSE: In the United States in particular, mortality rates have been higher for certain racial groups, including Blacks, Hispanics and Pacific Islanders. Higher mortality for these groups likely reflects poorer underlying health, lower access to health services and poorer quality of care when health services are accessed. This pattern may also be the case for Europe. However, for race to be a confounder it must be associated with BOTH the independent and dependent variables. While there is certainly evidence of an association between race and mortality, such evidence is lacking for the independent variable:time interval (days from the 22nd of January to the date when 100 first cases were reported). It is hard to imagine how the racial make-up of a country would affect the time between the 22nd of January and the date when the first 100 cases were reported. As such, it does not appear appropriate to control for a country’s racial make-up.

Further, we ran additional models using US data to see if the findings from Europe could be replicated in another context. They were replicated, despite a very different racial/ethnic make-up in the US compared to Europe (more details below).

How do you control for socio-economic differences?

RESPONSE: We estimate that the most important component of socio-economic differences in COVID-19 mortality is health care utilization, instead of indicators such as World Bank GDP per capita. As mentioned above we looked carefully at healthcare utilization patterns and included the new variable in the new model. Similar to the response above, we also ran additional models using US data to see if the findings from Europe could be replicated in another context. They were replicated, despite notable socioeconomic differences between European countries and states in the USA (more details below).

Why do you think there was a statistically significant difference in days 31-50 and not days 51-73?

RESPONSE: We think the non linear nature of the relationship is produced by the fact that in the days after the declaration of world pandemic (corresponding to 11thof March or 51st day in our model), countries were quick to introduce the drastic containment measures. The majority of countries, where pandemic started after mid March, benefitted similarly from the timely measures and the differences among them became insignificant. Google mobility reports confirm quick effects on social distancing of mid March measures. We have now indicated this on discussion, page 10, third paragraph.

We also tested the validity of the model with 50 states of United States of America. It shows a similar pattern (and very similar statistics for the two intervals), with the cut off only about one week later. We also applied the model using the 3rd of October mortality and the direction of the trend remained the same, although the association becomes linear (most probably confirming the weakening of the early effect of spring measures). Here below are the results of our model for USA 50states.

(a) Mortality per million in the U.S. on May 7th

Beta (95% CI) p-value

Time interval (46-59 days) -35.40 (-56.87, -13.93) 0.001

Time interval (60-70 days) -12.10 (-34.55, 10.36) 0.284

(b) Mortality per million in the U.S. on Oct 3

Beta (95% CI) p-value

Time interval (46-59 days) -35.61 (-67.03, -4.20) 0.026

Time interval (60-70 days) -39.21 (-72.06, -6.36) 0.020

We mention these extra analyses in the methods, results, and discussion.

Given that the virus can have an incubation period of up to 14 days post exposure, was there a significant change in viral transmission 14 days after individual countries imposed their containment measures? Was there a change in trend of time to 100 cases in the 14 days after Mar 11?

RESPONSE: We didn’t look at it in this analysis. The ecologic study methodology we have applied has its limitations. We acknowledge it in the revised paper.

Answers to Reviewer #2:

The modeling without validation is not reliable. The authors need to examine whether their model is consistent with independent (test) dataset/recent data or not. With such a small sample size, the associations are not unlikely and not very reliable.

RESPONSE: Thank you. This is an important point. We have limited the analysis to the dataset of European Region Countries for following reasons:

It encompasses a substantial number of countries (16) where early COVID-19 cases start circulating before 11th of March, when pandemic is officially declared.

There was a standard surveillance system put in place by the WHO

During the revision, we tested the validity of the model with 50 states of United States of America (where only 3 states had documented 100 cases before 11th of March). It shows a similar pattern (and very similar statistics for the two intervals), with the cut off only about one week later compared to Europe. We also applied the model using the 3d of October mortality and the direction of the trend remained the same, although the association becomes linear (most probably confirming the weakening of the early effect of spring measures). Here below are the results of our model for USA.

(a) Mortality per million in the U.S. on May 7th

Beta (95% CI) p-value

Time interval (46-59 days) -35.40 (-56.87, -13.93) 0.001

Time interval (60-70 days) -12.10 (-34.55, 10.36) 0.284

(b) Mortality per million in the U.S. on Oct 3

Beta (95% CI) p-value

Time interval (46-59 days) -35.61 (-67.03, -4.20) 0.026

Time interval (60-70 days) -39.21 (-72.06, -6.36) 0.020

We mention these extra analyses in the methods, results, and discussion.

We also encourage other researchers to apply it at country level, using regions as study unit.

We agree that the ecological study method is an issue, as it is the sample size, and have acknowledged it at limitations of revised paper on page 14, third paragraph.

We also agree that datasets based on official information published online, may have issues which can be only partially validated. Nonetheless, they remain an important source of information and are being used in a number of relevant publications.

(https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30581-8/fulltext)

We cite more relevant references in the revised manuscript.

--Page 5 last line: I don’t think model diagnostics are used to improve “model fit”? This is used mostly to check the underlying assumptions. Also, I don’t see any accuracy assessments done for the model.

RESPONSE: We elaborated more on model diagnostics and model fitting.

Model diagnostics were performed to examine normality and influential data points and adjusted R2 statistics were used to assess model performance. Model diagnostics showed that Italy, Belgium and Germany were influential for mortality analysis whereas Luxemburg and Island were influential for incidence analysis using Cook’s D criteria. Residuals for both multiple regression analysis were approximately Normal. Log-transformation of the two outcomes improved the Normality but adjusted R2 were smaller for mortality model. Interpretation of original measures of mortality and incidence were used for all models so that the interpretations were more meaningful.

We reflected this in the statistical methods section and the title of the figure.

The authors need to justify based on what criterion they have selected the first 100 cases as the main predictor. I mean, why not 90,80, 10? It may cause significant differences in the results. What is the scientific reason behind it?

RESPONSE: This is an important point. Thank You. In our model, this metric is crucial to quantify early or delayed start of COVID-19 community circulation.

As pandemic is underway and more research is carried out, the 100 first cases are used in relevant publications to classify early COVID-19 cases.

(https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30581-8/fulltext).

The indicator seems to represent a critical mass of cases documented during initial community circulation. A number of publications have described transmission dynamics in samples of first 100 COVID-19 cases demonstrating community circulation.

(https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7195694/)

(https://www.cdc.gov/mmwr/volumes/69/wr/mm6911e1.htm)

Still, it may seem arbitrary and its use is yet to be validated. We acknowledge this at revised paper limitations.

We also cite more relevant references in the revised manuscript.

Country-level age structure, racial diversity, and socio-economy should be considered as confounders.

RESPONSE: We agree this is an important point. To address this question, we looked at outpatient contact per person per year in the European Region. Upon observation, this variable does not appear to be a meaningful confounder. For example, countries of Ex Soviet Union have the highest per capita utilization of health care, and Nordic countries the lowest. Also, in Belarus, Hungary or Lithuania this indicator is much higher than in most European Union. Yet, utilization rates do not map onto COVID-19 mortality or incidence rates.

Nevertheless, because utilization patters could affect study findings, in the revised version of the paper, we have included this new variable in the model. Neither the patter nor the magnitude of association between time interval and mortality and incidence rates, change when including this variable.

Now, in the revised version of the paper we have included the new variable in the model. The pattern and the power of association don’t change.

Race: In the United States in particular, mortality rates have been higher for certain racial groups, including Blacks, Hispanics and Pacific Islanders. Higher mortality for these groups likely reflects poorer underlying health, lower access to health services and poorer quality of care when health services are accessed. This pattern may also be the case for Europe. However, for race to be a confounder it must be associated with BOTH the independent and dependent variables. While there is certainly evidence of an association between race and mortality, such evidence is lacking for the independent variable: time interval (days from the 22nd of January to the date when 100 first cases were reported). It is hard to imagine how the racial make-up of a country would affect the time between the 22nd of January and the date when the first 100 cases were reported. As such, it does not appear appropriate to control for a country’s racial make-up.

We proxy age structure with life-expectancy at birth.

What is the significance and broad impacts if the gap is filled? What is the added value of this work to public health? How the results can help public health decision-makers in practice.

RESPONSE: The study provides secondary evidence about the effect of public health measures taken in Europe after 11th of March, when pandemic was declared by WHO. It demonstrates that efforts to delay the early spread of the virus, may save daily an average 30 deaths per one million inhabitants.

We think the non linear nature of the relationship between early start and mortality is produced by the fact that in the days after the declaration of world pandemic (corresponding to 11th of March or 51st day in our model), countries were quick to introduce the drastic containment measures. The majority of countries, where pandemic started after mid March, benefit similarly from the timely measures and the differences among them become insignificant.

The results help public health professionals to better understand the pattern of pandemic spread and its relation to pandemic mortality. It also provides systematic evidence against speculations about pandemic reaching its peak since January in countries of East Europe, what explained according to them, the very low mortality in late April.

The resultswill support decision makers during pandemic to take early and swift public health measures in order to assure important health benefits in subsequent months.

We elaborated more extensively this issue in the discussion section of the revised version.

Submitted filename: Response to reviewers COVID time and mortality 15 Oct 2020.docx

Click here for additional data file.

23 Nov 2020

The lower COVID-19 related mortality and incidence rates in Eastern European countries are associated with delayed start of community circulation

PONE-D-20-15377R1

Dear Dr. Ylli,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Yury E Khudyakov, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: (No Response)

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) (Limit 100 to 20000 Characters)

"The authors addressed my comments"

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

25 Nov 2020

PONE-D-20-15377R1

The lower COVID-19 related mortality and incidence rates in Eastern European countries are associated with delayed start of community circulation

Dear Dr. Ylli:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Yury E Khudyakov

Academic Editor

PLOS ONE