Comparison of prediction accuracies between mathematical models to make projections of confirmed cases during the COVID-19 pandamic by country/region.

BACKGROUND: The COVID-19 pandemic caused >0.228 billion infected cases as of September 18, 2021, implying an exponential growth for infection worldwide. Many mathematical models have been proposed to predict the future cumulative number of infected cases (CNICs). Nevertheless, none compared their prediction accuracies in models. In this work, we compared mathematical models recently published in scholarly journals and designed online dashboards that present actual information about COVID-19.
METHODS: All CNICs were downloaded from GitHub. Comparison of model R2 was made in 3 models based on quadratic equation (QE), modified QE (OE-m), and item response theory (IRT) using paired-t test and analysis of variance (ANOVA). The Kano diagram was applied to display the association and the difference in model R2 on a dashboard.
RESULTS: We observed that the correlation coefficient was 0.48 (t = 9.87, n = 265) between QE and IRT models based on R2 when modeling CNICs in a short run (dated from January 1 to February 16, 2021). A significant difference in R2 was found (P < .001, F = 53.32) in mean R2 of 0.98, 0.92, and 0.84 for IRT, OE-mm, and QE, respectively. The IRT-based COVID-19 model is superior to the counterparts of QE-m and QE in model R2 particularly in a longer period of infected days (i.e., in the entire year in 2020).
CONCLUSION: An online dashboard was demonstrated to display the association and difference in prediction accuracy among predictive models. The IRT mathematical model was recommended to make projections about the evolution of CNICs for each county/region in future applications, not just limited to the COVID-19 epidemic.

Many mathematical COVID-19 models have been proposed in the past. None compared model effectiveness and accuracies between models. We applied 2 mathematical models in comparison to help readers understand the nature of mathematical models.

Three models of QE, modified QE (OE-m), and IRT were proposed to compare their accuracies in a short term and throughout the entire year of 2020, which were never seen before in the literature.

The way to visualize the comparison results between models using the forest plot is unique and modern, particularly using dashboards displayed on Google Maps that is worth introducing the features to researchers and readers in academics.

Introduction

The first infection case of COVID-19 was reported on December 30, 2019, in Wuhan city located in Hubei providence, China.[ The World Health Organization (WHO) recognized the disease as a pandemic on March 11, 2020.[ As of September 18, 2021, the world has accumulated >0.228 billion of the COVID-19 cases, threatening people's health, economic development, and social stability.[ We can observe an exponential growth of the disease based on the daily increment of confirmed cases.[

The need to build a prediction model during the COVID-19 epidemic

Based on the increasing COVID-19 data, it is necessary to analyze the evolution of the disease and create a prediction model to observe the trend of confirmed cases. Although many mathematical models[ have been suggested to predict the number of COVID-19 cases, none compared the prediction accuracy between models in the literature. The difficulty might be attributable to authors who are unfamiliar with algorithms developed by other authors. That is, the author who conducts a comparative study on 2 COVID-19 models should (and must) truly understand both predictive models thoroughly.

Furthermore, children and women of reproductive age might be disproportionately affected by the disruption of routine health services, particularly in low-income and middle-income countries (LMICs).[ The indirect effects of the COVID-19 pandemic on maternal and child mortality in LMICs should be properly modeled and early estimated using an appropriate way.

Two mathematical models are required for comparison in prediction accuracy

One of those mathematical prediction models has applied the quadratic equation (QE) to demonstrate the projected cases in Colombia, and deaths in Russia, India, and worldwide using the past 31-day data up to May 29, 2020. In which the constrained term is set at the middle point (i.e., P(x2,y2)) of the observations in the QE model[ that assumed the exponential growth during the COVID-19 epidemic. For instance, Colombia and India were increasing, Russia and the worldwide presented decreasing in a growth trend with exponential (i.e., increasing) and logarithmic (i.e., decreasing) phenomena prior to May 29, 2020.

Because a constrained term was set at the middle point (i.e., P(x2,y2)) of the observations to build the QE model,[ a modified QE model (named QE-m without constraint in the model) was conceived to compare whether the QE-m model earns smaller residuals than the QE model. The evidence is required for verifications.

Another study[ applied item response theory (IRT)[ to construct an ogive curve and determine the inflection point (IP) used for predicting the projected cases by country/region based on the cumulative number of infected cases (CNICs). We are motivated to compare the prediction accuracy among these 3 models (i.e., QE, QE-m, and IRT). The model accuracy is referred to R (i.e., explained variance in model = 1 – residuals/total squared deviations from the mean of observations) across all countries/regions.

An online dashboard developed to display a projection of future CNICs

The Florez and Singh[ stated that it is important to analyze the evolution of the disease in order to make decisions that tackle the growth rate of cases. An online dashboard was thus designed and developed in their work (called COVID-19 dashboard[), displaying a projection in order to estimate the future CNICs by country based on the actual observed CNICs. We are thus interested in mimicking the development of an APP for showing the actual CNICs and the projections of the future CNICs on an online dashboard.

The aims of this study

To achieve this projection of the growth rate of COVID-19 cases, we designed those 3 mathematical models (i.e., QE, OE-m, and IRT mentioned above) based on equivalently previous CNICs to conduct a comparative study.

Two phases are involved in this study, including comparison of prediction accuracy among the 3 models and application of ipcase-index[ and angle-index used to design an online dashboard laid on Google Maps.

Methods

Data source

We downloaded COVID-19 outbreak CNICs between January 1 and Febuary 16, 2021, as well as the entire year in 2020, from GitHub,[ a site that provides information on new CNICs in countries/regions around the world. All the downloaded data with 299 countries/regions, including the US States and provinces/metropolitan cities/areas in China, are publicly displayed on the website.[ Ethical approval is not necessary for this study because all the data are obtained via the Internet (see Supplemental Digital Content 1).

Two prediction models used in this study

The QE model

The QE is described in the left panel of Fig. 1. To perform the projection for estimating future cases, the QE model refers to a quadratic equation[ based on data of present-day and previous several days (e.g., 30 days). With the QE, projected cases for the next days up to several days (e.g., 10 days) can be obtained and displayed on a dashboard, similar to that[ in the previous study.[

Figure 1

Two typical predictive models used in this study (note that the 2 curves show the distinct difference with and without infection point on the curve).

In the QE model, let 3 points P1 = (x1, y1), P2 = (x2, y2), and P3 = (x3, y3) on the trajectory CNICs. As such, y1 denotes the corresponding number of cases on the day 1 (i.e., x1 = 0), y2 corresponds to the CNICs on the middle (e.g., x2 = 23 if the length of days is 47), and y3 corresponds to the CNICS on the present day (e.g., Febuary 16, 2021, in this study and x3 = 46). The 3 quadratic equations from (1) to (3) are defined at the top-left part in Fig. 1. The model parameters (i.e., a, b, and c) are derived from the 3 equations; see the calculations at the left part in Fig. 1. One example of the observed and expected CNICs in the United Kingdom is illustrated in a logarithmic (i.e., decreasing) growth trend at the bottom-right corner in Fig. 1.

The IRT model

The IRT-based COVID-19 model was proposed in the previous study.[ The probability function is present in Equation (11), shown at the top-right corner in Fig. 1. The parameters a and b would be calibrated for each country/region using Microsoft Solver Add-in.[

When viewing the 2 curves in Fig. 1, the different assumptions exist in the 2 models, including the sigmoid (or logistic) function for IRT and the exponential (or logarithmic) growth for QE.

Another QE-m model derived from the QE model

Due to a constrained term was set at the middle point (i.e., P(x2,y2) to create the QE model),[ the unconstrained QE model (i.e., QE-m) was thus proposed to compare the model accuracy in the 2 QE and IRT models. A hypothesis was made to the QE-m model having a smaller residual than the QE model due to no such constrained condition in the QE-m model. We adopted a similar technique of calibration for model parameters using the Microsoft Solver Add-in tool as the IRT model in the previous study.[ Three steps were applied to calibrate the model parameter a and b in the QE-m model:

Objective

To minimize the total residuals using the Microsoft function below:

where Oi denotes the observed CNIC and Ei represents the expected CNIC via the model equation (i.e., ), where c = y1 (=CNIC at day 1) according to Equation (10) in Fig. 1.

Parameters to estimation

Parameters (i.e., a and b) in Equation (13) are calibrated in the model.

Constrained terms

The parameter c is set at y1 according to the Equation (10) in Fig. 1.

Comparison of model residuals

The observed CNICs were compared with the corresponding expected CNICs using Equation (12) in Step A.

Perform the Solver add-in

The Microsoft Solver add-in[ was used to estimate the model parameters (Supplemental Digital Content 1). The QE projected curve can be plotted to predict the potential CNIC in the future.

Comparison of the 2 models

Two comparisons were made to evaluate the prediction accuracy in models, including association of R between models using correlation coefficient (CC) and difference in model R using paired-t test. (The t value for CC was calculated by the )[ and analysis of variance (ANOVA).

The R is defined in Equation (13):

where n is the observed days (e.g., 47 in this study), Oj is the observed CNIC at ith day, Ei is the expected CNIC at the ith day, and y-bar denotes the mean observed CNIC in a country/region.

The Kano diagram[ was used to interpret the association of R2 between models. Three parts are classified in color, including the one-dimensional category in the middle, the attractive part at the top, and the must-be area at the bottom.

An oline dashboard designed for displaying the projection of future cases

A choropleth map[ was used to compare the ipcase-index[ (based on IP days and the corresponding CNIC to represent the negative impact hit by COVID-19,[ see Equation (14)) for a specific country/region.

Another index named as angle-index is defined in Equation (15):

where both Degrees() and Atan() are derived from the functions in Microsoft Excel, the IP denotes the inflection point[ on the trajectory of ΔCNIC (e.g., ΔIP = 7 days, ΔCNIC = 27,100–11,177 = 15,934, ratio = 15,934/(7–1) = 2563, ). The angle-index is from 0 to 90, the higher means ODCOVID is severely hit by COVID-19 in a given country/region.

Statistical tools and data analysis using Kana diagram and forest plot

A visual representation displaying the comparison of prediction accuracy across continents/regions was plotted on the Kano diagram.[ The projections were drawn using QE and IRT models, respectively, on their dashboards. The forest plot[ was applied to display the difference in R2 between the 2 models during the entire year in 2020.

Results

Prediction Curves between the QE and IRT models

To know the growth trend of CNICs by country, a dashboard displays a projection for future CNICs based on the 2 mathematical models.

Figure 2 presents projected cases in Colombia, Russia, India, and Hebei Province in China. Black corresponds to the actual cases collected in data,[ while the green curve corresponds to the projected CNICs, and the yellow to the predictive CNICs. These projections performed on February 16, 2021, produced the quadratic equations shown in the left panel in Fig. 2, and other counterparts in the right panel using the IRT model.

Figure 2

Examples of model residuals based on the 2 models.

It is worth noting that the residual is huge in Hebei Province in China due to the observed CNICs not totally fitted to the QE model when compared with the IRT model in the right panel in Fig. 2. The reason might be the inflection point a at the earlier stage in China. We can be expected the pandemic situation with the form of ogive curve making the difference clearly in model residuals between the 2 study models. The longer period of infected days would be illustrated in the next section using the data in an entire year of 2020.

Comparison of prediction accuracies among the 3 models

We observed that the correlation coefficient was 0.48 (t = 9.87, n = 265) between the 2 QE and IRT models based on R2 when modeling CNICs dated from January 1 to February 16, 2021. Most bubbles (n = 235) in yellow are within the unidimensional category. In Fig. 3, a few bubbles in green (n = 19) and red (n = 11) are outside the upper and lower limit lines in the Kano diagram. Bubbles are sized by the R2 in the QE-m model. The top R2 are Cuba, Malaysia, Saudi Arabia, and others with R2 = 1.0. Readers are invited to scan the QR-code in Fig. 3 and click on the bubble of interest to see the respective trend curve on the IRT model.

Figure 3

Comparison of R-square for 2 models.

A significant difference in R2 was found (P < .001, F = 53.32) in mean R2 of 0.98, 0.92, and 0.84 for IRT, QE-m, and QE, respectively. The IRT-based COVID-19 model is superior to the QE-m and QE in R2 when modeling COVID-19 for countries/regions. In Table 1, we can see that the QE-m model (mean R2 = 0.92) is also superior to QE model (mean R2 = 0.84). The hypothesis is supported by the finding that the QE-m model having a smaller residual (or a greater R2) than the QE model.

Table 1

Correlation and difference among models.

Mean/Corr.	IRT	QE	QE-m
IRT	0.98	8.87	9.11
QE	0.48	0.84	8.13
QE-m	0.52	0.29	0.92

F = 53.32; P < .001; variance in upper; CC in bottom; mean CC across the diagonal. IRT = item response theory, QE = quadratic equation.

Comparison of ipcase-index and angle-index on Choropleth maps

Online dashboards were designed for distribution of ipcase and angle indexes in Figs. 4 and 5. Six colorful labels for regions are on the choropleth map. The darker means the negative impact severely hit by COVID-19. The top 3 of India, Brazil, and Russia are marked by 3 blue lines. Readers are suggested to scan the QR-code in Figs. 5 and 6. When the region is clicked, the growing trend of COVID-19 cases on either QE or IRT model using CNIC data from January 1 to Febuary 16, 2021, and the entire year in 2020, respectively, The results are almost similar using the ipcase-index and the angle index when oberving the choropleth maps in Figs. 4 and 5.

Figure 4

Distribution of ipcase-index on negative impact hit by COVID-19.

Figure 5

Distribution of angle-index on negative impact hit by COVID-19.

Figure 6

Comparison of model accuracy in R2 across continents and countries in the study 2 models.

Comparison of ipcase-index on a Choropleth map

Comparisons in R2 (in the entire year in 2020) are made in Fig. 6 using the mean difference method used in the forest plot. China earns the best-fit than other counterparts in continents and the United States using the IRT model. When comparing the model accuracy in R2 between the 2 models, the IRT model is superior to the QE model across all continents and the 2 countries in the United States and China.

From Fig. 6, we can see that the pandemic stationarity (e.g., China) is not suitable for the QE model, but for the increasing stage such as in the United States.

Online dashboards shown on Google maps

All those line plots in Fig. 2 would appear once the bubble in Fig. 3 or the region in Figs. 4 and 5 is clicked using the links.[ The Fig. 6 is referred to the linkp30. All of these dashboards are laid on Google maps.

Discussion

Principle findings

In the current study, we observed that the CC was 0.48 (t = 9.87, n = 265) on R2 between the 2 QE and IRT models when modeling the CNICs for countries/regions. As could be expected, a significant difference in R2 was found (P < .001, F = 53.32) in mean R2 of 0.98, 0.92, and 0.84 for IRT, QE-m, and QE, respectively. The IRT-based COVID-19 model has a substantially higher prediction power than the QE-m and QE models, particularly in a longer perfiod of infeted days(e.g., in an entire year). The advanteage of IRT model is obviously due to the ogive curve fitting to the COVID-19 situation better than the OE trend.

An online dashboard on projections of COVID-19 cases was displayed for each county/region using either QE or IRT-based mathematical models once the region of interest on the choropleth map has been clicked.[

Two effects of model R2 were found, including the optimization effect (=0.92–0.94 = 0.08 between QE and QE-m models; i.e., minimizing the residual in parameter estimation) and the assumption effect (=0.98–0.92 = 0.06 between QE-m and IRT models; i.e., exhbiting the differece in exponential and logistic growth of the disease outbreak).

What this finding adds to what we already knew

COVID-19 has required analyses of data regarding NCICs reported worldwide and by country/region using an online dashboard.[ Some COVID-19 dashboards developed by prestigious organizations, such as the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU),[ the WHO,[ Leszkiewicz personal dashboard,[ HealthMap,[ and the dashboard created by Schiffmann who is an 18-year-old high school senior from Washington State in the United States,[ are full of interest and charm with visualizations for reporting the current state of the COVID-19. Nevertheless, these dashboards lack important information and analysis on making a projection about the evolution of CNICs during the COVID-19 epidemic. As authors Ivanković et al[ stated in their reseach, only a handful of dashboards went further and employed predictive analytics by illustrating different future scenarios of “what could happen.” The lack of precision of predictive models and simulations early in the pandemic likely stunted their use. Use of both descriptive and predictive approaches to dashboard design and tighter links between infection control policies and their effects should be further explored into the next phases of the pandemic.

The COVID-19 dashboard[ was designed to offer additional valuable information (e.g., using a mathematical model to project future cases worldwide and by country). The QE mathematical model is based on the exponential growth of the disease with a QE instead of the sigmoid (or logistic) function with an IRT.[

Althouhg Liu et al[ found that the CNICs are well fitted by an exponential function in the 31 provincial-level regions in China, Lynch and Gore[ in their research found that exponential growth is not the best representation of case growth during early onset. Similarly, Adebowale et al[ in their study on the spread of COVID-19 outbreak in the first 120 days in Nigeria and 7 other countries found that the cubic polynomial model (CPM) provided the best fit for predicting COVID-19 CNICs across all the countries, but there was a clear deviation from the exponential growth model. Accordingly, the exponential growth is not totally suitable for the COVID-19 situation. Another more appropriate assumption for the CNIC trend is required for explorations.

A real-time policy dashboard to aid global transparency in response to COVID-19 has been expected to present the localized reasoning behind COVID-19 policy decisions and allow the global health community to provide further support to governments and international stakeholders.[ As such, dashboards barely presenting real-time descriptions of new daily cases and risk factors are insufficient.[ A reliable mathematical model used to predict COVID-19 cases is thus important and necessary to the epidemiology and government in the world.

The IRT-based COVID-19 model has been proposed in the previous study.[ Only 2 model parameters are required to be estimated, as did the QE model.[ The 2 unique difference between the 2 models (i.e., QE and IRT) are the 2 influential effects of optimization (=0.08) and assumption (=0.06) in model R2. Of them, the optimization effect between QE and QE-m models has been shown in Table 1.

Although a modeling study has been applied to early estimates of the indirect effects of the COVID-19 pandemic on maternal and child mortality in LMICs[ based on the assumption of severe acute respiratory syndrome at a 23.9% reduction in ambulatory care and a 35.2% reduction in inpatient care observed in Taiwan,[ the pandemic to affect the health of women and children is expected to yield accurate results through the IRT model along with the MS Excel module as we did in this study.

What IS implied and what should be changed

Two major topics were verified in this study, including whether the IRT-based COVID-19 model is better than the counterparts of QE and QE-m model and how a dashboard that makes projections of COVID-19 cases can be used for epidemiologists and policymakers.

Although numerous mathematical models[ have been applied to predict the CNICs, none compared the prediction accuracy between (or among) models and further developed an online dashboard to make projections of COVID-19 cases for each county/region as we have illustrated in this study. The difficulty might exist for authors who need to implement and compare the unfamiliar algorithms developed by others in their research. Few article authors are willing to provide their detailed experimental data and algorithms to readers who hope to replicate the study on their own. Nonetheless, relevant studies are encouraged to compare predictive models of COVID-19 or other epidemic crises using our suggested approaches provided in Supplemental Digital Content 1, including data, codes, MP4 videos, and the MS Excel module, which is rarely seen before in the literature.

Furthermore, >30 articles were searched in PubMed using the search string of “online dashboard.”[ In which, only 9 articles were invloved with the keywords “online dashboard and COVID-19”[ in title. We demonstrated the online dashboards of COVID-19 in all figures. Not only are the choropleth map and the Kano diagram displayed on dashboards, but the predicted curves are also laid on Google Maps.

The dashboard provided with IP days and the corresponding confirmed cases on the given ogive curve is modern and innovative. Visual displays on CNIC using the IRT model were designed and developed, particularly with the Kano model, to present the relationship between 2 (or more) variables on a diagram. In which, we can see that red bubbles are toward the X-axis feature, the green toward the Y-axis attribute, and the red meeting the requirement of one-dimensional characteristic.

Strengths of this study

The 3 main strengths of the current study include the model R2 was applied to compare the prediction accuracy shown on the Kano diagram, the 2 effects of optimation and assumption were evident to make the model residuals as small as possible using statistical methods (e.g., the Solver Add-in tool in Microsoft Excel), and an app was developed for understanding the growth trend of COVID-19 across countries/regions using the mathematical IRT model embedded in dashboards.

Limitations and future studies

Our study has several limitations. First, many case numbers of COVID-19 are originated from different statistical offices such as the CDC in the United States of America and WHO. This particular agency has provided scientifically highly relevant data since its inception. However, other countries’ related agencies may not have such a track record. Hence, the scientific soundness of this database needs to be established and verified further. The data source of this study was from GitHub,[ which is similar to the use of data in JHU.[ Fortunately, the JHU dashboard's 5 authoritative data sources include WHO, US CDC, National Health Commission of the People's Republic of China, European Centre for Disease Prevention and Control, and the Chinese online medical resource DXY.cn. The corresponding data repository is accessible as Google sheets in GitHub[ (similar to ours[).

Second, although we defined the CNIC as the proxy of negative impact hit by COVID-19 on countries/regions using the ipcase-index in comparison, research is suggested to include deaths to gain the contrast impact by COVID-19 using ipcase-index in the future.

Third, the Microsoft Solver add-in used to estimated IP days in the IRT model is not a unique approach. Many other methods might have better the model R2 than the IRT model. They are worthy of comparison in the future.

Fourth, visual dashboards are shown on Google Maps. However, these installments are not free of charge. For example, the Google Maps application programming interface (API) requires a paid project key for the cloud platform. Thus, the limitations of the dashboard are that it is not publicly accessible, and it is difficult to mimic by other authors or programmers for use in a short period of time.

Fifth, although IRT is common and popular in the educational and psychometric fields, many readers in public health are unfamiliar with IRT. The IRT model consists of 2 parameters that need some effort to understand through data and MP4 videos provided in Supplemental Digital Content 1. We ensure that readers who are familiar with MS Excel can easily replicate the study on their own in the future.

Sixth, only the period from January 1 to Febuary 16, 2021 was included in this study. The model validation and comparison would be examined further involving other periods of infected days in the future.

Finally, the 2 mathematical models[ combined with the QE-m model were compared in this study. Whether other mathematical models[ can yield smaller model residuals than the IRT-based COVID-19 model is required for further verification.

Conclusion

Two major topics are covered: confirming the R2 in IRT model better than the counterparts in the QE and QE-m models and presenting the visual dashboard available for making projections of COVID-19 cases on Google Maps. Two hypotheses are supported: the QE-m model has a lower residual than the QE model, and the QE (or QE-m) model has a higher residual than the IRT model. The IP days can only be calculated using an ogive curve and not quadratic or exponential functions. The ODCOVID can then be measured using the ipcase-index and the angle-index.

It is recommended that other COVID-related mathematical models be used to compare model precision with the IRT model, particularly in a longer period of infected days.

Besides the model R2 used for measuring model accuracy, using the mean absolute percentage error (MAPE) to calculate the prediction power on the basis of 2 time-frame scenarios (i.e., using training CNICs to predict the testing CNICs) between models (e.g., the residuals between the expected trend and the observed line shown in Fig. 2) is expected to be carried out in the future.

An app designed was developed to present the COVID-19 growth trend under the IRT model. An MP4 video in Supplemental Digital Content 1 contains more information about the application that can be clearly learned on the research process in this study. An online dashboard showing future COVID-19 cases for each county/region is recommended for epidemiologists and policymakers in research and practices and not just for the COVID-19 pandemic.

Acknowledgments

The authors thank Enago (www.enago.tw) for the English language review of this manuscript.

Author contributions

Kang-Ting Tsai developed the study concept and design. Yu-Tsen Yeh and Ju-Kuo Lin analyzed and interpreted the data. Willy Chou monitored the process of this study and helped in responding to the reviewers’ advice and comments. Tsair-Wei Chien drafted the manuscript, and all authors provided critical revisions for important intellectual content. The study was supervised by Willy Chou. All authors read and approved the final manuscript.

Conceptualization: Kang-Ting Tsai, Ju-Kuo Lin.

Formal analysis: Yu-Tsen Yeh.

Investigation: Willy Chou.

Methodology: Tsair-Wei Chien.