Comparison of prediction accuracies between two mathematical models for the assessment of COVID-19 damage at the early stage and throughout 2020.

BACKGROUND: The negative impacts of COVID-19 (ImpactCOVID) on public health are commonly assessed using the cumulative numbers of confirmed cases (CNCCs). However, whether different mathematical models yield disparate results based on varying time frames remains unclear. This study aimed to compare the differences in prediction accuracy between 2 proposed COVID-19 models, develop an angle index that can be objectively used to evaluate ImpactCOVID, compare the differences in angle indexes across countries/regions worldwide, and examine the difference in determining the inflection point (IP) on the CNCCs between the 2 models.
METHODS: Data were downloaded from the GitHub website. Two mathematical models were examined in 2 time-frame scenarios during the COVID-19 pandemic (the early 20-day stage and the entire year of 2020). Angle index was determined by the ratio (=CNCCs at IP÷IP days). The R2 model and mean absolute percentage error (MAPE) were used to evaluate the model's prediction accuracy in the 2 time-frame scenarios. Comparisons were made using 3 visualizations: line-chart plots, choropleth maps, and forest plots.
RESULTS: Exponential growth (EXPO) and item response theory (IRT) models had identical prediction power at the earlier outbreak stage. The IRT model had a higher model R2 and smaller MAPE than the EXPO model in 2020. Hubei Province in China had the highest angle index at the early stage, and India, California (US), and the United Kingdom had the highest angle indexes in 2020. The IRT model was superior to the EXPO model in determining the IP on an Ogive curve.
CONCLUSION: Both proposed models can be used to measure ImpactCOVID. However, the IRT model (superior to EXPO in the long-term and Ogive-type data) is recommended for epidemiologists and policymakers to measure ImpactCOVID in the future.

The Angle index was introduced for ImpactCOVID evaluation in the future.

A comparison of differences in indicators using forest plots is recommended for future research.

Determining IP days is important in epidemic research. The comparison was conducted between the difference in resulting and recommended IP days for future research and the ImpactCOVID assessed using the angle index.

1. Introduction

In the field of epidemiology, an “outbreak” refers to a sudden increase in occurrences of a disease at a particular time and place. For comparison, a “pandemic” is defined as a near-global disease outbreak in which multiple countries across the world are infected.[ The cumulative numbers of confirmed cases (CNCCs) are a commonly used indicator to assess the negative impacts of the COVID-19 outbreak (ImpactCOVID for short).[ However, this practice might be questionable because CNCCs do not involve the length of infected days to control COVID-19 (denoted by LID). This concept is reflected in the inflection point (IP) days on the CNCC curve, wherein the curvature changes its sign from an increasing concave (concave downward) to a decreasing convex (concave upward) shape or vice versa.[ Hence, combining CNCCs and IP-based LID for measuring ImpactCOVID is reasonable and necessary.

1.1. Literature review

1.1.1. Using IP and CNCCs to assess ImpactCOVID.

If CNCCs and IP-based LID are considered, then ImpactCOVID (= CNCCs ÷ LID) is similar to the journal impact factor (i.e., IF = citations ÷ publications). Many metrics, such as the h- and x-indexes, have been proposed to improve the drawbacks of impact factors.[ The h-index is determined by the maximum square that fits under an author’s citation curve when the numbers of citations are plotted in decreasing order, and the x-index is determined by the maximum area rectangle that fits under the curve.[ All excessive citations and publications are excluded from the metric computation.

The IPcase index, which uses the rectangular area (“core area”) multiplied by the IP days and the CNCCs, has been proposed to identify ImpactCOVID.[ However, this index has 2 drawbacks: it has an identical value but a distinct meaning (e.g., IPcase index = CNCC × IP = height × width = 100 = 4 × 25 = 25 × 4 = 10 × 10), and it does not denote the momentum of a sudden ImpactCOVID. Thus, the improved angle index is defined as follows:

where Degrees() and Atan() are derived from the functions in Microsoft Excel. For instance, ∆IP = 7 days, ∆CNCC = 27,100 – 11,177 = 15,934, ratio = 15,934/(7−1)= 2563, θ = Degrees(Atan(2653.8)) = 89.97. The angle index ranges from 0 to 90, wherein a high θ value means a great negative impact (“severely hit”) by COVID-19 in a given country or region. The premise is to determine the IP days (similar to the LID defined in the previous section) before calculating the angle index for a specific country/region.

1.1.2. Using mathematical models to determine the IP.

In general, the mean number of confirmed cases across varying periods (number of days) yields significantly different IP days, even though the daily number of confirmed cases (computed based on the previous 7 days) can be applied to estimate the observed IP days.[ In a similar situation, using a mathematical model to determine the expected IP days is more objective than traditionally observing the IP days on the mean number of confirmed cases; the latter is typically but irrationally employed in practice.[

Although many mathematical models[ have been proposed to predict the number of COVID-19 cases, none (except the one using item response theory [IRT][) have been applied to determine the IP days on the CNCC curve. Nonetheless, the differences in model accuracy (MA) and prediction power (PP) should be compared. MA and PP can be measured by the R2 model and mean absolute percentage error (MAPE), respectively (see the Methods section for details on R2 and MAPE).

Therefore, we applied MA and PP as indicators in the 2 COVID-19 models (i.e., exponential growth (EXPO) and IRT)[ and used the angle index to measure ImpactCOVID for each country/region.

1.2. Main Goals

This study aimed to compare the differences in MA and PP between the 2 proposed COVID-19 models, develop an angle index that can be objectively used to evaluate ImpactCOVID, compare the differences in angle indexes across countries/regions worldwide, examine the difference in determining the IP on the CNCC curve between the 2 mathematical models, and compare goodness-of-fit measures for the 2 models.[

2. Materials

2.1. Data source

COVID-19 data have been obtained from GitHub[ for each country/region[ (see Supplemental Digital Contents 1, http://links.lww.com/MD/G983 and 2, http://links.lww.com/MD/G984). The downloaded data are publicly available on the website.[ It was not necessary to obtain ethical approval for this study since all data were obtained from the GitHub website.

2.2. Introducing the mathematical models

2.2.1. The EXPO model.

In the study of natural science, nonlinear regression and iterative methods are commonly employed.[ As part of the COVID-19 prediction model, EXPO has been proposed,[ which is based on the daily growth rate (GR) of confirmed cases using Eq. 2:

where case and case are the daily number of confirmed cases on days n and n − 1, respectively. The GR can be modeled in Eq. 3, where a is a constant representing the growth rate at t = 0, β is an attenuation coefficient that indicates the effectiveness of government isolation and quarantine, and t represents the evolution of the epidemic.[ Based on Eq. 2, nonlinear regression and iterative methods can be constructed to predict the CNCCs using Eq. 4:

where CNCC_0 and CNCC are the CNCCs at t = 0 and the expected CNCC at t = k, respectively (see Supplemental Digital Content 1, http://links.lww.com/MD/G983).

2.2.2. The IRT Model.

The item response model (IRT) was proposed in 2021[ using Eq. 5:

The a and b parameters represent the discrimination (i.e., a slope from 0 to 4) and the difficulty (i.e., a value from −5 to 5, with the left value indicating that the outbreak started earlier and the right value indicating that it lasted longer).[ The number of infected days denoted by θ is standardized to range between −5 and 5 (see Supplemental Digital Content 2, http://links.lww.com/MD/G984).

2.2.3. Model parameter estimations.

The model parameters can be estimated by minimizing the residual using the Microsoft add-in tool.[ Details are provided in Supplemental Digital Contents 1, http://links.lww.com/MD/G983 and 2, http://links.lww.com/MD/G984 (e.g., executing the procedure, SolverSolve UserFinish: = True, ShowRef: = “ShowTrial,” using visual basic for applications, VBA).[

2.2.4. Analyzing model accuracy and prediction power.

Both R2 and MAPE statistics were applied to evaluate the model accuracy (MA) and prediction power (PP) in the 2 time-frame scenarios (see the training and testing samples shown in Fig. 1). We applied the model parameters calibrated in the training sample to predict the CNCCs in the testing sample by observing the 2 measures of R2 and MAPE. The R2 and MAPE are defined in Eqs. 7 and 8:

Figure 1.

Study flowchart.

where O denotes the observed CNCCs and O-hat is the mean CNCC in a given country/region. The model residual is computed by Eq. 8. E represents the expected CNCCs.

where O is the observed CNCC and E is the predicted CNCC. The absolute value in Eq. 8 is summed across all predicted points in days and divided by the number of fitted points n.

2.3. Three tasks using forest plots in model comparisons

The forest plot[ was used to examine the difference in R2 and MAPE. The first task was to compare the effects of R2 and MAPE in the 2 models at the early outbreak stage. The second task was to observe their difference in R2 and MAPE throughout 2020. The 3 tasks were to compare their IP days on vaccination data from January 1 to November 6 in 2021. The angle index defined in Eq. 1 was used to measure ImpactCOVID (see the example in the Introduction).

We used choropleth maps[ to present the angle indexes for countries/regions at an early stage and throughout 2020. The darker color in countries/regions indicates more severe impacts as measured by ImpactCOVID. On clicking a colored region, line plots that are used to predict the future CNCCs are generated.

2.4. Task 4: goodness-of-fit measures for the 2 models

Using the goodness-of-fit measures shown in Table 1, 5 indicators were compared between the 2 models.[ The area under the curve (AUC) is defined by the regression R2 (i.e., the determination coefficient computed by plotting the Pearson correlation coefficients between the 2 variables of the last 7-day predicted and observed confirmed cases in the 2 scenarios of the 20-day early outbreak stage and throughout 2020) and drawn on a scatter plot with the 95% control lines[ for countries/regions based on the IRT and EXPO mode ls.

Table 1

Comparison of goodness-of-fit measures for the models.

Indicator	Formula
AUC	=CC*CC
Residual	=∑(i = 1 to n)[(Oi– Ei)^2]
AIC	=-2 ln(L) + 2 k = n*ln(SSE/n)+2k
BIC	=2 ln(L) + ln(n)*k
Brier score	=∑(i = 1 to n)[(Oi – Ei)^2]/n

The residuals on the two 7-element variables were computed by Eq. 8. The other 3, including the Akaike information criterion,[ the Bayesian information criterion,[ and the Brier score,[ are compared. A smaller value means better modeling of the COVID-19 pandemic.

2.5. Statistical tools and data analysis

The mean and standard deviation (SD) were extracted to compare the standardized mean difference (SMD) in the forest plot. A significance level of type I error was set at 0.05.

Visual displays of the forest plot and choropleth map illustrate the comparison between MA and PP. The angle indexes were plotted online on Google Maps. The parameter estimation was executed in Microsoft Excel[ (Supplemental Digital Contents 1, http://links.lww.com/MD/G983 and 2, http://links.lww.com/MD/G984). The study flowchart is shown in Figure 1.

3. Results

3.1. Task 1

3.1.1. Model comparisons at the early outbreak stage.

An example of Guangdong in China and New York in the United States was used to illustrate the comparison. The results show that the EXPO model is superior to the IRT model when examining the trajectories of CNCCs at the early stage. As shown in the top panel in Figure 2, in Guangdong, China, model R2 = 1.0, 0.99 and MAPE = 0.02, 0.06 for the EXPO and IRT models, respectively. In comparison to New York (US), model R2 = 1.0, 1.0 and MAPE = 0.01, 0.29 for the EXPO and IRT models, respectively.

Figure 2.

Comparison of accuracy among the proposed models.

The following figure shows a comparison between R2 (at the earlier 20-day stage) and MAPE (at the following 7-day stage). We note that there is no significant difference in R2 or MAPE between these 2 models when measuring ImpactCOVID on continents at the early outbreak stage.

3.1.2. Comparison of Angle Indexes at the Early Outbreak Stage.

In Figure 4, we present the angle indexes based on the early outbreak stage of 20 days. Higher angle indexes are observed in Hubei Province (including Wuhan) in China as well as in New York and New Jersey in the United States. Darker colors indicate countries severely affected by COVID-19, including Iran and Turkey. The angle indexes were calculated using Eq. 1 below. The reader is encouraged to scan the QR code in Figure 4 to examine the Ogive curves for countries/regions when the color region is clicked.

Figure 4.

ImpactCOVID based on the angle index at the earlier stage. ImpactCOVID = impacts of the COVID-19 outbreak.

3.2. Task 2

3.2.1. Comparison of the EXPO and IRT models for 2020.

Modeling the CNCC data in the EXPO and IRT models to estimate their parameters in the long run revealed that the IRT model has significantly higher R2 and smaller MAPE values, as shown in Figure 5. Forest plots were generated by computing the respective pair statistics of the mean and SD values in R2 and MAPE, as shown in the upper and lower panels, respectively.

Figure 5.

Comparison of the R2 and MAPE values between the EXPO and IRT models using forest plots. EXPO = exponential growth, IRT = item response theory, MAPE = mean absolute percentage error.

Currently, only South America and the United States have the same R2 (>0.90) between the 2 models; see the upper panel of Figure 5. In other continents and in China, the IRT model has a higher R2 and a smaller MAPE than the EXPO model; see Figure 5 for a higher R2 in the left column (which favors the IRT) and a smaller MAPE in the right column (which also favors the IRT).

3.2.2. Comparison of angle indexes in the long run.

Figure 6 depicts the angle indexes based on the 2020 data. The most negatively impacted countries were India, the United Kingdom, and the United States (specifically California). In Figure 7, the angle is computed by the ACNCC (=CNCC at IP * CNCC at IP-6) divided by the LID (=6). In this regard, ImpactCOVID reflects the meaning of the disease outbreak since a 7-day incubation period is used to compute the angle index.

Figure 6.

Top 3 countries hit by ImpactCOVID based on the angle index (top) and the comparison of modeling effects between the EXPO and IRT models (bottom). EXPO = exponential growth, ImpactCOVID = impacts of the COVID-19 outbreak, IRT = item response theory.

Figure 7.

Comparisons of IP day yields by the 2 models using line-chart plots. IP = inflection point.

Comparison of R2 and MAPE across continents, China, and the US based on the 3 models. MAPE = mean absolute percentage error.

The reader is also invited to scan the QR code in Figure 6 to view the ogive curves for countries/regions of interest when the color region is clicked. For example, the EXPO and IRT models are used to compare the 2 ogive curves of Hubei Province (including Wuhan) in China. Even though the 2 curves appear similar, the projection curve in the IRT model (shown at the bottom of Fig. 6) has a higher R2 and a smaller MAPE than the EXPO model (0.99 vs 0.98 and 0% vs 0.2%, respectively).

3.3. Task 3

3.3.1. IP days in these 2 models on vaccination uptake per 100 people.

In Figure 7, line-chart plots were used to compare vaccination rates per 100 people. Based on Figure 8, it can be seen that the IRT model performed better than the EXPO model for the CNCCs with a given Ogive curve (e.g., China and Algeria) and different IP days resulted from the 2 models (Fig. 8A), which significantly influenced the computation of ImpactCOVID using the angle index.

Figure 8.

Comparisons of IP day yields by the 2 models across continents and IP comparison among continents. IP = inflection point.

3.3.2. Comparison of IP days using the IRT model.

Figure 8B shows that Europe has the shortest IP days, followed by the United States and South America. In contrast, Africa had longer IP days in 2021.

3.4. Task 4

3.4.1. Comparison of goodness-of-fit measures.

We compared the 5 goodness-of-fit measures in terms of the usefulness of the model. Only the AUC possesses the feature that a high score indicates a more appropriate model for the COVID-19 pandemic. According to Figure 9, the EXPO model is more appropriate for early outbreak stages, while the IRT model is more appropriate for the entire year. Note that all paired data obtained from the 2 models were standardized into ~N(0,1) due to significantly wider ranges among the 5 indicators. Thus, the SMD can be compared using the forest plot in Figure 9.

Figure 9.

Comparisons of goodness-of-fit measures for models in 2 scenarios.

3.4.2. Comparison of AUC indicators.

According to the scatter plot with the 95% control lines (Fig. 10), (1) the US states have higher AUCs (i.e., the determination coefficient in regression analysis) than the provinces/metropolitan cities in China. Most countries/regions lie along the 45-degree identity line. The AUC in Ukraine, for example, is 0.99, and in Taiwan, it is 0.96. The AUC of other countries can be examined by scanning the QR code and clicking on the bubble of interest on the dashboard we designed on Google Maps.

Figure 10.

Comparisons of AUC yielded by the 2 models for countries/regions on the scatter plot with the 95% control lines. AUC = area under the curve.

3.5. Online dashboards shown on Google Maps

The QR codes in the figures are linked to the dashboards. Readers are recommended to view the dashboards on Google Maps.

4. Discussion

4.1. Principal findings

We observed that (1) EXPO and IRT models had identical PP at the earlier outbreak stage; (2) the IRT model had a higher model R2 and smaller MAPE than the EXPO model in 2020; (3) Hubei Province in China had the highest angle index at the early stage; and (4) India, California (US), and the United Kingdom had the highest angle indexes in 2020. The IRT model is superior to the EXPO model when determining the IP on an Ogive curve.

4.2. Contributions of the study

Although the IP case index has been applied to examine the effective control of COVID-19,[ the angle index can accurately evaluate ImpactCOVID, as shown in Figures 4 and 6.

Many researchers[ have proposed the use of mathematical models to predict the number of COVID-19 cases, and others have investigated IP days during the COVID-19 pandemic.[ However, no one used the IP days to compare the ImpactCOVID or applied the angle index to inspect the ImpactCOVID in countries/regions.

The second contribution of this study is the comparison of mathematical COVID-19 models using forest plots. To date, only a few studies have compared the MA and PP between models because of a lack of familiarity with the algorithms proposed by other authors. In this work, we compared 2 mathematical models[ based on common conditions (e.g., the evolution of CNCC and ∆IP = 7 across countries/regions).

The EXPO model[ has been verified in several regions in China, including Wuhan in Hubei Province, Guangdong Province, and other parts of mainland China, during the early outbreak stage (from January 27 to February 18, 2020). For comparison, this study featured (1) 2 time-frame stages (i.e., the early 20-day outbreak stage and the entire year of 2020) and (2) all countries/regions hit by ImpactCOVID using the angle index.

The quadratic equation model (QE) model[ has been used to present the projected cases in Colombia and the deaths in Russia, India, and the rest of the world using past 31-day data up to May 29, 2020. In Tsai et al,[ the authors set the constrained term at the middle point (i.e., P(x2, y2)) of the observations with exponential growth during the COVID-19 epidemic and found that the IRT model is superior to the QE model.

Using less constrained parameters makes the model a good fit for the data.[ However, the one constrained term set at the middle point (i.e., P(x2, y2)) of the observations in the QE model[ yields a low MA, which has been verified in a previous study.[

Two other studies[ applied the IRT[ to construct an Ogive curve and determine the IP days used for predicting the projected cases in a country/region based on the CNCCs. Nonetheless, a sophisticated analysis to determine the IP days using the Newton–Raphson Iteration Method[ must be conducted in future studies.

4.3. Implications and recommendations

Numerous mathematical COVID-19 models[ and IP determinations[ have been proposed. However, none developed an angle index that can be used to measure ImpactCOVID and overcome the problem of the IP case index.[

Many online real- or near-real-time dashboards have been launched to track the worldwide spread of the COVID-19 outbreak.[ However, most of these tools are similar to other traditional websites[ and merely provide the same information as the WHO COVID-Dashboard.[ An accurate assessment of COVID-19 requires further mathematical analyses of global data.[ Although dashboards (e.g., JHU,[ WHO,[ and others[) have provided interesting visualizations for reporting the current state of COVID-19, these presentations lack important information on the disease outbreak using mathematical models to predict the projection of CNCCs in the future and understand COVID-19 trends[ further in data.

One study[ assessed 158 public Web-based COVID-19 dashboards and found that only a few dashboards employed predictive analytics by simulating various future scenarios. Thus, the imprecise predictive models and simulations early in the pandemic may have restricted their application. The current study presented predictive approaches with an online dashboard design that can benefit policymakers and epidemiologists during the COVID-19 epidemic.

4.4. Strengths of the study

First, the comparison of ImpactCOVID using the angle index in countries/regions can be applied to future relevant studies and is not limited to those merely focusing on the COVID-19 pandemic.

Second, MP4 videos on how to model the CNCCs and estimate the parameters for mathematical models[ have been provided to ordinary readers who are familiar with Microsoft Excel and hope to replicate the study in the future.

Third, the use of Microsoft Solver add-in to estimate the model parameters is a common approach that can be easily adopted by researchers.[ Data and model-building videos are provided in Supplemental Digital Contents 1, http://links.lww.com/MD/G983 and 2, http://links.lww.com/MD/G984. Approaches for the search of IP days that can also be used for computing the angle index have been previously studied.[

Fourth, the choropleth and forest plots used in this study can provide comprehensive insights into the difference in the comparison of various countries/regions or pair-panel statistics. In turn, these differences can be used by policymakers and decision-makers in visualizing their data.

Furthermore, an MS Excel module for drawing the forest plot is provided. Readers are recommended to watch the abstract video and the Excel module presented in Chien.[

4.5. Limitations and future studies

Our study has several limitations. First, only 2 models were compared for MA and PP assessment. Future investigations are required to study other COVID-19 models and further understand their differences and merits.

Second, only short- and long-term time periods during the COVID-19 pandemic were compared using the forest plot in the 2 models. Whether differences in MA and PP exist in the medium-term (or mid-term) epidemic must be investigated in the future.

Third, the case number is changeable and may vary day by day, particularly in countries undergoing second or third waves (peaks) in the ongoing pandemic. Thus, model parameters and angle indexes would also vary with time.

Fourth, the Microsoft Solver add-in is not unique in estimating model parameters. Many other methods and mathematical techniques should be used in making estimations and comparisons in the future, such as the Newton–Raphson iteration method[ for searching IPs on a given ogive curve.[

Fifth, visual dashboards on Google Maps are not free of charge, and a paid project key for using the Google cloud platform is needed. One limitation in using the dashboard is that it cannot be easily replicated by other authors or programmers for use in a short period of time.

Sixth, only 5 goodness-of-fit measures were applied to compare model usefulness (Fig. 10). Other indicators (e.g., Hannan–Quinn criterion[ and minimum description length[) could be employed in future studies.

Although IRT has superior MA and PP in the long run, other user-friendly mathematical models are also available for readers to understand the properties of exponential growth[ and quadratic equations[ in nature and for scientists to build improved COVID-19 models for further comparisons.

5. Conclusions

The 2 proposed models were compared according to their MA, PP, and 5 goodness-fit measures in 2 time-frame scenarios, namely, the early outbreak stage and throughout the entire year of 2020. We found that the IRT model is superior to the EXPO model in the long term. However, the EXPO model is better than the IRT model at the early stage because only the Ogive curves in the IRT model can appropriately generate the IP, which can then be used to compute the angle index. Both models are recommended so that readers can accurately project the CNCCs in other outbreak scenarios in the future. In addition, these models are not limited to the COVID-19 pandemic.

Author contributions

T.-W.C. developed the study. H.Y., K.T., C.Y., and W.C. analyzed the data. K.T. monitored the process of this study and helped respond to the reviewers’ advice and comments. T.-W.C. drafted the manuscript, and all authors provided critical revisions for important intellectual content. The study was supervised by K.T. All authors have read and agreed to the published version of the manuscript.

Acknowledgments

We thank AJE (American Journal Experts at https://www.aje.com/) for the English language review of this manuscript. All authors declare no conflicts of interest.