Forecasting stock market volatility with a large number of predictors: New evidence from the MS-MIDAS-LASSO model.

This paper explores the effectiveness of predictors, including nine economic policy uncertainty indicators, four market sentiment indicators and two financial stress indices, in predicting the realized volatility of the S&P 500 index. We employ the MIDAS-RV framework and construct the MIDAS-LASSO model and its regime switching extension (namely, MS-MIDAS-LASSO). First, among all considered predictors, the economic policy uncertainty indices (especially the equity market volatility index) and the CBOE volatility index are the most noteworthy predictors. Although the CBOE volatility index has the best predictive ability for stock market volatility, its predictive ability has weakened during the COVID-19 epidemic, and the equity market volatility index is best during this period. Second, the MS-MIDAS-LASSO model has the best predictive performance compared to other competing models. The superior forecasting performance of this model is robust, even when distinguishing between high- and low-volatility periods. Finally, the prediction accuracy of the MS-MIDAS-LASSO model even outperforms the traditional LASSO strategy and its regime switching extension. Furthermore, the superior predictive performance of this model has not changed with the outbreak of the COVID-19 epidemic.

Introduction

Financial market volatility is not only a key factor in assessing the risk of financial assets (see, e.g., Wang et al., 2015; Bee et al., 2016; Clements & Liao, 2017; Li & Wei, 2018; Ji et al., 2021) but also an important parameter in the pricing of financial derivatives and asset allocation (Graham & Harvey, 1996; Zhu & Ling, 2015), as well as an important factor in portfolio management (Ayub et al., 2015; Cederburg et al., 2020). Therefore, accurately predicting the volatility of the stock market is particularly important for investors when deciding on the sizes and timing of their investments.

Despite the many benefits of accurately predicting stock market volatility, improving predictive accuracy has proven challenging. First, the diversity of potential predictors poses a great challenge to the selection of predictors. For example, studies have determined the predictive abilities of macroeconomic and financial variables for stock market volatility (Nonejad, 2017; Paye, 2012). In recent years, with the introduction of economic policy uncertainty indices, some studies have also determined the predictive power of some economic policy uncertainty indicators (e.g., the economic policy uncertainty index, trade policy uncertainty index, monetary policy uncertainty index, equity market volatility index and geopolitical risk index) relative to stock market volatility (Alqahtani et al., 2020a; Gupta & Wohar, 2019; Li et al., 2020a; Olasehinde-Williams, 2021; Paye, 2012; Yu et al., 2018). In addition, studies have found evidence that the stock market can be driven by investors’ psychology (Daniel et al., 2002; Tseng, 2006) and determined the predictive power of some market sentiment indicators (see, e.g., Gupta et al., 2014; Perez-Liston et al., 2014; Oliveira et al., 2017; Jin et al., 2020; Liang et al., 2020a; Wang et al., 2020a). Other indicators, such as financial stress indices, have also been proven to have potential forecasting ability (Gupta et al., 2014; Singh, 2016; Sum, 2014). The predictive abilities of these potential predictors tend to change with changes in various external factors, making it more difficult to find stable predictors.

Additionally, it is not appropriate to consider only one predictor and ignore the valid information contained in other potential predictors; however, using a model with multiple predictors leads to the problem of identification and/or convergence (Asgharian et al., 2013; Li et al., 2020b). To solve these problems, some studies have used the key information from multiple variables extracted by dimensionality reduction methods, such as principal component analysis (PCA) and partial least-squares regression analysis (PLS) technology, to construct a forecasting model for achieving information integration (Asgharian et al., 2013; Cepni et al., 2019; Poncela et al., 2011), and they have discovered the ability of PCA- and PLS-based prediction methods to improve the accuracy of volatility prediction. In addition, forecast combination methods can also be used as information integration methods, and such approaches have been proven to be helpful for producing accurate and stable forecasts (Liang et al., 2020b; Paye, 2012; Yang et al., 2015; Zhang et al., 2020). In recent years, with the rapid increase in computing power and the emergence of big data, machine learning techniques have gained wide attention and application in the field of financial prediction. For example, Ma et al. (2018) and Zhang et al. (2019b) employed the least absolute shrinkage and selection operator (LASSO) method to forecast crude oil volatility with a large set of predictors, finding that the out-of-sample forecasting performance of LASSO is significantly better than that of models with individual predictors and combination approaches. Other studies have proven the superior forecasting effect of LASSO technology (Ng, 2013; Siliverstovs, 2015), and LASSO is also the focus of this paper.

In summary, finding the predictors with the most predictive power for stock market price volatility and forecasting methods that can better utilize the forecasting information from multiple predictors is of great importance to scholars, investors and policy makers. For these considerations, this paper focuses on finding the most powerful predictor from a large number of potential predictors, and explores the role of some newly constructed forecasting methods in improving the forecasting accuracy of stock market volatility by making full use of the forecasting information from a large number of potential predictors.

In terms of potential predictors, researches have shown that economic policy uncertainty indicators, market sentiment indicators and financial stress indices tend to contain richer and more interesting information than traditional macroeconomic (GDP, CPI, etc.) and financial variables (exchange rate, treasury bonds, etc.) (Asgharian et al., 2013; Gupta et al., 2014; Li et al., 2020b; Wei et al., 2017). Meanwhile, existing studies have also identified the advantages of economic policy indicators, market sentiment indicators and financial stress indices in volatility forecasting (Asgharian et al., 2013; Gupta et al., 2014; Li et al., 2020b; Wei et al., 2017). So, we refer to the work of Gupta et al. (2014) and forecast the US stock market volatility with economic policy uncertainty indicators, investor sentiment indicators and financial stress indices.

Refer to the existing literatures, we finalized the potential predictors included in the three categories of indicators. The economic policy uncertainty indicators considered in this paper include: (1) Global economic policy uncertainty index (GEPU); (2) US economic policy uncertainty index (EPU); (3) US monetary policy uncertainty index (MPU); (4) US trade policy uncertainty index (TPU); (5) US health care uncertainty index (HCU); (6) US equity market volatility index (EMV); (7) US infectious disease equity market volatility tracker (IDEMV); (8) Geopolitical risk index (GPR); and (9) US economic uncertainty related queries (EURQ). The market sentiment indicators considered in this paper include: (10) The CBOE volatility index (VIX); (11) ISEE sentiment index (ISEESI); (12) News sentiment index (NSI); and (13) US AAII retail investor sentiment index (AAII). The financial stress indices considered in this paper include: (14) The financial stress index provided by the Office of Financial Research (OFRFSI); (15) The financial stress index provided by St. Louis Fed. (STLFSI). All of these indicators are proved in the existing literatures to have significant impact on the price volatility of stock market, indicating that although not all of these indicators are directly related to the stock market, they all contain the important information that is significant enough to influence the stock market price volatility (Alqahtani et al., 2020b; Arouri et al., 2016; Bai et al., 2021; Bekaert & Hoerova, 2014; Das et al., 2019; Dua & Tuteja, 2016; Gupta et al., 2014; He et al., 2021a; Liang et al., 2020a; Piñeiro-Chousa et al., 2022; Smales, 2017; Wang et al., 2020a; Wen et al., 2022; Yang & Yang, 2021; Yu et al., 2018). Economic policy uncertainty indicators other than EMV and IDEMV capture the level of macroeconomic uncertainty caused by different types of economic policies or major events (Baker et al., 2016; Caldara & Iacoviello, 2018; Husted et al., 2018). In general, the release of economic policies or the occurrence of major events often lead to stock market price volatility by affecting real economic conditions (Al-Awadhi et al., 2020; Alqahtani et al., 2020b; Arouri et al., 2016; Bai et al., 2021; Bontempi et al., 2021; He et al., 2021a; Hoque & Zaidi, 2019; Megaritis et al., 2021; Wen et al., 2022; Yang & Yang, 2021; Yang et al., 2021), so exploring the predictive power of different economic policy uncertainty indicators is helpful for distinguishing the role of macroeconomic uncertainty caused by different types of economic policies or major events in driving stock market price volatility. Market sentiment is also closely related to stock market price volatility because it can reflect the investment intentions or expectations of market participants. It affects investors’ subjective judgment of their future returns, which in turn affects their investment behavior and ultimately causes stock market price volatility (Bekaert & Hoerova, 2014; Liang et al., 2020a; Oliveira et al., 2017; Perez-Liston et al., 2014; Shapiro et al., 2020). Financial markets are necessarily interconnected, so stock market price volatility is also subject to the stress of the whole financial market (Das et al., 2019; Gupta et al., 2014; Singh, 2016; Sum, 2014), this is also why we examined the predictive power of FSIs on stock market volatility.

Although the out-of-sample forecasting ability of most of these indicators for stock market volatility have been discussed by literatures (Alqahtani et al., 2020a; Gupta & Wohar, 2019; Gupta et al., 2014; Li et al., 2020a; Liang et al., 2020a; Olasehinde-Williams, 2021; Paye, 2012; Wang et al., 2020a; Yu et al., 2018), from the existing literatures, indicators that have significant impact on stock market volatility may not have appreciable out-of-sample predictive ability since different indicators usually impact the stock market price volatility with different intensities and paths (Liang et al., 2020a; Liu & Wang, 2020; Nonejad, 2017; Wang et al., 2020b). Meanwhile, some indicators, such as HCU and IDEMV, tend to have stronger impact on stock market volatility only because of the occurrence of certain events (e.g., COVID-19) (Bai et al., 2021; Li et al., 2020c). These are also why the out-of-sample predictive power of some of the potential predictors considered in this paper, i.e. HCU, IDEMV and EURQ, on stock market volatility has not yet been discussed.1 Besides, it also means that the predictive power of some of the potential predictors selected in this paper may not be robust. So, it is of certain implication for scholars and stock market participants to explore the role of some newly constructed forecasting methods in improving the predictive accuracy of stock market volatility by making full use of the predictive information of a large number of potential predictors. Considering that combination forecasting methods, dimensionality reduction methods and LASSO techniques are widely used in the studies about forecasting asset volatility with a large number of potential predictors, this paper also focus on comparing the role of these methods in improving the forecasting accuracy of stock market volatility. However, different from the existing literatures, we mainly compare the forecasting performance of these methods in the framework of MIDAS-RV model.

For years, several models including GARCH-class models, HAR-RV models and neural networks etc. are widely used in predicting the volatility of financial assets (Bekaert & Hoerova, 2014; Corsi, 2009; Litimi et al., 2019; Ma et al., 2019; Santos & Ziegelmann, 2014; Tian et al., 2017; Wang et al., 2020b; Yang et al., 2021). Among them, the HAR-RV model is the most commonly used in high-frequency realized volatility forecasting (Ma et al., 2019; Santos & Ziegelmann, 2014; Tian et al., 2017). However, different from most of these literatures, we mainly use the MIDAS-RV model and its several extensions to forecast stock market volatility. This paper is related to the work of Liu et al. (2020), who forecasted the monthly stock market volatility with macroeconomic and financial variables, as well as technical indicators using the AR based combination forecasting methods, dimensionality reduction methods and LASSO techniques. In contrast, we explore the abilities of 9 economic policy uncertainty indicators, 4 market sentiment indicators and 2 financial stress indices to predict daily stock market realized volatility. Besides, we construct a MIDAS-LASSO model by combining the LASSO technique with the MIDAS-RV method, in reference to Marsilli (2014), and compare its out-of-sample prediction performance with that of the MIDAS-RV-X model (the MIDAS-RV model extended by an additional predictor X), MIDAS-RV-PCA model (the MIDAS-RV model extended by PCA technology), MIDAS-RV-PLS model (the MIDAS-RV model extended by PLS technology) and some commonly used combination forecast methods (the mean, median, trimmed mean, DMSPE(1.0) and DMSPE(0.9), DMA and DMS). However, studies have found that a model with regime switching can be more efficient in forecasting asset volatility (Tian et al., 2016; Uddin et al., 2018; Wang et al., 2016b) than a model without regime switching, making us curious whether the MIDAS-LASSO model with regime switching can also improve the prediction accuracy for stock market volatility to a greater extent. Therefore, we further build a MIDAS-LASSO model with a Markov regime (MS-MIDAS-LASSO) and evaluate its prediction performance.

Our main reasons for constructing these extension models on the basis of the MIDAS-RV model are as follows. First, compared with the widely used HAR-RV model, the MIDAS-RV model can better reflect heterogeneity because the estimation process of the MIDAS-RV model primarily utilizes a specific function to determine the weights of the multi-order lagged RV series, while the HAR-RV model directly calculates the equally weighted average of the 1-week lagged and 1-month lagged RV series (Ghysels et al., 2006; Lu et al., 2020; Santos & Ziegelmann, 2014). Second, modelling asset volatility using the MIDAS-RV model and its extensions can better reflect the multi-order lagged effects of potential predictors on high-frequency volatility with fewer parameter estimates than the HAR-RV model, and is less prone to overfitting problems (Ghysels et al., 2006; Lu et al., 2020; Santos & Ziegelmann, 2014). Thirdly, studies have proved that the MIDAS-RV model can obtain more valid prediction information from predictors and has better prediction performance than the widely used HAR-RV model (Ma et al., 2019; Santos & Ziegelmann, 2014). Finally, as far as we know, there is no research that predicts stock market realized volatility by the MIDAS-LASSO model and its regime switching extension.

To seek the best predictors and forecasting method, we evaluate the absolute and relative forecasting performance of the models described above by various assessment methods, including the model confidence set (MCS) test, out-of-sample R2, direction-of-change (DoC) test, portfolio performance analysis and CumMSE difference analysis. In addition, the robustness of the evaluation results is checked by using an alternative benchmark model, alternative lag, and alternative rolling window. We also conduct some extension analyses. First, we compare the forecasting performance of the MIDAS-LASSO framework proposed in this paper (the MIDAS-LASSO model and MS-MIDAS-LASSO model) to that of the traditional HAR-LASSO framework (the HAR-LASSO model and MS-HAR-LASSO model). In addition, we explore the forecasting performance of the forecasting methods during high- and low-volatility periods. Finally, we also discuss the forecasting performance of the forecasting models during COVID-19.

In summary, this paper contributes to the extant literature in the following aspects. First, we comprehensively explore the predictive power of economic policy uncertainty indicators, market sentiment indicators and financial stress indices with respect to the realized volatility of US equity markets. Second, we forecast the realized volatility of the stock market by some MIDAS-RV extensions. In particular, we construct the MIDAS-LASSO model and its regime switching extension, which, to our knowledge, have not yet been used in the literature to forecast daily stock market realized volatility. Third, we further compare the prediction performance of our MIDAS-LASSO framework with that of the traditional HAR-LASSO framework. Finally, we discuss whether these forecasting models remain valid during high- and low-volatility periods and during COVID-19.

Some noteworthy findings are obtained in this paper. First, compared to the financial stress indices and the market sentiment indicators (except for the VIX), the economic policy uncertainty indicators (especially the EMV) and the VIX are more noteworthy. Among these predictors, VIX has the best predictive ability, but its predictive power has weakened during the COVID-19 epidemic, making EMV the best predictor during this special period. The financial stress indices and the market sentiment indicators except for VIX mainly play a role in enhancing the accuracy of stock market volatility in periods of low volatility. Second, across all considered forecasting methods, MS-MIDAS-LASSO has the best out-of-sample forecasting performance, followed by the MIDAS-LASSO model. The forecasting accuracy of the MS-MIDAS-LASSO model even outperforms the traditional HAR-LASSO model and its regime switching extension. In addition, the MS-MIDAS-LASSO model still has the highest prediction accuracy during the low-volatility period when the prediction accuracy of the MIDAS-LASSO model is significantly reduced. Finally, the predictive performance of the MS-MIDAS-LASSO model remained the best, even during the special period of the COVID-19 epidemic.

The remainder of the paper is organized as follows. Section 2 describes the related forecasting models. Section 3 summarizes the data utilized in this paper. Section 4 discusses the results of out-of-sample forecasts. Section 5 presents an extension analysis, and Sect. 6 concludes this paper.

Methodology

MIDAS regression framework

We start with a definition of the realized volatility (RV) on a certain trading day t. According to the definition of Andersen and Bollerslev (1998), the RV on day t is usually constructed as the sum of the squares of the returns of the intraday high-frequency trading data:where represents the realized volatility on day t, D represents the number of intraday return observations, and denotes the d-th intraday return on day t.

Then, the MIDAS framework and its several extensions are employed to forecast the RV of the S&P 500 index. We first define a benchmark model as:where denotes the h-day-ahead RV at time t, represents the i-order lagged RV, K is the largest order, and is the error item. In this paper, we consider the case when h = 1 and K = 66. is the weight term of . Since studies have found that the Beta polynomial is more suitable for time lags compared to other functions (Ghysels & Qian, 2019; Ghysels et al., 2007; Zhang & Wang, 2019), we define in Eq. (2) as a Beta polynomial which is usually used in the realized volatility forecasting researches (Ghysels et al., 2006; Lu et al., 2020; Ma et al., 2018):where f function in Eq. (3) is a Beta function which used to make sure the weight to be positive. The f function in Eq. (3) is based on the Gamma function and can be expressed as:where the Gamma function, , is defined as:

In this paper, in Eqs. (2) and (3) is set to be 1 following Ma et al. (2019) and Lu et al. (2020). The model described above uses the lagged RV as the predictor and thus can be used as a benchmark model to evaluate the relative predictive performance of our extended models.

In this paper, the maximum likelihood estimation method is used for the parameter estimation process for the MIDAS-RV model. Assuming that RV is independently and identically distributed and follow a normal distribution with mean and variance . When do a one-day-ahead volatility forecast, the likelihood function is:where

The estimated parameter, , and can then be obtained by solving the following optimization problem:

Using the parameter estimated at time t, the predicted RV of day t + 1, , can be calculated as:

To explore whether an additional predictor contains valuable predictive information for stock volatility, we further construct the MIDAS-RV-X model by adding a predictor of interest as an additional variable for the benchmark model above. Since the predictors we consider are sampled at different sample frequencies, we write the MIDAS-RV-X model as:where denotes a predictor on day t − i + 1 and denotes its original sampling frequency. In this paper, we consider the predictors sampled at daily, weekly and monthly frequencies, corresponding to , 5 and 22, respectively, and we disaggregate the predictors with low frequency to the daily frequency, inspired by Vinayagathasan (2014). The main purpose of disaggregating the low-frequency predictors to high-frequency series is to facilitate the construction of the following information integration-based prediction technology and make all forecasting methods have the same underlying structure so that the predictions are comparable.2

In this paper, a relatively large number of predictors are considered. However, studies have shown that in a data-rich environment, a forecasting model with only one predictor would ignore the valuable predictive information of other possible predictors, while a model with multiple predictors would cause the problem of identification and/or convergence (Asgharian et al., 2013; Li et al., 2020b). As a well-known statistical process for feature extraction and dimensionality reduction, principal component analysis (PCA) and partial least-squares regression analysis (PLS) technology can be used to collect the key information among multiple variables and thus are used in some of the existing studies to solve the above problems (Asgharian et al., 2013; Cepni et al., 2019; Poncela et al., 2011). Therefore, to forecast with more valuable predictive information, we further construct a MIDAS-RV-PCA model and a MIDAS-RV-PLS model. These two models obtain the key information/common factors from all additional predictors by principal component analysis (PCA) and partial least-squares regression analysis (PLS), respectively. The MIDAS-RV-PCA model can be defined as below:where represents the key information/common factors (here, we use the first principal component of all predictors) extracted from all additional predictors by PCA. The MIDAS-RV-PLS model can be defined as:where denotes the key information/common factors (here, we use the first principal component of all predictors) extracted from all additional predictors by PLS. For details about the PCA and PLS technique, see Yan et al. (2022) and He et al. (2021b). The parameters of MIDAS-RV-X, MIDAS-RV-PCA and MIDAS-RV-PLS models are still estimated by the maximum likelihood estimation method. But this time, in Eq. (6) are measured as:

Meanwhile, the RV of day t + 1 can be forecasted by:

In addition, studies have shown that prediction combination methods are helpful for producing more accurate and stable forecasts (Liang et al., 2020b; Paye, 2012; Yang et al., 2015; Zhang et al., 2020). Therefore, we further use the combination approach suggested by Rapach et al. (2010) to produce five combination volatility forecasts. Combination volatility forecasts can be mathematically measured as:where represents the h-day-ahead combination forecast of the RV, denotes the nth individual forecast, and is the combined weight of the nth individual forecast. The five considered combination methods include the following: (1) Mean combination. This method calculates the equal weighed average of the N individual forecasts. (2) Median combination. This method finds the median of the forecasts for the N individual models. (3) Trimmed mean combination. For this method, we first set the weights of the largest and smallest forecasts of the N individual models to zero and then calculate the weights of the remaining individual forecasts as . (4) Discount mean square prediction error (DMSPE) [including DMSPE (1.0) and DMSPE (0.9)]. For the DMSPE method, the of the nth individual forecast is measured as with , where t0 is the length of the initial training sample, represents a discount factor, and and are the true RV and the n-th RV forecasts, respectively. Following Li et al. (2020b)_ENREF_35 and Rapach et al. (2010), we consider two values of , i.e., 1 and 0.9, in this paper.

In addition to the combination prediction methods above, we also try to generate combination forecasts by the dynamic model averaging (DMA) and dynamic model selection (DMS) methods. The 1-day-ahead DMA and DMS processes start with the following time-varying parameter (TVP) model:where for represents a set of predictors, and . For an with N predictors, there are usually combinations for these predictors. Then, the DMA and DMS approaches can dynamically incorporate the M models as:

andwhere in Eq. (22) is the weight of model m, , and denotes that model m is selected at time t. m* in Eq. (13) indicates the model with the largest weight . To solve the problem of heavy computational complexity, we simplify the DMA and DMS processes using the Kalman filter method following Raftery et al. (2010).3

In this paper, we take the N individual predictions as and let M = N. By doing so, the computational processes can be further simplified, and the DMA and DMS process can dynamically combine the individual forecasts and be regarded as a dynamic forecast combination method (Wang et al., 2016a; Wei et al., 2020). Here, Eqs. (12) and (13) are rewritten as:and

Lasso-augmented MIDAS model and regime switching

To solve the problem that models with large number of predictors are prone to problems regarding multicollinearity and overfitting, we refer to the work of Marsilli (2014) and further combine the least absolute shrinkage and selection operator (LASSO) technology introduced by Tibshirani (1996) with the MIDAS-RV model, constructing a MIDAS-LASSO model. The LASSO belongs to the penalized regression model family since it operates by penalizing an optimization problem associated with the regression of a term that involves the -norm of the coefficients. Studies have shown that the LASSO tends to have a lower misspecification risk for forecasting models and has better out-of-sample forecasting accuracy than usual information criteria (Ng, 2013; Siliverstovs, 2015). Our MIDAS-LASSO is defined as below:where N denotes the number of additional predictors; when i = 1, . Since is set to 1, this MIDAS regression is solved by proposing the optimization problem below:where and are the shrinkage estimators and is an exogenous parameter that controls the strength of the LASSO penalization.

To solve this optimization problem using the maximum likelihood estimation method and do a one-day-ahead forecast, we need firstly to convert Eq. (27) as

To associate it with the maximum likelihood estimation, we letand multiply the function in Eq. (28) by a constant, , that is always greater than 0:

Then, adding the monotonically increasing exp in the function and transformation Eq. (30) to be:

Then, multiplying again by a constant greater than 0 and obtain the following optimization problem:

The final maximum likelihood estimation-based optimization problem can be obtained by further logarithmic the optimization problem of Eq. (32):

In Eq. (33), is called a log Posterior, is a standard log likelihood. Specially, is the prior density function when obey the prior distribution , it can convent the prior expressed of the original definition of Lasso into the maximum posterior estimation compatible with the maximum likelihood estimate. Using the parameter of MIDAS-LASSO model estimated at time t, the one-day-ahead forecast of RV can be measure as:

Studies have shown that models with regime switching may be able to achieve improved prediction accuracy to some extent (Tian et al., 2016; Uddin et al., 2018; Wang et al., 2016b). For this consideration, we further consider regime switching for the MIDAS-LASSO model, constructing an MS-MIDAS-LASSO model by considering high- and low-volatility regimes. Here, Eqs. (26) and (27) are rewritten as:andwhere is the state variable, and 1 denote the low- and high-volatility states, respectively, and are state parameters, and .

In this research, is assumed to follow a fixed two-state Markov process. Let and denote the probability of the occurrence of and :where and , the initial and are equal to 0.5. The Markov Switching process assumes that the state of the next moment is only relevant to the state of the previous moment. In other words, the state probability of the next moment is obtained only by the transition of the state probability at the previous moment. According to the literatures (BenSaïda et al., 2018; Lu et al., 2020; Ma et al., 2019), the transition probabilities are:

Since state is either transferred to or , there must be and . So, we just need to estimate the two transition probabilities, , and the other two transition probabilities are measured as and , respectively. Then, the prior probabilities (or forecasted probabilities) can be measured as and , .

The MS-MIDAS-LASSO is still estimated by the maximum likelihood estimation method, and the likelihood function is obtained by mixing the probabilities. As observed in Eq. (33), the likelihood function of MIDAS-LASSO consists of a standard log likelihood and a prior density function. Similarly, in this paper, we also let the likelihood function of MS-MIDAS-LASSO consists of a standard log likelihood and the prior density function. The standard log likelihood component is expressed as:where

Then, the log Posterior is expressed as:where

Solving the following optimization problem can get the parameter estimation results:

According to the in-sample estimation results of the model parameters at time t, we can use the following formula to obtain the predicted RV of time t + 1:where is the predicted probability of state occurring at time t + 1, besides,

It is worth noting that the likelihood function value, , can represent the probability density of the occurrence of a particular observation and also contains information about the probability of the occurrence of the state. So, refer to the work of Hamilton and Susmel (1994), we use the Bayes formula to update the posteriori probability in our forecasting process. The posteriori probability, and are updated by:

Data

In this paper, we try to explore the forecasting performance of the models discussed in Sect. 2 for the realized volatility of the US stock market. Following Liu et al. (2015) and Lu et al. (2020), we use the RV data of the S&P 500 index constructed by 5-min high-frequency returns obtained from the Realized Library of the Oxford-Man Institute of Quantitative Finance. The trajectory of the RV of the S&P 500 index is drawn in Fig. 1. Then, three classes of predictors are considered in this paper. The first nine predictors include economic policy uncertainty indicators, which were often discussed in recent literature (Bai et al., 2021; Baker et al., 2016; Caldara & Iacoviello, 2018; Gupta & Wohar, 2019; Husted et al., 2018; Li et al., 2020a). They are:

Fig. 1

Realized volatility of the S&P 500 index

(1) Global economic policy uncertainty index (GEPU) and (2) US economic policy uncertainty index (EPU): According to the existing literatures (Arouri et al., 2016; Hoque & Zaidi, 2019), GEPU and EPU affect stock market price volatility mainly through three channels. Firstly, they influence the decisions of economic agents regarding employment, consumption, savings and investment, and directly affect stock market participation and price volatility. Secondly, they annihilate the degree of economic freedom and the financing environment that affect stock market price volatility by reducing the amount of both international and local investors. Thirdly, they impact the commodity price and further affects stock market price volatility;

(3) US monetary policy uncertainty index (MPU): As the existing literatures suggest, MPU affect stock market volatility through two main channels (Wen et al., 2022). One the one hand, it affects stock market price volatility by influencing the trade channel and, consequently, the stock market. US monetary policy is generally accompanied by an appreciation or depreciation of the US dollar. In a contractionary monetary policy environment, an appreciating dollar will lead to an increase in import demand, which leads to a positive impact on economic and further increase stock prices. On the other hand, MPU can also affect stock market by influencing interest rates. A decrease in real interest rates leads to the decrease in the cost of capital and the increase in investment, which in turn has positive impact on stock prices;

(4) US trade policy uncertainty index (TPU): TPU also has significant impact on the stock market (He et al., 2021a). From a micro perspective, exporters are sensitive to TPU. If tariffs increase in a country, exporters from other countries are less likely to enter the market. Meanwhile, a reduction in TPU can change the mix of the export product, giving an advantage to higher quality products with lower prices. These both have direct impact on the stock market price performance of that exporter;

(5) US health care uncertainty index (HCU) and (6) US infectious disease equity market volatility tracker (IDEMV): The choice of these two indicators is related to the outbreak of the COVID-19. COVID-19 has a dramatic impact on the investment and business environment in countries around the world, which in turn has huge impact on stock markets of various industries (Al-Awadhi et al., 2020). Among the recently proposed uncertainty indices, HCU can capture the level of uncertainty brought to the national economy by the occurrence of various public health events and various health care policies, and IDEMV can capture the level of uncertainty brought to stock market price volatility by the occurrence of various public health events, both of which generally have significant impact on the stock market, especially during COVID-19 (Bai et al., 2021; Megaritis et al., 2021);

(7) US equity market volatility index (EMV): EMV can reflect the uncertainty level in the price volatility of the US stock market and is directly related to the volatility of the S&P 500 index (Alqahtani et al., 2020b; Yang & Yang, 2021);

(8) geopolitical risk index (GPR): Researches have pointed out that geopolitical risk is an important factor in shaping macroeconomic and financial cycles and has important impact on stock market price volatility (Yang et al., 2021). It can affect stock market in many ways. First, the negative impact of an increase in GPR on demand and supply channels can push up firms’ costs, which in turn affects stock prices. Second, as an uncertainty indicator, an increase in GPR can also delay the decision-making process of market participants. Third, an increase in GPR can also increase investment risk in stock market;

(9) US economic uncertainty related queries (EURQ): EURQ reflects the uncertainty which are similar to GEPU and EPU, but it constructed in a different way from GEPU and EPU. As discussed by Bontempi et al. (2021), it affects stock market price volatility with similar channels to GEPU and EPU.4

Since various market sentiments often directly determine the investment behaviour of stock market participants, studies have demonstrated the impact of market sentiment on financial market volatility (Oliveira et al., 2017; Perez-Liston et al., 2014). Besides, the existing literature finds that different types of sentiment indicators, such as stock market sentiment, media sentiment and investor sentiment, are differ in their impact on stock market price volatility (Liang et al., 2020a). For these considerations, we consider 4 additional market sentiment indicators reference to the existing literature (Liang et al., 2020a).

(10) CBOE volatility index (VIX): The VIX is commonly called the “fear index”, it is a measure of the market’s expectation of US stock market volatility over the next 30 days and is used as a stock market sentiment indicator in this work. Clearly, VIX is directly related to the US stock market (Bekaert & Hoerova, 2014);

(11) ISEE sentiment index (ISEESI): The ISEESI is measured by looking at the ratio of opening long call options to opening long put options purchased on the International Stock Exchange (ISE). Clearly, unlike the VIX, ISEESI quantifies stock market sentiment primarily by the investment behaviour of stock market participants;

(12) News sentiment index (NSI): The NSI is a high-frequency measure of media sentiment based on a lexical analysis of economics-related news articles (Shapiro et al., 2020). NSI is an important proxy variable for media sentiment and has also been proved to be an important driver of stock market price volatility (Liang et al., 2020a);

(13) US AAII retail investor sentiment index (AAII). The AAII used in this paper measures the difference between the percentages of individual investors who are bullish and bearish. AAII can reflect the expectations of individual investors in the US stock market and directly influence the investment behaviour of individual investors in the stock market.5

Finally, since financial markets are interconnected and stock market price volatility can also be impacted by the stress of the whole financial market (Das et al., 2019; Gupta et al., 2014; Singh, 2016; Sum, 2014). As the result of global economic integration, the stock markets of individual countries are subject to the stress of both the financial market of that country and the global financial market (Das et al., 2019; Singh, 2016). So, we further consider 2 financial stress indicators: (14) the financial stress index provided by the Office of Financial Research (OFRFSI); (15) the financial stress index provided by St. Louis Fed. (STLFSI). The OFRFSI is constructed from 33 global financial market variables, such as yield spreads, valuation measures and interest rates. The STLFSI is constructed from 18 weekly financial variables in the US: seven interest rates, six yield spreads, and five other indicators.6

Our original data contain three different sampling frequencies, i.e., daily, weekly and monthly. The daily sampled data include the RV of the S&P 500, EPU, IDEMV, VIX, ISEESI, NSI and OFRFSI. The weekly sampled data include the AAII and STLFSI. The monthly sampled data include the GEPU, MPU, TPU, HCU, EMV, GPR, and EURQ. All the predictors with low frequencies are disaggregated to the daily frequency. The daily data cover the period from January 3, 2006, to August 28, 2020.7 The results of the descriptive statistics are presented in Table 1. Table 1 shows that all variables except for the EPU, NSI and AAII are right skewed (the EPU, NSI and AAII are left skewed). The RV of the S&P 500, EPU, EMV, IDEMV, EURQ, VIX, OFRFSI and STLFSI have kurtosis values larger than 3, indicating that they have peaks and fat-tailed distributions. The J-B statistics reject the null hypothesis of normality are rejected for all variables. The ADF statistics reject the null hypothesis of the existence of a unit root. Finally, the Ljung-Box Q statistics reject the null hypothesis of no serial correlation for all variables.

Table 1

Descriptive statistics

	Mean	Std. dev	Skewness	Kurtosis	J-B	ADF	Q(5)	Q(22)
RV	13.129	10.777	3.497	22.131	63,653.104***	− 18.511***	10,827.086***	32,170.903***
GEPU	4.858	0.439	0.053	2.814	6.986**	− 2.714***	17,883.527***	72,263.201***
EPU	4.498	0.656	− 0.031	3.579	51.957***	− 26.392***	7003.411***	24,675.153***
MPU	4.198	0.593	0.149	2.368	74.814***	− 5.584***	16,621.704***	53,736.149***
TPU	4.191	1.150	0.828	2.931	421.632***	− 4.095***	17,439.682***	65,446.844***
HCU	4.936	0.683	0.201	2.787	31.720***	− 3.881***	17,426.299***	65,172.217***
EMV	2.955	0.344	1.165	4.849	1357.526***	− 5.509***	16,613.561***	53,568.761***
IDEMV	0.369	0.657	3.217	14.765	27,585.932***	− 22.110***	9930.593***	37,487.959***
GPR	4.501	0.460	0.537	2.643	196.660***	− 4.802***	17,092.840***	60,262.244***
EURQ	5.213	0.121	0.727	4.234	558.285***	− 5.655***	16,552.009***	52,736.040***
VIX	19.344	9.651	2.489	11.257	14,262.466***	− 6.330***	16,699.917***	60,377.204***
ISEESI	4.662	0.179	0.204	2.677	41.463***	− 5.646***	16,250.292***	49,892.449***
NSI	− 0.022	0.216	− 0.302	2.638	76.138***	− 3.325***	17,653.545***	66,505.854***
AAII	0.026	0.151	− 0.268	2.957	44.311***	− 11.754***	11,497.185***	22,937.370***
OFRFSI	0.115	4.930	2.517	11.043	13,814.270***	− 2.023**	18,140.304***	75,045.285***
STLFSI	0.039	1.282	3.328	17.294	38,144.932***	− 3.720***	17,617.577***	66,963.230***

This table lists the results of the descriptive statistics for the realized volatility (RV) of the S&P 500 index and 15 potential predictors. The “Std. dev” column contains the standard deviation results; the “J-B” statistics indicate the Jarque–Bera statistics for the null hypothesis of normality; the “ADF” column list the statistics for the augmented Dickey-Fuller unit root test; Q(5) and Q(22) represent the Ljung-Box Q statistics, which test up to the 5th and 22nd serial correlations

** and *** Indicate rejections of the null hypothesis at the 5% and 1% significance levels, respectively. Our daily data cover the period from January 3, 2006, to August 28, 2020

Empirical results

In this section, we evaluate the out-of-sample forecasting performance of the benchmark MIDAS-RV model, 15 MIDAS-RV-X models, MIDAS-RV-PCA, MIDAS-RV-PLS, the MIDAS-LASSO model, the MS-MIDAS-LASSO model, five combination forecasts [mean, median, TMC, DMSPE (0.9) and DMSPE (1.0)], DMA and DMS. We perform out-of-sample forecasts via the rolling window approach to avoid data overlap, and the length of the rolling window is 2000. Some of the forecasting results are drawn in Fig. 2. To assess the predictive performance, a model confidence set (MCS) test and the out-of-sample R2 are utilized. In addition, we also perform a direction-of-change test, portfolio performance analysis and CumMSE difference analysis. Finally, we also check the robustness of the predictive performance by considering alternative benchmarks, alternative lags for the MIDAS regression and alternative rolling window lengths.

Fig. 2

Results of out-of-sample forecasts

Out-of-sample forecasting performance evaluation

Evaluation results of the MCS test

In this subsection, the out-of-sample forecasting performance of the models of interest is evaluated by the model confidence set (MCS) test of Hansen et al. (2011). The MCS test can be used to find the “best” models by testing the null hypothesis of equal forecasting accuracy for any two models in a model set and eliminating the worst predictive models from sequentially. In our work, two test statistics are considered for the MCS process, i.e., (range statistic) and (semiquadratic statistic), where , , P is the length of the out-of-sample forecasts, and and are the loss series of models u and v (), respectively. For this MCS process, three loss functions are considered, i.e., the quasi-likelihood loss (QLIKE), mean square error (MSE) and mean absolute error (MAE). These loss functions are defined as: andrespectively, where is the i-th out-of-sample forecast and is the true RV corresponding to the i-th out-of-sample forecast. To find the “best” forecasting methods, the threshold p-value of and is set to be 0.1 following Tian et al. (2017), Zhang et al. (2019a) and Wang et al. (2020b). A p-value larger than 0.1 indicates that the corresponding model can survive the MCS test, and a larger p-value corresponds to a model with better out-of-sample forecasting performance.

The p-values obtained from the MCS test are listed in Table 2. Table 2 prediction shows first that the MS-MIDAS-LASSO model can pass the MCS test with a p-value equal to 1 under all loss function criteria, indicating that it has the best out-of-sample forecasting performance. The other prediction models cannot survive the MCS test under both the criteria of QLIKE and the MAE. However, under the MSE criterion, several individual models (the MIDAS-RV models extended by the MPU, EMV, IDEMV, VIX, NSI, OFRFSI and STLFSI), the MIDAS-LASSO model and several combination forecasts (MEAN, DMSPE (0.9), DMSPE (1.0), DMA and DMS) can also pass the MCS test with p-values smaller than 0.15 but larger than 0.1, implying their potential ability to improve the forecasting accuracy for stock market RV. Among these models, the MIDAS-LASSO model passes the MCS test under both the and statistics of the MSE, with an out-of-sample prediction performance that is second only to that of MS-MIDAS-LASSO.

Table 2

Results of the MCS test

	QLIKE		MSE		MAE
	T_R	T_SQ	T_R	T_SQ	T_R	T_SQ
Benchmark	0.000	0.000	0.070	0.087	0.000	0.007
GEPU	0.000	0.000	0.018	0.086	0.000	0.007
EPU	0.000	0.000	0.018	0.086	0.000	0.005
MPU	0.000	0.000	0.102	0.088	0.001	0.007
TPU	0.000	0.000	0.018	0.085	0.000	0.007
HCU	0.000	0.000	0.076	0.088	0.000	0.007
EMV	0.001	0.001	0.130	0.088	0.001	0.007
IDEMV	0.000	0.000	0.107	0.088	0.000	0.005
GPR	0.000	0.000	0.018	0.086	0.001	0.007
EURQ	0.000	0.000	0.018	0.086	0.000	0.007
VIX	0.010	0.010	0.130	0.088	0.011	0.011
ISEESI	0.000	0.000	0.018	0.086	0.001	0.007
NSI	0.000	0.000	0.102	0.088	0.000	0.007
AAII	0.000	0.000	0.070	0.088	0.000	0.005
OFRFSI	0.000	0.001	0.130	0.088	0.001	0.007
STLFSI	0.000	0.001	0.130	0.088	0.000	0.005
MIDAS-RV-PCA	0.000	0.001	0.076	0.088	0.000	0.007
MIDAS-RV-PLS	0.000	0.001	0.070	0.088	0.000	0.007
MIDAS-LASSO	0.001	0.003	0.131	0.131	0.008	0.007
MS-MIDAS-LASSO	1.000	1.000	1.000	1.000	1.000	1.000
MEAN	0.000	0.001	0.102	0.088	0.001	0.007
MEDIAN	0.000	0.000	0.070	0.088	0.000	0.007
TMC	0.000	0.000	0.076	0.088	0.001	0.007
DMSPE (0.9)	0.000	0.001	0.130	0.088	0.008	0.007
DMSPE (1.0)	0.000	0.001	0.130	0.088	0.008	0.007
DMA	0.000	0.001	0.102	0.088	0.008	0.007
DMS	0.001	0.001	0.130	0.088	0.008	0.007

This table reports the p-values of the MCS test obtained by 10,000 bootstrap simulations. The p-values larger than 0.1 are bolded, indicating that the corresponding models survive the MCS test. The p-values equal to 1 are bolded and underlined, indicating that the corresponding models have the best out-of-sample forecasting performance

Evaluation results in terms of the out-of-sample R2

To determine how much the MIDAS-RV extensions improve the prediction accuracy, the out-of-sample R2 () is used in this subsection to further evaluate the relative out-of-sample prediction performance of these MIDAS-RV extensions. calculates the percent reduction of the mean squared forecast error (MSFE) of the target model relative to that of the benchmark model and is widely used to evaluate the relative predictive performance of forecasting models of interest (Liu & Wang, 2020; Wang et al., 2020c; Wei et al., 2020). is defined as:where denotes the i-th RV forecast of the benchmark model, represents the i-th forecast of a model of interest, and is the true RV corresponding to the i-th out-of-sample forecast. According to the definition of , a positive indicates that the model of interest has better forecasting accuracy than the benchmark model. To further determine the statistical significance of the relative predictive performance of a target model, the Clark and West (2007) (CW) method is utilized to test the null hypothesis that the MSFE of a model of interest is no larger than that of the benchmark. The CW statistic can be statistically measured by defining: Then, the CW statistic is conveniently derived by regressing on a constant.

, the CW statistics and the corresponding p-values are presented in Table 3. Table 3 shows that the MS-MIDAS-LASSO model has the best out-of-sample forecasting performance, followed by the MIDAS-LASSO model, since they significantly improve the prediction accuracy by 20.102% and 16.547% over that of the benchmark, respectively, but the values of the other models are smaller than 10%. This result is consistent with that found in Table 2. Across all considered predictors, only four economic policy uncertainty indicators (the MPU, HCU, EMV, IDEMV) and the VIX can significantly improve the prediction accuracy of the model for stock market RV. Among them, the of the VIX and EMV are 8.410% and 6.273%, respectively, making them the best and second best predictors, respectively. Finally, MIDAS-PCA, MIDAS-PLS and all combination forecasts besides TMC can also significantly improve the forecasting accuracy of stock market RV, but they cannot beat the MIDAS-RV model extended by the VIX and EMV.

Table 3

Results of the out-of-sample R2 ()

	ΔRoos2(%)	Adjusted MSFE	p-value
GEPU	− 0.873	− 1.777	0.962
EPU	0.325	1.196	0.116
MPU	1.514***	2.407	0.008
TPU	− 1.493	− 2.489	0.994
HCU	0.518*	1.487	0.069
EMV	6.273***	3.224	0.001
IDEMV	0.363**	1.705	0.044
GPR	− 3.753	− 2.297	0.989
EURQ	0.045	0.756	0.225
VIX	8.410***	4.656	0.000
ISEESI	− 2.275	− 1.875	0.970
NSI	− 3.453	− 1.353	0.912
AAII	− 0.612	− 0.858	0.805
OFRFSI	− 4.333	− 0.927	0.823
STLFSI	− 3.134	− 1.447	0.926
MIDAS-RV-PCA	0.870***	2.359	0.009
MIDAS-RV-PLS	0.250***	2.637	0.004
MIDAS-LASSO	16.547***	4.028	0.000
MS-MIDAS-LASSO	20.102***	4.209	0.000
MEAN	1.412***	3.575	0.000
MEDIAN	− 0.786	− 1.468	0.929
TMC	0.605***	2.341	0.010
DMSPE (0.9)	1.724***	3.567	0.000
DMSPE (1.0)	1.550***	3.713	0.000
DMA	0.733**	1.821	0.034
DMS	2.790**	2.261	0.012

This table lists the results of the out-of-sample R2 (), the adjusted MSFE statistics and the corresponding p-values. The results with p-values smaller than 0.1 are bolded, and results with the largest are bolded and underlined

*, ** and ***Denote rejections of the null hypothesis at the 10%, 5% and 1% significance levels, respectively

Results of the direction-of-change test and portfolio performance analysis

Recently, the research of Degiannakis and Filis (2017) pointed out that the direction of change (DoC) is the key to the timing of trades and asset allocation, suggesting that it is also important for determining whether the direction of a volatility forecast is accurate. Therefore, we perform the direction-of-change test in this subsection to further evaluate the out-of-sample forecasting performance of all models of interest. The DoC test starts with measuring the percentage of forecasts that correctly predict the direction of the volatility change, which is defined by:where

Then, the nonparametric test proposed by Pesaran and Timmermann (1992) is used to test the null hypothesis that the DoC ratio of a target model is no larger than that of a random walk.

The results of the DoC test are listed in columns 2–4 of Table 4. The DoC results show that all forecasting models except the MIDAS-IDEMV, MIDAS-PCA and MIDAS-PLS models have greater success ratios than the benchmark. Among them, the MS-MIDAS-LASSO model performs best in forecasting the direction of the stock market RV, as its success ratio is the largest at 0.687. The MIDAS-LASSO model and MIDAS-VIX model take second place in the rankings regarding the forecasting ability with respect to the direction of the stock market RV, with success ratios of 0.673. This also suggests that the VIX can perform better than other predictors in terms of forecasting stock market realized volatility.

Table 4

Results of the direction-of-change (DoC) test and portfolio performance

	DoC			Portfolio performance
	Success ratio	PT statistic	p-value	AUoW
Benchmark	0.640	11.541	0.000	3.430
GEPU	0.644	11.896	0.000	3.430
EPU	0.640	11.604	0.000	3.433
MPU	0.646	12.069	0.000	3.438
TPU	0.646	12.028	0.000	3.414
HCU	0.640	11.599	0.000	3.429
EMV	0.651	12.652	0.000	3.498
IDEMV	0.631	10.932	0.000	3.443
GPR	0.644	11.787	0.000	3.393
EURQ	0.642	11.689	0.000	3.415
VIX	0.673	14.335	0.000	3.527
ISEESI	0.650	12.253	0.000	3.399
NSI	0.651	12.391	0.000	3.422
AAII	0.644	11.976	0.000	3.436
OFRFSI	0.650	12.420	0.000	3.429
STLFSI	0.647	12.249	0.000	3.441
MIDAS-RV-PCA	0.639	11.584	0.000	3.457
MIDAS-RV-PLS	0.635	11.249	0.000	3.457
MIDAS-LASSO	0.673	14.330	0.000	3.554
MS-MIDAS-LASSO	0.687	15.414	0.000	3.582
MEAN	0.649	12.282	0.000	3.444
MEDIAN	0.646	12.022	0.000	3.431
TMC	0.647	12.178	0.000	3.437
DMSPE (0.9)	0.650	12.366	0.000	3.445
DMSPE (1.0)	0.649	12.282	0.000	3.446
DMA	0.653	12.466	0.000	3.402
DMS	0.655	12.622	0.000	3.428

This table presents the results of the direction-of-change test and the evaluation results regarding portfolio performance. For the direction-of-change test, the success ratio, PT statistic and corresponding p-value are reported in this table for each model. The success ratios larger than that of the benchmark are bolded, and the largest success ratio is bolded and underlined. To evaluate the portfolio performance, the CER for a mean–variance function is reported. The CERs larger than that of the benchmark are bolded, and the largest CER is bolded and underlined

Existing research suggests that good asset volatility forecasts should be better applied in practice and have better portfolio performance. Therefore, we further measure the portfolio performance of all models of interest to evaluate their forecasting performance. Considering that Fleming et al. (2001)’s method for portfolio testing depends on both the return and volatility forecasts, we utilize the simple utility-based framework of Bollerslev et al. (2018) to quantify the portfolio performance. Bollerslev et al. (2018)’s method assumes that an investor with a mean–variance preference invests in an asset with time-varying volatility and a constant Sharp ratio. The expected utility can be obtained by:where is the expected RV, SR denotes the reward-to-risk ratio, represents the risk aversion of the investor and is the investor’s wealth at time i.8 Since is not available in practice, we measure the expected utility per unit of wealth by the forecasts of the RV for time i + 1 by: and report the average UoW (AUoW):

The AUoW results are reported in the last column of Table 4. Table 4 shows that the volatility forecasts yielded by the MS-MIDAS-LASSO model result in the highest expected utility at 3.582%, followed by the MIDAS-LASSO model with an expected utility at 3.554%. This further confirms the superior predictive performance of MS-MIDAS-LASSO for stock market volatility. In terms of our predictors of interest, the AUoW results show that the VIX and EMV have the best and second best predictive powers, respectively, compared to the other predictors because the MIDAS-VIX and MIDAS-EMV models produce AUoW values of 3.527% and 3.498%, respectively, and they can hardly be outperformed by other forecasting methods other than the MIDAS-LASSO and MS-MIDAS-LASSO models.

Results of the CumMSE difference method

We find from the empirical results above that the MS-MIDAS-LASSO model has the best out-of-sample forecasting performance, followed by MIDAS-LASSO. In this subsection, we further compare the relative forecasting performance of these two models by a cumulative MSE (CumMSE) method. The CumMSE for the i-th out-of-sample forecast is calculated as:

when i = 1 and . Then, we measure the CumMSE difference between each target model (MIDAS-LASSO and MS-MIDAS-LASSO) and the benchmark model (MIDAS-RV) to quantify the cumulative relative prediction performance. The smaller the CumMSE is, the higher the relative predictive performance of the corresponding model. The CumMSE difference between the MIDAS-LASSO model and MIDAS-RV model is drawn as the red area in Fig. 3, and the CumMSE difference between the MS-MIDAS-LASSO model and MIDAS-RV model is drawn as the blue area. Figure 3 shows that the MIDAS-LASSO model and MS-MIDAS-LASSO model generally have better prediction performance than the benchmark model, and MS-MIDAS-LASSO is better than MIDAS-LASSO in most cases.

Fig. 3

CumMSE differences

Robustness tests

Alternative benchmark

Our empirical results above are evaluated with MIDAS-RV as a benchmark. To further determine the robustness of these findings, we use the MIDAS-RV model with regime switching as a new benchmark, construct the MIDAS-RV-X, MIDAS-PCA and MIDAS-PLS models with regime switching and various MS-MIDAS-RV-X-based combination forecasts, and re-evaluate the out-of-sample forecasting performance of these models and the MIDAS-LASSO models (the MIDAS-LASSO model and MS-MIDAS-LASSO model). The evaluation results are listed in Table 5. Table 5 shows that the MS-MIDAS-LASSO model still has the best forecasting performance since it survives the MCS test with a p-value equal to 1 under all loss function criteria and has the largest and most significant at 20.573%. The predictive performance of the MIDAS-LASSO model follows closely. It passes the MCS test under both the and statistics of the MSE, and its is significant at 17.038%, but the values of the other models are less than 10%. In terms of our predictors of interest, the VIX and EMV still rank first and second in terms of their forecasting ability. This time, all the combination prediction methods can achieve statistically significant forecasting performance. These results confirm the robustness of the findings above.

Table 5

Results of the MCS test and the corresponding out-of-sample R2 values under an alternative benchmark

	QLIKE		MSE		MAE		ΔRoos2(%)
	T_R	T_SQ	T_R	T_SQ	T_R	T_SQ
Benchmark	0.000	0.000	0.193	0.094	0.002	0.012
GEPU	0.000	0.000	0.193	0.094	0.002	0.014	1.424***
EPU	0.000	0.000	0.193	0.094	0.002	0.007	0.557*
MPU	0.000	0.000	0.193	0.094	0.002	0.012	0.838**
TPU	0.000	0.000	0.096	0.083	0.002	0.012	− 0.909
HCU	0.000	0.000	0.133	0.091	0.002	0.010	0.394*
EMV	0.000	0.002	0.193	0.094	0.002	0.014	7.017***
IDEMV	0.000	0.000	0.193	0.094	0.002	0.009	− 1.344
GPR	0.000	0.000	0.106	0.087	0.002	0.014	− 3.924
EURQ	0.000	0.000	0.193	0.094	0.002	0.014	3.038**
VIX	0.007	0.007	0.193	0.094	0.027	0.027	9.366***
ISEESI	0.000	0.000	0.193	0.094	0.002	0.014	− 0.232
NSI	0.000	0.000	0.193	0.094	0.002	0.012	− 2.725
AAII	0.000	0.000	0.193	0.094	0.002	0.008	0.047*
OFRFSI	0.000	0.001	0.193	0.094	0.002	0.014	− 3.137
STLFSI	0.000	0.001	0.193	0.094	0.002	0.014	− 1.039
MIDAS-RV-PCA	0.000	0.001	0.193	0.094	0.002	0.013	− 0.419
MIDAS-RV-PLS	0.000	0.001	0.193	0.094	0.002	0.014	1.735***
MIDAS-LASSO	0.003	0.002	0.193	0.131	0.004	0.014	17.038***
MS-MIDAS-LASSO	1.000	1.000	1.000	1.000	1.000	1.000	20.573***
MEAN	0.000	0.001	0.193	0.094	0.002	0.014	3.021***
MEDIAN	0.000	0.000	0.193	0.094	0.002	0.014	0.922**
TMC	0.000	0.001	0.193	0.094	0.002	0.014	2.107***
DMSPE (0.9)	0.000	0.001	0.193	0.094	0.004	0.014	3.353***
DMSPE (1.0)	0.000	0.002	0.193	0.094	0.004	0.014	3.201***
DMA	0.000	0.001	0.193	0.094	0.002	0.014	2.270**
DMS	0.003	0.002	0.193	0.094	0.004	0.014	3.712**

This table reports the p-values for the MCS test obtained by 10,000 bootstrap simulations as well as the out-of-sample R2 values []. For the MCS test, p-values larger than 0.1 are bolded, indicating that the corresponding models survive the MCS test. The p-values equal to 1 are bolded and underlined, indicating that the corresponding models have the best out-of-sample forecasting performance. The statistical significance for is determined by Clark and West (2007)’s test method

*, ** and ***Denote rejections of the null hypothesis at the 10%, 5% and 1% significance levels, respectively. Significant values are bolded, and the largest is bolded and underlined

Alternative lag

In this section, we change the K value in the MIDAS-RV framework to perform another robustness check. Our analysis above let K be 66 (i.e., one quarter), but considering that the minimum initial sampling frequency of our predictors is 1 month, K is set to 22 in this subsection. The p-values of the MCS test and the corresponding values are reported in Table 6. The evaluation results show that the MIDAS-LASSO and MS-MIDAS-LASSO models have better forecasting performance than the other forecasting models since only these two models can survive the MCS test and have values larger than 10%. Among them, the MS-MIDAS-LASSO model has the best forecasting performance because it passes the MCS test with a p-value equal to 1 under all considered loss functions and has the largest and most significant at 21.719%. The values of the MIDAS-RV-X models further show that the VIX and EMV are the best and the second best predictors, respectively. This time, the MIDAS-RV-PCA and MIDAS-RV-PLS models, as well as all combination forecasts besides MEDIAN, can achieve statistically significant forecasting performance. These findings also confirm the robustness of the findings in Sect. 4.1.

Table 6

Results of the MCS test and the corresponding out-of-sample R2 values obtained under an alternative K

	QLIKE		MSE		MAE		ΔRoos2(%)
	T_R	T_SQ	T_R	T_SQ	T_R	T_SQ
Benchmark	0.000	0.000	0.070	0.071	0.000	0.001
GEPU	0.000	0.000	0.070	0.071	0.000	0.001	− 0.559
EPU	0.000	0.000	0.036	0.069	0.000	0.001	0.337
MPU	0.000	0.000	0.085	0.071	0.000	0.001	1.501***
TPU	0.000	0.000	0.070	0.071	0.000	0.001	− 1.232
HCU	0.000	0.000	0.070	0.071	0.000	0.001	0.571*
EMV	0.000	0.000	0.085	0.071	0.000	0.001	6.301***
IDEMV	0.000	0.000	0.085	0.071	0.000	0.001	− 0.387
GPR	0.000	0.000	0.070	0.071	0.000	0.001	− 1.363
EURQ	0.000	0.000	0.070	0.071	0.000	0.001	0.941*
VIX	0.003	0.005	0.085	0.071	0.016	0.013	8.577***
ISEESI	0.000	0.000	0.085	0.071	0.000	0.001	− 1.946
NSI	0.000	0.000	0.085	0.071	0.000	0.001	− 1.573
AAII	0.000	0.000	0.070	0.071	0.000	0.001	0.157
OFRFSI	0.000	0.000	0.085	0.071	0.000	0.001	− 4.332
STLFSI	0.000	0.000	0.085	0.071	0.000	0.001	− 2.881
MIDAS-RV-PCA	0.000	0.000	0.070	0.071	0.000	0.001	0.910***
MIDAS-RV-PLS	0.000	0.000	0.070	0.071	0.000	0.001	0.286***
MIDAS-LASSO	0.253	0.253	0.099	0.099	0.752	0.752	18.912***
MS-MIDAS-LASSO	1.000	1.000	1.000	1.000	1.000	1.000	21.719***
MEAN	0.000	0.000	0.085	0.071	0.000	0.001	1.789***
MEDIAN	0.000	0.000	0.070	0.071	0.000	0.001	− 0.050
TMC	0.000	0.000	0.085	0.071	0.000	0.001	1.048***
DMSPE (0.9)	0.000	0.000	0.085	0.071	0.000	0.001	2.069***
DMSPE (1.0)	0.000	0.000	0.085	0.071	0.000	0.001	1.920***
DMA	0.000	0.000	0.085	0.071	0.000	0.001	0.877**
DMS	0.000	0.001	0.085	0.071	0.000	0.001	3.265***

This table reports the p-values for the MCS test obtained by 10,000 bootstrap simulations as well as the out-of-sample R2 values (). For the MCS test, p-values larger than 0.1 are bolded, indicating that the corresponding models survive the MCS test. The p-values equal to 1 are bolded and underlined, indicating that the corresponding models have the best out-of-sample forecasting performance. Statistical significance for is determined by Clark and West (2007)’s test method

*, ** and ***Denote rejections of the null hypothesis at the 10%, 5% and 1% significance levels, respectively. Significant values are bolded, and the largest is bolded and underlined

Alternative rolling window length

Studies have shown that different rolling window lengths may yield different out-of-sample forecasts (Li et al., 2020b; Rossi & Inoue, 2012). Therefore, we change the length of the rolling window and re-evaluate the out-of-sample forecasting performance of our prediction methods of interest. Table 7 reports the evaluation results. Table 7 shows that the MS-MIDAS-LASSO model passes the MCS test with a p-value equal to 1 under all loss functions and increases the forecasting accuracy for stock market RV with the largest and most significant (19.212%). These results further suggest that the MS-MIDAS-LASSO model has the best prediction performance. MIDAS-LASSO increases the predictive accuracy a with significant value of 14.256% and is the second-best forecasting model. For the MIDAS-EMV and MIDAS-VIX models, their values are 6.317% and 6.504%, respectively, suggesting that the VIX is the best predictor and that the EMV is second. These findings further support the robustness of the findings in Sect. 4.1.

Table 7

Results of the MCS test and the corresponding out-of-sample R2 values obtained under alternative rolling window lengths

	QLIKE		MSE		MAE		ΔRoos2(%)
	T_R	T_SQ	T_R	T_SQ	T_R	T_SQ
Benchmark	0.000	0.001	0.231	0.168	0.112	0.166
GEPU	0.000	0.001	0.187	0.168	0.013	0.115	− 1.584
EPU	0.000	0.001	0.231	0.168	0.006	0.075	− 0.271
MPU	0.000	0.002	0.187	0.168	0.112	0.154	1.854**
TPU	0.000	0.001	0.187	0.168	0.112	0.247	− 1.908
HCU	0.000	0.002	0.187	0.168	0.013	0.125	0.859**
EMV	0.013	0.011	0.320	0.170	0.013	0.125	6.317***
IDEMV	0.000	0.002	0.231	0.168	0.006	0.073	0.793**
GPR	0.000	0.001	0.187	0.168	0.639	0.571	− 4.897
EURQ	0.000	0.001	0.187	0.168	0.112	0.163	0.423
VIX	0.195	0.195	0.320	0.170	0.639	0.571	6.504***
ISEESI	0.000	0.001	0.187	0.168	0.163	0.475	− 2.646
NSI	0.000	0.002	0.187	0.168	0.013	0.125	− 5.551
AAII	0.000	0.001	0.187	0.168	0.006	0.051	− 1.439
OFRFSI	0.000	0.002	0.187	0.166	0.013	0.111	− 7.337
STLFSI	0.000	0.002	0.187	0.168	0.006	0.051	− 5.654
MIDAS-RV-PCA	0.000	0.002	0.187	0.168	0.006	0.051	− 0.064
MIDAS-RV-PLS	0.000	0.002	0.187	0.168	0.006	0.051	− 1.515
MIDAS-LASSO	0.046	0.052	0.320	0.218	0.639	0.571	14.265***
MS-MIDAS-LASSO	1.000	1.000	1.000	1.000	1.000	1.000	19.212***
MEAN	0.000	0.002	0.231	0.168	0.163	0.475	1.108***
MEDIAN	0.000	0.002	0.231	0.168	0.112	0.163	− 0.993
TMC	0.000	0.002	0.231	0.168	0.112	0.264	0.231
DMSPE (0.9)	0.000	0.002	0.320	0.169	0.639	0.571	1.516***
DMSPE (1.0)	0.009	0.005	0.320	0.170	0.639	0.571	1.237***
DMA	0.000	0.001	0.187	0.168	0.006	0.051	− 3.586
DMS	0.000	0.002	0.187	0.168	0.006	0.073	− 1.217

This table reports the p-values for the MCS test obtained by 10,000 bootstrap simulations as well as the out-of-sample R2 values (). For the MCS test, p-values larger than 0.1 are bolded, indicating that the corresponding models survive the MCS test. The p-values equal to 1 are bolded and underlined, indicating that the corresponding models have the best out-of-sample forecasting performance. Statistical significance for is determined by Clark and West (2007)’s test method

** and *** denote rejections of the null hypothesis at the 5% and 1% significance levels, respectively. Significant values are bolded, and the largest is bolded and underlined

Extension analysis

Forecasting performance compared to that of the traditional HAR-LASSO framework

Our empirical results above demonstrate the superior forecasting performance of the MS-MIDAS-LASSO model. In this subsection, we further compare its predictive performance with that of the traditional HAR-LASSO framework and its regime switching extension. Before the comparison, we use the standard HAR-RV model as the benchmark, construct HAR-RV-X-, HAR-RV-PCA-, HAR-RV-PLS-, HAR-LASSO-, MS-HAR-LASSO models and various HAR-RV-X-based combination forecasts, and evaluate their out-of-sample forecasting performance. Meanwhile, we use the standard HAR-RV model as the benchmark and further evaluate the out-of-sample forecasting performance of MIDAS-RV model and its extensions. The evaluation results are reported in Table 8. Table 8 shows that the MS-HAR-LASSO model has the best out-of-sample forecasting performance in the HAR-RV framework, followed by HAR-LASSO. Interestingly, in the case with the HAR-RV-X model, the VIX and EMV remain the best and second-best predictors, respectively, which further demonstrates the robustness of the predictive power of the VIX and EMV. The evaluation results of MIDAS-RV model and its extensions show that the MIDAS-RV model has a significantly greater than 0, indicating that the out-of-sample prediction accuracy of the MIDAS-RV model surpasses that of the standard HAR-RV model. Also, most of the MIDAS-RV extension models yielded greater than those of the corresponding HAR-RV extension models. These findings further confirmed the advantages of the MIDAS model over HAR. The reason why the MIDAS model shows advantages are detailed in the Introduction section. Finally, Table 8 shows that the out-of-sample forecasting performance of the MS-MIDAS-LASSO model exceeds that of the MS-HAR-LASSO model, and the out-of-sample forecasting performance of the MIDAS-LASSO model is close to HAR-LASSO model, although it does not surpass that of the HAR-LASSO model.

Table 8

Results of the MCS test and the corresponding out-of-sample R2 values for the HAR extensions

	HAR-RV framework			MIDAS-RV framework
	ΔRoos2(%)	Adjusted MSFE	p-value	ΔRoos2(%)	Adjusted MSFE	p-value
RV				2.669***	3.081	0.001
GEPU	− 0.542	− 2.046	0.980	1.819***	2.913	0.002
EPU	0.124	0.661	0.254	2.985***	2.973	0.001
MPU	0.939**	2.122	0.017	4.144***	3.104	0.001
TPU	− 0.669	− 2.444	0.993	1.217***	2.683	0.004
HCU	0.274*	1.316	0.094	3.174***	3.041	0.001
EMV	6.031***	3.083	0.001	8.775***	3.410	0.000
IDEMV	1.933*	1.605	0.054	3.023***	2.409	0.008
GPR	− 0.764	− 2.256	0.988	− 0.983	1.569	0.058
EURQ	0.840	1.182	0.119	2.713***	2.587	0.005
VIX	12.043***	4.798	0.000	10.855***	4.650	0.000
ISEESI	− 1.034	− 1.836	0.967	0.455***	2.407	0.008
NSI	− 0.022	0.543	0.294	− 0.692	1.747	0.040
AAII	− 0.202	− 0.269	0.606	2.074***	2.970	0.001
OFRFSI	− 2.805	− 0.585	0.721	− 1.548	1.492	0.068
STLFSI	− 0.570	− 0.040	0.516	− 0.381	1.749	0.040
PCA	1.459**	2.015	0.022	3.516***	2.643	0.004
PLS	1.208**	2.322	0.010	2.913***	2.721	0.003
LASSO	19.841***	4.527	0.000	18.775***	3.937	0.000
MS-LASSO	20.057***	4.539	0.000	22.235***	4.268	0.000
MEAN	2.037***	3.918	0.000	4.043***	3.553	0.000
MEDIAN	0.278*	1.573	0.058	1.904***	2.855	0.002
TMC	1.094***	2.842	0.002	3.258***	3.304	0.000
DMSPE (0.9)	2.409***	3.915	0.000	4.348***	3.593	0.000
DMSPE (1.0)	2.202***	4.023	0.000	4.178***	3.603	0.000
DMA	4.257***	2.374	0.009	3.383***	2.561	0.005
DMS	5.724***	2.519	0.006	5.385***	2.661	0.004

This table reports the p-values for the MCS test obtained by 10,000 bootstrap simulations as well as the out-of-sample R2 values (). For the MCS test, p-values larger than 0.1 are bolded, indicating that the corresponding models survive the MCS test. The p-values equal to 1 are bolded and underlined, indicating that the corresponding models have the best out-of-sample forecasting performance. Statistical significance for is determined by Clark and West (2007)’s test method

*, ** and ***Denote rejections of the null hypothesis at the 10%, 5% and 1% significance levels, respectively. Significant values are bolded, and the largest is bolded and underlined

Next, we compare the forecasting performance of the HAR-LASSO, MS-HAR-LASSO, MIDAS-LASSO and MS-MIDAS-LASSO models by the MCS test. The evaluation results are presented in Table 9. Table 9 shows that although all four models can pass the MCS test under different loss functions, MS-MIDAS-LASSO exhibits the best out-of-sample prediction performance, as it survives the MCS test with a p-value equal to 1 under all loss functions. These findings further suggest that the MS-MIDAS-LASSO model can further improve prediction accuracy for stock market RV on the basis of MS-HAR-LASSO.

Table 9

Out-of-sample forecasting performance of the HAR-LASSO and MIDAS-LASSO models

	QLIKE		MSE		MAE
	T_R	T_SQ	T_R	T_SQ	T_R	T_SQ
HAR-LASSO	0.116	0.085	0.266	0.208	0.075	0.063
MS-HAR-LASSO	0.116	0.085	0.266	0.208	0.075	0.063
MIDAS-LASSO	0.002	0.002	0.266	0.208	0.002	0.002
MS-MIDAS-LASSO	1.000	1.000	1.000	1.000	1.000	1.000

This table reports the p-values for the MCS test obtained by 10,000 bootstrap simulations as well as the out-of-sample R2 values (). For the MCS test, p-values larger than 0.1 are bolded, indicating that the corresponding models survive the MCS test. The p-values equal to 1 are bolded and underlined, indicating that the corresponding models have the best out-of-sample forecasting performance

Forecasting performance during high/low-volatility regimes

Studies have shown that the forecasting ability of a model may vary under different volatility regimes (Li et al., 2020b; Wang et al., 2018). However, the MS-MIDAS-LASSO model is constructed precisely by considering both high- and low-volatility regimes, and therefore, it should have the best forecasting performance under both scenarios. To confirm this, the prediction performance of all forecasting methods of interest during high- and low-volatility periods is explored in this subsection. We distinguish the high- and low-volatility components by the median of the out-of-sample true volatility. Table 10 presents the evaluation results of the out-of-sample prediction performance during high-volatility regimes. During the high-volatility period, the MS-MIDAS-LASSO model and MIDAS-LASSO model still have better forecasting performance than the other models, and the MS-MIDAS-LASSO model performs better than the MIDAS-LASSO model. In terms of our considered predictors, the EMV and VIX have better out-of-sample forecasting ability than the other predictors, and the VIX performs better than the EMV.

Table 10

Results of the MCS test and the corresponding out-of-sample R2 values obtained during high-volatility regimes

	QLIKE		MSE		MAE		ΔRoos2(%)
	T_R	T_SQ	T_R	T_SQ	T_R	T_SQ
Benchmark	0.001	0.004	0.081	0.094	0.008	0.034
GEPU	0.001	0.004	0.052	0.094	0.005	0.034	− 0.901
EPU	0.001	0.004	0.046	0.094	0.002	0.029	0.253
MPU	0.001	0.004	0.151	0.094	0.008	0.034	1.510**
TPU	0.000	0.002	0.046	0.089	0.002	0.027	− 1.844
HCU	0.001	0.004	0.081	0.094	0.008	0.034	0.497*
EMV	0.039	0.077	0.151	0.094	0.090	0.103	6.965***
IDEMV	0.003	0.006	0.140	0.094	0.002	0.027	0.645**
GPR	0.000	0.000	0.046	0.086	0.002	0.017	− 4.951
EURQ	0.000	0.002	0.046	0.094	0.002	0.020	− 0.380
VIX	0.230	0.296	0.151	0.094	0.275	0.237	7.955***
ISEESI	0.000	0.001	0.046	0.083	0.002	0.023	− 3.191
NSI	0.001	0.004	0.052	0.094	0.005	0.034	− 4.046
AAII	0.001	0.005	0.112	0.094	0.008	0.034	− 0.539
OFRFSI	0.020	0.010	0.140	0.094	0.090	0.103	− 4.938
STLFSI	0.029	0.020	0.151	0.094	0.008	0.072	− 3.256
MIDAS-RV-PCA	0.003	0.006	0.081	0.094	0.005	0.034	0.799**
MIDAS-RV-PLS	0.003	0.005	0.052	0.094	0.004	0.033	− 0.224
MIDAS-LASSO	0.419	0.419	0.451	0.451	0.275	0.237	17.635***
MS-MIDAS-LASSO	1.000	1.000	1.000	1.000	1.000	1.000	19.658***
MEAN	0.001	0.005	0.112	0.094	0.008	0.072	1.169***
MEDIAN	0.000	0.003	0.046	0.094	0.005	0.034	− 1.079
TMC	0.000	0.003	0.081	0.094	0.008	0.036	0.353*
DMSPE (0.9)	0.001	0.005	0.151	0.094	0.109	0.104	1.478***
DMSPE (1.0)	0.003	0.006	0.151	0.094	0.109	0.103	1.302***
DMA	0.000	0.001	0.046	0.094	0.002	0.023	− 0.718
DMS	0.000	0.002	0.151	0.094	0.002	0.029	1.450**

This table reports the p-values for the MCS test obtained by 10,000 bootstrap simulations as well as the out-of-sample R2 []. For the MCS test, p-values larger than 0.1 are bolded, indicating that the corresponding models survive the MCS test. The p-values equal to 1 are bolded and underlined, indicating that the corresponding models have the best out-of-sample forecasting performance. Statistical significance for is determined by Clark and West (2007)’s test method

*, ** and ***Denote rejections of the null hypothesis at the 10%, 5% and 1% significance levels, respectively. Significant values are bolded, and the largest is bolded and underlined

The evaluation results regarding the out-of-sample predictive performance during low-volatility regimes are reported in Table 11. We find that the MS-MIDAS-LASSO model still has the highest forecasting accuracy, but the prediction accuracy of the MIDAS-LASSO model is substantially reduced, and its does not exceed the values of other forecasting methods such as MIDAS-PCA and DMSPE (0.9). This further suggests that the MS-MIDAS-LASSO model performs better than the MIDAS-LASSO model in capturing regime switching with respect to stock market volatility. In terms of the predictors, the VIX is still the best predictor, but the forecasting ability of the EMV decreases substantially and can no longer improve the forecasting accuracy for stock market RV. However, different from the findings in Sect. 4, the forecasting performance of the MIDAS-VIX model can no longer exceed that of DMA and DMS. Finally, although our findings in Sect. 4 find that the three market sentiment indicators other than the VIX and the financial stress indicators have nearly no ability to improve the predictive accuracy for stock market RV, during low-volatility regimes, we find that the ISEESI, NSI and OFRFSI have the ability to improve the forecasting accuracy.

Table 11

Results of the MCS test and the corresponding out-of-sample R2 values obtained during low-volatility regimes

	QLIKE		MSE		MAE		ΔRoos2(%)
	T_R	T_SQ	T_R	T_SQ	T_R	T_SQ
Benchmark	0.000	0.000	0.000	0.000	0.000	0.000
GEPU	0.000	0.000	0.000	0.001	0.000	0.000	− 0.557
EPU	0.000	0.000	0.000	0.000	0.000	0.000	1.140***
MPU	0.000	0.000	0.000	0.000	0.000	0.000	1.565***
TPU	0.000	0.000	0.000	0.005	0.000	0.000	2.495***
HCU	0.000	0.000	0.000	0.000	0.000	0.000	0.765***
EMV	0.000	0.000	0.000	0.001	0.000	0.000	− 1.581
IDEMV	0.000	0.000	0.000	0.000	0.000	0.000	− 2.842
GPR	0.008	0.010	0.077	0.079	0.029	0.028	9.856***
EURQ	0.000	0.000	0.003	0.009	0.000	0.000	4.874***
VIX	0.009	0.038	0.145	0.207	0.029	0.027	13.581***
ISEESI	0.000	0.003	0.009	0.026	0.001	0.003	8.126***
NSI	0.000	0.000	0.000	0.003	0.000	0.000	3.281***
AAII	0.000	0.000	0.000	0.000	0.000	0.000	− 1.435
OFRFSI	0.000	0.000	0.000	0.001	0.000	0.000	2.544***
STLFSI	0.000	0.000	0.000	0.000	0.000	0.000	− 1.749
MIDAS-RV-PCA	0.000	0.000	0.000	0.001	0.000	0.000	1.683***
MIDAS-RV-PLS	0.000	0.000	0.003	0.008	0.000	0.000	5.629***
MIDAS-LASSO	0.000	0.001	0.022	0.052	0.001	0.002	4.189***
MS-MIDAS-LASSO	1.000	1.000	1.000	1.000	1.000	1.000	25.147***
MEAN	0.000	0.000	0.000	0.001	0.000	0.000	4.162***
MEDIAN	0.000	0.000	0.000	0.000	0.000	0.000	2.534***
TMC	0.000	0.000	0.000	0.001	0.000	0.000	3.461***
DMSPE (0.9)	0.000	0.000	0.000	0.002	0.000	0.000	4.522***
DMSPE (1.0)	0.000	0.000	0.000	0.001	0.000	0.000	4.365***
DMA	0.009	0.038	0.389	0.253	0.391	0.401	17.202***
DMS	0.376	0.376	0.389	0.290	0.487	0.487	18.016***

This table reports the p-values for the MCS test obtained by 10,000 bootstrap simulations as well as the out-of-sample R2 values []. For the MCS test, p-values larger than 0.1 are bolded, indicating that the corresponding models survive the MCS test. The p-values equal to 1 are bolded and underlined, indicating that the corresponding models have the best out-of-sample forecasting performance. Statistical significance for is determined by Clark and West (2007)’s test method

***denotes rejections of the null hypothesis at the 1% significance level. Significant values are bolded, and the largest is bolded and underlined

The impact of the COVID-19 pandemic

Our out-of-sample forecasts cover the period after the outbreak of the COVID-19 epidemic. The outbreak of COVID-19 has had a great impact on the global economy and society; as a result, the US stock market has experienced turbulence. For example, the US stock market experienced four meltdowns in less than ten days from March 9 to March 19, 2020. The dramatic volatility of the US stock market during the COVID-19 epidemic poses a significant challenge to investors’ volatility forecasting since some effective forecasting methods may no longer be valid during this period. Therefore, it is also particularly important to examine the forecasting performance of the tested forecasting methods in this paper during the COVID-19 epidemic. Table 12 shows the evaluation results regarding the out-of-sample prediction performance of all models during the COVID-19 epidemic.9 We find that the MS-MIDAS-LASSO model can still pass the MCS test with a p-value equal to 1 under all three loss functions and can also significantly increase the forecasting accuracy with the highest , indicating that the superior forecasting performance of this model has not changed with the outbreak of the COVID-19 epidemic. This time, the MIDAS-LASSO model still has a smaller than that of MS-MIDAS-LASSO but larger than those of the other models, indicating that the prediction performance of this model is also unaffected by the COVID-19 epidemic. However, the predictive powers of the EMV and VIX have changed during this period. This time, the ability of the VIX to improve out-of-sample prediction accuracy diminishes, and the EMV replaces it as the best predictor. In fact, the outbreak of COVID-19 has caused great uncertainty in the global economy, trade, imports and exports as well as market sentiment, etc. The stock market price volatility during this period are also the result of multiple uncertainties. However, the VIX can only capture the changes of the stock market sentiment so its information content is not sufficient enough to predict stock market volatility during the COVID-19 pandemic. Meanwhile, since EMV quantifies multiple uncertainty information related to stock market price volatility during its construction (Baker et al., 2019), it is not surprising that it exhibits strong predictive power during COVID-19 pandemic.

Table 12

Results of the MCS test and the corresponding out-of-sample R2 values obtained during COVID-19

	QLIKE		MSE		MAE		ΔRoos2(%)
	T_R	T_SQ	T_R	T_SQ	T_R	T_SQ
Benchmark	0.002	0.050	0.380	0.369	0.777	0.827
GEPU	0.002	0.039	0.380	0.346	0.777	0.778	− 2.310
EPU	0.002	0.056	0.380	0.369	0.120	0.565	0.621
MPU	0.002	0.067	0.510	0.369	0.945	0.945	3.309**
TPU	0.002	0.047	0.380	0.337	0.777	0.753	− 2.885
HCU	0.002	0.065	0.380	0.369	0.777	0.827	1.485**
EMV	0.002	0.067	0.510	0.369	0.120	0.565	8.199**
IDEMV	0.002	0.067	0.380	0.369	0.120	0.389	1.481**
GPR	0.002	0.054	0.380	0.309	0.120	0.565	− 7.824
EURQ	0.002	0.055	0.380	0.369	0.120	0.565	0.990
VIX	0.002	0.067	0.380	0.369	0.120	0.529	4.575***
ISEESI	0.002	0.044	0.380	0.324	0.768	0.682	− 4.314
NSI	0.002	0.067	0.380	0.315	0.120	0.463	− 8.164
AAII	0.002	0.055	0.380	0.358	0.120	0.565	− 2.018
OFRFSI	0.002	0.067	0.380	0.304	0.120	0.565	− 10.727
STLFSI	0.002	0.067	0.380	0.298	0.120	0.457	− 9.122
MIDAS-RV-PCA	0.002	0.067	0.380	0.369	0.120	0.271	0.781*
MIDAS-RV-PLS	0.002	0.067	0.380	0.358	0.120	0.316	− 0.434
MIDAS-LASSO	0.035	0.067	0.510	0.446	0.120	0.565	14.635**
MS-MIDAS-LASSO	1.000	1.000	1.000	1.000	1.000	1.000	17.680***
MEAN	0.002	0.067	0.380	0.369	0.777	0.778	1.068*
MEDIAN	0.002	0.055	0.380	0.331	0.120	0.565	− 2.573
TMC	0.002	0.067	0.380	0.369	0.768	0.710	0.076
DMSPE (0.9)	0.002	0.067	0.510	0.369	0.777	0.827	1.622*
DMSPE (1.0)	0.002	0.067	0.380	0.369	0.777	0.805	1.194*
DMA	0.002	0.067	0.380	0.369	0.120	0.565	1.100
DMS	0.035	0.067	0.413	0.369	0.120	0.565	3.746**

This table reports the p-values for the MCS test obtained by 10,000 bootstrap simulations as well as the out-of-sample R2 values []. For the MCS test, p-values larger than 0.1 are bolded, indicating that the corresponding models survive the MCS test. The p-values equal to 1 are bolded and underlined, indicating that the corresponding models have the best out-of-sample forecasting performance. Statistical significance for is determined by Clark and West (2007)’s test method

*, ** and ***Denote rejections of the null hypothesis at the 10%, 5% and 1% significance levels, respectively. Significant values are bolded, and the largest is bolded and underlined

Conclusions

In this paper, we explore the effectiveness of 9 economic policy uncertainty indices, 4 market sentiment indicators and 2 financial stress indices in predicting the realized volatility of the S&P 500 index. We use several MIDAS-RV-X models to determine the predictive power of individual predictors and further construct PCA-, PLS- and LASSO-based MIDAS-RV extensions, a MIDAS-LASSO model with regime switching and various combination forecasts to find the optimal forecasting strategy. The out-of-sample forecasting performance of our forecasting models of interest is evaluated mainly by the MCS test and out-of-sample R2 statistics.

The empirical results first show that the economic policy uncertainty indicators (especially the EMV) and the VIX are more likely to produce better forecasting accuracy than the three market sentiment indicators and two financial stress indices. Among these predictors, the VIX has the strongest predictive power, followed by the EMV. Second, the combination forecasts and MIDAS-LASSO models usually have statistically significant relative forecasting performance. Across all forecasting models of interest, the MIDAS-LASSO model with regime switching has the best forecasting performance, followed by the MIDAS-LASSO model, indicating that LASSO technology can capture more valuable information from a large set of predictors than other information integration methods. The forecasting accuracy of MIDAS-LASSO with regime switching is also superior to that of the traditional HAR-LASSO method and its regime switching extension. When distinguishing between high- and low-volatility regimes, we find that the VIX is the best predictor under both scenarios. In addition, the economic policy uncertainty indicators other than GEPU either help predict high stock market volatility or low stock market volatility, but the market sentiment indices other than the VIX and the financial stress index (OFRFSI) exhibit statistically significant relative predictive performance only during low-volatility regimes. Across all considered forecasting methods, the MS-MIDAS-LASSO model still exhibits the best predictive performance during both high- and low-volatility regimes, but the MIDAS-LASSO model can outperform other forecasting methods except for MS-MIDAS-LASSO only during high-volatility regimes. We finally consider the impact of the COVID-19 epidemic. During the COVID-19 epidemic, the predictive power of the VIX diminished, and it was no longer the best out-of-sample predictor, while the predictive power of the EMV increased, making it the best predictor during this special period. However, the MIDAS-LASSO model with regime switching still exhibited the best forecasting performance across all considered forecasting models during the COVID-19 epidemic.

This paper has important practical implications. Firstly, the MIDAS-RV extension models (especially the MS-MIDAS-LASSO model) constructed in this paper are important extensions to existing volatility forecasting research methods and helps researchers and market participants to make full use of key information from multiple potential predictors to generate more accurate forecasts of asset volatility. Secondly, this paper explores the predictive validity of multiple indicators for stock market price volatility, which can help stock market investors to better utilize available information to predict possible stock market volatility and thus adjust their portfolios in time to reduce losses. Meanwhile, this paper is also helpful for derivatives market participants to make better use of the available information to price stock market derivatives more accurately and thus achieve higher returns. Finally, the empirical findings can also help the policy makers to make better use of the available information to formulate strategies timely to prevent and control possible risks in the stock market and make certain contributions to the stable and healthy development of the national economy.