Modeling Asymmetry in the Time-Distance Relation of Ordinal Personality Items.

In analyzing responses and response times to personality questionnaire items, models have been proposed which include the so-called "inverted-U effect." These models predict that response times to personality test items decrease as the latent trait value of a given person gets closer to the attractiveness of an item. Initial studies into these models have focused on dichotomous personality items, and more recently, models for Likert-type scale items have been proposed. In all these models, it is assumed that the inverted-U effect is symmetrical around 0, while, as will be explained in this article, there are substantive and statistical reasons to study this assumption. Therefore, in this article, a general inverted-U model is proposed which accommodates two sources of asymmetry between the response times and the attractiveness of the items. The viability of this model is demonstrated in a simulation study, and the model is applied to the responses and response times of the Temperament and Character Inventory-Revised, covering a broad range of personality dimensions.

Personality assessment is commonly conducted using self-report questionnaires in which subjects have to indicate to what extend a number of statements (items) describe their personality. To make inferences about personality, properties of the item statement are separated from the properties of the person. Next, the person and item properties can be quantified on the same underlying personality trait dimension (e.g., Kuncel, 1977). Kuiper (1981) demonstrated that the discrepancy between the person and the item is related to the time needed by the subject to answer the item. Specifically, it was shown that response times decrease for an increasing distance between the item and person position on the underlying trait dimension. Kuiper referred to this distance–time relation as the inverted-U effect. To investigate the inverted-U effect in dichotomous personality items, researchers relied on Item Response Theory (IRT) models (e.g., Ferrando, 2006) to infer the person positions on the trait (i.e., the latent trait parameter) and the item positions on the trait (i.e., the item attractiveness parameter). Next, the absolute difference between the latent trait and the item attractiveness was used in a separate, more traditional statistical analysis to test whether the hypothesized inverted-U effect holds. For instance, Ferrando (2006) correlated the item-person distances to the response times of a neuroticism and an extraversion scale and found all correlations to be negative which supports the inverted-U effect.

In approaches like these, individual differences in overall speed are not accounted for as the response times lack a general measurement model. Therefore, Ferrando and Lorenzo-Seva (2007a) proposed a multistep modeling approach to test the inverted-U effect using measurement models for both the response times and the (dichotomous) responses. In Ferrando and Lorenzo-Seva (2007b), a similar approach was presented for Likert-type scale personality items by treating the ordinal responses as approximately continuous. Using these IRT approaches, support for the inverted-U effect was found for both a dichotomous and a Likert-type scale questionnaire measuring neuroticism and extraversion.

The inverted-U approaches above are multistep procedures that first estimate the IRT parameters from the responses and then use these in subsequent modeling of the response times. Ranger (2013) and Molenaar et al. (2015) proposed single-step modeling approaches to test the inverted-U effect where, in the case of Likert-type data, the ordinal nature of the responses is explicitly taken into account. Both Ranger (2013) and Molenaar et al. (2015) applied the models to neuroticism and extraversion data and found support for the inverted-U effect.

Models for the inverted-U effect can be related to unfolding models (Coombs, 1964). Unfolding models are suitable for dichotomous or polytomous disagree-agree items in which the response process is proximity based (Roberts et al., 1999). That is, subjects are assumed to evaluate how closely a given statement matches their own opinion. Proximity-based items provide information on the absolute distance between the person and the item, but not on the direction of this distance. That is, if the distance between person and item is small, the probability of an “agree”-response is large, while if the distance is large (i.e., either the person is located above or below the item location), the probability of an “agree”-response is small. Earlier work has focused on the hyperbolic cosine model for dichotomous items (Andrich & Luo, 1993), but later, a generalized modeling framework was developed for polytomous items (Roberts et al., 2000).

Unfolding models and the inverted-U models considered here have in common that, from the observed data of a given item, the location of the person with respect to the location of the item can only be inferred if the response is (strongly) agree, or if the response time is small. That is, in an unfolding model, if the response is (strongly) agree, the person location is close to the item location. Similarly, in an inverted-U model, if the response time is large, the person location is close to the item location. However, if the response is (strongly) disagree (in an unfolding model) or the response time is small (in an inverted-U model), it can only be concluded that the person location is off the item location, but it cannot be concluded whether the person location is above or below the item location. As a result, the expected response function in the unfolding model, and the expected response time function in the inverted-U model are both symmetric and single peaked. Therefore, the inverted-U model is in essence an unfolding model for continuous items.[1]

Symmetry of the Inverted-U Effect

Although the inverted-U effect seems to be well established (at least for extraversion and neuroticism), in all approaches above it is assumed that the inverted-U effect is symmetrical. This assumption implies that (a) the response times decrease with the same rate to the left and right of the inflection point of the distance–time relation; and (b) the inflection point of the distance–time relation is located at 0. In the literature, this assumption has not been explicitly discussed or studied before. However, this assumption deserves attention for both substantive and statistical reasons as is argued below.

Substantive Perspective

There are a number of theories in which the question of asymmetry is interesting for our understanding of the processes underlying personality measurement. Below are some examples.

Bipolar traits

Fekken and Holden (1992) discuss how personality traits are most often bipolar with the one end of the dimension representing the opposite of the other end of the dimension (e.g., dependent—independent). Subjects can have two separate prototypes about the self, one for “dependent” and one for “independent”, formed on the basis of past experiences (Markus, 1977). Although this theory does not include the notion of “distance”, it can be inferred that if the decision processes of responding according to the “dependent”-prototype are the same as the decision process of responding according to the “independent”-prototype, similar response times are expected across the two processes. However, if the processes differ, different response times may arise for items that are located more toward the “independent”-end of the personality dimension as compared with the items that are more located toward the “dependent”-end of the dimension. This results in asymmetry in the inverted-U effect as the sign of the difference between person and item (i.e., whether the subject’s trait position is above or below the attractiveness) does matter. The exact direction of this effect (i.e., whether positive distances are associated with faster or slower responses) depends on the properties of the two decision processes.

Dual processing

A related idea is the theory on dual processing (Shiffrin & Schneider, 1977). According to this theory, faster responses reflect more automated processes that are proceduralized, parallel, and do not require active control, while slower responses reflect more controlled processes that are serial and require attentional control. This theory has been successfully applied to cognitive abilities (e.g., Goldhammer et al., 2014), but it may also be applied to personality. That is, if the inverted-U effect is explained from the dual processing theory, the dual processing theory implies that subjects are more likely to use controlled processes if the distance between item and person is small, while subjects may be more likely to adopt an automated process if the distance between the person and the item is large. Again, this explanation assumes that the automated processes for subjects on the higher end of the personality dimension are exactly the same as the automated processes for subjects on the lower end of the personality dimension, and it would be interesting to study this assumption to get insight in the processes underlying the responses to personality questionnaire items.

Diffusion model

Tuerlinckx and De Boeck (2005) and Tuerlinckx et al. (2016) showed how the diffusion model (Ratcliff, 1978) can account for the decision processes underlying personality questionnaire items. In this model, it is assumed that subjects use memory retrieval to accumulate evidence to justify either endorsing a personality statement (e.g., “I like meeting new people”) or rejecting it. This process is terminated, and a decision is made, if the amount of evidence retrieved exceeds a threshold. The diffusion model for personality includes the inverted-U effect naturally (see Tuerlinckx et al., 2016) as the rate of evidence accumulation (memory retrieval) is defined as the distance between the person and the item. If this distance is large (i.e., a high rate of evidence accumulation), retrieving relevant information from memory is easier and faster, and if the distance is small (i.e., a low rate of evidence accumulation), memory retrieval is more challenging and takes longer. Tuerlinckx and De Boeck (2005) and Tuerlinckx et al. (2016) explicitly assume the process to be symmetrical, that is, in deriving their model, they assume that an equal amount of evidence needs to be accumulated to reject the statement or to endorse the statement. However, it might be that for some personality constructs, it requires more evidence before a statement can be endorsed than before a statement can be rejected. This could be due to the sensitivity of the topic (e.g., insecurity, social desirability, neuroticism; some things are hard to admit but easy to deny), due to a reject response being associated with a different process than an endorse response, or due to differences in prototypes as discussed above (then the reject and endorse options correspond to the “independent”–“dependent” options). Nevertheless, different thresholds in the diffusion process model will result in an asymmetric process: different response times for responses endorsing the personality statement as compared with responses rejecting the personality statement. As a result, the presence of different thresholds introduces asymmetry in the inverted-U effect. Studying this asymmetry will thus inform us about the nature of the decision process as operationalized by the diffusion model. Note that although the diffusion model for personality as discussed above was developed for dichotomous personality items, the same predictions follow for Likert-type items.

Statistical Perspective

As discussed above, in modeling personality data, the inverted-U approach assumes symmetry of the effect. Statistically, it is valuable to be able to test this assumption (a) to ensure that the symmetric inverted-U model is appropriate for the data; (b) to quantify the size of possible violations; (c) to test the robustness of the results in the presence of any violations; and (d) to account for violations if they are significantly affecting the results. As the authors will demonstrate in the simulation study, bias may occur in the standard errors and parameter estimates of the symmetric inverted-U model if unmodelled asymmetry is present in the data. In practice, to be able to detect such unmodelled asymmetry and to investigate what effects it has for the application at hand, the methodology presented in this article is of importance.

The Present Study

Thus, statistical approaches are desired that enable studying asymmetry in the distance–time relation (a) to elaborate on theories of response processes in personality measurement and (b) to statistically test for the assumption of symmetry and account for possible violations. Therefore, in this article the viability of a one-step Bayesian implementation of a general inverted-U model is studied inspired by Ferrando and Lorenzo-Seva (2007a), Molenaar et al. (2015), and Ranger (2013) for Likert-type personality data. Within this model, two specific tests on the symmetry of the inverted-U effect are proposed and the biasing effects of neglecting asymmetry is demonstrated. In addition, the model from this article is applied to the responses and response times of the Temperament and Character Inventory–Revised (TCI-R; Cloninger, 1999) to test inverted-U effect and its symmetry in a broad range of personality constructs.

The outline of this article is as follows: First, the existing approaches to study the time–distance relation are formally presented. Next, a modeling approach to test the inverted-U effect and its symmetry is proposed in a Bayesian framework. Then, the viability of this approach and the effects of neglecting departures from symmetry are investigated in a simulation study. Finally, the approach is applied to the TCI-R and discussed in a general discussion.

Existing Modeling Approaches for the Inverted-U Effect

As discussed above, at first, the inverted-U effect was tested using more traditional statistical approaches. For instance, Holden and Fekken (1991) correlated the sum scores of a personality test with the response times of the endorsed items (which are expected to be positive in the case of an inverted-U effect), and with the rejected items (which are expected to be negative in the case of an inverted-U effect). In addition, Ferrando (2006) and Kuncel (1977) used IRT models to estimate the distance between the item and the person, and analyzed the relation between these distances and response times using correlations and the nonparametric sign-test.

Although valuable, the above approaches are suboptimal as they either do not separate measurement error, person effects, and/or item effects from the responses and/or response times, and they do not account for the uncertainties in the model parameter estimates in analyzing the relation between model-based distances and the response times (but see Ferrando, 2006, for a correction). To improve upon these aspects in studying the inverted-U effect, Ferrando and Lorenzo-Seva (2007a) proposed a simultaneous modeling approach in which (a) person and item effects are explicitly separated in the responses and response times; and (b) the response times are regressed on the distances between the person and the item in the same model. Specifically, if denotes the response of person to item and denotes the corresponding log-transformed response time, the model is given by

with , where is the logistic function, is a latent variable denoting the personality trait level for the person, is a latent variable denoting the overall speed of the person, is an intercept denoting the time intensity of the item (e.g., some items require more reading than others), is a discrimination parameter, is the item attractiveness, is a slope parameter that controls the slope of the inverted-U effect, and is the residual variance.

Likert-type Data

As it stands, the model above is only applicable to dichotomous personality items (yes-no). In the case of Likert-type items, Ferrando and Lorenzo-Seva (2007b) proposed an inverted-U model for the reciprocal response times and the ordinal items scores. In this model, the ordinal item scores are treated as approximately continuous to enable linear factor modeling. In this study, the focus is on the approach by Ranger (2013) and Molenaar et al. (2015) who used the Graded Response Model (GRM; Samejima, 1969) for the responses. That is, the item scores are explicitly treated as ordinal, that is,

where is the attractiveness parameter for category , with , where represents the total number of categories. In addition, both Ranger and Molenaar et al. adopted log-response times in the response time part of the model, used quadratic distances, and specified an overall parameter, that is,

where is the median for item which resembles the attractiveness parameter above. Below, is referred to as an “index” as in the present study is not a free parameter. However, is useful as an overall measure for the attractiveness of an item.

Focusing on quadratic distances in Equation 4 instead of absolute distances as in Equation 2 has the advantage that the resulting model has the form and can therefore be fit using conventional estimation procedures (see, e.g., Akrami et al., 2007; Molenaar et al., 2015; Ranger, 2013). Other possibilities besides the absolute and quadratic distances include regression of the log-response times on (Ranger, 2013; Ranger & Ortner, 2011) and (Meng et al., 2014). These approaches are discussed in the discussion section.

Testing the Symmetry of the Inverted-U Effect

In the present article, the following specification for the traditional (symmetric) inverted-U model from Equation 2 with replaced by , will be used

In addition, for the ordinal responses, the GRM from Equation 3 will be adopted. Within this symmetric inverted-U model for responses and response times a test on the symmetry assumption underlying the inverted-U effect will be proposed. Note that using the log-transformation for the response times in Equation 5 is intended to linearize the relation between the response times and the underlying speed variable , and to make the assumption of homoscedastic residuals more plausible. Note that as log(y) = log(|-y|) for , modeling the log-response times does not affect the symmetry of the inverted-U effect. That is, the inverted-U effect underlying the log-response times is still symmetric with its inflection point at a zero distance between the person and the item position.

In the top left plot in Figure 1, the expected log-response times of the model in Equation 5 are plotted as a function of the difference between the item and the person, . As can be seen, this function has an inflection point at with the absolute slope in the interval equal to the absolute slope in the interval . In principle, if the asymmetry is strong, this will be evident from the residuals of the symmetric model in Equation 5 (or from other model fit measures, e.g., Sinharay, 2018). However, consulting the residuals might be a suboptimal way to test for asymmetry as this approach is post hoc, exploratory, and lacks an explicit formal model for the asymmetry, making hypothesis testing challenging.

Figure 1.

Illustration of the distance–time relation for different configurations of the and parameter in the asymmetric inverted-U model, in all plots; in addition, the intercept and the speed variable are set to 0.

Therefore, an explicit statistical model is proposed which can be used to formally test whether the distance between the person and the item is symmetrical around . Specifically, two additional parameters that account for asymmetry are introduced: a difference in the absolute slope before and after the inflection point, and an inflection point that deviates from . Specifically, the following asymmetric inverted-U model for the response times is proposed:

In this model, and are the new parameters that account, respectively, for differences in the absolute slope before and after the inflection point, and for the exact location of the inflection point in the time–distance relation. Note that the above implies that

In Equation 6, two inequality constraints apply: is assumed to be larger than 0, and the absolute value of should be smaller or equal to . The reasons for these constraints are the following. First, the model is used to test the asymmetry of the distance–time relation. Therefore, the distance–time relation is assumed to hold (i.e., > 0). The second inequality constraint is imposed for theoretical and numerical reasons. The constraint ensures that the relation between distance and time retains an inverted-U shape. That is, if , the relation between is strictly increasing (if ) or decreasing (if ) across the person-item distance . Such an increasing relation is theoretically not desirable as the key of the inverted-U effect is that response times first increase and then decrease across the distance between the person and the item. In addition, numerically, if the inflection point parameter becomes unidentified as the distance–time relation approaches a linear function which does not have an inflection point. See Figure 1 for a graphical illustration of the relation between the log-response times and the distance between the item and the person for different parameter configurations.

In the model above, the latent variables and are uncorrelated, . That is, a possible covariance is modeled via because asymmetry in the inverted-U effect implies that and covary. That is, the term in Equation 6 that handles asymmetry in the inverted-U effect can be written as . If for all items, is equivalent to (this follows from a Cholesky decomposition of the covariance matrix of the latent variables; see Molenaar et al., 2015, p. 62). However, in practice, is not equal to 1 for all items, making connected but not identical to . In addition, also contributes to the covariance between and . That is, if the response time inflection point is larger or smaller than 0, slower responses are associated with respectively larger or smaller values of , which introduces an association between and .[2]

Thus, using the two new parameters and one can study whether a correlation between and as found in the symmetrical inverted-U model (or other item response theory approaches, e.g., the hierarchical model; van der Linden, 2007) is due to asymmetry in the inverted-U effect and/or an response time inflection point other than 0 . A final and related note is that—due to the above way in which the correlation between and is treated in the model—the asymmetric model is not nested within the symmetric model. However, as will be shown in the simulation study below, the models can be validly compared using the Deviance Information Criterion (DIC) fit index.

Estimation of the Model

Below, a procedure to fit the symmetric and the asymmetric inverted-U models to data is outlined. First, the item parameter in vector and the person parameters for person in vector are collected. Then, by assuming independence of and conditional on and , and by assuming a normal distribution for conditional on and , the likelihood function of the model given by Equations 3 and 6 above is

where (.) is the standard normal density function, denotes the number of items, is determined from Equation 3 and is given in Equation 5 for the symmetric inverted-U model and by Equation 6 for the asymmetric inverted-U model.

The model above can be fit in a frequentist item response theory framework by specifying prior distributions for the random person parameters in , numerically integrating over these random person variables, and maximizing the resulting approximate likelihood function over the unknown parameters in η0 for a given sample. In the present article, however, a Bayesian item response theory framework is adopted in which prior distributions for all parameter in η1p and η0 are specified. First, a bivariate normal distribution for the vector is adopted to reflect the distribution of and in the population of test takers, that is

This parameterization can be derived by a Cholesky decomposition of the covariance matrix of and (see, e.g., Fox et al., 2007). Note that in the asymmetric model as discussed above. The prior specification for and above ensures that and that the covariance matrix of η1p is positive definite, which identifies the model. No additional identification constraints are needed as the scale of is identified by the unit of the log-response times. Note that due to the parametrization above, besides for the symmetric model, the hyperparameter are estimated instead of . For and , a normal distribution with mean 0 and variance and an inverse-gamma distribution with shape and scale parameter 0.1, respectively, were used.

For the item parameters in , the following priors were specified to reflect the relative uncertainty about the parameters before any actual data are analyzed: First, independent normal distributions with mean and variance for and were used. The prior of is censored below 0. The prior of is censored below and above . In addition, the improper prior specification for was used as discussed for Equation 3 above. For , an inverse-gamma distribution with shape and scale parameters equal to was used. Note that a normal prior with mean 0 and variance 10 for was used instead of, for instance, a log-normal or gamma prior which are strictly positive, as in personality questionnaires contra-indicative items may occur which result in . In the simulation study below, the viability of the symmetric and asymmetric model is demonstrated in terms of parameter recovery. The model above is implemented in freely available software package OpenBUGS (Thomas et al., 2006). The syntax file is given in the Online Supplemental Materials #1. Using this implementation, samples are drawn from the posterior parameter distributions of the symmetrical and asymmetrical inverted-U models. In the below (simulation study and application) the means of these posterior distributions are used as point estimates of the parameters in the models. In addition, the posterior standard deviations are used as estimates for the standard errors of the parameters.

Simulation Study

Design

Responses and response times were simulated for 20 items using Equations 3 and 6 with 5 response categories . Either 500 or 1,000 subjects were used. In addition, was chosen to be either -0.05, 0, or 0.05, and δ was chosen to be either 0, 1, or 2. These choices above resulted in a fully-crossed 2 × 3 × 3 design. For the other parameters, the following true values were used: For the response parameters in Equation 3, = 1 for the odd items and = 2 for the even items were used. For the category attractiveness parameters , the overall attractiveness parameters are set to increasing, equally spaced values between -2 and 2. Next, the actual parameters were then determined by adding, respectively, -1, -0.25, 0.25, and 1 to these overall attractiveness parameters, so that 4 category attractiveness parameters are obtained for each item. Note that by doing so, the items increase in their overall attractiveness.

For the response time parameters in Equation 6, and were used for all . In addition, and 0.10 were used. The above choices resulted in raw response times between roughly 1 and 60 s over all conditions. Finally, the residual variance was approximately 60% of the total variance over all conditions. Parameter equaled 0.10 in all conditions. This choice was based on the results of Ferrando and Lorenzo-Seva (2007a) who found estimates for this parameter of 0.155 for an extraversion scale and 0.116 for a neuroticism scale.

Fifty replications were conducted within each condition. To the data within each replication, the symmetric and asymmetric inverted-U models are fit. Bayesian implementation discussed above is used to draw 10,000 samples from the posterior parameter distribution of which 5,000 are omitted as burn-in. Pilot simulations have demonstrated that this number was enough to ensure convergence of the sampling algorithm.

Results

Parameter Recovery: Symmetric Inverted-U Model

The results concerning parameter recovery of the symmetric inverted-U model in Equations 3 and 5 can be found in Online Supplemental Materials #2. Parameter recovery is generally good. The discrimination parameters and attractiveness parameters show a slight shrinkage effect. That is, for these parameters, the estimates are slightly pulled toward their prior means. This shrinkage effect for the item parameters has previously been found for the graded response model in Equation 3 by Kieftenbeld and Natesan (2012).

Parameter Recovery: Asymmetric Inverted-U Model

The results concerning parameter recovery of key parameters ρ1, ρ2, and δ from the asymmetric inverted-U model in Equations 3 and 6 can be found in Table 1. The parameters seem unbiased. Most importantly, the new parameters and are adequately recovered for all conditions in the simulation study. In the Online Supplemental Materials #3, results are reported for the item parameters. These results are generally good. See the figures in Online Supplemental Materials #4 and #5 for boxplots of the estimates of the discrimination parameters in the different conditions in the simulation study. Again a shrinkage effect is noticeable. Note that this shrinkage effect is comparable to the shrinkage affect found in the (traditional) symmetric inverted-U model (discussed above).

Table 1.

Mean (and Standard Deviations) of the Posterior Mean Estimates of the Inverted-U Parameters for the Asymmetric Inverted-U Model and the Symmetric Inverted-U Model.

Data		Asymmetric model			Symmetric model
ρ_2	δ	ρ_1	ρ_2	δ	σ_θτ	ρ_1
		N = 500
0	0	0.101 (0.005)	−0.001 (0.008)	0.023 (0.126)	−0.005 (0.015)	0.100 (0.005)
	1	0.101 (0.004)	0.000 (0.009)	0.990 (0.134)	−0.047 (0.016)	0.090 (0.005)
	2	0.099 (0.007)	−0.001 (0.008)	1.948 (0.124)	−0.083 (0.016)	0.064 (0.005)
0.05	0	0.100 (0.005)	0.048 (0.009)	0.013 (0.123)	0.063 (0.017)	0.098 (0.005)
	1	0.099 (0.007)	0.049 (0.009)	0.961 (0.125)	0.023 (0.013)	0.088 (0.006)
	2	0.100 (0.008)	0.050 (0.010)	1.964 (0.146)	−0.014 (0.017)	0.064 (0.005)
−0.05	0	0.102 (0.006)	−0.049 (0.010)	−0.014 (0.135)	−0.068 (0.016)	0.103 (0.006)
	1	0.100 (0.006)	−0.049 (0.009)	0.975 (0.130)	−0.113 (0.017)	0.090 (0.005)
	2	0.098 (0.006)	−0.047 (0.011)	1.984 (0.153)	−0.151 (0.015)	0.065 (0.004)
		N = 1,000
0	0	0.100 (0.003)	−0.001 (0.006)	−0.013 (0.078)	−0.001 (0.011)	0.100 (0.003)
	1	0.100 (0.004)	0.000 (0.006)	0.985 (0.090)	−0.048 (0.010)	0.088 (0.003)
	2	0.100 (0.005)	0.000 (0.006)	1.994 (0.107)	−0.087 (0.011)	0.064 (0.004)
0.05	0	0.101 (0.003)	0.048 (0.005)	−0.024 (0.088)	0.069 (0.012)	0.100 (0.003)
	1	0.100 (0.005)	0.049 (0.008)	1.003 (0.093)	0.021 (0.010)	0.088 (0.004)
	2	0.100 (0.006)	0.051 (0.006)	1.984 (0.103)	−0.013 (0.009)	0.065 (0.003)
−0.05	0	0.100 (0.004)	−0.049 (0.007)	0.009 (0.088)	−0.070 (0.013)	0.101 (0.004)
	1	0.100 (0.004)	−0.050 (0.007)	0.984 (0.091)	−0.117 (0.011)	0.090 (0.004)
	2	0.099 (0.005)	−0.048 (0.006)	2.007 (0.113)	−0.153 (0.009)	0.065 (0.003)

Note. True value for ρ is equal to 0.10 in all conditions.

Bias in the Symmetric Inverted-U Model

To study the biasing effects of neglecting asymmetry in the inverted-U effect, the parameter estimates of the symmetric inverted-U model in the different conditions of the simulation study are considered. First, with respect to the inverted-U parameter , results are in Table 1. Parameter is also considered as there is a relation between this parameter and and as discussed above. As can be seen from the table, for and , parameter 1 is well recovered because there is no misfit (the inverted-U effect is symmetrical in this condition). However, for increasing values of , parameter becomes underestimated. In addition, it can be seen that, indeed, parameter captures part of the effect of . From the conditions in which , it can also be seen that is also affected by . That is, if the time–distance inflection point is not at 0, this affects the covariance between and .

With respect to the parameter estimates of the latent variables and , and the attractiveness parameters , results indicate that there is no noticeable bias (results not tabulated). However, the discrimination parameters are systematically biased. See the figures in Online Supplemental Materials #5 and #6 for boxplots of the estimates of the discrimination parameters in the symmetric inverted-U model for the different condition in the simulation study. From the figures it is clear that in the and condition, the discrimination parameters are adequately recovered up to the shrinkage effect discussed above. This is to be expected as for this condition, the inverted-U effect does not depart from symmetry. However, for the other conditions—in which the inverted-U effect departs from symmetry—the discrimination parameters are systematically biased. That is for the discrimination parameters are overestimated for the less difficult items and underestimated for the more difficult items. This effect is larger if and smaller for . From the condition in which it can be inferred that has a similar effect but much smaller if is positive (i.e., ), and has an opposite (but much smaller) effect if is negative (i.e., ). Therefore, the largest biasing effects can be seen in condition and as the two effects strengthen each other. In addition, the bias is less in condition and as the two effects suppress each other. Note that the bias in the discrimination parameters is a function of the average distance between person and item. That is, for Item 1 (the “easiest” item) and Item 20 (the most “difficult” item), the absolute distance between person and item is the largest if averaged over persons. Therefore, for these items, the misfit is largest.

The discrimination parameters are an important determinant of the standard errors of . As the above indicates that the discrimination parameters become biased in the presence of asymmetry, the standard errors of (as approximated by the posterior standard deviation of θ) in the different conditions in the simulation study are studied. To this end, the function between the standard errors of θ and the true values for θ are estimated using a 6th order polynomial.[3] See the figures in Online Supplemental Materials #7 and #8 for the results of the symmetric and asymmetric inverted-U model in the different conditions of the simulation study. It can be concluded that for the and condition (symmetric inverted-U effect), the standard errors in the two models are the same across the -range. However, for the other conditions (asymmetric inverted-U effects), the standard errors are biased. That is, for , or , and , standard errors are biased upward in the symmetric model for higher values of , and biased downward for lower -values. This effect is larger if as compared with . If and , the effect is similar but much smaller, and if and , the effect is reversed. In addition, if and , the effect on the lower -values cancels out, leaving only the standard errors of the higher -values to be overestimated in the symmetric inverted-U model. Note that the biasing effects on the standard errors across correspond to the biasing effects on the discrimination parameters across . That is, if discrimination parameters are underestimated for larger -values, standard errors are biased upward for higher values of . For the standard errors of , no effect is found. To see how the bias in the standard errors affects the confidence intervals of , the coverage rates of the 99% confidence intervals were consulted. A small effect was found between the symmetric and asymmetric model if the data were asymmetric, with the confidence intervals based on the asymmetric model have a slightly more accurate coverage rate (and the coverage rates of the symmetric model being either slightly too small or too large depending on the condition). However, these differences were small.

Model Fit

Finally, results (not shown) suggest that the DIC (Spiegelhalter et al. (2002) can validly be used for model selection. That is, in all conditions in which the symmetric model was used to generate the data, the symmetric model was correctly identified as the best fitting model. In the conditions in which the asymmetric model was used to generate the data, the asymmetric model was identified as the best fitting model in all or most cases (e.g., in 72% of the cases for , and , and in 100% of the cases for and ).

Conclusion

Besides a slight shrinkage effect on the item parameters, parameter recovery seems to be adequate for both the symmetric and asymmetric inverted-U models. Most importantly, key parameters and seem unbiased. If asymmetry in the distance–time relation is neglected, results showed that the discrimination parameters and the standard errors of are upward or downward biased depending on the direction of the asymmetry. However, this bias hardly affects the confidence intervals of . In addition, the inverted-U parameter from the symmetric model is biased downward if asymmetry is neglected.

Application

Data

The data comprises the responses and response times of 1,901 subjects to a computerized online administration of the 235 items of the TCI-R (Cloninger, 1999). The subjects are between 14 and 80 years of age with a mean of 27.50 (SD: 11.96). In addition, 63.9% of the participants are female.

The TCI-R is based on a seven-dimensional psychobiological personality theory (Cloninger et al., 1993) in which four temperament dimensions are distinguished: novelty seeking (NS), harm avoidance (HA), reward dependence (RD), and persistence (PS) and three character dimensions: self-directedness (SD), cooperativeness (C), and self-transcendence (ST). Each dimension consists of 3 (ST), 4 (NS, HA, RD, and PC), or 5 (SD and C) subscales. Most of the subscales are bipolar, for example, NS1 (the first subscale of the novelty seeking scale) measures “exploratory excitability” versus “stoic rigidity.” Each subscale on its turn consists of 5 to 11 items with a 5-point Likert-type scale. Response times are generally between 4 and 60 s. Response times smaller than 1 s and larger than 60 s were excluded, which constituted 1.1% of the total data.

Analysis

As each subscale consists of few items, the analysis was conducted at the scale level. That is, the symmetric model from Equations 3 and 5 and the asymmetric model from Equations 3 and 6 are fit to the items of each of the seven scales separately. The model specification as discussed above is used. In addition, similarly as in the simulation study, 10,000 iterations are used with 5,000 iterations as burn in. To ensure convergence of the sampling algorithm, the trace plots of the parameters are consulted together with the Gelman-Rubin statistic for each parameter (Gelman & Rubin, 1992) based in two independent chains.

Results

All chains seemed converged with trace plots that varied randomly around a stable average. The largest Gelman-Rubin statistic that was observed equaled 1.01 (commonly values above 1.05 are taken as an indication of nonconvergence). Table 2 contains the posterior means and 95% highest posterior density (HPD) intervals for parameter (symmetric inverted-U model) and , and (asymmetric inverted-U model) together with the value of the DIC fit index for both models.[4] As can be seen from the results of the symmetric model, evidence for the inverted-U effect was found as none of the 95% HPD intervals of in the symmetric model contains 0. From the results of the DIC it can be seen that for scales NS and ST, the asymmetric model fits better than the symmetric model. From the 95%-HPD intervals of and , it appears that for both scales, this is mainly due to a nonzero parameter. From the posterior means of it can be concluded that for both scales, the slowest response time is before a 0 distance.

Table 2.

Posterior Means (and the 95% Highest Posterior Density Intervals) of ρ1, ρ2, and δ in the Asymmetric Distance Model for the Different Subscales of the TCI-R Together With the DIC Fit Indices for the Asymmetric Model (DICAS) and for the Symmetric Model (DICS).

Scale	DICAS	DICS	ρ_1	ρ_2	δ
NS	225209.2	225211.9	0.037 (0.031; 0.043)	0.006 (0.000; 0.013)	−0.193 (−0.393; −0.029)
HA	208276.3	208256.3	0.039 (0.034; 0.044)	0.004 (−0.003; 0.011)	−0.147 (−0.363; 0.043)
RD	186609.9	186573.0	0.037 (0.030; 0.043)	0.003 (−0.003; 0.009)	−0.192 (−0.438; 0.106)
PS	209809.3	209808.9	0.044 (0.039; 0.048)	−0.002 (−0.009; 0.005)	−0.011 (−0.198; 0.165)
SD	258516.2	258513.0	0.042 (0.036; 0.047)	0.001 (−0.006; 0.008)	0.542 (0.346; 0.738)
C	214174.1	214150.7	0.035 (0.030; 0.040)	0.002 (−0.004; 0.007)	0.431 (0.155; 0.743)
ST	167031.1	167043.8	0.054 (0.050; 0.059)	0.002 (−0.003; 0.007)	−0.251 (−0.445; −0.052)

Note. For the DIC, for each scale, the best value is in boldface. TCI-R = Temperament and Character Inventory–Revised; DIC = Deviance Information Criterion; NS = novelty seeking; HA = harm avoidance; RD = reward dependence; PS = persistence; SD = self-directedness; ST = self-transcendence.

For scales SD and C, the parameter seems to also depart from 0 as judged by their HPD intervals. However, as the DIC favors the symmetric model for these scales, it can be concluded that this effect in can be accounted for by the correlation between and in the symmetric model.

Conclusion/Discussion

For two scales, evidence for asymmetry of the inverted-U effect was found. For these subscales, it can be included that the response process for the one end of the scale differs from the response process of the other end of the scale. For instance, respondents on the lower end of Novelty Seeking (the NS scale) have been characterized by “stoic rigidity”, “reflection”, “reserve”, and “regimentation”, while respondents on the upper end of Novelty Seeking scale have been characterized by “exploratory excitability”, “impulsiveness”, “extravagance”, and “disorderliness” (see Mochcovitch et al., 2012). As our results indicate that there is asymmetry in the inverted-U effect, this suggests that the underlying response process is different for impulsiveness, extravagance, and so on, on the one side and rigidity, reflection, and so on, on the other side. On the other hand, Reward Dependence (the RD scale), for instance, is characterized by “indifference”, “aloofness”, “distance”, and “independence”, on the one side and “sentimentality”, “open to warm communication”, “attachment”, and “dependence”, on the other side. For this scale, no asymmetry was found, indicating that for this construct, there is no important (detectable) difference in the response process across its scale.

General Discussion

In the present study, the assumption of symmetry in the inverted-U effect underlying response times to personality items was tested. To enable these tests on symmetry, the modeling focussed on the absolute distance formalization by Ferrando and Lorenzo-Seva (2007a, 2007b). However, other formalizations have also been proposed in the literature. That is, Ranger and Ortner (2011) proposed a model in which the log-response times are not regressed on the absolute distance between person and item, but on the probability of the given response. In addition, Meng et al. (2014) regressed the log-response times on the sum of the squared category probabilities. Although these models have some desirable properties (see Ranger & Ortner, 2011), testing for symmetry in such models is more challenging as the relation between the probability of a given response and the log-response times can be multimodel for ordinal item responses. As a result, the focus of this study was on the formalization by Ferrando and Lorenzo-Seva. However, the approaches by Ranger and Ortner and Meng et al. (2014) are amenable to the present ideas in principle.

Most of the existing approaches to model the inverted-U effect have focused on item invariant inverted-U parameters (i.e., in this study: ). However, Ferrando and Lorenzo-Seva (2007b) noted that there may be item specific effects, which would require item specific inverted-U parameters (i.e., . In our case, this would require , , and to be specified as item specific parameters. As a result, the model will become more complex, and parameter estimation will be more challenging. by using item invariant parameters, one can probably capture the most important patterns in the data. That is, the three item invariant inverted-U parameters from the present study (, , and ) enable a theoretical test on the presence of the inverted-U effect and possible asymmetry in the construct of interest. In addition, they think that these parameters account for the most important effects of asymmetry on the item parameters and the standard errors. However, if researchers are interested in item specific effects, it is straightforward to adapt the BUGS code in the Online Supplementary Materials, and make , , and item specific parameters.

In this article, bias was found in the discrimination parameters if there is unmodelled asymmetry in the time–distance relation. As the discrimination parameters are in essence parameters in the response model, and not in the response time model, question arises whether a two-stage procedure shouldn’t be adopted in which the response model is fit first, and those results are then being used in the response time model. Advantage is that the discrimination parameters will not be biased. However, disadvantage is that estimation of the model parameters in general, including the standard errors, do not optimally benefit from the relation between the response and the response times. Therefore, the present approach would be ideally preferred as parameters are unbiased, and parameter estimation utilizes all information available in the responses and responses times

Click here for additional data file.

Supplemental material, sj-docx-1-apm-10.1177_0146621621990756 for Modeling Asymmetry in the Time–Distance Relation of Ordinal Personality Items by Dylan Molenaar, Sandor Rózsa and Natasa Kõ in Applied Psychological Measurement