Time for Tele-TTO? Lessons Learned From Digital Interviewer-Assisted Time Trade-Off Data Collection.

The preferred mode of administration of time trade-off (TTO) in large-scale valuation studies is face-to-face (personal) interviews facilitated by a trained interviewer. Geographical, financial or situational constraints could complicate personal TTO interviews. When facing such constraints, the use of digital interviews, in which trained interviewers facilitate through videotelephony software (i.e. tele-TTO) may be considered. This paper aims to guide researchers in how to approach tele-TTO interviews and discusses their advantages and disadvantages. The main advantages of tele-TTO compared to personal TTO are decreased need for travel and increased flexibility of interview scheduling, which could reduce costs and may foster representative sampling. Possible disadvantages of tele-TTO are partial loss of visual cues, complications with building rapport and possible selection effects that result from differences in interview preparation. Furthermore, the paper reports on lessons learned from a project in which both personal TTO and tele-TTO interviews were conducted. The results of this project suggest that although they require a different recruitment and interview process, tele-TTO interviews are feasible and provide flexibility to the interviewer. Furthermore, tele-TTO interviews yield largely similar results. Future research should explore the role of possible selection effects and respondents' perspective on tele-TTO interviews.

Key Points

Introduction

The time trade-off (TTO) method, which requires trading off length and quality of life, generally produces a large amount of inconsistent or implausible responses [1]. To improve data quality, the methods used to operationalise TTO in large-scale valuation studies have developed substantially in the past decades [2]. Whereas early valuation studies relied on props to illustrate TTO [3], this valuation method is now often operationalised digitally. For example, the latest valuation protocol for EQ-5D-5L prescribes the use of computer-assisted face-to-face (personal) interviews (CAPI) for TTO using highly standardised software [4], although alternative software packages exist [5]. The mode of administration, however, has not changed substantially for decades; TTO interviews in large-scale valuation studies are completed with costly and time-intensive face-to-face data collection (e.g. in public places or through home visits, see [2, 4, 6–9]). Although, in methodological studies, TTO can also be operationalised online, this in most cases reported as a limitation in study design (e.g. [10, 11]). Other pilot work compared data quality between face-to-face CAPI and self-completing TTO in groups with interviewers supervising, and found face-to-face CAPI to be superior [12]. Hence, it appears that face-to-face CAPI is seen as the gold standard for health utility measurement, at least in the context of valuation of EQ-5D [2, 4, 13].

However, geographical, financial or situational constraints could complicate the use of face-to-face CAPI. Such constraints could include limited budget, populations in remote places, sampling across large distances or even social distancing policies as implemented during the COVID-19 outbreak. In such circumstances, one could consider to use videotelephony software (e.g. Zoom, Skype or Microsoft Teams) as an alternative mode of administration. The rapid technological development of such software and increased access to devices that support their use has already led to experimentation with videotelephony as an alternative means of data collection for qualitative research. In this context, the costs and benefits of videotelephony have been discussed extensively [14-20]. Online modes of administration have also been explored extensively in patient-reported outcome measures [21-23]. In the context of health preference measurement, however, to date only digital self-completed methods were compared to personal CAPI in several studies (e.g. [24-26]). Norman and colleagues [26] suggested comparing digital interviewer-assisted TTO to face-to-face CAPI interviews (henceforth: personal TTO interviews) in future research, but to my knowledge no such work has been published. As such, little guidance exists to consider if TTO interviews facilitated with videotelephony (henceforth: tele-TTO interviews) are feasible and provide data of sufficient quality.

To address this gap in the literature, this paper discusses the main advantages and disadvantages of tele-TTO interviews as an alternative to personal TTO interviews, based on the extant findings reported in studies on qualitative research methods. Next, the paper discusses the findings and lessons learned from a project in which both personal TTO and tele-TTO interviews were completed, with a specific focus on selection effects and issues with building rapport. Finally, several suggestions for future research are provided.

Possible Advantages of Tele-TTO Interviews

Earlier work by others investigating the use of videotelephony for qualitative interviews suggests that it may have several advantages [14-20], of which the following appear relevant for tele-TTO. First, the use of tele-TTO enables interviews with respondents and interviewers in a different physical location, i.e. the need for travel is removed. This could be an advantage for valuation studies performed in countries with geographically dispersed populations, and could also facilitate sampling in remote areas. When sampling should yield populations representative of the general public (e.g. EQ-5D valuation studies), it appears advantageous to remove barriers related to travel by respondents or interviewers. Currently, in such valuation studies, sampling often occurs in urban areas (e.g. [6-9]), although some valuation studies have explicitly sampled in rural areas [27, 28], as it was shown that TTO valuations differed between respondents sampled from urban and rural areas [27, 29]. Digital interviews could, thus, reduce the efforts involved with data collection in rural areas, which may be relevant for health state valuations in lower-middle income countries. Furthermore, removing or reducing the need to travel may also lower costs of data collection, as in principle no travelling time and expenses need to be incurred by either interviewer or respondent.

Second, the use of videotelephony allows flexible interview scheduling. That is, digital interviewing allows interviewers to work with a flexible schedule to accommodate respondents’ preferred time for interviews [17]. For example, respondents in full-time work may not be available for personal interviews during daytime, but interviewers may be hesitant to schedule personal interviews in the evenings or at night for health and safety concerns. As representative sampling is important in many valuation studies using TTO ([e.g. valuation of EQ-5D instruments: [4, 13]), the flexibility in scheduling offered by tele-TTO may be beneficial and enhance interviewer safety. Furthermore, this flexibility also pertains to the location from which interviewers and respondents conduct the interview. If both select locations in which they feel comfortable and are alone, this could be seen as advantageous to the interview process, as it enables them to freely speak their mind. Nonetheless, this increased flexibility may also lead to disruptions of the interview when tele-TTO respondents are interviewed in noisy or distracting environments [17, 18].

Possible Disadvantages of Tele-TTO Interviews

However, earlier work on videotelephony for qualitative interviews [14-20] also suggests several possible disadvantages of which the following are relevant for tele-TTO. First, tele-TTO involves a video connection, and as such, interviewers and respondents will in principle only see a partial image of each other through a video connection. Earlier work in qualitative research often mentioned this as a disadvantage [17, 18], as visual observation of respondents during interviews can be a source of data. That is, it may be relevant to report on the effect some parts of the interview have on respondents’ body language. Whether this affects tele-TTO is unclear, as in large-scale valuation studies, which involve quantitative analysis, such aspects of the interview are not reported (e.g. [6, 7, 9]). Experienced TTO interviewers may, however, use visual cues in their interviews, by determining from body language if respondents are active and engaged, or perhaps if the content matter is evoking negative emotions. As such, although it is not a source of data, the loss of visual cues may yield lower data quality as monitoring respondents’ engagement with and understanding of TTO tasks may be more difficult. However, this same lack of physical proximity and visual cues may foster anonymity and for some respondents increase the degree to which they are willing to share views that may not be socially desirable [17]. For tele-TTO, this could, for example, affect sharing of views on euthanasia or end of life, which are known to affect health state valuations [10, 30].

Second, the use of digital interviewing complicates building rapport [16-19], i.e. it hampers the creation of a pleasant atmosphere in the interview (for a theoretical discussion of what rapport entails, see [31]). It has, for example, been suggested that physical contact (e.g. handshakes) and sharing a drink are pivotal in establishing rapport [19], and neither is possible with tele-TTO. Additionally, the use of videotelephony introduces possible disruptions of interview flow, owing to lags in bandwidth or audio-visual problems. Both could harm rapport [18, 19], as the interview process feels less natural. Although rapport is generally seen as a precondition for successful qualitative interviewing [31], whether this affects tele-TTO is unclear, as these interviews yield quantitative data. Nonetheless, a lack of rapport could negatively influence interviewers’ and respondents’ experience during the tele-TTO interviewing process.

Third, tele-TTO interviews require different (and perhaps more) preparations by both the interviewer and respondent than personal interviews. In preparation of personal TTO, the interviewer should (inter alia) learn the interview protocol, determine the location for the interview and (possibly) recruit respondents. The main preparation for respondents involves navigating and travelling to the interview location. Although tele-TTO interviews may reduce the time and costs involved with travelling, respondents are required to prepare working audio-visual hardware and videotelephony software. Willingness and ability to prepare for and complete digital TTO interviews may depend on digital literacy and could yield selection effects [17], in which parts of society (e.g. older respondents or those without access to audio-visual hardware) are underrepresented. Furthermore, earlier work suggests that digital interviews have more no-shows [17] and a lower response rate than personal interviews [15]. This is in line with conclusions by Norman et al. [26] for digital self-completed TTO. Hence, tele-TTO introduces possible biases owing to increased non-response and selection effects.

Tele-TTO Applied in Practice

The degree to which these possible advantages and disadvantages affect TTO interviews is, however, unknown. This paper, therefore, reports on data and lessons learned from a project in which tele-TTO interviews were necessary because of social distancing related to the COVID-19 outbreak. Data collection for this project ran between March and May 2020 in the Netherlands. Recruitment was handled by a marketing agency, who used quota sampling for age and sex, and all interviews were performed by the first author. The project was set up to involve only personal TTO interviews, and indeed the first 36 interviews performed were completed face to face. However, as closures of public facilities were imminent in March, preparations for tele-TTO interviews were undertaken. These preparations involved exploring the technical possibilities in various videotelephony software and a discussion with the marketing agency responsible for recruiting respondents before data collection started.

After such closures were announced on 13 March (in the Netherlands), all remaining respondents that had already signed up to take part in this project were informed of the tele-TTO interview’s process and the preparation required of them. This enabled the completion of 113 tele-TTO interviews. The interview design and proprietary software were based on the EuroQol Valuation Technology (EQ-VT) protocol [4], and as such composite TTO tasks were completed in random order for the following six health states, described by EQ-5D-5L: 11211, 13313, 35332, 22434, 24443 and 55555 (see Appendix A for more details). The personal TTO interviews were completed with both the interviewer and respondent seated at the same table. Tele-TTO interviewers were completed via Zoom with audio and video connection, enabled through a shared screen. In both modes of administration, the interviewer operated the TTO software, and the respondents stated their preferences verbatim. Figure 1 shows a screenshot of a tele-TTO interview from the interviewer perspective.1

Fig. 1

Interviewer view of tele-time trade-off (TTO), with Dutch labels for TTO (A and B are 10 years in perfect health and 10 years in a wheelchair, respectively)

It is important to note that this project was not designed to compare tele-TTO and personal TTO interviews, it aimed to replicate and extend the findings reported in Lipman and colleagues [32]. The lessons learned reported in this paper should thus be cautiously interpreted for (at least) two reasons. First, the types of data collected were determined before moving from personal to tele-TTO interviews. As a result, no information was collected on, for example, sign-ups, response rates, cancellations and respondents’ experiences during the interview. Although quantitative comparisons of such interview characteristics are warranted, in this paper, the single interviewer’s subjective recollection of events is the only source of data. Second, the mode of administration was not assigned randomly; digital TTO interviews were completed later in time. Such non-random assignment may have influenced the data collected for several reasons, as later interviews could have been subject to influences that earlier interviews were not subjected to. For example, the continued development of the public health crisis related to COVID-19 may have affected later interviews. Furthermore, both the interviewer and respondents may have accumulated experiences that could influence TTO interviews. Over time, the interviewer would have become more experienced, which could have affected tele-TTO more strongly than personal TTO. Respondents may also have become more experienced with videotelephony software over time, as a result of reliance on these software packages in other contexts, for example, working from home or contact with distant relatives. With these possible caveats in mind, the following lessons could be learned from this project.

Lessons Learned from Tele-TTO

First, when tele-TTO interviewing commenced, it appeared that this method of interviewing indeed had a higher risk of respondents cancelling on short notice or not showing up at all2. In this project, this could have been explained by respondents overlooking information provided on the change from personal to digital interviews. Deakin and Wakefield [17] also suggest that respondents cancelling or deliberately not showing up for digital interviews is a risk in digital interviews, and suggested reaching out to respondents beforehand. Similar suggestions have been made in the contexts of survey research [33], suggesting that combining reminders of different forms is beneficial. Indeed, when respondents were called a day beforehand as a reminder, in addition to a standard e-mail, no shows appeared to occur less frequently.2 Furthermore, this call enabled questions about video telephony and/or assisting respondents with preparation. Although it may be recommended to use this reminder call to introduce respondents to the content of the interview and the TTO method, in this project, reminders were provided by the marketing agency rather than the interviewer who could provide this introduction.

Second, tele-TTO indeed offered additional flexibility in scheduling and performing interviews. In multiple cases, respondents cancelling on short notice were immediately replaced by other respondents who happened to be available for tele-TTO interviews2. This would not necessarily have been possible when interviewers or respondents would have been required to travel. However, the lack of proximity did result in some interviews with audio and video problems, which would not have occurred in personal interviews, but these could also be addressed flexibly. For example, if it improved audio connection, video connection was dropped (as in [17]), maintaining only the necessary audio and screen share connections for completing the interview.3 Finally, respondents who experienced audio problems received a call on their telephone, whilst maintaining video connection via their personal computer.

Third, tele-TTO interviews appeared to be considerably more demanding than personal TTO interviews for the interviewer. Perhaps, this is related to difficulties with building rapport with respondents, fewer visual cues available to determine if respondents understood the task and were sufficiently engaged, and disruptions related to connection problems. Although this reflects my subjective experience (but see also [34]), it appears beneficial to schedule fewer subsequent tele-TTO interviews or ensure sufficient time for breaks (and delays) in between interviews.

Fourth, in this project, the respondents that completed tele-TTO interviews were significantly younger (mean = 40.30, standard deviation = 14.89) than those completing personal TTO interviews (mean = 50.25, standard deviation = 15.55), t(56.7) = 3.38, p = 0.001. Furthermore, respondents in tele-TTO were significantly less likely to be married than respondents who completed personal TTO interviews, with 21% and 41% of respondents married respectively (Chi-squared test, p = 0.02). No significant differences were found in terms of education level, parental status, household income and religion. Hence, this suggests that the use of tele-TTO interviews may introduce selection effects.

Finally, no differences in outcomes were observed between the 113 tele-TTO interviews and 36 personal TTO interviews. Appendix B shows the distribution of TTO data collected, which appears largely similar, and both overall and separately for each health state. A series of analyses comparing the central tendencies, variances and distribution of the data collected with both modes of administration confirms this conclusion. Furthermore, Appendix B shows that no differences were observed in data quality for tele-TTO and personal TTO interview outcomes.

Time for Tele-TTO? On Selection Bias and Rapport

The lessons learned from this ongoing project indicate that tele-TTO might be a feasible alternative to personal TTO, as the outcomes of the interviews as well as the quality of the data appeared to be mostly unaffected. Nonetheless, this project showed that using videotelephony may introduce possible selection effects and issues with building rapport, both of which warrant additional scrutiny.

When (as in this project) no measures are taken to control for selection effects, the use of tele-TTO may introduce selection bias. That is, in this project, respondents that took part in personal interviews were older and more likely to be married than those completing tele-TTO. Age and marital status have been found to affect outcomes of health state valuations in earlier work [35-37], with such evidence also existing for education level [38]. As a result, generally, in large-scale applications of TTO, samples are recruited representatively in terms of such demographics (e.g. [6, 7, 9]). Hence, although tele-TTO may decrease costs associated with sampling in remote areas, this benefit may be offset by complicating such representative sampling. This is especially problematic, as evidence exists for differences between age groups in access to the internet [39] and/or ability to effectively use the internet [40]. This inequality in digital access or literacy is often referred to as the ‘digital divide’ [40], and could explain the age-related selection effects for tele-TTO. Furthermore, income and education level also contribute to differences in internet access or digital skills [40, 41]. One may expect this digital divide to decrease because of COVID-19, with people all ages relying on digital media to connect with others. However, some scholars have argued that COVID-19 has a risk of increasing the digital divide, as pre-existing inequalities in digital access or literacy could be exacerbated [42-44]. Hence, further exploration of inequalities in the ability to take part in tele-TTO is crucial, in particular in lower-middle income countries [45, 46], where such inequalities may be even larger than in OECD countries.

In this project, no data were collected on respondents’ or the interviewer’s experiences during personal- and tele-TTO interviews. As such, although it is seen as a key concern in qualitative work [19, 34], it is not possible to determine if the respondents and interviewer experienced differences in rapport. Weller [34] investigated this question in qualitative longitudinal research, in which videotelephony was introduced in a project with young-adult respondents who had relied on home visits for data collection in earlier phases. When comparing interview experiences, Weller [34] notes that the rapport experienced during these videotelephony interviews could be similar or at times even better (see also [17]). For example, some respondents experienced the interview as more personal and felt less pressure. However, this appeared to be dependent on the quality of the audio-visual connection. For example, whereas personal interviews often commenced by exchanging pleasantries (which are important for establish rapport), digital interviews often commenced with ensuring the connection was of sufficient quality [34]. Furthermore, when the audio quality impeded the interview, both interviewer and respondent had to invest considerable energy understanding each other and avoid interruptions, rather than thinking through their answers [34]. The latter seems of particular importance for tele-TTO, as in TTO where the use of mental shortcuts, rather than thinking through the answer, is often seen as a problem [47, 48]. Hence, further research should explore whether tele-TTO affects interviewer and respondents’ experiences, as it appears that these are heterogeneous [34]. It could also be explored if a hybrid data collection is feasible, in which the decision between tele-TTO and personal TTO is left up to the respondents. Such strategies were used successfully in qualitative research [17].

Conclusions and Additional Suggestions for Future Research

Admittedly, the results of this ongoing project should be carefully interpreted as it was not designed with the goal of comparing tele-TTO and personal TTO interviews. Hence, a study that has this explicit goal is a first paramount step for future research. Furthermore, future research may explore the role of alternative devices and software. Most tele-TTO interviews for this project were completed with a Zoom connection on a laptop. Perhaps the feasibility of tele-TTO interviews depends on the software used, as the options available to interviewers and the steps required for respondents to connect may differ. Performing tele-TTO interviews on mobile phones may require changes to the operationalisation of TTO (e.g. increased font size). Such changes could be worthwhile, as access to mobile phones and wireless connection is growing worldwide [49].

In conclusion, this paper suggests that tele-TTO may be a feasible, flexible, and low-cost alternative to personal TTO interviews, with some evidence suggesting that both modes of administration yield mostly similar outcomes. However, tele-TTO may suffer from selection effects and the use of videotelephony software may complicate the interview process. As situational constraints may prohibit the use of personal TTO interviews (e.g. COVID-19), additional research studying when, why, how, and for whom tele-TTO interviews should be used appears valuable.

When geographical, financial or situational constraints complicate the use of face-to-face time trade-off (TTO) data collection, the use of digital interviews facilitated by trained interviews and videotelephony software is a feasible alternative.

Advantages of digital TTO interviews compared to face to face are the decreased need for travel and increased flexibility of interview scheduling, which could reduce costs and may foster representative sampling. Possible disadvantages of tele-TTO are partial loss of visual cues, complications with building rapport and possible selection effects that result from differences in interview preparation.

A comparison of personal and digital TTO interview data suggests that the results are similar for both modes of administration.

Table 1

Health states used for tele-time trade-off (TTO) and personal TTO interviews

11211	13313
No problems in walking about	No problems in walking about
No problems washing or dressing myself	Moderate problems washing or dressing myself
Slight problems doing my usual activities	Moderate problems doing my usual activities
No pain or discomfort	No pain or discomfort
Not anxious or depressed	Moderately anxious or depressed
35332	22434
Moderate problems in walking about	Slight problems in walking about
No problems washing or dressing myself	Slight problems washing or dressing myself
Moderate problems doing my usual activities	Severe problems doing my usual activities
Moderate pain or discomfort	Moderate pain or discomfort
Slightly anxious or depressed	Severely anxious or depressed
24443	55555
Slight problems in walking about	Unable to walk about
Severe problems washing or dressing myself	Unable to wash or dress myself
Severe problems doing my usual activities	Unable to do my usual activities
Severe pain or discomfort	Extreme pain or discomfort
Moderately anxious or depressed	Extremely anxious or depressed

Table 2

Measures of central tendencies and variance for TTO utilities

State	Personal TTO interviews				Tele-TTO interviews
	M	SD	Median	Q1/Q3	M	SD	Median	Q1/Q3
11211	0.96	0.09	1	1/1	0.96	0.06	1	0.95/1
13313	0.75	0.35	0.82	0.72/0.95	0.81	0.2	0.9	0.75/0.95
35332	0.43	0.6	0.68	0.43/0.81	0.55	0.44	0.7	0.4/0.8
22434	0.25	0.56	0.43	0.18/0.61	0.22	0.6	0.4	0.1/0.65
24443	− 0.06	0.69	0.18	− 0.91/0.51	− 0.25	0.65	− 0.4	− 0.95/0.35
55555	− 0.64	0.43	− 0.8	− 1/− 0.5	− 0.7	0.42	− 0.9	− 1/− 0.6
All	0.28	0.72	0.55	− 0.5/0.9	0.27	0.74	0.5	− 0.5/0.9

M mean, Q1 first quartile, Q3 third quartile, SD standard deviation

Table 3

Modal utilities, proportion of respondents with utilities equal to that mode and Chi-squared tests for difference in distributions

State	Personal TTO interviews		Tele-TTO interviews		χ^2	P-value
	Mode	Proportion (%)	Mode	Proportion (%)
11211	1.0	77.8	1.0	64.9	1.38	0.23
13313	1.0	16.7%	1.0	17.5	<0.001	1
35332	0.7	13.9%	0.8	13.2	<0.001	1
22434	0.5	13.9%	−1.0	12.3	<0.001	1
24443	−1.0	16.7%	−1.0	24.6	0.61	0.43
55555	−1.0	38.9	−1.0	43.0	0.078	0.78

TTO time trade-off

Table 4

Data quality in personal time trade-off (TTO) and tele-TTO interviews, with quality indicators adapted from [51]

Problematic responder types	Personal TTO (n = 36)		Tele-TTO (n = 113)		χ^2	P-value
	N	%	N	%
All 6 health states same utility	0	0	0	0	–	–
One or more health state(s) ranked the same as 55555	8	22.2	32	28.3	0.25	0.62
One or more health state(s) ranked strictly lower than 55555	3	8.3	3	2.6	1.07	0.30
11211 same utility as 55555	0	0	0	0	–	–
Fewer than 4 distinct values	1	2.8	7	6.1	0.13	0.72
Utility of −1, −0.5, 0, 0.5 or 1 for all 6 health states	0	0	0	0	0	–
No negative utilities	6	16.7	13	11.4	0.29	0.59
No use of half-year increments in TTO	7	19.4	31	27.4	0.55	0.46