Measuring Resource Utilization: A Systematic Review of Validated Self-Reported Questionnaires.

A variety of methods may be used to obtain costing data. Although administrative data are most commonly used, the data available in these datasets are often limited. An alternative method of obtaining costing is through self-reported questionnaires. Currently, there are no systematic reviews that summarize self-reported resource utilization instruments from the published literature.The aim of the study was to identify validated self-report healthcare resource use instruments and to map their attributes.A systematic review was conducted. The search identified articles using terms like "healthcare utilization" and "questionnaire." All abstracts and full texts were considered in duplicate. For inclusion, studies had to assess the validity of a self-reported resource use questionnaire, to report original data, include adult populations, and the questionnaire had to be publically available. Data such as type of resource utilization assessed by each questionnaire, and validation findings were extracted from each study.In all, 2343 unique citations were retrieved; 2297 were excluded during abstract review. Forty-six studies were reviewed in full text, and 15 studies were included in this systematic review. Six assessed resource utilization of patients with chronic conditions; 5 assessed mental health service utilization; 3 assessed resource utilization by a general population; and 1 assessed utilization in older populations. The most frequently measured resources included visits to general practitioners and inpatient stays; nonmedical resources were least frequently measured. Self-reported questionnaires on resource utilization had good agreement with administrative data, although, visits to general practitioners, outpatient days, and nurse visits had poorer agreement.Self-reported questionnaires are a valid method of collecting data on healthcare resource utilization.

INTRODUCTION

With increasing healthcare costs, economic evaluation is being used more frequently, to curb costs without decreasing the quality of health care provided.[1] This has placed demands on researchers, who are tasked with finding data for a broad range of health technologies and pharmaceuticals. Economic evaluations require valid costing estimates to produce useful data. To calculate such estimates, it is first necessary to know the quantities of resources utilized, for example, number of doctors’ visits or number of days in hospital. Once resource utilization is known, costs of the resources can be calculated.

Administrative data, often collected by hospitals, governments, or insurance companies, are 1 source to identify resource utilization.[2,3] However, administrative data are often limited and may be unobtainable in a timely, cost-efficient manner. Additionally, they may fail to capture important resources; they do not capture indirect resource utilization, for example, lost productivity, travel, or caregiver time.[3] Inputs such as these are necessary for economic evaluations that adopt a societal perspective.[4]

Self-reported questionnaires are 1 alternative to administrative data, for obtaining resource utilization data. Two types of self-reported questionnaires are often used. Self-reported resource questionnaires may target a specific population, for example, assessing resources used by individuals with rheumatoid arthritis. Specific questionnaires will often be designed around the particular needs of this patient population (eg, care by a rheumatologist). In contrast, resource utilization questionnaires may broadly target a general population, rather than a population subgroup. Although these questionnaires may not collect the depth of information that a targeted questionnaire does, they are widely applicable across clinical conditions. Both specific and general questionnaires can be useful for capturing resource utilization data for economic analyses.

Self-reported questionnaires are at risk of distortion and bias, such as inaccurate recall and bias due to social desirability.[5] Bias in self-reported questionnaires may impact the validity of results. Validity of self-reported questionnaires is therefore an important consideration when choosing a tool for collecting resource utilization data.

Despite the potential usefulness of these alternative measures for supporting economic evaluations, to our knowledge, there are no systematic reviews that summarize these instruments, and asses their validity and their relative merits. The objective of this study, therefore, was to identify validated self-report healthcare resource use instruments and to map the attributes of these instruments. This synthesis will aid researchers in selecting the most appropriate tool for measuring resource utilization within their own research.

METHODS

A systematic review of published literature was conducted. An information specialist developed the search strategy, and ran each search. Five databases were searched: the Health and Psychosocial Instruments Database (1985-April 10, 2014), MEDLINE (1946-November 20, 2015), EMBASE (1974-November 20, 2015), PsycINFO (1887-November 20, 2015), and CINAHL (1937-November 20, 2015). Keywords such as “utilization,” “healthcare utilization,” and “health service utilization” were combined with terms such as “questionnaire,” “survey,” “self-report”, “validity,” “valid,” and “reproducible.” The search strategy was limited to English-language results only; no other limitations or filters were used (see search, supplemental content, which shows MEDLINE search strategy). The detailed search strategies for each database are available from the authors upon request.

All abstracts retrieved were independently reviewed in duplicate (LEL, FC). To ensure that all relevant literature was captured, abstracts included by either reviewer proceeded to full-text review. All studies included after abstract review were reviewed in full text (LEL, FC). Inclusion criteria were as follows: reports on a self-reported health service resource use questionnaire; at least 1 question was validated; original data; included only adult participants; and the questionnaire being assessed must be publicly available. Studies which failed to meet any of these criteria were excluded from analysis. Independent reviewers assessed the full texts in duplicate; any discrepancy between reviewers was resolved through discussion and consensus.

A standardized data extraction form, developed a priori, was used to extract data from each study. Two independent reviewers completed data extraction (LEL, FC); any discrepancy between reviewers was resolved through discussion and consensus. Name of the questionnaire, target population, number of questions and time to complete the questionnaire, method of validation, validation findings, and type of resource utilization included in the questionnaire were extracted from each study when available. Resource utilization types were categorized under 3 headings: health system use (general practitioners, specialists, nurses, and rehabilitation or physiotherapy practitioners, and Emergency Room and inpatient stays); medication use (prescribed medication, over-the-counter medication, and supplemental medication); and other resources used (caregiver time, complementary therapy, travel time, out-of-pocket expenses, productivity, and other resources not previously captured). To our knowledge, there is no quality assessment for validation studies, and therefore, quality assessment was not possible.

The methods used in conducting this systematic review adhere to the standards outlined by the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement checklist.[6] Since this research relied solely on publicly available information, ethics committee review was not required.

RESULTS

Of the 2343 abstracts identified, 2297 did not meet the inclusion criteria. The remaining 46 abstracts were reviewed in full text. An additional 31 studies were excluded during full-text review: 6 did not have a questionnaire available; 9 did not assess validity; 8 did not present a questionnaire; 3 were only available as abstracts; 3 were duplicates; 1 was not an adult population; and 1 was unavailable in full text. Ultimately, 15 studies, each reporting on a distinct validated resource utilization instrument, were included (Figure 1).

FIGURE 1

Flowchart of included and excluded studies.

The included studies were conducted in a variety of countries: 4 in the United States[7-10]; 3 in the Netherlands[11-13]; 2 in Germany[14,15]; and 1 each in the United Kingdom (UK),[16] Sweden,[17] Canada,[18] Italy,[19] Belgium,[20] and New Zealand[21] (Table 1). Broadly, 6 of the studies assessed resource utilization surveys for patients with a chronic condition (eg, epilepsy, irritable bowel disease, osteoarthritis)[8,9,11,14,18,21]; 5 targeted mental health service utilization[10,13,15,16,19]; 3 were intended to capture resource utilization from a general population[7,12,20]; and 1 was used to assess healthcare resource utilization in adults over 75 years of age.[17]

TABLE 1

Characteristics of Included Studies

Some of the included studies validated all of the questions in the resource utilization questionnaire; however, most validated only a subset of the questions. Table 2 summarizes the resources assessed in each of the included questionnaires. The results of these validation studies are summarized, by target population, below.

TABLE 2

Outcomes Assessed by Each Included Questionnaire

General Populations

Three studies assessed the validity of self-reported resource use questionnaires for a general population, using administrative data as the reference standard.[7,12,20] The length of the questionnaires varied: 1 included 3 questions,[12] 1 included 38 questions,[7] and 1 study did not report this information.[20] All 3 assessed the utilization of different healthcare resources (Table 2). No single questionnaire included all types of health resources. Reijneveld[12] included inpatient stays; rehabilitation or physiotherapy services; and use of prescribed medication; the name of the questionnaire was not reported. Peersman et al[20] assessed utilization of general practitioner care and visits to specialists using the Belgian National Health Interview Survey, and Zuvekas and Olin[7] assessed inpatient stays and utilization of general practitioners using the Medical Expenditure Panel Survey.

The validity varied by instrument and by type of health resource. Zuvekas and Olin found that participants accurately reported the frequency (kappa: 0.89) and duration of inpatient stays, but significantly under-reported the frequency of emergency department use (kappa: 0.76) and visits to a medical office (kappa: 0.55).[7] Reijneveld[12] found general overestimation of resource utilization (kappa: 0.77) and physiotherapy services (kappa: 0.71). Agreement was lowest for prescription medication (kappa: 0.58). Peersman et al found that physician visits were under-reported with a mean of 0.89 for self-reported contacts compared with 0.94 for registered contacts (kappa: 0.65).[20] Women or those who reported that they had a health limitation were most likely to over-report health service utilization.[20] Peersman et al[20] found that agreement was lower for specialist visits, with a kappa of 0.62.

Chronic Conditions

Six studies assessed the validity of self-reported resource utilization questionnaires for populations with chronic conditions, the nature of which varied widely. The questionnaires were of different lengths, ranging from 3 to 38 question items[8,9,11,14,18,21] (Table 1). Pinto et al[21] included questions on the most number of resources (emergency room stay; inpatient stay; general practitioner, specialist, nurse, and homecare visits; rehabilitation or physiotherapy services; prescribed, over-the-counter, and supplemental medication; and also complementary therapy, travel time, out-of-pocket expenses, and productivity). The remaining 5 questionnaires only included a subset of these resources (Table 2).

The validation approach varied, although the majority of the instruments were validated against administrative data. Lairson et al[9] assessed concordance between self-reported resource use and administrative data for individuals with epilepsy, and reported an overall kappa of 0.75. Pinto et al[21] assessed the validity of the Osteoarthritis Cost and Consequences Questionnaire with administrative data, and found that broadly, database costs agreed with self-reported costs for all health resource use measured (overall cost pc = 0.606). A study by Longobardi et al[18] assessed concordance between a self-reported questionnaire and administrative data among individuals with inflammatory bowel disease, and noted over-reporting in inpatient nights by 25% to 35% and under-reporting in physician visits by 35% to 45%. Lastly, Schweikert et al[14] compared self-reported data from patients requiring cardiac rehabilitation after a coronary event to administrative data, and found overall agreement; the best agreement was found for inpatient days (0.90) and the worst agreement was for cost of medical intake (0.57).[14]

The remaining 2 studies compared nonadministrative data sources to self-report. Ritter et al[8] compared self-reported data with provider records for patients in a chronic disease self-management program, and found over-reporting of hospitalizations and under-reporting of physician visits. Van den Brink et al[11] examined the concordance of a self-reported questionnaire with a daily dairy for individuals with rectal cancer, and found no significant difference between the diary and questionnaire.

Mental Health Service Utilization

Five studies validated questionnaires specifically geared towards measuring mental health service utilization.[10,13,15,16,19] Validated questionnaires included the following: the TiC-P,[13] the Client Service Receipt Inventory,[16] the Los Angeles Epidemiologic Catchment Area Survey,[10] the Client Socio-demographic and Service Receipt Inventory,[15] and the Intervista Costi Assistenza Psichiatrica[19] (Table 1).

Both registry and administrative data sources were used for validation. Bouwmans et al[13] compared self-reported questionnaire data with occupational registry data, including psychotherapist visits, physician visits, visits to social worker, use of medication, and absence from work, for patients with a diagnosed psychiatric disorder. This study found that agreement (kappa) was 0.597 for GP visit, 0.711 for social worker visit, 0.839 for use of medication, and 0.795 for hospital day care treatment.[13] Mirandola et al[19] found that the correlation concordance coefficient for agreement between self-reported resource use and data from a register was ρc < 0.5 for outpatient costs, day-patient costs, and community costs. Correlation worsened for those with schizophrenia (ρc = 0.279), and was better for those with affective disorders (ρc = 0.691), other diagnoses (ρc = 0.741), and neuroses (ρc = 0.550).[19] Byford et al[16] compared self-report and administrative data, reporting higher agreement for visits to general practitioner (ρc = 0.631), but poorer agreement for all other services types (ρc < 0.40), including emergency room visits, inpatient stays, nurse visits, and visits to specialists. Golding et al[10] also assessed validity using administrative data for visits to mental health professionals; this study found few differences between administrative and survey data, although individuals tended to over-report the use of publicly funded services. Heinrich et al[15] found that the concordance was highest between self-reported questionnaire data and hospital records (ρc = 0.8432), and lowest for day-care costs (ρc = 0.2977).

Older Adults (75+ Years of Age)

Only 1 study, by Carsjo et al,[17] specifically assessed the validity of self-reported resource utilization tools for individuals over 75 years of age (Table 1). This study assessed the validity of 5 questions on inpatient stays, general practitioner visits, care from a nurse, and home care.[17] When self-reported resource utilization was compared with administrative sources, this study found that the frequency of homecare, hospital, and general practitioner visits did not significantly vary; however, a statistically significant difference was found in the frequency of nurse visits in those 75 to 84 years old (P < 0.05).[17]

DISCUSSION

To conduct an economic evaluation, resource utilization data must be collected. When factors prohibit the use of administrative data, self-reported questionnaires may be used. In this systematic review, we have described 15 validated questionnaires. The included instruments focus on a variety of patient populations.

When selecting a questionnaire, a number of elements should be considered: what information you are trying to collect, whether the questionnaire is appropriate, and whether the questionnaire is valid.[22] These general conditions translate to resource utilization questionnaires.

It is necessary to first determine what resource utilization data are necessary to inform the economic evaluation,[22] for example, whether family and patient resources such as time will be included.[23] Within a patient population, no identified instruments captured all categories of resource use. We found that the most frequently measured resources were inpatient stays[7-9,11-19,21]; visits to a general practitioner[7-9,11,13-18,20,21]; and Emergency Room utilization.[7-9,15,16,19,21] Particularly lacking are studies looking at nonmedical resources such as travel time, out-of-pocket expenses, and productivity. We found that when these questionnaires were included, they were frequently not validated. To assist researchers in collecting these data, future work on validating nonmedical self-reported questions would be valuable. Additionally, instrument developers should ensure that broad -spectrum resources can be assessed using their instrument.

Our findings suggest that an instrument often only has a subset of validated questions. Recognizing that an instrument of choice may not be validated for all types of resources consumed, researchers should also consider which healthcare resource is likely to be the cost driver in their study, and should choose a questionnaire that has validated that resource. Self-reported measures may introduce bias differentially, depending on the type of healthcare resource most frequently used by the patient population.

The included studies showed that some resources are more likely to be over or under-reported than others. Self-reported questionnaires tended to accurately estimate the number of hospitalizations and Emergency Room visits, but underestimate the number of times individuals saw a general practitioner. This suggests that some resource utilization estimates are more accurate than others. An understanding of the severity and directionality of errors is necessary for authors using self-reported questionnaires.[20] Methods of mitigating self-reporting bias, such as indirect questioning,[24] reducing the length of assessment interval,[25] interviewer administration,[25] momentary sampling,[26] and anonymity,[25,27] have been suggested by other studies. Some of these methods may be useful for resources that are known to be frequently over or underestimated, such as general practitioner visits.

A few limitations merit comment. Many of the studies only validated only a subset of the resource utilization questionnaire. For these studies, we are limited to reporting the validity of the questions assessed, and cannot make claims about the validity of the entire questionnaire. Only having validation results from a subset of questions limits our ability to draw conclusions about the questionnaire as a whole. Additionally, due to the heterogeneity of survey populations, methods of validation, and survey questions, quantitative synthesis was not appropriate. If further validation studies using similar methods, populations, and assessing similar questions were to be published, quantitative synthesis could offer useful information.

CONCLUSIONS

The 15 validated healthcare resource utilization questionnaires range in breadth and in target population. Results from these validation studies suggest that self-reported questionnaires are a valid method of collecting data on healthcare resource utilization. However, when using self-reported resource utilization questionnaires, it is necessary to understand which types of resources are most frequently under and over-reported; some resource utilization data are more accurate than other data. In the validated resource utilization questionnaires, questions on nonmedical resource utilization are lacking. These types of data are usually not available from administrative data sources, yet represent an important and potentially large proportion of resource use. Future research should focus on developing tools to accurately measure these resources.