Health-Related Resource-Use Measurement Instruments for Intersectoral Costs and Benefits in the Education and Criminal Justice Sectors.

BACKGROUND: Intersectoral costs and benefits (ICBs), i.e. costs and benefits of healthcare interventions outside the healthcare sector, can be a crucial component in economic evaluations from the societal perspective. Pivotal to their estimation is the existence of sound resource-use measurement (RUM) instruments; however, RUM instruments for ICBs in the education or criminal justice sectors have not yet been systematically collated or their psychometric quality assessed. This review aims to fill this gap.
METHODS: To identify relevant instruments, the Database of Instruments for Resource Use Measurement (DIRUM) was searched. Additionally, a systematic literature review was conducted in seven electronic databases to detect instruments containing ICB items used in economic evaluations. Finally, studies evaluating the psychometric quality of these instruments were searched.
RESULTS: Twenty-six unique instruments were included. Most frequently, ICB items measured school absenteeism, tutoring, classroom assistance or contacts with legal representatives, police custody/prison detainment and court appearances, with the highest number of items listed in the Client Service Receipt Inventory/Client Sociodemographic and Service Receipt Inventory/Client Service Receipt Inventory-Children's Version (CSRI/CSSRI/CSRI-C), Studying the Scope of Parental Expenditures (SCOPE) and Self-Harm Intervention, Family Therapy (SHIFT) instruments. ICBs in the education sector were especially relevant for age-related developmental disorders and chronic diseases, while criminal justice resource use seems more important in mental health, including alcohol-related disorders or substance abuse. Evidence on the validity or reliability of ICB items was published for two instruments only.
CONCLUSION: With a heterogeneous variety of ICBs found to be relevant for several disease areas but many ICB instruments applied in one study only (21/26 instruments), setting-up an international task force to, for example, develop an internationally adaptable instrument is recommended.

Key Points for Decision Makers

Introduction

For economic evaluations to be a useful tool to inform policy makers regarding the allocation of scarce resources, accurate measurement and valuation of all relevant costs and benefits related to an intervention is key. Costs and benefits related to healthcare interventions that occur outside the healthcare sector, i.e. intersectoral costs and benefits (ICBs) [1, 2], have been shown to be a vital component in economic evaluations applying a societal perspective [3-6]. At the same time, taking a societal viewpoint in economic evaluations is recommended by guidelines for the use of economic evaluation in several countries [7], amounting to a proportion of approximately two-thirds of the analyzed countries in 2010 [8], and most recently also including the US [9]. The societal perspective implies that all benefits and costs that are directly or indirectly caused by the intervention (if significantly present) should be included in the economic evaluation, independently of who incurs them [10]. These costs and benefits can occur outside the healthcare system, including the education and criminal justice sectors, and household and leisure activities, in addition to productivity loss [6].

In recent research, generic methods for valuing ICB unit costs have been developed and applied in the context of the Dutch education and criminal justice system [11]. However, the issue of how to measure such intersectoral resource use in trial-based economic evaluations and cost-of-illness studies with standardized resource-use measurement (RUM) instruments has received little systematic scientific attention to date [12]. This might hinder the consideration of relevant ICB resource use in economic evaluations in the first place [9], and consequently lead to biased results from a societal perspective [13]. It also potentially creates heterogeneity in the measurement due to variabilities in considered cost components, which in turn impairs the comparability of cross-study cost estimates.

In addition, it is unclear to what extent the quality of existing ICB RUM instruments in terms of their psychometric properties has been evaluated. Sound instruments are pivotal to adequately capture the impact of healthcare interventions in other sectors. A review of health-related RUM instruments that led to the set-up of the Database of Instruments for Resource Use Measurement (DIRUM) in 2011 [14] showed that limited evidence is available regarding the validity, and especially reliability, of the listed instruments [15]. This presumably also applies to instruments measuring ICBs. One major exception includes the measurement of lost productivity, for which both a number of RUM instruments have been developed, validated, and the quality of their validation evidence critically appraised in recent publications [16, 17]. Similarly, regarding ICBs in the household and leisure sectors, instruments to measure informal care have been collected and/or assessed in previous research [18-20]. This is not yet the case for the education or criminal justice RUM instruments.

Against this background, this research focuses on health-related ICBs linked to resource use in the education sector (i.e. related to the impairment of educational achievement of individuals [2, 9]) and criminal justice sector (i.e. related to the costs of crimes [2, 9]), and has two aims. Our main aim was to provide a systematic overview of the characteristics of current RUM instruments used in health economic analyses that include ICB items capturing the impacts on the education and criminal justice sectors. Second, we sought to determine the existence of psychometric quality assessments of these instruments. This will not only provide practical guidance for researchers to identify sound instruments for ICB RUM in specific disease areas but also help detect disease areas for which such measurement in economic evaluations might be crucial. In addition, it will provide an evidence-base to potentially pioneer the development of standardized new items to measure relevant costs and benefits outside the healthcare sector.

Methods

Identification of Intersectoral Cost and Benefit (ICB) Instruments

Databases

To identify instruments with relevant ICB items, a stepwise approach was adopted. First, the open-access repository DIRUM (http://www.dirum.org) [14], listing 77 RUM instruments in July 2016, was hand-searched. DIRUM contains full texts of health-related RUM instruments, as well as information and/or references about respective instrument qualities. DIRUM has a search function allowing the selection of specific ‘items of resources being measured’ (e.g. criminal justice service, educational, other non-National Health Service), however, all of the 77 listed instruments were examined in detail.

Second, as DIRUM is not exhaustive [21], an extensive systematic literature search was conducted. Herein, the adopted research approach rests on the assumption that existing ICB instruments that measure such resource use are cited and/or listed in economic evaluations and cost-of-illness studies that were conducted from a societal perspective. Seven electronic databases were searched in July 2016, including MEDLINE (PubMed), EMBASE (Ovid), Social Science Citation Index (SSCI; Web of Science), PsycINFO, Econlit, Education Resources Information Centre (ERIC) and CINAHL (EBSCOhost). This covers two general medical literature databases (MEDLINE, EMBASE) [22], which, in combination with CINAHL and PsycINFO, are considered appropriate to identify economic evaluations [23]. SSCI, Econlit and ERIC were included to incorporate a social science, education and economic perspective, respectively. No date or language restrictions were applied (electronic supplementary Appendix Table A1). Both search strategy and database choices were discussed and agreed with the project team and an information scientist.

Inclusion and Exclusion Criteria

Eligibility criteria of the identified studies, framed around PICOS [24], included all non-institutionalized and institutionalized age groups of the population (P), regardless of the intervention (I), comparators (C) and outcomes (O), set up as a trial-based (non-simulation/non-model) full economic evaluation (i.e. comparing both costs and outcomes of the intervention group and at least one comparator) or as a cost-of-illness study, adopting a societal perspective (i.e. including costs in both the healthcare sector and other sectors) as the study design (S) and measuring ICBs in their analysis, published as a full paper or report. Consequently, study exclusion criteria included no original research, no full economic evaluation or cost-of-illness study, based on model or simulation, and not adopting a societal perspective. Studies building on nationwide population surveys were excluded, whereas articles focusing on the description of RUM instruments were also screened. All studies fulfilling the inclusion criteria irrespective of language were reviewed for referenced patient/carer-reported ICB RUM instruments in the education or criminal justice sectors (and information on relevant psychometric evidence). Instruments either had to measure education or criminal justice resource use, or the article had to refer to a specified (published) instrument that could potentially include such items. Relevant information about the instrument was extracted.

Retrieval of ICB Instruments

Full texts of instruments identified through DIRUM were readily available from the DIRUM website [14]. Full texts of the instruments identified through the database search of economic evaluations and cost-of-illness studies, if not included in the identified publication itself, were retrieved via two channels. A Google (Scholar) search was conducted first. If unsuccessful, the (corresponding) authors of the articles were then contacted via email, and a full version of the utilized instrument (and, in a second step, information about the psychometric properties) was requested. No language restrictions were applied in terms of the included instruments, and translators were used for extraction. Instruments had to be available as a full version (for free).

Identification and Retrieval of Studies Assessing Instrument Properties

Regarding psychometric properties that assess the quality of the identified instruments, this review focuses on their validity and reliability. Validity captures the degree to which an instrument measures what it intends to measure, while reliability refers to the ability of an instrument to do this in a consistent manner [25]. To identify studies assessing the instruments in this respect, the references of the studies listed in DIRUM under ‘instrument qualities’ were investigated. For the instruments included based on the systematic literature review, secondary database searches were carried out based on the names of the instruments and other relevant additional information found in the initially reviewed studies. The same databases were searched as for the identification of the instruments.

Screening of Studies and Instruments

The methodology and reporting of the systematic review were generally set up to be consistent with the Guideline for Conducting Systematic Literature Reviews in Economic Evaluation [24], the methods proposed by the Cochrane Collaboration [26], and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [27]. The PRISMA checklist is presented in electronic supplementary Appendix Table A2. No review protocol was registered. Titles and abstracts (level 1) were screened by SM. In addition, AŁ conducted a second independent screening of the titles and abstracts for 10% of all studies. Differences in inclusion (level 1) were found in approximately 20% of the studies, but did not affect the final inclusion of unique instruments, for which close to no disagreement was observed. This approach also seems justifiable given that all studies that did not explicitly contradict the inclusion criteria on level 1 were assessed further, independently of actually mentioning a (specific or non-specific) instrument in the title or abstract. Full-text screening of the articles (level 2), instrument extraction, and the additional search for psychometric evidence was independently conducted by two reviewers (SM, AŁ) and any disagreement resolved by discussion against the inclusion criteria. The instrument extraction sheet, which partly follows the taxonomy [15] designed for the set-up of DIRUM, was discussed with the project team and piloted on 10% of the included instruments.

Results

Of 3637 unique studies identified in the systematic review of the literature, 167 full texts were included for assessment of the instruments mentioned, cited or listed in these studies (Fig. 1). Following the screening of the full instruments, a total of 55 instruments were identified, resulting in 26 unique RUM instruments with ICB items in the education and/or criminal justice sectors after deduplication (Table 1); 8 instruments were retrieved based on the DIRUM search, 13 instruments were found based on given references in economic evaluations and cost-of-illness studies, and 5 instruments came up in both searches. Of the 26 instruments, five were included following email correspondence with the study authors as full texts were not publicly available. Twenty-one of the 26 instruments were found to be used only once, i.e. in one study, for one country.

Fig. 1

PRISMA flowchart of electronic database search and DIRUM search. PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses, ICB intersectoral costs and benefits, DIRUM Database of Instruments for Resource Use Measurement. Asterisk refers to the instruments included from the DIRUM search

Table 1

Instrument administration and content

Q name, administration mode	Language, origin	Disease category	Target population	Person filling in instrument	Recall period (ICB)	Healthcare RUM in instrument	ICB: education sector	ICB: criminal justice sector	Psychometric evidence, pilot testing	Reference and instrument identification
Mental health (general)
Bodden 2008: diary	Dutch, Netherlands	Anxiety disorder	Children, adolescents	Parents	2 weeks (prospective)	✓a	✓		Validitya [28]	[28]; SLR and E
Client Service Receipt Inventory (CSRI): originally interview, paper-based instrument	English, UK, translated into various other languages	Mental health	Adults	Patient/carer; researcher	3 months	✓a		✓	Validitya [73, 74 ]; validity of Italian version [75], pilot tested	[29]; D and SLR
Client Sociodemographic and Service Receipt Inventory—European Version (CSSRI—EU): paper-based instrument	English and several other languages, Europe	Schizophrenia/mental health	Adults	Researcher	3 months	✓a		✓	Validitya [42] [76] [73, 74 ]; reliability of German version [77] (but no details)	[42]; D and SLR
Alcohol-related disorders
Alcohol: Evaluating Stepped care in Older Populations Study (AESOPS) Instrument: paper-based instrument	English, UK	Alcoholism	Adults	Patient	6 months	✓		✓		[30]; D and SLR
Health Service Utilization Inventory (HSUI), modified: interview	English, Canada	Fetal alcohol spectrum disorder	Adults	Parents	12 months	✓a	✓	✓	Validitya [78]	[50]; SLR and E
Parrott 2006: paper-based instrument	English, UK	Alcoholism	Adults	Patient	6 months	✓		✓		[36]; SLR and E
Sommers 2011: interview	English, USA	Alcoholism and substance abuse	Adults	Researcher	12 months	✓	✓	✓		[44]; SLR and E
European Addiction Severity Index (EuropASI): interview	English and several other languages, Europe	Substance abuse	Adults	Researcher	1 month/lifetime	✓a		✓	Pilot tested; validated in several countries (e.g. France, Germany, Czech Republic) [54], see, for example, Dutch version [79]	[54, 80]; SLR and E
Developmental disorders
Child and Adolescent Services Assessment (CASA) child interview/parent interview, version 5: interview/paper-based instrument (mixture)	English, USA	Developmental problems; mental health	Children, adolescents	Researcher/parent	3 months	✓a	✓	✓a	Reliability,a validity (for healthcare RUM only) [43] [57, 58]	[43]; SLR
Client Service Receipt Inventory Children’s Version (CSRI-C): originally paper-based instrument	English, UK	Developmental, psychosocial and learning problems	Children	Parent/carer; researcher	12 months; per week	✓	✓		Pilot tested [81]	[31]; D
Studying the Scope of Parental Expenditures (SCOPE): web-based survey	English, Canada	Developmental disorders	Children	Parent	6 months; per week		✓	✓	Pilot tested [49]	[49]; SLR
Self-Harm Intervention, Family Therapy (SHIFT) Parent or Carers Questionnaire Booklet: postal instrument	English, UK	Developmental, psychosocial and learning problems	Adolescents	Parent	3 months	✓	✓	✓		[38]; D
Self-Harm Intervention, Family Therapy (SHIFT) Young Person Questionnaire Booklet: postal instrument	English, UK	Developmental, psychosocial and learning problems	Adolescents	Patient	3 months	✓	✓	✓		[39]; D
Chronic disease
Client Service Receipt Inventory for Children with Diabetes (CSRI-CD): paper-based instrument	English, UK	Diabetes	Children, adolescents	Researcher (with child or parent and child)	1 month	✓	✓			[32]; D
Client Service Receipt Inventory for Adolescents with Chronic Pain (CSRI-Pain): paper-based instrument	English, UK	Chronic pain	Adolescents	Parents	12 months	✓	✓	✓	Pilot tested [82]	[33]; SLR
SubCutaneous Insulin: Pumps or Injections (SCIPI) RUM: paper-based instrument	English, UK	Endocrine and metabolic	Children, adolescents	Researcher	3 months	✓	✓			[37, 55]; D
Wetterneck 2006: interview	English, USA	Chronic hair pulling	Adolescents, adults	Researcher	3 months	✓	✓			[45]; SLR
Other diseases
Aygören-Pürsün 2014: web- or paper-based survey	English, multicountry (Spain, Germany, Denmark)	Hereditary angioedema	Adolescents, adults	Patient	Specified based on attack; 6 months	✓	✓		Pilot tested [53]	[53]; SLR
Cost of Trauma Instrument (COTI): mailed questionnaire/telephone interview	English, Australia	Orthopedics and trauma	Adults	Patient (parent for child)	12 months	✓	✓	✓		[48]; D and SLR
HUGS V: Hemophilia Costs and Impact of Disease Study, Version 2: interview	English, USA	Hemophilia	Children, adolescents	Parent (patient < 18 years of age): version 2	1 month	✓	✓			[46]; SLR and E
Huy 2009: interview	English, Cambodia	Dengue fever	Children/other household members	Parent/researcher	Fever episode	✓a	✓a		Validitya (but no details given), pilot tested [51]	[51]; SLR
MAGnesium NEbuliser Trial In Children (MAGNETIC) questionnaire: postal instrument	English, UK	Lungs and airways	Children, adolescents	Parent	1 month	✓	✓		Pilot tested [34]	[34]; D
North of England and Scotland Study of Tonsillectomy and Adeno-tonsillectomy in Children (NESSTAC) Parent’s Questionnaire: postal instrument	English, UK	Ear, nose and throat	Children, adolescents	Parent/carer	3 months	✓	✓		Pilot tested [35]	[35]; D
SafetY and Cost Effectiveness of Adalimumab in Combination with MethOTRExate (SYCAMORE): diary and questionnaire	English, UK	Eyes and vision	Children, adolescents	Researcher (questionnaire); patient (diary)	3 months (prospective)	✓	✓			[41] [56]; D
The Tool to Estimate Patients’ Costs: paper-based questionnaire	English, Kenya	Tuberculosis	Adults	Researcher	During treatment	✓	✓		Pilot tested [83]	[52]; D and SLR
Work Productivity and Activity Impairment Questionnaire plus Classroom Impairment Questions: Specific Health Problem Version 2.0 (WPAI + CIQ:SHP, V2.0)	English, USA	Adaptable to specific diseases/health problems	Adolescents, adults	Patient	1 week		✓a		Validitya [60]	[47]; SLR

Q questionnaire, D Q full-text retrieved via search in DIRUM, SLR Q identified via systematic literature review, E Q full-text received following e-mail correspondence with study author(s), ICB intersectoral costs and benefits, RUM resource-use measurement, DIRUM Database of Instruments for Resource Use Measurement, ✓ indicates yes (included)

aPsychometric evidence (partly) assessed for these sections of the instrument (for details, see column ‘Psychometric evidence’)

A descriptive overview of the characteristics of the included instruments is presented in Table 1. With the exception of the Dutch [28] RUM instrument, all instruments were published in English, mostly referring to the UK healthcare system [29-41], the US [43-47], Australia [48], Canada [49, 50], Cambodia [51] and Kenya [52]. Three instruments [42, 53, 54] were designed for multicountry studies, and were published in English and several other European languages. Mostly, instruments were used for research related to mental health problems or developmental disorders [28, 29, 31, 38, 39, 42, 43, 49], alcohol-related disorders or substance abuse [30, 36, 44, 50, 54], chronic conditions [32, 33, 37, 45] or other specific diseases [34, 35, 41, 46–48, 51–53]. In total, 14 instruments refer to children and/or adolescents [28, 31–35, 38, 39, 43, 46, 49, 51, 55, 56], three instruments refer to adolescents and adults [45, 47, 53], and nine instruments refer to the adult population only [29, 30, 36, 42, 44, 48, 50, 52, 54]. Although many instruments were designed for, or used in, multiple administration modes and forms of recording, 9 [29, 43–46, 48, 50, 51, 54] of the 26 instruments were administered in an interview set-up, including direct contact with a researcher.

ICB Instrument Content

All but two instruments not only include ICB-specific items but also a variety of healthcare utilization RUM questions. Education-related RUM items are included in 21 instruments, most frequently in instruments designed in the context of various specific disease areas [34, 35, 41, 46–48, 51–53], developmental disorders [31, 38, 39, 43, 49] and chronic conditions [32, 33, 45, 55]. Criminal justice RUM items are included in 13 instruments, most frequently related to developmental disorders [38, 39, 43, 49], alcohol-related disorders or substance abuse [30, 36, 44, 50, 54] and mental health in general [29, 42]. Seven instruments include items from both the education and criminal justice sectors. Extracted details on the instrument content regarding, for example, item wording are presented in the electronic supplementary Appendix Table A3.

As shown in Fig. 2, ICB items referring to the education sector mostly cover time missed at school due to the specific disease (15 items), followed by extra need for tutoring activities (six items) and classroom assistance (six items). Attendance at a special/boarding school is captured by five items, as is social/school functioning. ICB items referring to the criminal justice system most commonly refer to the use of lawyers or general legal assistance (11 items), police custody or prison detainment (9 items), appearances in criminal or civil courts (6 items), injuries (5 items), police contacts (5 items), or probation/correction services (5 items) [Fig. 3].

Fig. 2

Education-related ICB items in the instruments. Instrument references by ICB items: (1) absenteeism from school [28, 32, 34, 35, 39, 41, 43–45, 47, 51, 53, 55]; (2) tutoring [31, 33, 41, 43]; (3) classroom assistance [31, 33, 38, 39, 49, 56]; (4) special school/boarding school [31, 33, 38, 39, 43, 49]; (5) school functioning [35, 45, 46]; (6) social functioning [46]; (7) adolescence/school counsellor [38, 39, 50]; (8) special school teacher [33, 43]; (9) special needs statement [31, 38]; (10) special class [43]; (11) school dropout [48, 52]; (12) other educational services [38, 39] [31, 35, 47–49, 51]. As more than one item per topic may be contained in an instrument, the number of references do not necessarily add up to the numbers indicated in the figure. ICB intersectoral costs and benefits

Fig. 3

Criminal justice-related ICB items in the instruments. Instrument references by ICB item: (1) lawyer/legal assistance [33, 36, 38, 39, 49, 50]; (2) police custody/prison detainment [29, 30, 42, 44, 50, 54]; (3) criminal/civil court appearance [29, 30, 42, 44, 50]; (4) injury [36]; (5) police contact [29, 38, 39, 42, 50]; (6) probation/correction services [38, 39, 43, 50, 54]; (7) aggressiveness/violence [36, 54]; (8) traffic accident [36, 44]; (9) psychiatric assessment in custody [29, 42]; (10) property damage [38, 48, 54]; (11) youth offending team [38, 39]; (12) other services [29, 42] [36, 38, 44, 54]. As more than one item per topic may be contained in an instrument, the number of references do not necessarily add up to the numbers indicated in the figure. ICB intersectoral costs and benefits

Psychometric Properties of ICB Items

The psychometric properties were assessed for seven of the identified instruments, namely the Child and Adolescent Services Assessment (CASA) interview [43], Client Service Receipt Inventory (CSRI) [29], Client Sociodemographic and Service Receipt Inventory—European Version (CSSRI—EU) [42], cost diary by Bodden et al. [28], Work Productivity and Activity Impairment (WPAI) instrument [47], Health Service Utilization Inventory (HSUI) [50], and European Addiction Severity Index (EuropASI) [54]. In case of one instrument [51], validation was mentioned but no further details were given. In the economic evaluations citing the included instruments, most authors provided details on the development of the instrument, including, for example, the development team or information on which other instrument the relevant instrument is based on. Pilot testing of the instrument was reported for nine instruments.

Psychometric assessment of ICB-related items could be confirmed for two of these seven instruments; for the CASA instrument [43], both reliability and validity were determined. Test–retest reliability was found to be high for the most intensively used services, including juvenile justice; however, services used in the child’s natural setting, for example school services, were found to be reported with low reliability [57]. The assessment of concurrent validity (i.e. correlation of service use measured with two instruments) was restricted to mental health service use only, and, again, varied for individual services [58]. Construct validity and reproducibility of the general WPAI Questionnaire was established in earlier research [59]. For the WPAI Questionnaire plus Classroom Impairment Questions: Specific Health Problem (WPAI + CIQ:SHP, version 2.0) specifically, psychometric evidence is available for its allergy-specific (AS) first version (the WPAI + CIQ:AS), testing its discriminative and evaluative validity [60]. Except for time missed from the classroom, allergic rhinitis symptoms were found to be well-correlated for most impairment measures (discriminative validity). The same conclusion applies to the correlation analysis on the change in symptoms and change in time missed from the classroom (evaluative validity).

Discussion and Conclusion

Conducting economic evaluations from a broader societal perspective rather than from a narrow healthcare viewpoint is increasingly acknowledged as the gold standard [61], and also manifests in pharmacoeconomic guidelines across Europe [7] and in the US [9]. To be able to consider health-related impacts of interventions in the education or criminal justice sectors in practice, sound instruments measuring relevant resource use are a prerequisite. This review identified a total of 26 unique instruments with a variety of such ICB items. Most frequently, ICB items in the education sector measured resource use due to school absenteeism, tutoring activities, classroom assistance and school or social functioning. Resource use related to the criminal justice sector most commonly captured legal assistance, police custody or prison detainment, criminal or civil court appearance, injuries, police contacts or the use of probation/correction services. Psychometric evidence was found to be examined for seven instruments that, with two exceptions [43, 47], focused only on healthcare RUM items, hence generally leaving out ICB items from their assessment. For the two exceptions, validity (discriminative and evaluative) was assessed for one instrument (WPAI + CIQ:AS [47]), and reliability (test–retest) was assessed for another (CASA [43]).

From a methodological perspective, compared with patient-reported outcome measures, methods of RUM within economic evaluations have been less of a focal point [21]. Specifically, a culture of psychometric validation of RUM instruments for self-reported measurement is lacking in health economics [21, 62]. In a review of UK Health Technology Assessment (HTA) program funds trials, only approximately 30% of the studies were found to report on validation of their RUM data [63]. This review confirms such lack of psychometric evidence, also specifically for ICB items. Indeed, establishing validity, in particular for ICB in the education or criminal justice sectors, might prove even more challenging than for non-ICB items [64]. For example, it might prove difficult, in practice, to get access to alternative data sources (e.g. administrative information) for items such as the number and/or duration of police contacts, psychiatric assessments, received legal assistance, or the number of traffic tickets (electronic supplementary Appendix Table A3) to establish some degree of criterion validity [15]. This is a drawback, especially given that criminal justice resource use is likely a sensitive topic and potentially rather prone to, for example, measurement error [36]. For instance, earlier research suggests that a social desirability bias is more likely to be present in interview situations with a researcher than, for example, self-administered questionnaires [64, 89]. This issue could thus also be relevant for existing ICB items, given that an interview-type administration mode was used for 9 of the 26 identified instruments. Lack of psychometric assessment was also found with regard to piloting of the instruments, although pilot-testing is another critical step in ensuring reliability [63, 65]. For 9 of the 26 instruments included in this review, pilot-testing was reported, which is comparable to the proportion identified in the UK HTA review [63].

A recent systematic literature review by Leggett et al. [12] also analyses RUM instruments. RUM items outside the healthcare sector, including travel expenses, out-of-pocket costs and productivity losses, were concluded to be particularly rare in the 15 identified instruments. Other ICB items were not discussed in the review, which was generally restricted to publicly available, validated instruments designed for adult populations only. Overlap between the instruments identified by Leggett et al. [12] and this research is consequently limited to the CSRI instrument only. The lacking overlap is also likely due to this review’s indirect search approach (Sect. 2.1), contrasting the common method of directly identifying validated instruments through literature search in earlier reviews [12, 16, 66].

Strengths and Limitations

This review is the first to give a comprehensive overview of instruments including ICB items for measurement of resource use in the education or criminal justice system. It discloses both a heterogeneous variety of existing ICB RUM items in these sectors and reveals a lack of psychometric evaluation of these instruments. The latter finding strongly supports the choice of the adopted indirect search strategy. Relevant new instruments identified in the course of this review will be fed into DIRUM, which will help future researchers identify and retrieve instruments with ICB items in a more efficient manner. Indeed, the fact that the majority (21 of 26) of the included instruments were found to be specifically designed for and used in one study only points out substantial efficiency potential for researchers through sharing instruments. Finally, in most cases, the identified ICB items are part of broader instruments; in this sense, this review also provides an overview of existing international instruments for healthcare RUM, albeit not being the initial aim.

Note though that several limitations apply to the instrument identification strategy. Firstly, instruments that generally contain ICB items, but were used in an economic evaluation from a healthcare perspective only, were potentially overlooked. However, all instruments that were specified in the studies were checked for relevant ICB RUM items, even if inclusion of such elements was not to be expected from the respective economic evaluation or cost-of-illness analysis. At the same time, the high number of studies using non-specified, non-referenced, non-listed instruments detected in this review is striking. For example, of all 167 studies included for instrument assessment, approximately one in five articles (n = 31) reported using, for example, some ‘(adapted) standardized’ instruments or a general ‘economic’ instrument, and failed to provide more details. This lack of methodological transparency in instrument reporting, which was also seen in earlier research [64], hampers not only the potential comparability of these economic evaluations but also impairs the quality assessment of the measurement of the included cost components.

Secondly, this review was restricted to instruments mentioned in published health-related economic evaluations or cost-of-illness studies, and thus health services research in general. Looking at other disciplines such as, for example, education research, economic evaluations seem to be applied rather sparsely [67]. However, additional RUM instruments might be found, for example, in the literature on education program impacts [66]. Future research might thus want to consider looking into instruments developed in other disciplines. Trial registries could be an additional channel to identify more recent ICB measurement instruments.

Thirdly, comparable to the review by Leggett et al. [12], a quality assessment of the psychometric evidence was not carried out. This is a limitation since not only the validation findings themselves are crucial but also the quality of the methodology applied [16]. However, given the currently limited evidence published on instrument quality for ICB items (2 of 26 instruments), at this point such an assessment would be premature. Once more relevant studies have been conducted, the quality of the validation studies could be checked, e.g. by applying the COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments) checklist [68], as was done by recent research [16, 66, 69].

Research and Policy Implications

When choosing an RUM instrument for use in a trial-based economic evaluation, it is crucial that this instrument covers the domains that are needed to capture the real-world economic consequences of an intervention in the specific disease area [42]. Based on the included instruments, ICB RUM items in the education sector seem particularly relevant not only for age-related developmental disorders affecting children and adolescents but also chronic diseases, including diabetes and chronic pain. In contrast, criminal justice service resource use seems more important in the fields of alcoholism and substance abuse, and developmental disorders, as well as mental health in general. Future economic analyses, particularly in these fields, should thus consider measuring ICBs from the education and criminal justice sectors, respectively.

Against the backdrop of the recently developed ICB classification scheme [2], the majority of ICB items in the education sector are captured by the existing instruments. Most items are found in the CSRI–Children’s Version (CSRI-C) [31], Studying the Scope of Parental Expenditures (SCOPE; based on CSRI) [49] and Self-Harm Intervention, Family Therapy (SHIFT) instruments [38, 39]. Regarding the criminal justice sector, most ICB items are included in the CSRI [29], CSSRI [42], SHIFT [38] and the instrument developed by Sommers et al. [44]. However, compared with the ICB classification scheme, existing instruments lack ICB items regarding, for example, child maltreatment, sexual assaults and crime consequences on victims, which should be taken into account when developing new items.

Besides validating existing ICB items, future research could focus on the development of a harmonized new instrument with a broad variety of relevant ICB elements. Indeed motivated by the fact that RUM instruments lack validation, RUM items from the DIRUM database are currently being reviewed by Thorn et al. [62] with the aim of developing a standardized RUM instrument for the UK. This instrument will focus on the health and social care sector and will exclude ICBs. Therefore, the first step in developing a new ICB instrument would include a literature search to identify the main cost-driving elements from relevant economic evaluations in a specific disease area [21]. This was selectively performed in prior research [11, 70]; however, given that the inclusion of ICBs does not seem to have a long tradition [71], these empirical studies might be missing such cost elements for this very reason [72]. Thus, collating ICBs that were mentioned in studies but not necessarily measured appears more reasonable [2]. The second step in the development of a new instrument [21] includes the identification of existing instruments, which was also one of the main aims of this review. Following this step, focus groups with healthcare professionals, experts in the education/criminal justice sectors and patients to discuss these instruments should be organized to develop a first version of a new instrument. Many instruments identified in this review were specifically developed for use in one study only, which potentially implies considerable duplication of work happening across studies. Therefore, bundling individual efforts could be a more cost-effective strategy overall. Setting-up an international task force to support these activities by, for example, reflecting on and exploring different sources of, for instance, electronic data as a means of validation, and also developing an internationally adaptable, validated instrument, could be the next step. Such an initiative should take into account the state of the art of RUM classifications by detailing the key components of an RUM [64, 71]. Following a structured taxonomy such as the one developed by Ridyard et al. [64], by providing a description of the data source, who completes the instrument, the administration mode, and the methods and medium of recording, clarity and methodological transparency could be established in the development of such a new instrument.

Data Availability Statement

The authors declare that the data supporting the findings of this study are available within the article and its supplementary electronic information files. Full instruments not directly retrieved from the literature but following email communication with the instrument developer may be directly requested from them based on the information provided (Table 1, ‘Reference and instrument identification’). Relevant instruments will be included in DIRUM (http://www.dirum.org).

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 44 kb)

Health-related interventions often incur costs and benefits outside the healthcare sector (i.e. intersectoral costs and benefits [ICBs]) which, if relevant, should be considered in economic evaluations and cost-of-illness studies from the broader societal analytical perspective to arrive at unbiased conclusions.

Reliable and valid instruments to measure such resource use in these sectors are pivotal. Based on a systematic review of existing instruments used in applied economic evaluations, this paper provides an overview of generic and disease-specific resource-use measurement instruments in the education and criminal justice sectors, including their psychometric properties. Relevant instruments will be included in the Database of Instruments for Resource Use Measurement (DIRUM; http://www.dirum.org).

Many instruments identified in this review were specifically applied in one study only, potentially implying considerable duplication of work across studies. Hence, bundling individual efforts could be a more cost-effective strategy overall, and setting-up an international task force to support these activities, e.g. by fostering methodological transparency and developing an internationally adaptable, harmonized instrument including relevant ICB items, is thus suggested.