Evaluation of the Victorian Healthy Homes Program: protocol for a randomised controlled trial.

INTRODUCTION: The evaluation of the Victorian Healthy Homes Program (VHHP) will generate evidence about the efficacy and cost-effectiveness of home upgrades to improve thermal comfort, reduce energy use and produce health and economic benefits to vulnerable households in Victoria, Australia. METHODS AND ANALYSIS: The VHHP evaluation will use a staggered, parallel group clustered randomised controlled trial to test the home energy intervention in 1000 households. All households will receive the intervention either before (intervention group) or after (control group) winter (defined as 22 June to 21 September). The trial spans three winters with differing numbers of households in each cohort. The primary outcome is the mean difference in indoor average daily temperature between intervention and control households during the winter period. Secondary outcomes include household energy consumption and residential energy efficiency, self-reported respiratory symptoms, health-related quality of life, healthcare utilisation, absences from school/work and self-reported conditions within the home. Linear and logistic regression will be used to analyse the primary and secondary outcomes, controlling for clustering of households by area and the possible confounders of year and timing of intervention, to compare the treatment and control groups over the winter period. Economic evaluation will include a cost-effectiveness and cost-benefit analysis. ETHICS AND DISSEMINATION: Ethical approval was received from Victorian Department of Human Services Human Research Ethics Committee (reference number: 04/17), University of Technology Sydney Human Research Ethics Committee (reference number: ETH18-2273) and Australian Government Department of Veterans Affairs. Study results will be disseminated in a final report and peer-reviewed journals. TRIAL REGISTRATION NUMBER: ACTRN12618000160235. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

It is the first randomised controlled trial of its kind within Australia and contains an economic evaluation as part of the trial design.

There is potential for performance bias with respect to some self-reported secondary endpoints because participants are not blinded to the intervention. However, the intervention is blinded to the study team.

The primary outcome measure of average daily temperature within the home, while unblinded to participants, is objective.

The study is pragmatic in that not all households receive the same upgrades. While this means that effects cannot be attributed to a specific type of upgrade, it does reflect more accurately how the intervention will be used in practice.

The minimum time between control and intervention households receiving their upgrades is 3 months which may not allow for longer time-based effects to be captured. However, there is a trade-off between having a longer interval and the management of delivery of this government programme in a fair and ethical way.

Introduction

Poor housing quality is associated with inefficient energy use, adverse living conditions and increased risk of morbidity and mortality.1–5 People living in such homes are more likely to have respiratory and cardiovascular disease, including asthma and stroke.1–3 6 Vulnerable people, including the elderly, individuals with disability or chronic illness and those with low incomes, are at higher risk due to greater exposure to poor housing conditions.7 This can lead to poorer quality of life for the individual, as well as higher health system costs to society, exacerbating existing health inequities.3 8 9 The impacts of poor housing occur at the individual level but have flow-on effects for society by placing extra pressure on the health system, and other economic effects like decreased productivity and increased carbon emissions.10 11

Cobenefits arise from addressing the key health risks associated with poor quality housing.12 Improving thermal insulation and energy-efficient heating through home thermal comfort and energy efficiency (TCEE) upgrades can increase indoor temperatures and indoor air quality which may lead to improved health,1 while simultaneously reducing energy expenditure and decreasing carbon emissions.12 13 The WHO Housing and Health Guidelines14 recommends an indoor temperature of at least 18°C for the general population and potentially higher for vulnerable groups including the elderly and those with existing health conditions.

Several studies have examined the impact of home upgrades.8 9 15–18 Installing insulation in the homes of low-income New Zealand households resulted in a net benefit of $NZ1574 per household, with approximately two-thirds of this benefit due to reductions in hospital admissions. Grimes et al report that every $1 invested in a home insulation programme resulted in a benefit to society of $3.88, where 99% of these benefits were attributable to health savings.16 Although more energy-efficient and thermally comfortable homes may lead to changes in health, decreased energy expenditure is not a foregone conclusion because residents may change their behaviour and use heating more frequently when the home is more energy efficient.19–21

Three international studies have used a randomised controlled trial (RCT) or longitudinal design to examine the effects of home upgrades on health.8 22 23 These studies showed significant impacts of housing upgrades on decreased wheezing,8 lower respiratory tract symptoms for children with asthma22 and lower rates of emergency admissions for cardiovascular conditions.23 Currently, there is no evidence of health benefits in an Australian context. Results across countries may vary due to differences in climate, the quality of the housing stock and existing levels of energy efficiency.24 There is also limited evidence of the economic benefits of residential internal TCEE upgrades within Australia. Recommendations from the WHO report14 call for further research into vulnerable population groups who spend more time at home, interventions and policies targeted at raising indoor temperatures and the need to explore a range of health outcomes. Our research trial meets all these objectives and will address this important evidence gap.

The main research question of the Victorian Healthy Homes Program (VHHP) evaluation is: do TCEE home upgrades increase winter indoor temperatures? Furthermore, do these increases in indoor temperature result in improvements in health and quality of life of householders, as well as reductions in individuals’ healthcare utilisation and energy consumption?

The overall objectives of the evaluation of the VHHP are to:

Measure the energy benefits of the VHHP.

Measure the health benefits of the VHHP.

Measure the costs and outcomes of the VHHP with an economic evaluation of the programme.

Methods and analysis

The VHHP team

Sustainability Victoria (SV), Australian Energy Foundation (AEF) and researchers from the University of Technology Sydney (UTS) are responsible for leading, delivering and conducting the research components of the VHHP, respectively. The programme delivery partner, AEF, is experienced in working with people from disadvantaged and vulnerable backgrounds and has the responsibility for the recruitment, survey data collection, installation and collection of data loggers, undertaking home assessments and organising the home upgrades. The programme research partner, UTS, is responsible for all other aspects of the research including training AEF project team in the conduct of Good Clinical Practice and reporting the health and energy benefits and conducting an economic evaluation of the VHHP. SV initiated and funded the programme and research, set overall programme policy and coordinated programme delivery and research.

Study design

This clinical trial protocol follows the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guidelines (see online supplemental file 1 and SPIRIT checklist).25 26 Reporting of trial outcomes will follow the Consolidated Standards of Reporting Trials guidelines.27–29 The trial was prospectively registered on anzctr.org.au. Amendments have been made to the protocol because of Victorian Government policy changes, including changes made in response to the COVID-19 pandemic (online supplemental file 2).

This study has a staggered parallel group RCT design, where households rather than individuals are randomly assigned to one of two study groups. All households receive the home TCEE upgrade. The intervention group receive the upgrade prior to winter and the control group receive the upgrade after winter (figure 1). Recruitment is staggered by local government area (local government area) in each of the two geographic regions of the programme (Western Melbourne and the Goulburn Valley in Victoria, Australia—figure 2). This enables a pragmatic balance between the logistical constraints of implementing a large-scale home upgrade programme and the scientific design requirements needed to conduct an outcome evaluation.

Figure 1

Trial design for the Victorian Healthy Homes Program (VHHP).

Figure 2

Map of Victoria showing the nine local government areas involved in the Victorian Healthy Homes Program (VHHP).

Participants and setting

The eligibility criteria for participants are listed in box 1. Approximately 800 households are recruited from the local government areas of Brimbank, Hobsons Bay, Maribyrnong, Melton and Wyndham where the average daily minimum and maximum winter temperatures are 5.4°C–13.2°C, respectively.30 In addition, 200 households are recruited from the regional local government areas of Campaspe, Greater Shepparton, Moira and Strathbogie where the average daily minimum and maximum winter temperatures are 3.4°C and 3.3°C, respectively.30 These local government areas were selected by SV based on social or economic disadvantage and less favourable health outcomes compared with other parts of Victoria (figure 2).

Adult ≥18 years of age.

Living in a single family dwelling of any housing tenure type (home ownership, private rental or managed by state housing authority) within one of the designated local government areas of Western Melbourne or Goulburn Valley (refer to figure 2).

Is low income defined as having one of the Australian Government welfare cards.*

At least one person is receiving home care support services through local council or community organisations.

Has lived in the current home for at least 1 year† and plans to remain there for another 2 years or more.

Mental capacity to provide informed consent.

Treatment

Each of the households in the VHHP receives a preupgrade and postupgrade Victorian Residential Efficiency Scorecard (VRES) assessment of their home by a qualified assessor.31 The preupgrade VRES assessment involves a visit to the home whereas the postupgrade VRES assessment is a desktop assessment, amending the initial assessment based on the upgrade. The assessments provide a variety of metrics including the overall star rating out of 10, which reflects the modelled energy costs of the home. The final decision on which upgrade measures are performed in each home is based on three factors: (1) subjective (perceived) winter thermal comfort considerations from householders, (2) subjective assessment by the energy liaison officer of measures that can be introduced to improve winter thermal comfort, and (3) objective assessment of measures and recommended energy efficiency remediation actions from the VRES.

After the assessment, each household receives a home TCEE upgrade (of value up to $A3500). As it is the winter period that we are studying, control group upgrades are undertaken as soon as practical after winter to avoid increasing inequity between the groups. There are various possible combinations of home upgrade measures that can be undertaken to improve home energy efficiency and warmth, the effects of which are assessed as a package using VRES and are not being assessed individually. Both physical and behavioural aspects are considered when deciding on the upgrade, but priority is given to upgrades that maximise the efficiency with respect to thermal warmth improvement given the budget constraints. Recommendations are made by the VRES assessor and the final decision is made by the manager of the VHHP at AEF. The range of upgrades available is ceiling and underfloor insulation, draught sealing external doors, space heating which can include reverse-cycle air conditioning or replacement of gas heater, upgrades to lighting and internal window coverings. The full list of home upgrade options from which these upgrades were drawn and their specifications can be seen in online supplemental file 3. This list was compiled by experts in home energy efficiency taking into consideration the Australian context of often poorly insulated houses, budget constraints and ease of installation with least possible disruption to occupants.

The upgrade is paid for by the Victorian Government through SV. AEF is responsible for safety checks of homes prior to the upgrade and for arranging appropriate certification of all work. AEF staff check all upgrades and are the first point of contact for participants with any issues related to the upgrade and are responsible for addressing and reporting any problems that arise.

Recruitment and randomisation

Screening and consent

Prior to and during the recruitment periods for the VHHP, the nine participating local government areas disseminate promotional materials about the programme to potentially eligible householders within their jurisdiction. Interested householders complete an expression of interest and are then contacted by AEF staff by telephone to assess their eligibility, answer questions about the programme and the research and to arrange a home visit with an interviewer. Informed written consent is also sought from study participants to gain access to their energy and administrative health data during the first home visit and prior to commencing the interview. Consent forms are stored securely and are separate from any participants’ data to ensure confidentiality.

Randomisation procedures

The randomisation sequence is a 1:1 scheme using random permuted blocks and stratified by local government area. It was created at UTS using the Ralloc command in Stata V.15.0.32 AEF are supplied with the group allocation outcome after participant consent is obtained and all baseline assessments are completed. Therefore, neither the householders nor the delivery partner is informed which arm of the study a home is randomly assigned to until after consent has been provided and the baseline data have been collected (including home assessment). After data collection is complete, the random allocation is provided to analysts in a coded form so that primary and secondary analyses will be conducted blinded to group allocation.

Blinding

The trial is single blinded because we are unable to blind the households from the timing of the home upgrade. All intention-to-treat and per-protocol analyses will be conducted with group assignment in coded form only so that analysts are blinded to the household’s intervention status. Unblinding will only occur after all analyses are complete. No interim analyses for efficacy are planned.

Outcomes

Primary outcome

The primary outcome is average difference in temperature in the home between the intervention and control groups over winter. Winter is defined as the period from 22 June to 21 September, in line with the astronomical winter in Victoria. Temperature is measured every 30 min using a data logger installed in the main living area of the home. All households in both the intervention and control groups will be used for the analysis of the primary outcome.

In addition to the mean indoor temperature over the entire winter period, additional subsets of temperature by time of the day will be tested for differences. This includes mean temperatures across mornings (08:00–12:00), afternoons (12:00–17:00), evenings (17:00–22:00) and overnight (22:00–08:00). We will also investigate the percentage of time each household is exposed to cold (below 18°C) and hazardous (below 16°C and humidity above 65%33) indoor conditions.

External temperature and humidity data at 30 min intervals will be provided by the Bureau of Meteorology for the nearest weather stations to allow comparison of indoor temperatures with external conditions, and to check for significant differences in seasonal severity.

Secondary outcomes

Multiple secondary outcomes will be examined (table 1) and include the following.

Table 1

Outcome measures for VHHP

Data	Description	Data source
Primary outcome	Average daily temperature within the home	30 min interval readings from data logger during winter
Secondary (household level) outcome	Change in average daily humidity within the home	30 min interval readings from data logger during winter
	Change in amount of mould or mildew	Self-reported survey
	Change in amount of damp or condensation	Self-reported survey
	Change in thermal comfort	Self-reported survey
	Household energy costs	Self-reported survey
	Total daily household energy billing consumption	Data held by electricity and gas distributors
Secondary (individual level) outcomes	Health-related quality of life including health utilities	Self-reported surveys (SF-36, EQ-5D-5L and ASCOT)
	Respiratory symptoms	Self-reported survey
	Absenteeism from school or work	Self-reported survey
	Healthcare utilisation: GP visits Specialist visits Diagnostic tests Medicines prescribed	Medicare data (Services Australia)Pharmaceutical Benefits Scheme (PBS) data (Services Australia)
	Hospital admissions	Victorian Admitted Episodes Dataset (VAED)
	Emergency department presentations	Victorian Emergency Minimum Dataset (VEMD)
Economic evaluation	Costs of the VHHPAggregate measures from the outcomes above	Internal records, invoices, staff salaries and surveys

ASCOT, Adult Social Care Outcomes Toolkit; EQ-5D-5L, 5-Level version of EuroQol-5 Dimension Quality of Life Measure; GP, general practitioner; SF-36, 36-Item Short Form Survey; VHHP, Victorian Healthy Homes Program.

Quality of life

Quality of life data will be collected before and after winter using three established instruments: the 5-Level version of EuroQol-5 Dimension (EQ-5D-5L),34–36 the 36-Item Short Form Survey (SF-36)37–39 and the Adult Social Care Outcomes Toolkit (ASCOT).40 The SF-36 Physical Component Summary Scale and Mental Component Summary Scale (MCS) are key secondary outcome measures and the MCS is used in the sample size calculation. The EQ-5D-5L and SF-36 will be used to test for differences in health-related quality of life and the ASCOT to test for difference in social care outcomes between the intervention and control groups. In addition, the EQ-5D-5L and ASCOT will be used to calculate health utilities which will be converted to quality-adjusted life-years (QALYs) to be used in the economic evaluation.

Gas and electricity consumption

Gas and electricity consumption data for each dwelling will be provided by gas and electricity distribution network operators. Electricity data are provided as consumption (kWh) in 30 min intervals while gas consumption data are provided using metre readings every 2 months. A method will be developed to interpolate gas consumption for each dwelling over the winter period. Energy consumption data are sought for the 2 years prior and 1 year after study participation and will be analysed to test for the differences between the control and intervention groups with respect to both total consumption and costs.

Health service utilisation and costs

Healthcare use and costs will be identified from administrative health data including the Medicare Benefits Scheme (MBS), Pharmaceutical Benefits Scheme (PBS), hospital admissions and emergency department presentations. Data will be requested for the 3 years before and up to 1 year after the winter following recruitment. Data will be analysed both to establish the cost differences between the control and intervention groups and to quantify differences in amount of service utilisation and health events such as cardiovascular disease episodes. Seasonal differences will be accounted for.

Statistical analysis

Analysis will be by intention to treat41 and will use regression methods appropriate to the type of outcome variable (continuous, count, categorical) and its distribution. All analyses will include adjustment for local government area (stratifying variable for randomisation) and the year of recruitment (as the severity of the winter may affect the impact of the home upgrades). The extent of clustering by household will be assessed for individual-level outcomes. Analytical approaches such as mixed models or weighted generalised estimating equations will be used to account for this as required. These approaches will also provide valid estimates of the presence of any types of missing data. As the COVID-19 pandemic may impact on the extent to which some households randomised to the intervention group receive their intervention at the appropriate time, a per-protocol analysis will also be undertaken.

Economic evaluation

A trial-based economic evaluation will be conducted to determine the costs and benefits of the VHHP. The economic evaluation will analyse both outcomes and cost data from the primary and secondary outcomes in the intervention and control groups. Resources needed for the implementation of the VHHP will be identified, quantified and valued, and compared against the health and energy cobenefits. The economic evaluation will be both a cost-effectiveness analysis (CEA) and a cost-benefit analysis (CBA). The CEA will produce evidence on the incremental cost per unit of outcome or the incremental cost-effectiveness ratio. The outcome measure is the QALYs, estimated using the quality-of-life scores from the EQ-5D-5L questionnaire. This outcome will be compared with the costs of the VHHP minus the potential monetary cost savings gained through programme participation. Potential cost savings are through reduced healthcare use and improved energy efficiency. In addition, the CBA will model the costs and benefits of the VHHP in monetary terms. Results will be expressed in terms of a ratio of cost per benefit. For example, for every $1 invested in VHHP, the return to society will be $x.

The evaluation will measure the downstream healthcare resource effects for up to 1 year after intervention. It will also estimate longer term outcomes through modelling techniques which will incorporate economic benefits and potential savings in energy use and healthcare costs compared with the cost of the home upgrades. Sensitivity analyses will be undertaken to explore the robustness and validity of cost-effectiveness data and test any assumptions which were used in the economic model. The economic evaluation will also include extrapolation of future potential energy savings, taking into account changes in consumption patterns, prices and depreciation of home upgrades.

Sample size calculation

A total of 1000 households are being recruited (500 per group) which will provide sufficient power to estimate effects for primary and secondary endpoints. All households will be used in the analysis of both primary and secondary outcomes. Sample size calculations are based on two study endpoints, one household measure (indoor temperature) and one individual measure (the SF-36 MCS score). This is because the sample size needed for the primary outcome is much smaller than the secondary outcomes and we wanted to ensure we could detect differences in both outcomes.

To detect a significant difference in the primary endpoint, average indoor daily temperature of 0.81°C between the intervention and control groups with a two-sided 5% significance level and a power of 90%, a sample size of 125 households per group is required, given an anticipated 15% loss of households to follow-up, based on effect sizes and variances found previously.42 43 Specifically, the South East Councils Climate Change Alliance study,43 conducted in 160 homes (80 retrofits vs 80 controls) in six Victorian councils east of Melbourne, Australia, reported that homes that underwent retrofits had a statistically significantly higher average daily temperature during winter months compared with those not (+1.9°C; 95% CI not reported). It should be noted that changes in indoor temperature are not linear in their effects and that we are interested in absolute temperature and thus the number of homes above the 18°C threshold set by the WHO. A change from 17.2°C to 18.01°C is therefore much more meaningful than a change from 21°C to 21.81°C even if both are significant.

To detect a difference in our secondary endpoint of 1.51 norm-based points in the MCS score between the intervention and control groups with a two-sided 5% significance level and a power of 80%, a sample size of 475 households (950 participants) per group is required, assuming an average household size of two adults, 20% loss to follow-up and using an intracluster correlation of 0.1.22 Because of the paucity of published studies in this area, Intraclass correlations (ICC) estimates were not readily available. Therefore, SV requested the ICC estimates calculated by the Housing and Health Research Program at the University of Otago, Wellington, where similar RCT studies have been undertaken.

Data collection and follow-up

Survey data are being collected with face-to-face interviews at two visits: the baseline (first study visit) and a follow-up visit after winter. The interviewer conducts one household survey as an interview, with the primary adult householder, using an iPad to access the online survey via the Qualtrics platform. Each adult household member completes the self-reported individual survey using the iPad if possible, if not then an interview is performed. After the visit, the interviewer uploads the household and individual surveys to the Qualtrics survey platform using a secure connection. After a review of data quality, data are transferred from the Qualtrics platform to the Secure Unified Research Environment44 system for analysis. There is no data monitoring committee for this trial; however, both SV and UTS staff will perform ongoing checks on data quality and completeness in consent forms and surveys.

During the first visit, the interviewer installs a small HOBO UX100-003 Temperature/Relative Humidity data logger in the main living area of the home to record temperature and humidity throughout winter at 30 min intervals. The devices are fixed to the wall at standing head height (to be out of reach of small children due to the risks from the button batteries), and in a location not directly affected by direct sunlight, drafts or heat from appliances. The data logger is then being removed by the interviewer at the follow-up visit. AEF, the delivery partner, is responsible for ensuring the retention of participants and have communication and risk procedures in place to ensure regular contact and follow-up with participants.

Healthcare utilisation and costs are collected through administrative claims data. Services Australia’s data will provide detailed records on medical services and pharmaceutical consumption covered by the MBS and PBS, respectively. The Victorian Admitted Episode and Emergency Department data sets provide detailed records on hospital use and costs.

Programme costs will be gathered in two parts: upgrade costs and administering costs and follow previous costing guidelines.45–47 For the home upgrades, the costs for each home will be determined by invoices received by AEF for the work undertaken, which will be a maximum of $3500 per home. Administering costs will be gathered from both SV and AEF with a series of questions and tables about resource use, including types of resource and quantity. Staff time will be partitioned according to the percentage of time involved in the VHHP. Further, the percentage of time spent by AEF and SV staff on research-related tasks will be excluded from the economic evaluation and as such CBA will incorporate ‘delivery only’ costs.

Modifications to the protocol

Several modifications to the original protocol after trial commencement have been necessary. These are outlined in online supplemental file 2. The most significant of these changes are due to the COVID-19 pandemic, which coincided with the largest stage of household recruitment.

Ethics and dissemination

The VHHP is funded by SV. The study is being conducted in accordance with the National Health and Medical Research Council (NHMRC) guidelines for the ethical conduct of human research.

Ethical approval for the original study protocol was received from the Victorian Department of Human Services HREC (reference number: 04/17) and the UTS HREC (reference number: ETH18-2273) on 2 August 2017 and 20 April 2018, respectively. This manuscript describes the original protocol together with the amendments to the protocol which were reviewed and approved by these HRECs. All protocol changes were noted on anzctr.org.au and can be seen in the online supplemental file 2.

Study results will be disseminated in a final report as well as submitted for publication in peer-reviewed journals and presented at national and international conferences and will be shared with research participants.

Trial status

Recruitment began on 22 February 2018 and was completed by 19 March 2020. Data collection is still ongoing at the time of manuscript submission.