Has COVID-19 changed carer's views of health and care integration in care homes? A sentiment difference-in-difference analysis of on-line service reviews.

Closer integration of health and social care is a policy priority in many countries. The COVID-19 pandemic has reinforced the necessity of joining up health and social care systems, especially in care home settings. However, the meaning and perceived importance of integration for residents' and carers' experience is unclear and we do not know whether it has changed during the pandemic. Using unique data from on-line care home service reviews, we combined multiple methods. We used Natural Language Processing with supervised machine learning to construct a measure of sentiment for care home residents' and their relatives' (measured by AFINN score). Difference-in-difference analysis was used to examine whether experiencing integrated care altered these sentiments by comparing changes in sentiment in reviews related to integration (containing specific terms) to those which were not. Finally, we used network analysis on post-estimation results to assess which specific attributes stakeholders focus on most when detailing their most/least positive experiences of health and care integration in care homes, and whether these attributes changed over the pandemic. Reviews containing integration words were more positive than reviews unrelated to integration in the pre-pandemic period (about 2.3 points on the AFINN score) and remained so during the first year of the pandemic. Overall positive sentiment increased during the COVID-19 period (average by +1.1 points), mainly in reviews mentioning integration terms at the beginning of the first (+2.17, p-value 0.175) and second waves (+3.678, p-value 0.027). The role of care home staff was pivotal in both positive and negative reviews, with a shift from aspects related to care in pre-pandemic to information services during the pandemic, signalling their importance in translating integrated needs-based paradigms into policy and practice.

Introduction

SARS-CoV-2 has provided a large shock to health and care systems around the world. In particular, the care home sector, a setting particularly vulnerable to SARS-CoV-2, has experienced high morbidity and mortality in England[1], and internationally[2]. In response to the rising community transmission of SARS-CoV-2, governments in many countries implemented multiple prevention and mitigation policies[3, 4]. Especially during the first year of the pandemic, populations in many countries were advised (or forced) to avoid contact with those outside their household. To limit outbreaks and excess mortality in care homes specifically, many governments enacted additional restrictions, including bans on visiting or forbidding or strictly limiting physical contact between residents and their families, limiting access of healthcare workers and of residents flowing in and out of hospitals[1, 2, 5].

While these policies were designed to protect populations and care home residents from physical consequences of contracting the virus, there were also likely to be negative spillover effects. Accumulating international evidence suggests that prolonged limitation of human contact between care home residents and their families adversely affects quality of life, wellbeing and mental health of residents and residents’ families[6], [7], [8], [9].

The cumulative effect of being in a high-risk population, compounded by restrictive policies, would, therefore, intuitively be expected to negatively influence resident and their carer's views towards care home services. However, Covid-19 has also pushed forward the health and care ‘integration’ policy agenda, with increasing calls for improved co-ordination among actors caring for vulnerable people (see for England, [10, 11]).

Integrated care has been a global health systems policy priority since well before the pandemic[12]. While ‘integration’ is an ambiguous term with many meanings[13] which can be focused on either processes or outcomes[14], the English NHS has adopted a ‘patient-centred’ definition of the term. This definition encompasses a series of ‘I’ statements. For example, “I tell my story once”[15]. This comes from the recognition that a patient/service user's direct experience of integrated care is likely to be unconcerned with the organisational boundaries observed by health and care professionals and researchers, e.g. ‘primary care/secondary care/social care’. Rather, users, and their carers’, are likely to be most concerned that their needs are met: they are listened to, have questions answered, share in decisions, are treated with empathy and compassion, and transitions between professionals and services are smooth[16].

Some integrated health and social care models have previously shown potential benefits[17], especially when designed in care home settings. For instance, by slowing the rise in emergency hospital admissions among care home residents[18], and generally on improving experience of care[19]. As such, integration policies might have the potential to counterbalance some of the negative effects of the pandemic and the mitigation policies. However, very little is known about which specific aspects users and relatives focus on most in terms of describing their overall experience of integrated care.

With a mix of potentially negative and positive influences on patient experiences caused by Covid-19 and the various responses to it, we have very limited evidence on whether users and carers overall preferences have changed during and after the pandemic[20, 21]. Considering hospitals, patients have generally reported an improvement in their experience and satisfaction as a result of operating changes in health facilities and the ways of consultation[22], [23], [24]. In the case of care homes, however, we do not currently know whether and how these experiences have been modified. Furthermore, we do not know whether their individual experience of integrated care has altered this overall perception.

The availability of timely and reliable information on residents’ and their families’ perceptions is a critical component in supporting service improvement. However, in England, as in many other countries, inspections from the independent health and social care regulator stopped during the pandemic[25]. Data collection on residents’ and residents’ families’ views also faced challenges, with formal study fieldwork, for example surveys or qualitative studies, severely disrupted[26]. The few exceptions, qualitative studies conducted via telephone or videocall, reported findings from relatively small, convenience samples[27], which do not necessarily reflect the average trends.

In this paper, we aimed at filling this gap by making innovative use of online reviews submitted to the largest online platform used to record users’ perception around care home services in the UK. To our knowledge, this is the first study using this type of unstructured text data on a quantitative basis for the analysis of the long-term care sector. We focused on a devolved city-region in England, Greater Manchester, to remove sources of ecological bias, and combined multiple methods, including Natural Language Processing with supervised machine learning, sentiment analysis, difference-in-difference analysis, and network analysis on post-estimation results, to answer three sets of research questions:

How has overall sentiment on care home services changed during the first year of the Covid-19 pandemic (with respect to the pre-pandemic period) for care home residents and their families?

Is there any significant difference in sentiments of reviews associated with integration compared to those which are not? What, if any, impact has Covid-19 had on this sentiment gap?

Which specific attributes do stakeholders focus on most when detailing their most/least positive experiences of health and care integration in care homes? Have these attributes changed over the pandemic?

The paper is structured as follows. Section 2 describes the data and the methods employed in this study. Section 3 presents results, while Section 4 concludes.

Data and Methods

Institutional Settings

In England, the long-term care sector consists of primarily independent sector providers of care, with the large majority of them being for-profit[1, 28].

On 24 March 2020, in response to Covid-19, the general population in England was ordered not to leave their home except for “essential” reasons[29]. This law included bans on visiting care homes, later clarified to only allowed in exceptional circumstances such as at the end of life. Further stringent infection control rules and regulations were also introduced during the pandemic, including limiting access of healthcare workers in care home settings, limiting residents flowing in and out of hospitals, and forbidding, or strictly limiting, physical contact between patients/residents and their families[5]. The national lockdown was eased from early July 2020, immediately followed by the introduction of a testing strategy for staff and residents in care homes caring for older people and those with dementia. A series of piecemeal regional policies followed, until additional national restrictions (including the “rule of six” – limiting social gatherings to a maximum of six people - and curfew) were re-imposed in September 2020, a new hospital discharge approach, and the expansion of the testing programme for staff and residents of all care homes (August). A formal ‘three tier system’ of restrictions determined at local level was then imposed from early October 2020[30], before a second national lockdown was re-imposed on the 5th of November[31].

The mix of national and local variation in policies and Covid rates makes it difficult to conduct any long-term coherent analysis at the national level over this period. Instead, we focus on a single large metropolitan area, Greater Manchester. This region also holds devolved health and social care responsibility and budgets, but still with variation in deprivation and local policy and implementation across its 10 localities supplying about 17,400 care home beds. It was one of the most strictly controlled regions in England in terms of Covid-19 policy over 2020 because of relatively high transmission rates, which means we would expect any effects on user experience from the mix of direct viral effects and policy spillovers to be most pronounced.

Data Source

The core source of data comes from carehome.co.uk - the main online platform with information for the majority of care home and home care providers operating in the United Kingdom. For the English market, this website refers to about 15,000 care homes. Importantly for our study, this site gives information about the service experience of users and their relatives through reviews posted online on a daily basis.

We collected all reviews (2,195) submitted for 242 care homes offering services to older people with/out dementia that were operating in Greater Manchester, over a 24-month period: 14 months before (January 2019 - February 2020) and the first calendar year (10 months, from March 2020 to December 2020) after the official start of the Covid-19 pandemic in England.

Online reviews were linked at a care home-level with the CQC register of active care homes in March 2020. There were 445 registered care homes offering services to older people with/out dementia that were operating in Greater Manchester (see Appendix 1 for an assessment of the sample selection bias of our study).

Methods

We used a combination of existing methods to de-construct and analyse unstructured textual data. We used natural language processing with supervised machine learning to construct a measure of sentiment for care home residents' and their relatives’, difference-in-difference analysis to examine whether experiencing integrated care altered these sentiments by comparing changes in sentiment in reviews related to integration (containing specific terms) to those which were not. Finally, we used network analysis on post-estimation results to assess which specific attributes stakeholders focus on most when detailing their most/least positive experiences of health and care integration in care homes, and whether these attributes changed over the pandemic. With full details provided in Appendix 2, we summarise here the most salient aspects of our analysis.

Outcome measure

We used the content of the written reviews to extract the underlying sentiment – the positive or negative opinions towards the care home services[32] - and synthesise this into a score. In doing so, we followed natural language processing techniques used in other fields to derive opinion scores from text (see e.g.,[33, 34, 35]) and employed the AFINN algorithm[26] to extract the sentiment score from the words used by the reviewers (with each included word scored between -5 to +5, where higher scores indicate a more positive sentiment).

Exposure variable

The English NHS has adopted a ‘patient-centred’ definition of the term co-developed by National Voices (the coalition of health and social care charities in England)[15, 16]. We developed a list of ‘integration terms’ from a National Voices document focused on how patients would describe integration[16], to differentiate reviews focused on ‘integration’ from other reviews. Reviews were coded as related to ‘integration’ if they contained at least one of these listed terms.

Analysis

To formally examine the presence of differentiated sentiment in reviews containing integration terms and those which did not in the pre- and during Covid-19 period, we used a care home fixed-effect estimator and a difference-in-differences analytic design without (unadjusted) and with covariates (adjusted) and by reviewer types. The identification of a net impact of the pandemic on users’ and relatives’ sentiments relies on the standard “parallel trends” assumption. That is, time trends in the sentiment of reviews mentioning integration words and those which did not would have been parallel in the absence of Covid-19. We checked whether the trends in the sentiments were parallel in the pre-Covid-19 period by visual inspection and by estimating our main model with a linear time trend, and with a flexible monthly-specific trend. We also implemented a version of our main model that was able to capture differential responses over the first 10 months of the pandemic period. This allowed us to estimate a flexible sentiment response path during the Covid-19 period.

Reviews in the lower and upper tertiles of the predicted sentiment score were used to explore the occurrence of specific integration terms in post-estimation analysis. We measured the occurrence of integrated terms before and during the pandemic and took the difference to highlight changes in the occurrence of specific terms. We then used a network diagram analysis to visually describe the associations between the top eight most used integrated terms and other words within the reviews.

Results

Descriptive statistics

There were similar numbers of reviews in comparable pre-Covid months to post-Covid months (see Appendix Figure 1 , panel A). During the first wave of Covid-19, however, there was a slightly lower number of reviews than expected from historical trends. On the other hand, there were a higher number of reviews than the previous year during the 2nd national lockdown (from 5th November).

Figure 1

Sentiment trend over period of analysis by review types

Notes: Locally weighted regressions (bandwith=0.8) of the sentiment score on months. The null hypothesis of parallel trends in the pre-Covid period was not rejected at conventional statistical levels when assuming a linear trend (p-value = 0.862), nor allowing for monthly-specific trends (p-value = 0.617).

Where pre-Covid-19 there was no difference in trends in comparable months, the percentage of reviews related to integration increased dramatically over the Covid-19 period, with two spikes at the beginning of the first and second Covid-19 waves and a drop to the March 2020 level that occurred in the middle (August), coinciding with the period where the testing programme was expanded to all adult care homes (see Appendix Figure 1, panel B). Similar trends were found in the average number of integrated terms used in reviews pertaining to integration (see Appendix Figure 1, panel C).

Figure 1 shows sentiment trends for reviews mentioning, or not mentioning, integration terms. Overall, sentiment exhibited a positive trend and continued to increase over the beginning of the first part of the Covid-19 period, but then declined after the re-opening after the first lockdown and during the second Covid-19 wave. The sentiment was higher, by roughly 2 points on the AFINN score, in the reviews containing integration terms than in the review unrelated to integration in the pre-Covid period and remained so in the post-Covid period (see also Appendix Figure 2 for comparison of kernel densities).

Figure 2

Net change of sentiments in reviews containing integration words over the Covid-19 period

Notes:estimates obtained from equation 2 (see Appendix 2). Vertical bars are 95% confidence intervals.

Econometric analysis

Results of the main econometric specification are reported in Table 1 . In line with the descriptive analysis reported in Figure 1, the sentiment score was significantly higher (more positive) in the reviews containing integration terms than in the reviews unrelated to integration in the pre-Covid period by about 2.3 points on the AFINN range for the overall adjusted model. The result is largely driven by the high sentiment scores of the reviews left by daughters, sons or spouses of the resident.

Table 1

Difference-in-Difference estimates

	Overall		Review submitted by
	Unadjusted	Adjusted	Female relative	Male relative	Other relatives/friends	Resident
Review pertaining to integration (vs not pertaining)	2.746⁎⁎⁎	2.300⁎⁎⁎	2.454⁎⁎⁎	2.547⁎⁎⁎	2.365	-1.185
	(0.427)	(0.425)	(0.580)	(0.861)	(1.889)	(1.125)
Covid-19 period (vs pre-Covid)	1.006⁎⁎⁎	1.125⁎⁎⁎	1.791⁎⁎⁎	0.135	-0.394	0.923
	(0.369)	(0.413)	(0.581)	(0.741)	(1.633)	(1.371)
Review pertaining to integration submitted in the Covid-19 period	0.238	-0.0380	-0.439	0.727	-1.459	-0.973
	(0.664)	(0.673)	(0.861)	(1.409)	(2.829)	(1.658)
Review from a female relative (reference category)
Review from a male relative		-1.388⁎⁎⁎
		(0.345)
Other relative/friends		-1.491⁎⁎⁎
		(0.511)
Resident		-2.000⁎⁎⁎
		(0.554)
Number of Reviews	2,195		1,354	542	169	130

Notes: estimates from equation 1 reported (see Appendix 2). Standard errors in parentheses. Significance Level: * p < 0.10

** p < 0.05

*** p < 0.01.

During the Covid-19 period, the general sentiment expressed in reviews increased significantly at 5% level by about 1.1 points on the AFINN scale for the overall adjusted model. The result was largely driven by the significant and large increase in the sentiment score of female relative reviewers (+1.8 points).

The third row of the table provides the difference-in-differences estimates ( in equation 1). Over the Covid-19 period, the overall sentiment score associated with reviews pertaining to integration very slightly decreased compared to those not pertaining to integration (-0.04 for the overall adjusted model), but not significantly at conventional statistical levels. The effect was largely driven by reviews submitted by other relatives/friends and by residents themselves, although none of the coefficients in sub-group analyses were significant.

We also assessed the presence of changes over time in the Covid-19 period with relevant estimated parameters plotted in Figure 2. It shows that higher (more positive) sentiments were expressed in reviews mentioning integration terms only at the beginning of Covid-19 wave 1 (March: 2.172, p-value=0.175), in July 2020 (0.872, p-value=0.617), and just before the beginning of wave 2 (September: 3.678, p-value 0.027), with perhaps a subsequent decline in October (-2.749, p-value 0.056). No significant differences were found through the rest of the period of analysis.

Post-estimation analysis

As shown in Figure 3 , in the pre-Covid period, “support”, “team” and “hospit” were by far the most used integration stems (>50%) in both the most positive and least positive reviews. “Team” and “support” remained top integration stems also during Covid-19, but other stems like “regular”, “contact”, “discharg”, “prevent” and “inform” leveraged significantly in the most positive reviews in the post-pandemic period. On the other hand, terms referring to other care sectors, such as “doctor”, “servic”, and “hospit” decreased in frequency for all reviews submitted during the Covid period. The occurrence of “inform” more than doubled in frequency in the least positive reviews post-pandemic, with stems like “decis” “team”, “nurs” and “talk” also increasing in frequency in the least positive reviews.

Figure 3

Most frequent integration stems over top and bottom positive reviews before (upper panel) and during (bottom panel) the Covid-19 period

Notes: Percentages of occurrence of a given integrated stem over the total integrated stems within a given group.

Topic modelling identified the top salient topics that dominated reviews related to integration. From our graphical analysis in Figure 4 , we extracted two main findings. First, the central role certain words played in the reviews. In particular, staff was the most paired term across the nodes that composed the network (associated with about 2,400 words on average before and after Covid-19). This suggests that most reviews alluded explicitly to issues linked directly to staff. This result may be expected given the labour-intense nature of the care home sector. Likewise, team is the most central integration-specific term (paired with 1,056 words on average). The second main result related to how integrated stems were associated with each other, with the edges between nodes and their thickness relating to frequency of associations. Thus, staff and team showed the strongest association before and during Covid (appearing together in reviews 43 times, and 39 times respectively). Echoing the results above, staff was also related to terms such as team and hospital before Covid, while its associations with terms such as informed and regular strengthened during the pandemic. Support remained strongly associated with staff throughout.

Figure 4

Most frequent associations with integration words pre- and during Covid-19

Discussion

Principal findings

Using online reviews submitted to the main online UK platform for care home services, we estimated the impact of Covid-19 on care home resident's relatives’ sentiments, and whether experiencing integrated care altered these sentiments. We showed how a combination of different methods enabled the de-construction of unstructured textual data to inform the policy debate on the reviewer's perception around integrated care services pre- and during the first year of the Covid-19 pandemic.

We found that sentiments around care home services exhibited a positive trend pre-Covid-19 that continued during the beginning of the first part of the Covid-19 period before declining after the re-opening from the first lockdown and during the second Covid-19 wave to reach a level just marginally higher than pre-Covid-19 by the end of 2020. This finding of increasing care positive sentiment post-Covid fits with previous analysis from the hospital sector[22]. In that respect, the Covid-19 shock to sentiment appeared to act mainly in the shorter term in care homes. This was possibly related to the initially exponential growth waves of cases which caused the drastic initial policy responses (and, maybe even more due to the policy response to the initial spike itself). The more negative spillovers of these policies, as well as the fatigue of responding to the pandemic itself, might have dampened any initial effects over the longer term.

Motivated by the significant policy push of health and care integration policies, we assessed users and carers’ perception around “integration” in care home settings and whether it has changed during the pandemic. We found that, on average, reviews containing integration terms were more positive that those which were not, signalling a significantly positive perception of integration mainly by residents’ spouses and sons. Over the Covid-19 period, this gap in positiveness remained almost stable, with idiosyncratic increases at the beginning of Covid-19 waves one (although not significant at conventional levels) and two (statistically significant), signalling the presence of potentially limited shorter-term effects once again. Importantly, we noted a shift in reviews’ focus from aspects related to external care and support services, that were common in the pre-pandemic period, to regular information services during the COVID-19 period. This might signal the pivotal role of care workers in informing relatives when residents flowing in and out of hospitals and care home visiting (of doctors and residents’ relatives, etc.) were restricted. This might also indicate a shift from importance of ‘vertical’ integration pre-Covid towards ‘horizontal’ (i.e., within care home) integration during Covid. It likely also signals the additional physical and psychological stress that care workers have had to take on to cope with the pandemic.

Limitations

When extracting evidence for policy and practice from reviews data, two factors are relevant. First, their internal validity against attribution bias (how they reflect the real view of the reviewers and the real service delivered); conformity bias (the feeling to act due to the (in)actions of others); and perception bias (reviewing situations based on own expectations and/or incorrect/distorted assumptions). Under our difference-in-difference setup, we have assumed these (along other unobservable characteristics) remained constant before and during the covid-19 pandemic and between types of reviews.

Second, the external validity and generalisability of results against the sample selection occurred at the reviewer level (differences in the probability of submitting an online review) and at the care home level (differences in the probability of reviewing services). While we were not able to assess the former source of selection, we noted in our sample an under-representation of small homes, with quality below the inspector's (CQC) threshold. On the other hand, we controlled for heterogeneity in care provider by using a fixed-effect (at care home level) estimator. The focus on only Greater Manchester care homes also allowed us to negate the differential effects of local policies and Covid-19 case rates over the period.

While recognising the limitations of reviews data, these are an available source in a period where other data collection processes were severely limited. They may also complement quality inspection data on aspects not necessarily reported in formal inspections.

There is also no ideal list of integration terms for defining our exposure variable, particularly for capturing the way residents and relatives might describe these interventions. Nevertheless, we identified and drew on the best available source available to us from a patient perspective[16], and constructed our own measure from this.

Strengths in relation to other studies

There is already substantial evidence of the direct impacts of Covid-19 on care homes, particularly in terms of morbidity and mortality of residents[1, 2]. More recently, there is evidence of the indirect effects, for example on measures of loneliness and mental health[9]. We add to this literature by examining a measure of experience of care, sentiment, and showing that, contrary to the above, on average this appears to have been increased over the first calendar year of the pandemic, with respect to comparable pre-Covid-19 months. Particularly, experience of integrated care appears to have been associated with higher sentiment throughout, and somehow protective in the shorter-term for initial experience of shocks. However, the specific focus of residents and their relatives appears to have evolved somewhat from pre- to post-Covid-19, from a focus on interactions with other services and support, towards a focus on information from staff in the care home itself.

However, as sentiment also increased in reviews which did not contain integration words, we do not rule out the possibility of other integration initiatives that, by improving on information flows and provided care, have been positively perceived by residents and relatives but are not as directly realised or described as such by users and their carers. However, our study showed the focal point which care home staff appear to play in driving sentiment for residents and their relatives, and it might be that their additional effort over the Covid-19 period was recognised leading to the initial boost in sentiment we found.

The central role of staff fits with previous studies in other care settings, where, e.g. staff reported as a key determinant of patient satisfaction[36] and perceptions of care in English hospitals[37]. In care homes specifically, “empathic staff” has also been identified more generally as the top priority for improving residents’ satisfaction[38].

Our findings also ring true to the findings from the limited qualitative interviews undertaken with care home stakeholders over the Covid-19 period. Information transfer, particularly lack thereof, was identified as a key determinate for residents’ and their carers’ experience over the first calendar year of the pandemic[27]. Our study adds to these findings with a significantly larger dataset and use of advanced quantitative methods to record stakeholder experience systematically in a longitudinal manner.

Implications of the study and further research

Care homes have, until recently, been a fairly neglected part of the health and care system in terms of research[39]. This might in part be due to the fragmented nature of the sector in many countries, with a particularly high prevalence of small private providers caring for a mixture of self-funding and publicly-funded residents, largely seen as separate from the research-intensive healthcare sector. We have contributed to the expanding health literature that uses online user-generated reviews[40], [41], [42], [43], [44], [45], [46] by using innovative approaches to obtaining and analysing care home online reviews data.

In terms of policy, the integrated care agenda has been escalating in many countries over the past decade. Our results suggest a clear perception of care home residents’ carers’ of the important role of the staff in mitigating the adverse effects induced by the pandemic. Specifically, our findings emphasise the importance of the care home workforce and, therefore, for policy to ensure a robust, sustainable, healthy workforce to provide these services[47]. This should not be overlooked or neglected as policymakers seek to pursue vertical integrated care initiatives. They are an essential channel for translating integrated needs-based health and care paradigms into policy and practice.

Further research should examine whether these findings hold in other regions, or with other data sources for a more full picture, and to examine generalisability further.

Conclusions

Reviews by residents, and their carers’, are an important source of information for monitoring care home quality and experience. Care home experience appears, on average, to have been kept high over the initial months of the pandemic, particularly for those experiencing integration, and likely as a result of the extra effort of care home staff and their adaptation to providing additional information and support needs. Supporting this workforce in turn, and ensuring it is sustainable, should be a policy and research priority moving forward, as should additional work to determine particularly effective integration initiatives.

Declaration of Competing Interest

none.