Determinants of regulatory compliance in health and social care services: a systematic review protocol.

Background: The delivery of high quality health and social care services is a fundamental goal for health systems worldwide. Identifying the determinants of quality is a complex task as there are a myriad of variables to choose from. Researchers in this field have assessed a range of organisational and environmental factors (for example: staff composition, facility ownership, facility size) for an association with various quality metrics. Less attention has been paid to the determinants of compliance with quality regulation. Identifying the determinants of compliance has the potential to improve regulatory processes and can inform quality improvement initiatives undertaken by service providers and policy makers. This protocol describes a systematic review which will review literature from a wide range of study designs and sources to develop an overview of the determinants of regulatory compliance in health and social care services.
Methods: A wide range of study designs and grey literature will be sought for this review. Searches will be conducted using PubMed, MEDLINE, PsycInfo, SocINDEX and CINAHL databases. The studies included in the review will be subject to quality appraisal with reference to the collection of tools available from the Joanna Briggs Institute. Data extraction will be informed by the Consolidated Framework for Implementation Research (CFIR). A narrative synthesis will be conducted on the barriers, facilitators and factors associated with compliance, with reference to the concepts mapped onto the CFIR. GRADE-CERQual will be used to grade the overall body of evidence.
Conclusion: The findings of this review will be useful to regulators to inform regulatory policy and practice. Service providers and policy makers may also use the findings to inform quality improvement initiatives aimed at improving compliance and quality across a range of health and social care services. Copyright:

Introduction

The delivery of high quality health and social care services is a fundamental goal for health systems worldwide. Quality is a somewhat nebulous term that can be difficult to define. In health and social care, the following have been proffered — by the European Commission and the Institute of Medicine — as definitions of quality: “health care that is effective, safe and responds to the needs and preference of patients” ; “Quality of care is the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge” .

Quality of care is often conceptualised with reference to Donabedian’s framework : structure, process and outcome . Donabedian "defined structure as the environment in which healthcare is provided, process as the method by which healthcare is provided and outcome as the consequence of the healthcare provided" . To use social care as an example: the number of staff working in a centre would represent structure; the frequency with which a person has an assessment of need would represent process; and the degree to which a person has autonomy to make decisions about their care would be an outcome.

Quality of care in health and social care settings is variable. Variation in quality may be the result of a structural component as demonstrated in a study that assessed the effect of changes to staffing levels on the prevalence of pressure sores in nursing homes . It may be process-related as found in a prospective evaluation of simulated emergency department triage which found a high degree of variability in the processes of triage and measurement of vital signs . Outcomes are also subject to variation such as found in an analysis of in-hospital mortality in non-cardiac surgical patients across Europe, which found that mortality varied across the countries studied .

Regulation is one response to variability in quality: authorities establish a set of norms or standards to benchmark quality and then assess the extent to which organisations and individuals meet these standards. Failure to comply with regulations may lead to sanctions such as intensified surveillance, or even revocation of license to operate (typically through a registration or licensing system). Regulation is a common feature in a wide range of sectors: finance, environment, transport and healthcare. Selznick (1984) offers a useful definition of regulation where it is described as: “sustained and focused control exercised by a public agency over activities which are valued by a community” .

Regulations in health and social care are wide-ranging but typically cover aspects such as, hygiene, governance and management, documentation/records, care practices, staffing, training . As with components of quality, regulations can also be conceived of as falling into the categories of structure, process and outcome. By way of example, in the context of staffing and patient experience: a regulation specifying the staffing ratio represents structure; a regulation specifying the supervision or development of staff in the humanity of care represents process; and a measure of patient experience represents the outcome.

Compliance, in a regulatory context, can be understood as “behavior fitting expectations communicated to regulatees regarding how the former should or should not behave in a given domain” . Compliance is generally articulated by a regulator along a continuum of ‘compliant’ to ‘not compliant’. For example, the Care Quality Commission (CQC), regulator for health and adult social care in England, has four levels of compliance: outstanding, good, requires improvement, inadequate . These ratings are determined according to the professional judgment of a CQC inspector and assessed at the level of individual care components; services also receive an overall rating.

Determining the level of compliance with a specific regulation requires varying degrees of effort and evidence-gathering on behalf of an inspector. Evidence can be generated from speaking with residents and staff, reviewing documentation and records, and observing practices as they happen. For example, assessing compliance with a structural requirement, such as a requirement that a person in charge should have “a minimum of 3 years’ experience in a management or supervisory role in the area of health or social care” , is a relatively straight-forward task of identifying the appropriate documentation. Judging whether a service has admitted residents in “a competent, equitable, timely, and respectful manner” , requires the inspector to undertake several tasks: speak with recently-admitted residents or their representatives, review admission records, speak with staff involved in admitting new residents. Assessing compliance in a regulation which specifies an outcome, such as one that seeks to “ensure respect for the personal privacy of each person in care” , requires the inspector to speak with residents and staff, review documentation and observe care practices on-site.

As with quality, compliance with regulations is also variable. This is evidenced in a number of reports by regulators in the health and social care sectors . Various factors have been assessed for their association with compliance in health and social care settings such as: ownership, facility size, patient feedback, staff levels and competencies, availability of amenities, location, presence of an ombudsman and patient/resident characteristics . In the case of some of the above, there is a danger of a type of circular reasoning because it may be a factor that is regulated. For example, if there is a regulation requiring managers to have a certain qualification, then it is difficult to construe this as a determinant because it is a pre-requisite for compliance.

Describing the reasons and potential explanations for the variability in compliance with regulations is the key focus for this systematic review. The authors have found no such review to date and this represents a gap in knowledge. Our review will use the Consolidated Framework for Implementation Research (CFIR) to categorise the determinants of compliance described in the literature.

The CFIR is an overarching typology used in implementation science . The CFIR was developed by including “constructs from a synthesis of existing theories” and is concerned more with what works where and why as opposed to simply ‘what works’ . There are five domains within the CFIR: intervention characteristics, outer setting, inner setting, characteristics of the individuals involved, and the process of implementation. Each of these have multiple constructs within each domain.

The CFIR may be applied to regulation by mapping any barriers, facilitators or factors associated with levels of compliance. In this sense, regulation is perceived as the intervention and compliance is the outcome. By way of example, the points below illustrate how regulation and compliance can, hypothetically, be mapped onto the constructs within the five CFIR domains:

intervention characteristics: do regulatees regard regulations as being evidence-based?

outer setting: are there incentives/disincentives that encourage the regulatee to comply?

inner setting: what resources (e.g. staffing) are available to the regulatee to achieve compliance?

characteristics of the individuals: what knowledge and beliefs do senior managers in regulatees have towards the regulator?

process of implementation: do inspectors act as external change agents to foster compliance?

Beyond mapping the barriers, facilitators or factors associated with levels of compliance our review will use theory to aid interpretation. It is not possible at this juncture to be definitive in what theories will be applicable, as this will be informed by the nature and design of the studies included for the systematic review. As such, the following sections will describe some potential theories that may serve to elucidate the material that is mapped on to the CFIR domains and constructs.

Various theories have been posited in the context of regulation and on the means by which regulators seek to ensure compliance . The disposition or modus operandi of a regulator can be conceptualised as being plotted along a spectrum. At one end are those that are intolerant of any form of non-compliance and quick to deploy punitive measures. At the other end of the spectrum there is a greater acceptance that compliance can fluctuate and the regulator will adopt a more consultative approach which seeks to coax providers into compliance .

Other theories look to characterise the disposition of regulatees and their attitude towards compliance . Non-compliant organisations may be ‘political citizens’, generally agreeing with the goals of regulation but objecting to the prescriptions of the regulator in terms of how this is to be achieved. Or, they may simply be ‘organisationally incompetent’ and fail to understand or manage the demands of the regulations or the regulator . Some organisations may fully subscribe to the goals of regulation and be ‘model citizens’ that strive to meet or exceed the standards that have been set. Others pay ‘lip service’ to these goals and do the minimum to satisfy the regulator, giving the appearance of compliance .

The regulator can also be interpreted by organisational actors as being either an ally, threat or obstacle . The regulator being perceived as a threat can mean the threat is at the level of an individual (their job or esteem) or at the level of the organisation (profits or reputation). As an ally, the regulator is perceived as competent and regarded as encouraging an organisation into compliance, offering advice and support to achieve common goals . The regulator as obstacle is seen as lacking authoritative technical expertise in the particular field or where “compliance requirements…are inadequately connected to the underlying regulatory goals” .

As set out above, compliance is not only influenced by structural or organisational factors such as the size of a facility or who owns it. It is also an outcome of the nature of engagement between regulator and regulatee and contingent on their respective dispositions towards compliance. Normalisation process theory (NPT) represents a theory within which to understand these relationships and contingencies. NPT facilitates “systematic exploration of why some processes lead to a practice becoming successfully (or not) embedded (i.e. normalised) and sustained, by attempting to understand the intervention in relation to the work that people do” . NPT facilitates an exploration of what factors are associated with the successful integration and alignment of an organisation’s goals with those of regulation. Such an approach has been adopted elsewhere but the author has found no studies that have used NPT as a theory to aid understanding of regulatory compliance in organisations. NPT may be particularly useful in the fifth CFIR domain: process.

The findings of this review will be of benefit to regulators as they may inform changes to regulatory policy and practice. In addition, service providers and policy makers may use the findings to develop quality improvement initiatives to improve rates of compliance and, ultimately, provide a better quality service.

Research question: What are the determinants of regulatory compliance in health and social care services?

The protocol will describe the:

process for a comprehensive search for relevant articles

eligibility criteria for the inclusion of such articles

method for screening articles for inclusion

appraisal method for assessing the quality of individual studies

approach to data extraction, synthesis and appraisal of the overall body of evidence.

Protocol

Criteria for inclusion

The phenomena of interest are the determinants of regulatory compliance in health and social care services.

There are no limits on the articles for inclusion in terms of publication date or language.

Articles — either qualitative, quantitative or mixed-methods — will be included if they:

Describe factors or characteristics that are related to regulatory compliance. Specifically, this refers to regulations that are mandated by government or other state authorities. A wide range of constructs will be considered for inclusion including, but not limited to, the following: service characteristics (size, location, model of care, ownership); organisational characteristics (culture, management/governance structure, maturity); service user characteristics (age, disability type, disease/illness); nature of engagement (punitive, adversarial, collaborative).

Discuss barriers or facilitators to regulatory compliance for health and social care services.

Are focused on quality of care in health and social care services and use regulatory compliance as an outcome measure.

Studies will be excluded if they:

Analyse regulatory compliance in a field other than in a health or social care setting or service.

Analyse compliance with clinical guidelines or other evidence-based methods for managing care that are not underpinned by the potential for regulatory sanction where there is a failure to comply.

Use an outcome measure that is not equivalent to regulatory compliance in accordance with the definitions set out above. For example: adherence to voluntary standards or codes of conduct; where failure to comply does not result in regulatory sanctions of enforcement; compliance concerning individuals as opposed to organisations as is the case with regulations for specific health care professionals.

Types of study to be included

There will be no specific limitations on the types of study considered for inclusion. Given the nature of the research question and the topic under review it is anticipated that the studies will generally fall into the categories of: cross-sectional designs for quantitative studies; and ethnographic or focus group/interview designs for qualitative studies. Preliminary searches have found studies that use compliance data from regulators coupled with cross-sectional data on services which are sourced either directly through surveys or via national repositories such as the Centers for Medicare and Medicaid Services (CMS) in the USA.

Search methodology

The CIMO (Context, Intervention, Mechanisms, Outcome) framework for developing a search strategy will be used for this systematic review (see Table 1). The search terms in the framework are set out in Table 2 below.

Table 1.

Context, Intervention, Mechanisms, Outcome (CIMO) framework for this study.

Context

Any health or social care service (e.g. hospital, nursing home, residential disability service) which is regulated as an organisation. This excludes services that are provided by individual professionals such as dentists or general practitioners.

Intervention(s)

Regulation is the intervention. The term ‘regulation’ may differ in a given context but it is generally taken to mean a process of external evaluation by an independent or statutory agency which is underpinned by enforcement powers.

Mechanism

Factors influencing, or determinants of, compliance; barriers and facilitators to compliance; nature of engagement between regulator and regulatee.

Outcome

Main outcome: regulatory compliance. Additional outcome(s): Other measures that are consistent with compliance will be included (e.g. in the USA, equivalent terms for non-compliance may be a ‘deficiency’ or ‘violation’).

Table 2.

Key search terms.

Context	“healthcare system*” OR “health care system*” OR “care system*” OR “social care” OR ”healthcare service*” OR ”health care service*” OR ”social care service*” OR “hospital*” OR “health care setting*” OR “healthcare setting*” OR “social care setting” OR “residential facilit*” OR “care facility*” OR “nursing home*” OR “residential care” OR “long-term care” OR “long term care” OR “disabilit*” OR “disability service” OR “care home” OR “aged care” OR “aged-care” OR “mental health service” OR “mental health centre” OR “mental health facilit*” OR “psychiatric service” OR “psychiatric centre” OR “psychiatric facilit*” OR “addiction service” OR “addiction centre” OR “addiction facilit*” OR “drug- treatment centre” OR “drug-treatment service” OR “drug-treatment facilit*” OR “drug treatment centre” OR “drug treatment service” OR “drug-treatment facilit*” OR “homecare” OR “home care” OR “domiciliary” OR “primary care” OR “community care” OR “respite care” OR “specialist care” OR “live-in care” OR “live in care” OR “homeless service*” OR “homeless shelter*”
Intervention	“regulation” OR “regulator*” OR “inspect*” OR “enforcement” OR “licens*” OR “certification” OR “withdrawal”
Mechanisms	“factor*” OR “barrier*” OR “facilitator*” OR “enabler*” OR “determinant*” OR “characteristic*” OR “indicator*” OR “association*” OR “relationship*” OR “cause*” OR “engagement” OR “attitude*” OR “predictor*”
Outcome	“compliance” OR “non-compliance” OR “violat*” OR “deficienc*” OR “sanction*” OR “citation*” OR “failure*” OR “failing*”

Information sources

Searches will be carried out on the following databases: PubMed, MEDLINE, PsycINFO, CINAHL and SocINDEX. In addition, the bibliographies of the included full-text articles will be hand searched for relevant articles. Forward citation searching will also be carried out to identify other potential material for inclusion. The search terms for one electronic database (PubMed) are provided in Table 2.

Searches will also be conducted on established grey literature databases including OpenGrey System for Information on Grey Literature in Europe and OpenSIGLE. Targeted searches will also be carried out on websites of regulatory organisations and Government agencies/departments involved in health and social care regulation, identified by referring to the Organisation for Economic Development and Cooperation’s (OECD) resources on regulatory policy internationally .

Software

The software used for screening articles is the online tool Covidence and the bibliography manager is EndNote X8.2 by PDF Tron™ Systems Inc.

Screening

All references returned by the search terms from each information source will be imported into Covidence. Duplicate references will be removed. Two researchers will independently screen the titles and abstracts of each of the retrieved articles against the inclusion/exclusion criteria using Covidence. Any disagreements on inclusion/exclusion will be resolved, in the first instance, by discussion. Any disagreements not resolved by discussion will be adjudicated on by a third author. Full-text review and bibliography searches will be performed by PD. The search strategy and study selection process will be reported in accordance with the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement . A PRISMA flow diagram will be generated.

Quality appraisal

As referenced above, a wide range of study designs will be considered for inclusion in the systematic review. Preliminary searches have found studies that are entirely quantitative or qualitative as well as mixed-methods. Quality appraisal tools will be selected dependent on the type of study. As such, the suite of appraisal tools made available by the Joanna Briggs Institute will be used . Grey literature will be appraised for quality with reference to Tyndall’s checklist which assesses the following aspects: authority, accuracy, coverage, objectivity, date and significance (AACODS) .

Two reviewers will independently assess the quality of articles selected for data extraction. Any disagreements on quality appraisal will be resolved by consensus or, if necessary, a third author will be consulted for final decision.

Data extraction

Two reviewers will independently carry out data extraction of all articles deemed eligible for inclusion, using a data extraction table (see Extended data ). The data to be extracted includes general information (title, author(s), publication date) as well as specific data under each of the CFIR domains. The data extraction template has been piloted and refined using studies retrieved during preliminary searches. The extracted data will be compared to ensure agreement and identify any discrepancies; disagreements will be resolved by discussion. Any disagreements not resolved by discussion will be referred to a third author for arbitration or, where appropriate, through contact with the study authors.

Data synthesis

Due to the wide range and heterogeneity of the studies that will be returned through the search strategy, a narrative synthesis will be performed. Narrative synthesis uses text and illustrations to describe, compare and combine heterogeneous qualitative findings and quantitative results . This approach places the focus on the interpretive synthesis of the narrative aspects of research findings, as opposed to any attempt to synthesise findings in a quantitative manner, such as in a meta-analysis.

The extracted data will first be tabulated in accordance with the CFIR and its respective domains and constructs. The data will then be summarised in narrative form, incorporating elements of Popay et al.’s methodology of narrative synthesis, and in line with the CFIR domains and constructs in addition. Overall confidence in the evidence will be appraised with reference to GRADE-CERQual .

Dissemination of information

The systematic review will be submitted to an academic journal on completion. Conference abstracts arising out of the systematic review will also be submitted to appropriate conferences for presentation. The findings of the systematic review will be circulated and presented to regulation staff in the Health Information and Quality Authority, Ireland. Review findings will be circulated to other regulators in Europe through the Supervision and regulation Innovation Network for Care (SINC).

Study status

Database searches using the search terms outlined in Table 2 have commenced.

Strengths and limitations

To the best of the author’s knowledge, this review will be the first to systematically assess the determinants of regulatory compliance. In addition, the methodological approach (including a wide range of study designs; synthesising the data using narrative synthesis with reference to the CFIR) allows for a comprehensive exploration of what factors are associated with the successful integration of regulatory requirements with an organisation’s goals. The use of GRADE-CERQual in appraising the quality of the overall body of evidence will aid knowledge users in establishing which determinants are appropriate for inclusion in any quality improvement initiatives.

In terms of limitations, it is possible that some relevant studies may not be retrieved due to the multiplicity of terms used in the literature to refer to regulation and compliance. This has been ameliorated with reference to resources from a range of countries to ensure that equivalent words and phrases are used in the search terms.

Conclusion

This protocol describes the methodological approach for searching, synthesising and quality appraisal of the available literature to answer the research question, what are the determinants of regulatory compliance in health and social care services? The findings of the systematic review will be of interest to organisations working in a regulatory capacity across diverse fields and may also inform quality improvement initiatives for service providers and policy makers in the health and social care sector.

Data availability

Underlying data

No data are associated with this article.

Extended data

Figshare: Supplementary File 1 - Data Extraction Tool.docx https://doi.org/10.6084/m9.figshare.13546664

This project contains the following extended data:

Supplementary file 1. This data extraction tool provides for the extraction of data for the systematic review. It includes general information about studies in addition to fields specific to the Consolidated Framework for Implementation Research (CFIR).

Reporting guidelines

Figshare: PRISMA-P checklist for ‘Determinants of regulatory compliance in health and social care services: a systematic review protocol’ https://doi.org/10.6084/m9.figshare.13553882.v2

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

I’ve reviewed the revised article. I appreciate the efforts that the authors put to address the reviewer’s suggestions and feedback. I have no further comments.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

NA

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Thank you for the opportunity to review this protocol. This protocol describes a systematic review that aims to provide an overview of determinants of compliance with quality regulation in health and social care services. The focus of the proposed study represents an understudied and important topic. As evidenced in the current version, a number of changes have been made to strengthen the protocol. I have some minor comments that may help to further improve the paper:

Introduction:

p.3, paragraph 2- I suggest the authors give some more details about the Donabedian's framework (e.g., what do the three domains- structure, process, outcome- refer to), instead of just providing an example for each.

p.3, paragraph 5- Need citation for “Regulations in health and social care are wide-ranging but …training.”

Theoretical framework:

p.4- The inclusion of CFIR framework and potential theories is appreciated as it helped to improve clarity. However, I believe the theoretical framework section can be developed more clearly and in more details. For example, I am wondering if the question listed in each of the five domains is comprehensive enough. Take outer setting domain as an example, you currently list “are there incentives/disincentives that encourage the regulatee to comply” as the question. But how about contextual level factors that could also affect regulation yet not necessarily providing incentives/disincentives? Please at least clarify whether these questions are exhaustive or are provided just as examples.

Table 2:

p.6- Consider add “predictor*” to “Mechanisms”.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

NA

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

We would like to thank Ms Wang for her valuable additional suggestions. Revisions have been made and are individually detailed below. We believe the revisions, informed by your comments, enhance the systematic review protocol greatly.

Italic text: reviewer’s comments.

Bold Text: response from the Authors

p.3, paragraph 2- I suggest the authors give some more details about the Donabedian's framework (e.g., what do the three domains- structure, process, outcome- refer to), instead of just providing an example for each.

We thank the reviewer for drawing attention to this oversight. The text has been amended to include a quotation and reference which we consider addresses the reviewer’s comment. See below:

Donabedian "defined structure as the environment in which healthcare is provided, process as the method by which healthcare is provided and outcome as the consequence of the healthcare provided".

Reference: Rademakers J, Delnoij D, de Boer D. Structure, process or outcome: which contributes most to patients' overall assessment of healthcare quality?. BMJ quality & safety. 2011 Apr 1;20(4):326-31.

p.3, paragraph 5- Need citation for “Regulations in health and social care are wide-ranging but …training.”

We thank the reviewer for drawing attention to the need to provide a reference for this statement. To address this, the sentence is now followed by citations to four documents which outline regulations in various jurisdictions and settings. See references below: p.4- The inclusion of CFIR framework and potential theories is appreciated as it helped to improve clarity. However, I believe the theoretical framework section can be developed more clearly and in more details. For example, I am wondering if the question listed in each of the five domains is comprehensive enough. Take outer setting domain as an example, you currently list “are there incentives/disincentives that encourage the regulatee to comply” as the question. But how about contextual level factors that could also affect regulation yet not necessarily providing incentives/disincentives? Please at least clarify whether these questions are exhaustive or are provided just as examples.

Government of Ireland. Care and Welfare of Residents in Designated Centres for Older People Regulations 2013. Dublin, The Stationery Office; 2013. Available from:

Welsh Government. Statutory Guidance - For service providers and responsible individuals on meeting service standard regulations. Wales, Welsh Government; 2019. Available from:

Government of New Zealand. Health and Disability Services (Safety) Act 2001. Wellington, New Zealand: Government of New Zealand; 2001.

Government of British Columbia. Community Care and Assisted Living Regulation. Victoria, British Columbia: Queen's Printer; 2004.

Thank you for the opportunity to clarify this important section of the protocol. The questions referred to by the reviewer are only intended for illustrative purposes. This is stated in the preceding sentence where it is noted that they are included by way of example. The preceding sentence is as follows: “ By way of example, the points below illustrate how regulation and compliance can, hypothetically, be mapped onto the constructs within the five CFIR domains:”.

As such, it is our intention to leave this section unchanged.

p.6- Consider add “predictor*” to “Mechanisms”.

This suggestion is welcome and we agree that this word should be added to the search terms in order to generate a more comprehensive list of results. Table 2 has now been revised to include this term.

I would like to thank the authors for their comprehensive and considered responses to my original comments. I have no further comments to make.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Regulation of health and social care services.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

I think this could be a really insightful review, although tricky to conceptualise and operationalise. The idea of using Normalisation Process Theory (NPT) and the Consolidated Framework for Implementation Research (CFIR) is good. I wish I had thought of it!

However, I am not totally clear from this protocol as to how NPT and CFIR fit together. I am not sure whether it is just a case of needing to make things a bit clearer for readers, or whether there is something more fundamental. I can see NPT as relating particularly to implementation processes, but it is not clear to me exactly how NPT facilitates understanding of other CFIR categories. Another example or two might be instructive? Ideally perhaps you might include a table to show how different elements of NPT and CFIR complement each other?

Related to this, in your Data Extraction and Data Synthesis sections I can't see any mention of NPT, only of CFIR. Shouldn't NPT also be part of your data extraction and analysis?

I also found your bullet points about how CFIR may be applied to regulation and compliance difficult to understand. I think there could be alternative ways of mapping regulation and compliance onto CFIR, so perhaps some further explanation would be helpful? For example, is the process of implementation more than this? E.g. sharing of / publicising standards? You have separated out the regulator as the only actor involved in implementation. But what about, for example, the regulated organisation incorporating those standards into its own QA or QI reporting or processes? You may have put this under Inner Context, but why?

One further thought is whether/where patients/service users (and perhaps other stakeholders) appear, in view of "tripartite" regulation? Perhaps they are part of the outer setting?

I also have a few small suggestions regarding the practical details of the protocol.

Excluding voluntary standards or codes of conduct may be a grey area, in that some "voluntary" accreditations may be mandated by ministries of health and hence might be viewed as regulation.

I wonder if there is something missing from the exclusion/inclusion criteria? By "regulatory compliance" you don't just mean compliance with standards subscribed to by regulators, but also the process of regulation (inspection frequencies, nature, sanctions etc.) - as per your "implementation process" category above. However, as I understand your inclusion criteria, they could apply to E.g. NICE guidelines that regulators also assess against. There will be lots of papers that assess implementation/impact of NICE guidelines without any reference to regulator use of those guidelines. If these are included then the scope of the review could be very large (and difficult to determine - you would need to know which guidelines regulators refer to). So perhaps you need something more explicit in your criteria about active regulation being part of the context which included research studies consider? Reading on I see that perhaps you have this covered under the Intervention part of the CIMO framework for the study, but it doesn't come across in the description in the main text of the study protocol.

Table 2: Key search terms

You might perhaps consider adding:

Context: primary care; home / domiciliary care

Intervention: certification

Mechanisms: withdrawal (of licence)

Are you going to do any citation searching in addition to keyword searching? I think the Mechanism element of CIMO is tricky for this search, and you might miss some research as a result.

Finally, I have some small points about some of the wording in the Introduction section (although they might possibly indicate a need to be a bit clearer conceptually?):

Paragraph 3. I did not find the process example particularly illuminating, and wonder if you can find something more obviously related to process. I appreciate that the boundaries between structure and process can be fuzzy, though. The structure and process examples relate these to outcomes. The outcomes example relates outcomes to countries. It might perhaps be better to put this example first, and then to put the others which relate outcome variation to structure and process?

Paragraph 4. "typically fall under" is a bit ambiguous. The structure, process, outcome framework could be applied to any set of standards; it is an empirical question whether governments organise their standards in this way (explicitly or implicitly). This would be strengthened by a supporting reference. If this isn't available, perhaps add an example or two to help back this up?

Paragraph 4. "must then comply with regulations...." Strictly speaking, be assessed as complying with regulations, and avoid potential sanctions or other potential negative consequences? Think, for example of reputation (as you highlight later) and "sunshine regulation".

Column 2, Paragraph 1: CQC as an example of compliant/non-compliant continuum works better with regard to the continuum aspect than the compliant/non-compliant aspect, although CQC assessment does in some ways equate Good with compliant and Requires Improvement with non-compliant. There is also the point that CQC assessments are based on professional judgement and not purely on legal standards, which may complicate things?

Column 2, Paragraph 2: "requires a little more interrogation and evidence gathering". And also a value judgement, although measures of competence etc. might possibly be set out?

Column 2, Paragraph 2: "requires a lot of evidence gathering as well as observation". I don't see how this example is different to admitting in a respectful manner?

Also a point perhaps that observation is one way of generating evidence? By evidence gathering perhaps you mean documentary evidence? Could be clearer?

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Regulation of health and social care services.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

We would like to thank Mr Boyd for his valuable and constructive comments. Revisions have been made and are individually detailed below. We believe the revisions, informed by your comments, enhance the systematic review protocol greatly. The principal amendment is a reconfiguration of the section outlining the theoretical framework for the systematic review.  Any minor revisions highlighted in-text and not explained were made in accordance with or as a result of the changes outlined in this document.

Italic text: reviewer’s comments.

Bold Text: response from the Authors

I. I am not totally clear from this protocol as to how NPT and CFIR fit together. I am not sure whether it is just a case of needing to make things a bit clearer for readers, or whether there is something more fundamental. I can see NPT as relating particularly to implementation processes, but it is not clear to me exactly how NPT facilitates understanding of other CFIR categories. Another example or two might be instructive? Ideally perhaps you might include a table to show how different elements of NPT and CFIR complement each other?

Related to this, in your Data Extraction and Data Synthesis sections I can't see any mention of NPT, only of CFIR. Shouldn't NPT also be part of your data extraction and analysis?

We thank the reviewer for drawing attention to this important element of the protocol. We agree that the manner of interaction between the CFIR and NPT could have been explained more clearly and this feedback has aided our thinking in this regard. In response, we have restructured the section in the main body of the text that sets out the theoretical framework. The use of the CFIR as a mapping device for the identification of enablers and barriers, has been introduced earlier in the section and the theory now comes later. In addition, the text makes clearer that NPT is one of several potential theoretical frameworks which can aid understanding and contextualising the enablers and barriers mapped in the CFIR. The other theories discussed in the main text around how regulators and regulatees behave and interact with one another are now included alongside NPT as several possible theories to help interpret the study’s findings.

In addition to these edits, two other references to the use of the CFIR in conjunction with NPT (in the abstract and in the strengths/limitations sections) have been edited to remove reference to the NPT and leave open the possibility of using several theories, including NPT to aid explanation of the findings.

As to the comment regarding the data extraction and data synthesis, we believe the above response helps explain why NPT is not included in any of the data extraction methodology. Moreover, the choice of theory to aid data synthesis is not yet determined. It may draw on one of the several theories outlined in the main body of the protocol or rely on other theories that are identified in the literature. The choice of theories will be dependent on the enablers and barriers identified during the review.

The new section outlining the theory is set out below:

“ Describing the reasons and potential explanations for the variability in compliance with regulations is the key focus for this systematic review. The authors have found no such review to date and this represents a gap in knowledge. Our review will use the Consolidated Framework for Implementation Research (CFIR) to categorise the determinants of compliance described in the literature.

The CFIR is an overarching typology used in implementation science

The CFIR may be applied to regulation by mapping any barriers, facilitators or factors associated with levels of compliance. In this sense, regulation is perceived as the intervention and compliance is the outcome. By way of example, the points below illustrate how regulation and compliance can, hypothetically, be mapped onto the constructs within the five CFIR domains: Beyond mapping the barriers, facilitators or factors associated with levels of compliance our review will use theory to aid interpretation. It is not possible at this juncture to be definitive in what theories will be applicable, as this will be informed by the nature and design of the studies included for the systematic review. As such, the following sections will describe some potential theories that may serve to elucidate the material that is mapped on to the CFIR domains and constructs.

intervention characteristics: do regulatees regard regulations as being evidence-based?

outer setting: are there incentives/disincentives that encourage the regulatee to comply?

inner setting: what resources (e.g. staffing) are available to the regulatee to achieve compliance?

characteristics of the individuals: what knowledge and beliefs do senior managers in regulatees have towards the regulator?

process of implementation: do inspectors act as external change agents to foster compliance?

Various theories have been posited in the context of regulation and on the means by which regulators seek to ensure compliance

Some studies have analysed the disposition of regulatees and their attitude towards compliance

The regulator can also be interpreted by organisational actors as being either an ally, threat or obstacle

As set out above, compliance is not only influenced by structural or organisational factors such as the size of a facility or who owns it. It is also an outcome of the nature of engagement between regulator and regulatee and contingent on their respective dispositions towards compliance. Normalisation process theory (NPT) represents a theory within which to understand these relationships and contingencies. NPT facilitates “systematic exploration of why some processes lead to a practice becoming successfully (or not) embedded (i.e. normalised) and sustained, by attempting to understand the intervention in relation to the work that people do”

II. I also found your bullet points about how CFIR may be applied to regulation and compliance difficult to understand. I think there could be alternative ways of mapping regulation and compliance onto CFIR, so perhaps some further explanation would be helpful? For example, is the process of implementation more than this? E.g. sharing of / publicising standards? You have separated out the regulator as the only actor involved in implementation. But what about, for example, the regulated organisation incorporating those standards into its own QA or QI reporting or processes? You may have put this under Inner Context, but why?

We thank the reviewer for identifying the improvement opportunity in which the bullet points seek to show how CFIR might be applied to regulation and compliance. We have amended the text to use new examples and make specific reference to the terminology used within each of the CFIR domains. The revised introductory text and bullet points are as follows:

“The CFIR may be applied to regulation by mapping any barriers, facilitators or factors associated with levels of compliance. In this sense, regulation is perceived as the intervention and compliance is the outcome. By way of example, the points below illustrate how regulation and compliance can, hypothetically, be mapped onto the constructs within the five CFIR domains: Furthermore, we agree with the reviewer’s point on the regulator being the only actor involved in implementation. We have, therefore, included reference to the regulatee and to service users in the examples above.

intervention characteristics: do regulatees regard regulations as being evidence-based?; what cost is incurred by the regulatee in complying?

outer setting: what are the demographics of those whose needs are served by the regulatee?; are there incentives/disincentives that encourage the regulatee to comply?

inner setting: what resources (e.g. staffing) are available to the regulatee to achieve compliance?; is the culture within the organisation positive towards the regulator?

characteristics of the individuals: what personal attributes do individual inspectors exhibit when confronted with non-compliance?; what knowledge and beliefs do senior managers in regulatees have towards the regulator?

process of implementation: are there specific staff appointed to champion regulatory compliance within regulatees?; do inspectors act as external change agents to foster compliance?”

III. One further thought is whether/where patients/service users (and perhaps other stakeholders) appear, in view of "tripartite" regulation? Perhaps they are part of the outer setting?

The reviewer raises a legitimate point on the inclusion of patients and service users and where they might feature in the context of the CFIR. We agree that there should be reference to service users and their potential to influence the level of regulatory compliance. The section of the protocol outlining how regulation and compliance can be mapped onto CFIR (see revised bullet points in II. above) now includes an example of how the demographics of the service users can impact on the ability to comply.

IV. Excluding voluntary standards or codes of conduct may be a grey area, in that some "voluntary" accreditations may be mandated by ministries of health and hence might be viewed as regulation.

The reviewer raises an important distinction regarding the grey area of accreditation, codes of conduct and voluntary standards. We agree that this requires further explanation in order to clarify the distinction between what is ‘regulated’ and what is ‘monitored’ or aspired to as best practice. As such, we have included a sentence in the exclusion criteria to explicitly state that voluntary standards/codes will excluded where they are not backed up by regulatory sanction. The revised text is set out below:

“Specifically, this refers to regulations that are mandated by government or other state authorities.”

V. I wonder if there is something missing from the exclusion/inclusion criteria? By "regulatory compliance" you don't just mean compliance with standards subscribed to by regulators, but also the process of regulation (inspection frequencies, nature, sanctions etc.) - as per your "implementation process" category above. However, as I understand your inclusion criteria, they could apply to E.g. NICE guidelines that regulators also assess against. There will be lots of papers that assess implementation/impact of NICE guidelines without any reference to regulator use of those guidelines. If these are included then the scope of the review could be very large (and difficult to determine - you would need to know which guidelines regulators refer to). So perhaps you need something more explicit in your criteria about active regulation being part of the context which included research studies consider? Reading on I see that perhaps you have this covered under the Intervention part of the CIMO framework for the study, but it doesn't come across in the description in the main text of the study protocol.

We thank the reviewer for identifying that regulatory compliance can encompass many considerations and guidelines that complement regulation. We agree that this should be addressed to clearly state what is meant by ‘regulatory compliance’. We have addressed this in the inclusion/exclusion criteria by being explicit about what ‘regulatory compliance’ means and also excluding any studies which focus on compliance with clinical guidelines that are not supported by regulatory enforcement. The revised text is underlined below:

“Studies will be excluded if they: VI.  Table 2: Key search terms You might perhaps consider adding:

Analyse regulatory compliance in a field other than in a health or social care setting or service.

Use an outcome measure that is not equivalent to regulatory compliance in accordance with the definitions set out above. For example: adherence to voluntary standards or codes of conduct

Context: primary care; home / domiciliary care

Intervention: certification

Mechanisms: withdrawal (of licence)

We thank the reviewer for the suggested additional search terms. On reflection, we agree that the terms require expanding to capture a wider range of health and social care services. In terms of the ‘Context’ field above, we have included the terms primary, home and domiciliary care as suggested. In addition, we have decided to include further terms related to the following services which would commonly be subject to regulation: mental health, addiction, homeless services, respite, community care and specialist care.

In addition, we agree that the word certification merits inclusion in the ‘Intervention’ field and have done so. Finally, under ‘Mechanism’, we disagree that the term ‘withdrawal’ should be included here. The terms included in this field relate to phenomena that may promote, inhibit or be in some way associated with compliance (for example, ‘barriers’, ‘indicators’, ‘cause’). The term ‘withdrawal’ is more applicable as a regulatory intervention. As such, we have decided to include it in the intervention field.

Context

“healthcare system*” OR “health care system*” OR “care system*” OR “social care” OR ”healthcare service*” OR ”health care service*” OR ”social care service*” OR “hospital*” OR “health care setting*” OR “healthcare setting*” OR “social care setting” OR “residential facilit*” OR “care facility*” OR “nursing home*” OR “residential care” OR “long-term care” OR “long term care” OR “disabilit*” OR “disability service” OR “care home” OR “aged care” OR “aged-care” OR “mental health service” OR “mental health centre” OR “mental health facilit*” OR “psychiatric service” OR “psychiatric centre” OR “psychiatric facilit*”  OR “addiction service” OR “addiction centre” OR “addiction facilit*” OR “drug-treatment centre” OR “drug-treatment service” OR “drug-treatment facilit*” OR “drug treatment centre” OR “drug treatment service” OR “drug-treatment facilit*” OR “homecare” OR “home care” OR “domiciliary” OR “primary care” OR “community care” OR “respite care” OR “specialist care” OR “live-in care” OR “live in care” OR “homeless service*” OR “homeless shelter*”

Intervention

“regulation” OR “regulator*” OR “inspect*” OR “enforcement” OR “licens*” OR “certification” OR “withdrawal”

Mechanisms

“factor*” OR “barrier*” OR “facilitator*” OR “enabler*” OR “determinant*” OR “characteristic*” OR “indicator*” OR “association*” OR “relationship*” OR “cause*” OR “engagement” OR “attitude*”

Outcome

“compliance” OR “non-compliance” OR “violat*” OR “deficienc*” OR “sanction*” OR “citation*” OR “failure*” OR “failing*”

The reviewer’s suggestions here have been valuable and should result in a more comprehensive search strategy from studies on a wide range of settings. The updated search terms are set out below:

VII. Are you going to do any citation searching in addition to keyword searching? I think the Mechanism element of CIMO is tricky for this search, and you might miss some research as a result.

The protocol makes reference to hand-searching the reference list of included studies in order to identify any further studies for inclusion. However, we agree that the protocol should go further in terms of citation searching as we had intended to complete forward citation, however we did not make it explicit in the previous version. On foot of this comment from the reviewer, we have amended the protocol to now also specify that forward citation searching will be carried out, see addition below.

“Forward citation searching will also be carried out to identify other potential material for inclusion.”

VIII. Paragraph 3. I did not find the process example particularly illuminating, and wonder if you can find something more obviously related to process. I appreciate that the boundaries between structure and process can be fuzzy, though. The structure and process examples relate these to outcomes. The outcomes example relates outcomes to countries. It might perhaps be better to put this example first, and then to put the others which relate outcome variation to structure and process?

The reviewer is justified in querying the applicability of the example used above. We agree and have therefore removed this example and replaced the text with an example that is more closely related to how a process can influence an outcome. See revised example below:

“It may be process-related as found in a prospective evaluation of simulated emergency department triage which found a high degree of variability in the processes of triage and measurement of vital signs.”

While acknowledging the suggestion that the outcome example could be listed first, we disagree that it should be restructured. We feel it is more appropriate to retain the logical sequence of structure/process/outcome and have, therefore, retained this order in the paragraph.

IX. Paragraph 4. "typically fall under" is a bit ambiguous. The structure, process, outcome framework could be applied to any set of standards; it is an empirical question whether governments organise their standards in this way (explicitly or implicitly). This would be strengthened by a supporting reference. If this isn't available, perhaps add an example or two to help back this up?

The reviewer is correct in identifying this ambiguity. We agree that is needs revision. We have rephrased this section to remove the ambiguous language. We would also like to draw attention to the examples of regulations framed as structure/process/outcome included in paragraph 5.

X. Paragraph 4. "must then comply with regulations...." Strictly speaking, be assessed as complying with regulations, and avoid potential sanctions or other potential negative consequences? Think, for example of reputation (as you highlight later) and "sunshine regulation".

We thank the reviewer for clarifying this nuance in terms of what it means to be ‘compliant’ and have changed the terminology used in this section to better reflect the reality, see revised text below:

“Regulation is one response to variability in quality: authorities establish a set of norms or standards to benchmark quality and then assess the extent to which organisations and individuals meet these standards. Failure to comply with regulations may lead to sanctions such as intensified surveillance, or even revocation of license to operate (typically through a registration or licensing system).”

XI. Column 2, Paragraph 1: CQC as an example of compliant/non-compliant continuum works better with regard to the continuum aspect than the compliant/non-compliant aspect, although CQC assessment does in some ways equate Good with compliant and Requires Improvement with non-compliant. There is also the point that CQC assessments are based on professional judgement and not purely on legal standards, which may complicate things?

We thank the reviewer for drawing attention to the manner in which the CQC assesses and rates compliance. While the CQC do not explicitly use the term ‘compliant’ in their rating system, we are of the view that the continuum concept still holds up. As the reviewer points out, the CQC rating system can be mapped quite readily on to more traditional judgments of compliant and not compliant. Therefore, we have retained the example of CQC in this section.

We agree with the reviewer’s second point here. We have addressed the point on professional judgment and have included further text to clarify this, see below with new text underlined:

“These ratings are determined

XII. Column 2, Paragraph 2: "requires a little more interrogation and evidence gathering". And also a value judgement, although measures of competence etc. might possibly be set out?

The reviewer correctly identifies this passage as requiring improvement and agree that it should be amended. We have edited the text to use different examples and be more explicit about what forms of evidence are required by inspectors to make judgments. The revised paragraph is set out below:

“Determining the level of compliance with a specific regulation requires varying degrees of effort and evidence-gathering on behalf of an inspector. Evidence can be generated from speaking with residents and staff, reviewing documentation and records, and observing practices as they happen. For example, assessing compliance with a structural requirement, such as a requirement that a person in charge should have “a minimum of 3 years experience in a management or supervisory role in the area of health or social care” , is a relatively straight-forward task of identifying the appropriate documentation. Judging whether a service has admitted residents in “a competent, equitable, timely, and respectful manner” , requires the inspector to undertake several tasks: speak with recently-admitted residents or their representatives, review admission records, speak with staff involved in admitting new residents. Assessing compliance in a regulation which specifies an outcome, such as one that seeks to “ensure respect for the personal privacy of each person in care” , requires the inspector to speak with residents and staff, review documentation and observe care practices on-site.”

XIII. Column 2, Paragraph 2: "requires a lot of evidence gathering as well as observation". I don't see how this example is different to admitting in a respectful manner?

See response to previous comment XII above.

XIV. Also a point perhaps that observation is one way of generating evidence? By evidence gathering perhaps you mean documentary evidence? Could be clearer?

See response to previous comment XII above.