Revalidation and quality assurance: the application of the MUSIQ framework in independent verification visits to healthcare organisations.

OBJECTIVES: We present a national evaluation of the impact of independent verification visits (IVVs) performed by National Health Service (NHS) England as part of quality assuring medical revalidation. Organisational visits are central to NHS quality assurance. They are costly, yet little empirical research evidence exists concerning their impact, and what does exist is conflicting.
SETTING: The focus was on healthcare providers in the NHS (in secondary care) and private sector across England, who were designated bodies (DBs). DBs are healthcare organisations that have a statutory responsibility, via the lead clinician, the responsible officer (RO), to implement medical revalidation. PARTICIPANTS: All ROs who had undergone an IVV in England in 2014 and 2015 were invited to participate. 46 ROs were interviewed. Ethnographic data were gathered at 18 observations of the IVVs and 20 IVV post visit reports underwent documentary analysis. PRIMARY AND SECONDARY OUTCOME MEASURES: Primary outcomes were the findings pertaining to the effectiveness of the IVV system in supporting the revalidation processes at the DBs. Secondary outcomes were methodological, relating to the Model for Understanding Success in Quality (MUSIQ) and how its application to the IVV reveals the relevance of contextual factors described in the model.
RESULTS: The impact of the IVVs varied by DB according to three major themes: the personal context of the RO; the organisational context of the DB; and the visit and its impact. ROs were largely satisfied with visits which raised the status of appraisal within their organisations. Inadequate or untimely feedback was associated with dissatisfaction.
CONCLUSIONS: Influencing teams whose prime responsibility is establishing processes and evaluating progress was crucial for internal quality improvement. Visits acted as a nudge, generating internal quality review, which was reinforced by visit teams with relevant expertise. Diverse team membership, knowledge transfer and timely feedback made visits more impactful. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

Our study presents original empirical data relating to an important aspect of quality improvement (QI) and clinical governance, namely the revalidation and appraisal system in England.

This study represents an original application of a pre-existing QI framework, the Model for Understanding Success in Quality (MUSIQ) framework, to a new QI context and activity; that of independent verification visits (IVVs).

The study also contributed to understanding how the MUSIQ framework operates. Our data suggest that the interaction between the microsystems is underemphasised in the original model, and our study thus indicates areas in which the model could be refined and improved.

First, the study is limited by its scope. Visits in this context were targeted specifically towards the microsystems involved in appraisal and revalidation. In contrast, other forms of healthcare quality assurance such as that carried out by the Care Quality Commission (CQC) delivers whole system inspection.

Second, the essentially interpretive nature of qualitative data may limit any more general and quantifiable conclusions that can be drawn from this study. In mitigation, however, the fact that the team used the MUSIQ model to frame the data collection and used semistructured data collection tools gave coherence to the study. This was facilitated by the use of three qualitative methods for triangulation.

Introduction

Institutional inspection is writ large in quality assurance processes.1 2 Visiting is an accepted form of performance verification in many public and professional services,3–5 and quality assurance occurs against a cultural backdrop of increasing transparency, in which the public expects to know more about the conditions under which their public services operate.6

There are, however, concerns about the effectiveness of visiting. High profile cases in which visit teams failed to detect higher than average standardised mortality ratios led to criticism of inspection in the second Francis report,7 and at the extreme ‘appalling issues in healthcare were not detected by visiting teams’ (ref. 8, p. 106). Systematic clinical governance, healthcare inspection and regulation began in 2002,9 and significant developments have since taken place, such as publicly available metrics which can improve performance;8 however, there is still concern that the significant resources put into inspection may not reap a proportionate impact.10

Despite the imperative to visit, the regulatory burden associated with visiting, and the concerns about the effectiveness of visits, little empirical work exists investigating the utility and impact of visits in healthcare. Moreover, firm conclusions about their impact cannot be drawn, due to the paucity of relevant robust data.7 11 12 Questions remain as to whether visits should be risk-based or routine-based; the appropriate frequency of visits; the correct balance of depth and breadth, self-assessment and external review; and what constitutes consistency in judgements.8 The fluidity of National Health Service (NHS) organisations and different approaches to organisational visits have also made it difficult to understand the impact of inspection regulation.13

Studies from quality assurance visits in education—for example, Office for Standards in Education, Children's Services and Skills (Ofsted) reports of evaluations of children's educational services—have shown a mixed picture. Ofsted teams have been criticised for poor-quality visit teams, making spurious judgements and being overly limited by a quality framework regarded as encouraging teachers to perform for the test and even manipulate data rather than to innovate and raise standards locally.5 Visits have been reported to result in unwanted organisational behaviour and outcomes included gaming, strategic de-compliance and stress.14 15 There is also concern that organisations resistant to improving do not respond positively to the provision of external support,16 thus visits may do little or worsen poorly performing organisations, with sanctions probably only causing temporary solutions14 and impacting negatively on staff morale.17

More positively, research—again mainly from education rather than healthcare—shows that although visits can have a variable impact on performance, they do improve self-assessment4 and organisational visits probably drive change indirectly.18 19 This may be through the phenomenon of the ‘nudge’, where change is initiated on receipt of a visit notification. What has been shown to deliver change in these cases is feedback, local leadership at the organisational level, and the capacity of the organisation to improve and be guided by clear outcomes and standards.7 14 Research also showed that expertise is typically varied across visiting teams and the organisations being visited,20 and therefore a ‘one size fits all’ approach may not work; rather, a tailored approach based on a needs assessment,20 21 and a targeting of the correct part of the system22 is more effective.

In summary, there is uncertainty regarding the right methods for performing organisational visits in healthcare and about the impact of visits and their ability to achieve positive change. This study aimed to explore in depth the impacts and outcomes of visiting on a variety of healthcare organisations, focussing on the independent verification visits (IVVs) undertaken in England as part of NHS England's quality assurance of medical revalidation.3 23 24

Research questions were:

Which contextual factors were important in the success of IVVs?

Which mechanisms of action effected change?

What was the impact of the IVVs on designated bodies (DBs)?

Methods

Context

Medical revalidation was introduced in the UK in 2012 and aims to improve patient care by ensuring that doctors are fit to practice. It has required a step change in appraisal and clinical governance.25 Responsibility for implementing revalidation lies with the responsible officer (RO)—a senior doctor—at a DB. DBs are organisations where doctors are employed and include hospital trusts, general practices, locum agencies, private companies and charities. Quality assurance of revalidation is the remit of NHS England, due to the prescribed connection of ROs to higher level ROs who are the regional medical directors of NHS England, for which regional revalidation teams act as independent verifiers ensuring that DBs' self-assessment returns, the annual organisation audit (AOA), and other organisational data are sufficient and valid. Independent verification has subsequently been introduced to support and ensure that DBs are performing their statutory duties around appraisal and governance. IVVs are part of this process. The value of visiting derives from the claim that ‘experience has shown that inaccuracies can be introduced when self-reporting mechanisms are the sole source of information’ (ref. 23, p. 11) as well as the opportunity to share best practice.

Methodology

We conceptualise revalidation as a quality improvement (QI) intervention and applied the Model for Understanding Success in Quality (MUSIQ) framework as our analytic frame, subsequently modified in light of our findings to more accurately represent the data (see figure 1). MUSIQ allowed for the complexity of a system such as revalidation and the IVV initiative, and accounts for high degrees of contextual variation.26 MUSIQ, derived from empirical and expert knowledge, outlines 25 contextual factors necessary for successful QI implementation and enables researchers to describe and evaluate these influences (ref. 26, p. 14). MUSIQ identifies three domains to evaluate: (1) the details of the setting; (2) the environment (the culture, the readiness and capacity for change) and (3) leadership (including effective teamwork and systems at the strategic and operational levels of the organisation). MUSIQ describes direct and indirect influences on QI implementation. Direct factors are the QI team and microsystems which have the biggest impact. A microsystem is a discrete operational unit that is tasked with a specific activity within a larger organisation. In the present context, this refers to the RO and their immediate team. Indirect factors include macro-level organisational factors and the external environment. In the context of our evaluation, the external environment included the drivers for medical revalidation (medical scandals, patient safety and the general medical council (GMC)).

Figure 1

MUSIQ: Model for Understanding Success in Quality (adapted from Kaplan et al. 2012). QI, quality improvement.

Data gathering

We used a mixed method qualitative approach, comprising semistructured interviews, rapid ethnography and documentary analysis of the visit reports, in order to gain detailed and holistic understanding of the visit. This allowed triangulation of three data sources enabling a comprehensive review of the visits.

A semistructured interview schedule was devised and informed by the MUSIQ framework. We ensured that the schedule was developed to reflect the contextual factors: setting, environment and leadership as well as the specific lines of enquiry about appraisal and revalidation. Telephone interviews were recorded and professionally transcribed. A rapid ethnographic approach27 informed the IVV observations and were carried out to build a deeper understanding of social interactions and technical events. Rather than the traditional long-term ethnography associated with, for example, anthropology, rapid ethnography is useful for a more pragmatic ‘describe and feedback’ in time-bound studies involving multiple sites as in this case.28 Likewise we developed a semistructure observation schedule for the ethnography, documenting; the context and setting; key stakeholders present; structure of the day; the areas of appraisal and revalidation covered during the visit and interactions between visitors and the DB to examine environment and leadership.

Documentary analysis of the visit reports was carried out, against which interview and observation data were triangulated. Content analysis was applied the reports.

All ROs who had undergone an IVV in England in 2014 and 2015 were invited to take part. Table 1 gives a breakdown of those contacted and the response rates. Based on table 1 and GMC figures regarding the number of ROs in England at the time,29 ∼7% of all ROs took part in the study. It is difficult to say precisely how accurate this percentage is at present, as the number of ROs may have changed since the study was carried out.

Table 1

Number of RO interviews and response rates across the four regions

Region	ROs contacted from 2014 visits	ROs contacted from 2015 visits	Total requests for interview	No. interviewed	No. from observed visits	% response rate
London	7	4	11	9 (including one appraisal lead)	3	82%
South	21	13	34	21 (including five visiting RO's)	5	62%
East and the Midlands	10	7	17	8 (including one appraisal lead)	3	47%*
North	7	6	13	11	5	85%
Total	47	30	75	49	16	69%

*One RO was responsible for multiple DBs and the Midlands and East performed their own evaluation for part of this project. RO, responsible officer.

Data analysis

Thematic content data analysis30 was applied to all three data sets. The approach was inductive and deductive, allowing the research team to explore the relevance of the contextual factors in MUSIQ while also allowing for emerging themes. A coding framework was developed by consensus (AG, TW, KW) and applied to all the data. A team of researchers (AG, IG, KW, RV, AR), coded the interview data using NVivo QSR 10©.31 A similar approach was taken to the ethnographic data and visit reports by two researchers (AG, TW).

Ethics

The project was deemed a service evaluation by REDACTED. We undertook the project according to ethical research principles described by the British Educational Research Association.32 All ROs were given the opportunity to opt out. All participants were fully informed and consented. All materials were anonymised and were held confidentially in compliance with the Data Protection Act 1998. The research team signed confidentiality agreements with NHS England and agreed to their standard terms of reference. The organisations visited consented to the research team being present.

Results

Characteristics of the data

Ethnographic data were gathered from 18 IVVs, amounting to ∼15 full days of observation material (see table 2). Fifty-two one-to-one telephone interviews ranged from 13 to 51 min duration, and included 16 follow-up interviews with 46 ROs and other members of the DB who were observed during the ethnography (see table 3). Twenty IVV reports were analysed (see table 4).

Table 2

Characteristics of the designated bodies (DBs) observed

Region	NHS trust	NHS foundation trust	Independent	Locum agency	Total
North	1	3	0	1	5
Midlands and East	1	1	1	2	5
London	0	1	1	1	3
South	1	3	1	0	5
Total	3	8	3	4	18

NHS, National Health Service.

Table 3

Number of RO interviews in each region. Interviews resulted from requests based on the 2014 and 2015 visits, plus requests made during observations of visits as part of the current study

Region	No. in response to requests based on 2014 and 2015 visits (response rate)	No. from observed visits	Total
North	11 (85%)	5	16
Midlands and East	8 including one appraisal lead (47%*)	3	11
London	9 including one appraisal lead (82%)	3	12
South	21 including five visiting RO's (62%)	5	26
Total	49 (69%)	16	65

*One RO was responsible for multiple designated bodies (DBs) and the Midlands and East performed their own evaluation for part of this project.

RO, responsible officer.

Table 4

Reports analysed according to region and organisational type

Region	NHS trust	NHS foundation trust	Independent	Locum	Total
North	1	2	2	1	6
Midlands and East	3	R*	R*	1	4
London	0	2	2	1	5
South	2	2	1	0	5
Total	6	6	5	3	20

*Redacted report does not contain this information.

NHS, National Health Service.

Themes

Figure 2 presents the common coding framework used for all three data sets. Three main themes describe the personal context, the organisational context and the impact of the visit, from notification through to final outcomes. Participant quotes are coded by location (North—N; Midlands and East—ME; London—L; South—S) and a number, for example, RO number 1 from the North is ‘NRO1’.

Figure 2

Coding framework for the data. DB, designated bodies.

The personal context

This theme describes how participants' personal context affected their perception of the visit. It encompasses two subthemes: medical leadership and motivation for the role.

Medical leadership

ROs and lead appraisers were experienced doctors with a track record in medical leadership, and their role in revalidation was a natural progression from another senior leadership role. The demands of the role were noted, with one explaining that ‘…it became apparent to all the medics…that this was a significant task’ (NRO1).

Motivation

Given that many ROs had been involved in piloting revalidation and many had educational leadership backgrounds, it is perhaps unsurprising that there was a general consensus about the positive benefits of appraisal and revalidation.

Revalidation is the most important development that has happened in medical career progression and medical management.

(MERO6)

Appraisal was felt to be particularly important, with most agreeing it is not just a ‘tick box’ exercise but encourages meaningful self-reflection and opportunities for ongoing development. Positive perceptions of revalidation included its ability to bring objectivity and rigour to the appraisal system; its positive impact on the maintenance of accurate records of practice; and that it formalises the assessment process and identifies any concerns or deficiencies in a clinician's practice, thus enabling them to learn from mistakes. As well as being a tool for self-reflection, revalidation was perceived by some—but not all—as being a tool to uncover those who were not performing. In particular the ROs affiliated to locum agencies or private organisations claimed that revalidation has left fewer ‘floating lone guns on the periphery’ (NRO1).

While many ROs felt revalidation was important, scepticism regarding the adequacy of the system caused a lack of willingness to engage with the process at times. Some ROs felt better systems for recording performance were needed, and that the minimum acceptable standard is too low. Similarly, some ROs thought either that it will take too long before there is real evidence of impact from revalidation, or that the process is unlikely to stop outlying ‘bad’ doctors:

Is it going to stop Dr Shipman? No…because all you need to do is be able to manipulate the system.

(MERO5)

For some practising in the private sector, there was concern about whether revalidation had much added benefit to the performance visits from other, for example, CQC, commissioners, ministries. There was a sense of fatigue from the ‘endless visit industry’.

The more I am involved in it, the more I've been exposed to it, I perceive it to be of diminishing value and increasing time and effort.

(SRO7)

Nevertheless, most participants viewed their RO responsibilities simply as part of their leadership role. They gained personal satisfaction from knowing they were running a functional organisation, or from the work itself. ROs in non-NHS organisations felt that the revalidation process served to reward the organisation as it helped to integrate NHS and non-NHS organisations. There were demotivating factors, however. For example, the considerable burden of the role, such that for one RO, it was detrimental rather than rewarding:

I've had complaints made against me to the GMC by consultants when I had to deal with issues relating to revalidation…there are significant risks to your personal career.

(SRO19)

The organisational context

The second theme describes the variability in the organisations, teams and resources within DBs.

The characteristics of the designated body and the revalidation team

DBs varied considerably in terms of their location (and spread, in the case of large DBs), population size, number of doctors, financial turnover and range of services (see tables 1 and 2). DB revalidation teams varied, reflecting the variability of DBs. A team could consist of an RO plus any of the following; medical director, lead appraiser, HR representative, IT support, education representatives, project manager, revalidation manager and administrative support. Some teams had just an RO with part-time administrative support. Several teams were fluid, with staff changes occurring and some members working part-time on this process among other duties in the organisation.

Resources and data infrastructure

The main focus of criticism was that the appraisal and revalidation process was administratively burdensome, trawling huge numbers of doctors to catch a few. Administrative support appeared crucial and the IVV sometimes provided a useful intervention which led to increased support. Most respondents were satisfied with the IT resources that they used, either using existing software (smaller DBs) or buying in new software (larger DBs). Several stated that their boards were generally supportive in approving the use of further resources for this process when needed:

So, say for example when the whole revalidation kicked off we-I brought a paper then and identified that this would be-require additional resources um, there's a collective groan because they'd be all short of money but the resources were provided and I've never had any difficulty with that subsequently.

(NRO1)

ROs who considered themselves well resourced had a good revalidation team in place, good access to IT resources and support from boards or senior management. Where ROs were poorly supported, this was usually due to lack of time and administrative support. A common word used to describe resourcing was ‘adequate’, with many respondents feeling that although they can do the work with the resources available, they could do a better job with more. Resources and supportive senior management were key to RO perceptions that they and their teams could effect change and several ROs stated that their boards were generally supportive in approving the use of further resources when needed. However, change was a slow process for some teams, limited only to what was in the teams' control and not extending to wider organisational concerns.

The trigger and its impact

This theme looks at the mechanisms of action involved in the IVV, from the notification of the visit, the processes involved at the visit, to any changes made as a result.

The nudge

The request to visit triggered a range of institutional responses, this is a concept known as the ‘nudge’. About a quarter of ROs discussed how notification of a visit prompted them to act. Actions included gathering data, setting up or refining systems or databases, checking compliance with policy, and reflecting on their strengths and weaknesses. Two ROs described how knowledge of an impending visit encouraged them to do the things that they were aware they should do but might not have done, or might have delayed. One RO explained how they used a forthcoming visit to persuade others to provide them with data they needed and to assess the competency of their staff.

…it also, you know, gave me the stick to get us to perform better in other areas …I think we had the opportunity to talk around why we thought our system was better than the standard one…things like that. And it is good to be told that you're actually doing things quite well, as well as being told, look, this will not do, you need to improve.

(NRO8)

Knowledge transfer

Regional revalidation teams, ROs and their DBs learnt about and from each other, and relationship building occurred through the IVVs. The regional revalidation leads took a supportive educational approach in the IVV process but were mindful of tensions between governance and education. The visits also enabled knowledge transfer that occurred between the ROs and their teams and regional revalidation teams, but where the visit engaged more widely, with appraisers, doctors and human resources, for example, knowledge transfer became more effective and impactful, raising the status of appraisal and revalidation in the organisation. The following excerpt from an observation schedule about the risks of employing locums, exemplifies how knowledge is shared between those present at a visit and how system and process changes follows:

[There] was initial reluctance for the Trust to consider taking on another administrative task, however, the risks of not doing so were highlighted. Trusts were empowered to check their locum agencies. [By the] end of meeting the RO had already decided this needed to happen and was making suggestions about how it might work.

(Observation schedule)

Visitors shared best practice from their knowledge of other DBs, such as how to set up systems, conduct appraisals, and the visits were observed to provide a narrative benchmarking process. That being said, many ROs felt they learnt little from their visit, but were reassured that they were doing things as they should. Visiting ROs (VROs) were another mechanism by which knowledge transfer could be enhanced. VROs described the opportunity to participate in IVVs as an educational activity, by being able to learn from the visit and contribute to supporting the visited DB.

I felt that will be a learning opportunity for me rather than verification visit to be honest. I wanted to know how other people are doing from inside…you learn and you take experiences from this visit.

(SRO10)

Additionally, the visit report was considered by most ROs as a useful source of knowledge that, in most instances, should be shared.

Regulatory gaze

The observational data revealed that appraisals tended to be the main focus of the IVV. The visit team's lines of enquiry focused on meeting satisfactory numbers being performed and on the quality of appraisal. The second most common area for discussion concerned governance, for example, regarding the transfer of information between DBs and the use of locum agencies. Some ROs commented on an undue emphasis being placed on certain areas over others, but agreed that the topics were all reasonable to cover. While all teams had senior representatives, some visit teams and DBs had a greater diversity of participants than others, including lay representation and a visiting RO. This was observed to influence the regulatory gaze by broadening key lines of enquiry. In some instances, issues discussed with key informants were subsequently raised again with other members of the DB to check if understanding was consistent between all parties involved.

Feedback and decision making

Many ROs found that verbal feedback during the visit was helpful, particularly when the visiting team made practical suggestions. One RO explained how concrete practical advice from the visit team could be used to effect change within their organisation.

…if you take to the Board something that says you were under resourced, well they got endless people coming to say they are under resourced and really it would be much more helpful if people said for an organisation, with this many doctors, we would recommend a team of such and such…

(NRO6)

Perceptions varied, however, in relation to the amount of immediate feedback. Where feedback was absent or unstructured, this was criticised by the RO. Variation in quantity and quality of feedback was also sometimes noted by the research team at observations.

[The] Trust was very keen for feedback and requested it on several occasions. Eventually the team gave very outline feedback saying that performance was generally good, there was much good practice, and there were no specific concerns.

(observation schedule)

Written reports were sometimes reported as problematic by RO interviews who wanted them straight after the visit in order to make changes quickly. ROs appreciated written feedback, many perceived it as fair, and felt that the regional visiting teams had gone out their way to make the reports balanced, and ‘well structured and supportive’.

Some reports included a numeric score which generated several comments. ROs disappointed with the marks challenged the numeric scoring which had sometimes resulted in other metrics being added or the rating being changed. Even those with good marks found the judgements hard to rationalise:

The scores don't bother me, they're all green…But I have to say, I've done the job for quite a long time, I know what my fellow ROs are doing…if we're getting a three, there's going to be a lot of people not getting a three.

(SRO11)

However, ROs were equally critical when they did not receive an overall rating or recommendations. One reported:

When you get an external audit report that gives you a summary conclusion… with a rating system and recommendations, but in reality this visit was neither one nor the other, it wasn't an audit.

(MERO1)

The documentary analysis concurred with ROs views. Reports were largely descriptive accounts of the IVV and suggested areas for development.

System and process change

The IVV provided insights about how to improve practices and process associated with appraisal and revalidation, and motivated revalidation teams to improve. Examples included: changes to appraisal systems or processes, for example, changing when appraisals would be scheduled; bringing in new databases; training appraisers; and bringing appraisals more in line with job planning.

Because you never know whether you've been selected because you're seen to have a good system, or because you're, you're seen to have, you know, a system that requires improvement…But we did receive some information that we were seen as a trust with reasonably good practice, so that was quite reassuring. So we calmed down then and felt that it was a very useful opportunity for, you know, for our system to be reviewed, for us to learn from it.

(LRO4)

Observations confirmed that IVVs directly resulted in changes to systems and processes. For example, HR approaches changed in relation to locum employment, improvements to the quality of appraisal summaries, and to training, as mentioned earlier.

Outcomes from the visit

There was variability regarding whether ROs reported visits as having an impact. About a third said that the visiting team had set priorities for them to change; a third said they had not, and the remaining third said that they had but that the priorities were in fact set by themselves. Changes included the appointment of extra administrative or educational staff following the visit and greater leverage and credibility when taking ‘issues to the board’ (SRO19) which had persuaded the board to release funds. Getting to know the revalidation teams and feeling able to contact them for future issues were also considered important outcomes of the visits.

For some in the private sector, there was concern about whether revalidation had added benefit to the performance visits from other bodies, for example, the CQC, commissioners or ministries.

Discussion

Summary of main findings

This study highlighted how organisational visiting may facilitate or undermine QI within healthcare settings. Visits can raise the status of appraisal and revalidation within DBs and support ROs. In turn, the visit teams gain insight in to the organisations by talking through the processes and outputs of appraisal and medical revalidation, since this helped them to understand the detail behind the quantitative data which monitors revalidation.

Learning and knowledge transfer proved to be the dominant activity for revalidation leads and ROs and the expertise of the visit teams was critical for their success. However, there was also resistance, either because of mistrust concerning the benefits of appraisal and revalidation, or because of the increasing administrative burden. This acted to reduce learning opportunities and de-legitimised expert knowledge offered by the visit team.

The extent and quality of feedback varied. Feedback at the time of the visit or in the form of the report was something most organisations were eager to receive. The observations and interviews indicated that its provision was not uniform across the DBs. However, feedback while largely demanded was in certain circumstances rejected. The qualitative nature of the data and professional judgements made by some of the visit team teams appeared to underpin the contestability of judgements. For example, it tended to be rejected if did not fit with the prevailing beliefs of the organisational system, or the relation between qualitative judgements and numerical scores were unclear.

Although the narrative nature of the report reflected that the visit took place and documented its content, it less frequently gave a clear indication of actions or timelines for improving performance. The reports reviewed as part of this evaluation avoided making absolute statements that explicitly ‘signed off’ or unequivocally verified the DB. Quantitative data has been shown to improve performance (Bevan 2011), and as such the new CQC model provides quantitative and qualitative data which can be used for rating against its clinical domains as well as a textual description supporting its judgements.

Original contributions

This study represents an original application of the MUSIQ framework to a QI activity; that of IVVs. The MUSIQ framework highlighted the importance of the microsystem at work, since all features (as illustrated in figure 1) were shown to be valid in this context. The MUSIQ framework provided a schema for analysing context. In this study the external environment, the organisation, the QI team and the microsystem all played influential roles in implementing appraisal and revalidation. Organisational leadership, senior management and trust boards were influential in providing the HR and infrastructure required for effective implementation of revalidation. The triggering event and its subsequent report on the DB's status were in turn used by the microsystem to engage support downstream. In this respect, leadership at the organisational and microsystem level drove important changes.

Our study also contributed to understanding how the MUSIQ framework operates. Our data, however, suggests that the interaction between the microsystems is underemphasised in the original model. Parts of the microsystem exert an influence on each other, and the QI outcomes are therefore driven by the initial trigger event feeding its way through the system as a whole: interactions occurring between parts of the microsystem also affected the QI outcomes. The linearity featured in the MUSIQ framework therefore does not account for this interactive element and recognise the often oscillating nature of progress. In our study knowledge transfer across the microsystem and between the trigger (the visitors) and the RO and their teams was the primary catalyst for QI. A further consideration when using the model in evaluating organisational visits is the models current of conceptualisation of the trigger. The trigger in this study was a team of visitors, in fact a further microsysytem, which was introduced to the context and the complexity of this additional system is at present outside of the MUSIQ framework.

Limitations

A first limitation of this study relates to its scope. Visits in this context were targeted specifically towards the microsystems involved in appraisal and revalidation. In contrast, other forms of healthcare quality assurance such as that carried out by the CQC delivers whole system inspection. It is beyond the remit of our study, therefore, to draw any conclusions about the adequacy of IVVs on a macro scale. The essentially interpretive nature of qualitative data, may limit the conclusions that can be drawn from this study. In mitigation, however, the fact that the team used the MUSIQ model to frame the data collection and used semistructured data collection tools gave coherence to the study. This was facilitated by the use of three qualitative methods for triangulation.

Conclusion

In returning to our research questions, we summarise the conclusions of this study. In relation to the first question about which contextual factors were important in the success of IVVs we can say that influencing the teams whose prime responsibility is to set up processes and evaluate progress—this is to say, the microsystems—through knowledge dissemination involving expert visit teams appeared crucial. Organisational resistance limits QI. Regarding the second research question about which mechanisms of action effected change, we conclude that visits acted as a ‘nudge’ for DBs to self-assess but external judgements made by the visit team were often contested. Finally, in relation to our third research question about the impact of IVVs on DBs, we conclude that while there was a high degree of satisfaction with the visits, tangible impacts and outcomes were less apparent.

Organisational visits can only be effective if frontline teams and senior leadership teams at DBs are fully engaged in delivering the highest possible standards of care. Several salient conclusions can be drawn from the analysis of the IVV context and the application of the MUSIQ framework to it. Dealing first with the IVV data, there are good grounds to conclude that the IVV system is, so far, a success for visitors and visitees. The newness and complexity of the system as a whole, however, has caused some procedural inconsistencies, resourcing, and technology streamlining issues which should be remedied to improve the system and ensure its integrity.

The MUSIQ framework was useful for interrogating the IVV system. Its hypotheses and assumptions, and its description of the micro/macro-environments mapped approximately with the data. Modification is needed, however, to derive a truly perspicuous representation of what was found. We suggest, moreover, that some of the changes which would improve its correlation with the IVV context would also enhance its explanatory power more generally, if applied to other data sets.