Quality assurance and quality improvement using supportive supervision in a large-scale STI intervention with sex workers, men who have sex with men/transgenders and injecting-drug users in India.

BACKGROUND: Documentation of the long-term impact of supportive supervision using a monitoring tool in STI intervention with sex workers, men who have sex with men and injection-drug users is limited. The authors report methods and results of continued quality monitoring in a large-scale STI services provided as a part of a broader HIV-prevention package in six Indian states under Avahan, the India AIDS Initiative.
METHODOLOGY: Guidelines and standards for STI services, and a supportive supervisory tool to monitor the quality were developed for providing technical support to STI component of large-scale HIV-prevention intervention through 372 project-supported STI clinics. The tool contained 80 questions to track the quality of STI services provided on a five-point scoring scale in five performance areas: coverage, quality of clinic and services, referral networks, community involvement and technical support.
RESULTS: The tool was applied to different STI clinics during supportive supervision visits conducted once in every 3 months to assess quality, give immediate feedback and develop a quality score. A total of 292 clinics managed by seven lead implementing partners in six Indian states were covered in 15 quarters over 45 months. Overall quality indicators for the five performance areas showed a three- to sevenfold improvement over the period.
CONCLUSION: It was possible to improve quality over the long-term in STI interventions for sex workers, men who have sex with men and injection-drug users using an interactive and comprehensive supportive supervision tool which gives on-the-spot feedback. However, such an effort is time-consuming and resource-intensive, and needs a structured approach.

Background

The control of STIs among high-risk groups such as sex workers (SWs), men who have sex with men and transgenders (MSMs/TGs), and injecting-drug users (IDUs) is an important strategy for containing HIV infection.1 It is a challenge to provide acceptable, accessible and high-quality sexual-health services to control and treat STIs among high-risk groups.2 Establishing standards against which quality can be measured (quality assurance) and developing approaches to solve the problems affecting the quality (quality improvement) are recognised as two important means to ensure high-quality services.3 4 Several approaches to evaluate and enhance the quality of STI services are reported using different methodologies such as mystery patients, record reviews, patient exit interviews, management tools and supervisory measures.5–10 The managerial approach of supportive supervision involving joint problem-solving, record review and observing clinical practice has consistently shown a moderate to marked effect in quality improvement.5 11 12 However, the reported experience of using these various methodologies to measure quality was a one-time assessment rather than pursuing measures to improve quality over a long period.11 13–16 In addition, most reports employ methodologies to assess a single aspect of service delivery, from a clinical, managerial or community perspective rather than combined assessments.17 In this paper we report on the experience and results of 4 years of work to monitor and improve STI service quality of high-risk groups using a supportive supervision tool that incorporated community, clinical and management perspectives.

Methods

Avahan, the India AIDS Initiative, has supported the HIV-intervention programme in six HIV high-prevalence states in India through various grants awarded beginning from December 2003 to August 2005. As of December 2007, the programme provided HIV-prevention services to about 280 000 SWs and MSMs through a pyramidal virtual organisational structure consisting of a central STI team, seven lead implementing partners who subgranted to over 140 local non-governmental organisations (NGOs).18 The services provided through project supported clinics, peer educators and out-reach workers included STI and primary HIV care, condom provision, behaviour change communication and community mobilisation to build capacity for community ownerships. The STI clinic services were scaled up, starting from December 2004. Details of early scale-up are available in a previous publication.19

The delivery of STI services worked at three levels.19 At the grass-root level, NGOs organised services to SWs, MSM/TGs and IDUs through designated clinic settings such as static clinics, mobile clinics in a vehicle and health camps. Some static clinics were located at private general practitioner clinics who were identified by the community as preferred service providers. At the middle level, the seven state lead implementing partners provided technical and management support, established a supervisory system and gave logistical assistance to facilitate overall STI service delivery. At the central level, an STI capacity building team was responsible for ensuring high quality and standardised STI services which were provided uniformly across the seven lead partners and their subgrantees. Clinical operational guidelines, standard operating procedures and corresponding supervisory handbooks were developed to support standardisation and supervision.20 21 The central team conducted trainings, provided support and mentoring to mid-level supervisors, monitored overall activities and readjusted guidelines based on the experiences, monitoring data and operation research. Overall, the capacity-building system was responsible to facilitate the delivery of uniform high-quality services for STIs management. Counselling and basic HIV management services were added later.

The supervisory system was two-tiered: a state and a central level. The state-level supervisory team consisted of staff from the lead implementing partner who conducted systematic, routine periodic visits to the designated clinics supported and managed by them. Every clinic was covered by these visits. The supervisory visit schedules could be adjusted to respond to the need for state supervisors based on the requirements of individual clinic. Frequent visits were given to clinics which needed more support for quality improvement such as newly started clinics and those with new staff. The central STI capacity-building team conducted ‘dipstick’ supervisory visits to different clinics supported by each lead implementing partner once in every 3 months. Dipstick supervision consisted a visit by central supervisors accompanied by state supervisors to preselected clinics. Around 10% of existing clinics under each lead implementing partner were selected in each quarter in consultation with state clinic supervisors. Different clinic clusters were visited by the central team in each quarter. Clinic staff were aware of supervisory visit plans beforehand at least a week earlier.

The central STI capacity-building team developed a clinic quality monitoring tool in a participatory manner as a component of regular supervision, for which guidelines were outlined in the supervisory handbook.21 This tool was used as part of regular supervision and assisted in monitoring the quality of clinical services against the prescribed standards presented in Clinic Operational Guidelines and Standards.19 20 The purpose of the tool was twofold: (1) to serve as a checklist to help clinic supervisors from the state lead implementing partners to support, monitor and improve the overall quality of service delivery during their periodic visits; and (2) to track quality of STI services over time.

This tool assessed five performance areas of STI clinical services, as shown in the table 1. The coverage performance area measured accessibility, acceptability and contact coverage, which were the three important domains related to provision of STI services in programme settings where exclusive services are made available to SWs, MSM/TGs and IDUs.22 23 The coverage component of the tool was devised to explore various structural and environmental barriers for service uptake such as geographical distance, mobility, work limitations and social stigma, so that actions could be taken to enhance the coverage.24 The quality of clinic and services performance area measured 10 clinic and service components (see table 1) such as correct treatment, counselling, infection control, confidentiality, drug stock and record keeping measured against the defined standards.20 The referral network measured the availability of adequate referral network and the use of STI clinics as an entry point for HIV testing and treatment services. The community involvement performance indicator measured the involvement of the community in the clinic service delivery, which is an explicit component of the programme to facilitate client-orientated services, community ownership and sustainability.25 The technical support performance indicator measured the adequacy of supportive supervision provided to the clinic staff by the lead implementing partner level technical and management team. The tool documents the observations in yes/no or numerical form (percentage and numbers) for around 80 observations including interviews, clinical observations, record reviews and data analysis. The yes/no questions were scored 0 or 1. Numerical answers were converted to an ordinal score between 0 and 5. Based on the number and type of questions under each enquiry area, a mean score between 0 and 5 for each of the five enquiry areas could be calculated. For example, coverage component had five numerical questions (table 1). Based on percentage of coverage, each question was given a score between 0 and 5. The mean score for these five questions was calculated to obtain the score for coverage. In a similar manner, the mean score between 0 and 5 for each of the five enquiry areas was calculated using the supervisory tool. Some of the subcomponents of the five main performance indicators in the tool have undergone refinement over the period based on field experiences. Such adjustments were mostly improvements in the definition of subcomponents and its measurement methodologies so that the observations could be documented quantitatively. The clinic quality monitoring tool is a participatory tool which guides the supervisors through a systematic supportive supervision session requiring about 2–3 h per clinic assessment with the active involvement of the service providers.

Table 1

Summary of the clinic quality-monitoring tool used to measure the quality against defined standards21

Question (Indicator)	Main components of composite scale	Methodology
1. Are the services accepted and accessed? (Coverage)	Clinic service uptake Clinic orientation to sex workers Clinic orientation to provide STI services Monthly check-ups—uptake Presumptive treatment for STIs—uptake	Quantitative analysis of clinic and programme data (calculated in percentage)
2. Whether adequate clinical care, effective drugs and preventive support were given? (Quality of clinic and services)	Clinic staffing and training (five subcomponents) Clinic set-up (five subcomponents) Performance at clinic encounter (five subcomponents) Performance on counselling (five subcomponents) Performance on correct treatment (review of 10 randomly selected records from previous 3 months) Infection control and waste management (five subcomponents) Drug and condom supply (five subcomponents) Ethical standards and confidentiality (five subcomponents) Completeness of patient records (review of 10 randomly selected records from previous 3 months) Clinic documentation and reporting (five subcomponents)	Provider interview, provider observation and review of records selected randomly
3. Is the clinic an entry point for prevention to care continuum? (Referral network)	Availability of referral directory Documentation of referrals Formal linkages with HIV testing Formal linkage with other referrals Periodic meeting with referral sites	Providers interview and record review
4. Are MSMs and SWs taking active role in managing and delivering the services? (Community involvement)	Drop-in centre utilisation Involvement of community members in clinic activities Involvement of community in STI follow-up in the field Periodic meeting between clinic and outreach staff Recommendations of meetings are acted upon	Staff interview and clinic observations
5. Are we providing adequate supportive supervision to the clinics? (Technical support)	Availability of designated supervisor Frequency of clinic visit by supervisor Overall clinic load per supervisor Use of supervisory tool Joint action plan implementation	Review of scale-up information, field observations, interviews and consensus score

During the supervisory visits, the central team documented the observations on the clinic quality-monitoring tool with active participation of the clinic staff. Two outcomes were generated from the tool. First, the central STI capacity building team generated a technical report and recommendations based on the observations leading to immediate follow-up actions by the clinic to improve the service quality. The state supervisory team followed up the corrective actions during their subsequent visits. The second, clinic quality-monitoring tool and technical reports were periodically reviewed by central STI capacity building team to generate an objective report using an ordinal score of 0–5 for each of the above five enquiry areas at the clinic level. For each of seven state lead implementing partners, quarterly clinic quality scores were generated under five performance indicators. The scores were calculated by a central STI capacity building team by averaging the scores for all clinics visited under the respective partner in that quarter. Final quarterly quality scores for five performance indicators were obtained by averaging the score for all the seven state lead partners. Whenever a visit to a state lead implementing partner was not conducted in a particular quarter, the clinic quality score from the previous quarter was carried forward, assuming no change in the quality in that quarter. The statistical significance of the quality score trend over the time was assessed based on the correlation coefficient using SPSS (SPSS, Chicago).

Results

The total number of static project owned clinics managed by the seven state lead partners under Avahan was around 372 by October 2008 compared with 106 in February 2005 (table 2). During the past 45 months, the central support team had made 15 rounds of visits to state lead partners to cover 292 clinics and provided onsite support to clinic staff and state supervisory team. The scoring exercise was carried out by the central team for all quarters except the 2nd, 3rd, 5th and 14th quarters. On 14 occasions, the visit to one or more state lead partners was not made in a particular round, and the clinic quality score from previous quarter was carried forward, assuming no change in the quality in that quarter. Thus, the analysis presented here includes 292 clinic visits conducted in 11 rounds in 4 years.

Table 2

Score for five quality indicators measured on a scale of 0–5 using a clinic quality monitoring tool from 292 clinics between April 2005 and October 2008

	April 2005–June 2005	July 2005–September 2005	April 2006–June 2006	October 2006–December 2006	January 2007–March 2007	April 2007–June 2007	July 2007–September 2007	October 2007–December 2007	January 2008–March 2008	April 2008–June 2008	October 2008–December 2008
Total no of clinics	106	190	304	300	394	395	342	333	361	372	372
Clinics visited	20	31	28	33	45	24	32	23	17	25	14
1. Coverage	0.48	2.10	2.67	2.97	3.30	3.24	3.39	3.35	3.46	3.48	3.24
2. Quality of clinic and services	1.57	2.88	3.92	3.80	3.77	3.69	4.15	4.42	4.44	4.49	4.27
3. Referral network	1.00	2.71	3.14	3.00	3.21	3.07	3.03	3.13	3.13	3.51	3.80
4. Community involvement	1.57	2.86	3.29	3.29	3.00	2.57	3.40	3.06	3.40	3.86	4.21
5. Technical support	1.43	2.43	2.71	3.00	3.07	2.93	3.67	3.75	3.85	4.12	4.14

All five performance indicators of STI clinic services showed consistent improvement over the 45 months of observation, as summarised in the table 2. The performance indicator with the largest increase was the coverage component which increased sevenfold over a period of 45 months, to reach 3.24. The quality of clinic and services attained the score of 4.27 followed by community involvement at 4.21, technical support at 4.14 and referral networks 3.80 by October 2008.

The time-trend analysis of the overall mean score for all five performance indicators showed a threefold increase in the score in three phases of progress (figure 1). The quality score improvement trend is statistically significant with a correlation coefficient of 0.88 (p<0.001). The first year beginning April 2005 had an accelerated quality improvement; the second and third years (April 2006–January 2008) showed a slow but continued to increase, whereas in the fourth year (April 2008) onwards, it appears to have stabilised.

Figure 1

Quality of overall STI clinic services measured from 292 clinic visits using clinic quality monitoring tool in the scale 0–5.

Discussion

We have shown that quality improvement in STI clinic services is an important component of a large-scale STI intervention for high-risk groups. In addition, a supervision-monitoring tool could be used simultaneously for real-time quality improvement. The clinic quality monitoring tool was used in a large-scale HIV/STI-prevention programme setting among SWs, MSM/TGs and IDUs in six Indian states over a period of 45 months to monitor and document quality improvement in various aspects of STI services.

This experience reaffirmed the usefulness of supportive supervision in improving the quality of clinical services.5 11 26–28 For example, primary healthcare operational research in 12 countries concluded that supervisory system is an essential component of quality improvement.29 The experience also confirmed the value of a locally appropriate tool for structured supervision. Methodical supervision using checklists has resulted in quality improvement in practice of primary health treatment guidelines in Nepal and diarrhoea management in Nigeria.26 In the Philippines, the use of a supervisory tool improved indicators in a primary health facility by 42 per cent in the experimental group compared with 18% in control group.30

There are three aspects of the quality-monitoring tool used in this report that address some of the shortcomings in previous publications. First, most reports on the impact of supervision noted effects after a short period of observations or as pilot studies.11 26 In the experience reported here, dipstick supervision by an external team over a long period coupled with ongoing supervision and technical support by the organisation responsible demonstrated short-term rapid gains in quality and long-term maintenance. Second, this tool covers three aspects of quality improvement: client or patient, managerial and clinical.17 Other studies have measured quality improvement of services through client satisfaction surveys, whereas the Avahan programme focussed on community involvement in design and implementation of STI services.18 The community involvement approach used in Avahan takes the client perspective further to engage the client in clinical service quality improvement. Community hubs, called ‘drop-in centres’ usually attached to the clinics, were established to provide a space for community interaction. The set-up is intended to progressively involve community in clinic activities such as receptionists, clinical assistance, infection control management and conducting exit interviews to provide feedback on community perceptions on clinic services and management. In addition, community involvement has a larger dimension in making the HIV/STI intervention more effective.31 Finally, this tool has attempted to monitor service coverage based on the framework of an effective coverage in health systems described by Tanahashi.22 This coverage model envisages five domains: availability, accessibility, acceptability, contact and effective coverage, and recommends managerial decisions and health policies needed to enhance the coverage in all these domains.22 23 The clinic quality monitoring tool reported here addressed issues related to accessibility, acceptability and contact coverage in a setting where STI services were made available to the high-risk and marginalised populations. Further, it explored various barriers for providing quality STI services and attempted to resolve them.24

There were several limitations in the study. The clinics visited by central team at any quarter were not representative of all existing Avahan clinics due to possible selection bias, as they were selected in consultation with state lead implementing partners. This bias might have directed the visits to better performing clinics. However, the central team visited most of the clinics over the period limiting the bias over the long term. On 14 occasions (14/77=18%) the visit to a state lead partner was not made in a particular round; the quality performance score from the previous round was carried forward, assuming no change in that quarter, which might have underestimated the overall performance score. On the other hand, the fact that clinic staffs were aware about the supervisory visits may have bolstered the quality in two general ways. First, there could be a conscious effort made by clinic staff to display better quality, and second, there could be an improvement in the performance just because the staff were being observed.32 However, as patient records for review were selected randomly from the previous 3 months, such overestimation is limited. This tool was developed for Avahan-supported static clinics and not suitable for mobile clinics, health camps and other public-funded clinics. It was difficult to apply this tool in a busy private practice or other preferred service provider settings due to their time constraints. External technical support has limited influence over the service delivery of private practitioners. Modification of the tool for different settings is necessary. It is recognised that this tool is too time-consuming to be utilised in routine supervision by the national AIDS control programme. Going forward, the key elements of the supervision tool will be incorporated into a simpler supervision and monitoring tool.

In summary, the experience of long-term use of a clinic quality monitoring tool for supportive supervision and monitoring has shown that clinical services in Avahan had improved and sustained quality in all five aspects of clinic performance (coverage, quality, referral networks, community involvement and technical support). The experience demonstrated the usefulness of the participatory monitoring tool to track and provide quality improvement inputs in real time over a long period that facilitated immediate feedback at the time of supervision. Such a comprehensive monitoring and quality improvement effort in the STI programme could be resource-intensive and time-consuming, and needs a structured approach. Cost-effectiveness, impact of reduced comprehensiveness of supervision areas and replicability of such an approach in limited resource settings need to be the focus of future work.

Continued quality improvement over a long period is possible in sexually transmitted infection interventions for high-risk groups.

Locally appropriate supportive supervision tool is important to ensure quality improvement in clinics providing STI services.

Combined clinician, patient and managerial perspectives are essential components in the supportive supervision tools.