A comparison of hierarchical cluster analysis and league table rankings as methods for analysis and presentation of district health system performance data in Uganda.

In 2003, the Uganda Ministry of Health introduced the district league table for district health system performance assessment. The league table presents district performance against a number of input, process and output indicators and a composite index to rank districts. This study explores the use of hierarchical cluster analysis for analysing and presenting district health systems performance data and compares this approach with the use of the league table in Uganda. Ministry of Health and district plans and reports, and published documents were used to provide information on the development and utilization of the Uganda district league table. Quantitative data were accessed from the Ministry of Health databases. Statistical analysis using SPSS version 20 and hierarchical cluster analysis, utilizing Wards' method was used. The hierarchical cluster analysis was conducted on the basis of seven clusters determined for each year from 2003 to 2010, ranging from a cluster of good through moderate-to-poor performers. The characteristics and membership of clusters varied from year to year and were determined by the identity and magnitude of performance of the individual variables. Criticisms of the league table include: perceived unfairness, as it did not take into consideration district peculiarities; and being oversummarized and not adequately informative. Clustering organizes the many data points into clusters of similar entities according to an agreed set of indicators and can provide the beginning point for identifying factors behind the observed performance of districts. Although league table ranking emphasize summation and external control, clustering has the potential to encourage a formative, learning approach. More research is required to shed more light on factors behind observed performance of the different clusters. Other countries especially low-income countries that share many similarities with Uganda can learn from these experiences.

Key Messages

League tables have been used in Uganda and elsewhere to present information on performance of health systems, yet they have some challenges.

Cluster analysis can be used to present performance data and can alleviate some of the challenges of league tables.

Cluster analysis highlights comparison of similar entities and a learning approach.

Introduction

Low-income countries (LICs) are trying to improve the health of their populations in the context of limited resources. Performance assessment, referring to the measurement and use in decision making of the extent to which various aspects of the health system meet their key objectives, is a key tool for improving health systems and subsequently contributing to improvements in population health. Performance assessment should make it possible to make comparisons within a system across time and different levels, and between systems and across various settings (World Health Organization 2000; Smith 2002; Loeb 2004). In the recent past, efforts were made by governments and agencies in low-and middle-income countries (L/MICs) to develop comprehensive health system performance assessment (HSPA) frameworks (Barron ; Boerma 2013; Ministry of Health 2011).

The analysis and presentation of performance data does matter since it contributes to whether the information is used in decision making or not (Freeman 2002; Gysels ; Gibberd 2005; Hildon ). League tables have long been used to present performance data in industry when certain goods and services are produced by competing organizations. They are also used in sports and by international agencies like the United Nations Development Programme that ranks all the United Nations member countries on an index of development annually (United National Development Programme 2013). Adab defined league tables as ‘a technique for displaying comparative rankings of performance indicator scores of several similar providers’. League tables are commonly used, where a standard against which to judge performance has not been set (Adab ; Marshall ). In public health, league tables have been used by WHO in the World Health Report since 2000; by the United Kingdom National Health Services and in some LICs, like Burkina Faso and Uganda (World Health Organization 2000; Marshall ; Foro 2013).

In 1999/2000, the Uganda Ministry of Health (MoH) published a National Health Policy (NHP) and Health Sector Strategic Plan (HSSP) which included reforms with implications for performance assessment. One of the reforms was decentralization, which provided for different responsibilities, namely policy formulation, strategic planning, resource mobilization and allocation and overall monitoring at the national level; and operational planning and management of delivery of services at the district level. The second reform sector wide approach to health development (SWAp) supported common arrangements amongst sector stakeholders for priority setting, supervision and monitoring (Ministry of Health 1999, 2000).

The first Uganda District League Table (DLT) was prepared in 2003, covering the period July 2002 to June 2003, and was included in the 2003 Annual Health Sector Performance Report (AHSPR). This was the first time various district performance parameters were presented and compared; previous reports had only provided aggregate national sector performance. Since then the DLT is prepared and published every year in the AHSPR (Ministry of Health 2003; Murindwa ). The number of districts increased from 56 in 2003, to 69 in 2006, 80 in 2007 and 112 in 2012 (Ministry of Health 2013). The DLT objectives were indicated as to enable comparison of performance between districts and therefore determine good and poor performers; to provide information to facilitate the analysis behind good and poor performance so as to enable corrective measures; to increase local government ownership for achievements and to encourage good practices. Appropriate corrective measures were indicated to include increasing available resources or more frequent and regular supervision to the district.

The DLT is compiled primarily using health management information system (HMIS) and community-level data. The HMIS in Uganda is the main information system for the health sector including data from public and private not-for profit (PNFP) health facilities but not from other private health facilities (Kintu ; Mandelli and Giusti 2005). The DLT also includes data collected through surveys for items for which data are not routinely available—for instance, data on household latrine coverage (Ministry of Health 2003). The DLT includes input, process and output indicators. The indicators selected for inclusion in the DLT were based on the HSSP indicators, with emphasis on those deemed to reflect system-wide performance (Ministry of Health 2003). The indicators are shown in Table 1, with information on year of collection and whether or not they are included in the DLT rank computation. The DLT presents district performance against each single indicator and a composite indicator computed by weighting some of the indicators (see Table 1). The composite index is used to rank the districts in performance from the first (best performer, with highest score) to the last one (worst performer, with least score) (Ministry of Health 2003, 2008c; Komakech 2005; Boerma 2013). In 2006, following the development of a new strategic plan, adjustments were made in the indicators to include in the DLT and in the computation of the composite index (Table 1). The DLT for 2008 can be accessed as a supplementary file. A limited analysis of the DLT data beyond the ranking was done using some individual output indicators and the composite indicator. Some district characteristics were noted to have a bearing on district performance as shown by the DLT ranking, including the presence of a hospital in the district (positive), recently designated district status, recent experience of conflict and belonging to certain regions/cultural grouping (negative) (Ministry of Health 2003, 2006a).

Table 1.

Uganda DLT indicators 2003–10

	Indicator/item	Type of indicator	Numerator	Denominator	Data collection		DLT rank computation
					2003–05	2006–10	Years	Weight factor
1	Population	Descriptive, absolute number			√	√
2	No. of health sub-districts	Descriptive, absolute number			√	Not in 05/06
3	No. of hospitals	Input, absolute number			√	X
4	Total number of health units (excluding hospitals)	Input, absolute number			X	√
5	Total number of health units	Input, absolute number			√	08/09 to 09/10
6	Total (public) funding to health sector per capita	Input, per capita	Govt. health budget funding to district incl. development, wage, non-wage; to public & PNFP units	All population (mid-year)	In 02/03 only	X
7	Approved posts filled by trained health personnel	Input, proportion	Trained health personnel in approved posts by cadre in public and PNFP facilities	Staff norms by cadre, by health facility	√	X	2003–05	5
8	District HMIS Outpatient returns submitted timely	Process, proportion	No. HMIS Outpatient monthly returns submitted by district timely	All expected (12, months)returns	√	√	2003–10	5
9	District HMIS outpatient returns submitted complete	Process, proportion	No. of HMIS outpatient monthly returns including all health units	All expected (12, months) returns	√	X	2003–05	5
10	PHC funds spent on drugs at NMS & JMS	Process, proportion	Funds spent on drugs at NMS and JMS	Indicative budget for drugs	√	√	2003–10	10
11	Quarterly requests submitted timely	Process, proportion	Quarterly requests submitted timely	All expected (Four, quarters) reports	In 02/03	X	2003–05	10
12	PHC funds disbursed that are expended	Process, proportion	PHC funds expended	PHC funds disbursed from Ministry of Finance	Not in 02/03	√	2006–10	5
13	FDS flexibility gain	Process, proportion	Funds gained by district health sector from other sectors	District health sector conditional grant	X	√	2006–10	5
14	Children <1 received 3 doses of DPT as per schedule (DPT3)	Output, proportion	Children <1 year that have received DPT 3	Estimated no. of children <1 year (mid-year)	√	√	2003–10	12.5
15	Govt &PNFP OPD utilization per person per year	Output, per capita	Visits for new episode of illness at OPD of a Govt. or PNFP facility	All Population (mid-year)	√	√	2003–10	12.5
16	Pit latrine coverage	Output, proportion	Households with latrines	All households	√	√	2003–10	7.5
17	Deliveries in Govt and PNFP health facilities	Output, proportion	Mothers delivering in Govt. and PNFP health facilities	Expected pregnant women in population (mid-year)	√	√	2003–10	12.5
18	Proportion of TB cases notified compared with expected	Output, proportion	TB cases notified	Expected no. of TB cases in population (mid-year)	√	√	2003–10	10
19	Pregnant women receivingsecond dose Fansidar for IPT (IPT2)	Output, proportion	Pregnant women receiving 2nd dose of Fansidar/SP	Pregnant women –receiving ANC	√	√	2003–10	10
20	HIV/AIDS service availability	Output, composite	HCT + PMTCT + ART by level	Relevant beneficiary population for each indicator	X	√	2006–10	10

ANC, antenatal care; ART, antiretroviral therapy; DPT3, third dose of diphtheria, pertussis and tetanus vaccine; HCT, HIV counselling and testing; HMIS, health management information system; IPT, intermittent presumptive treatment of malaria with sulphadoxine pyrimethamine (Fansidar); JMS, joint medical stores; NMS, national medical stores; OPD, outpatient department; PHC, primary health care; PMTCT, prevention of mother-to-child transmission of HIV; PNFP, private not for profit; SP, sulphadoxine pyrimethamine.

The DLT is published every year in the AHSPR that is presented at the annual health sector stakeholder forum the joint review mission (JRM), and biennially at the national health assembly (NHA). The JRM and NHA include representatives from the public sector, donors, PNFP providers, civil society organizations and academia. Public officials include those from the national and district levels, with political, administrative and technical managers participating. There is public recognition with presentation of plaques to the 10 best performing districts according to the DLT ranking (raised to 15 with the increase in number of districts in 2007). In addition, mention is made of the bottom ranked 10 districts (later 15), consistently good (or poor) performers, and those that would have shown marked improvement (or worsening).

The Uganda DLT has been in place for >10 years, providing a source of district health systems performance data and used to support decision making by some stakeholders (Ministry of Health 2003, 2006a, 2008c). However, a number of challenges have been noted with the DLT. There are concerns that the DLT is not a fair method of comparing performance given the different contexts in the districts and that the rank of the district is not capturing the complex realities of districts. These are some of the factors that may have contributed to the less-than expected use of the DLT for decision making by health system managers (Komakech 2005; Ministry of Health 2008c). The doubling of the number of districts over the last decade has markedly increased the data points in the DLT, further increasing complexity. For example, the DLT for 2011 covers 10 pages, just the table excluding any narrative. The Uganda health sector stakeholders have a decade of experience in district HSPA that can be utilized to inform the development/adjustment of HSPA frameworks within the country and by countries with similar context. This also provides an opportunity for exploring alternative and/or complementary ways of analysing and presenting district health system performance data.

Another approach that has been used for analysis and presentation of health system performance data is cluster analysis. Clustering has been used to group many data points into fewer categories to facilitate comparison among similar entities and support decision making. In health it has been used extensively in biomedicine, in genetics, asthma and cancer studies and in psychology (Tavazoie ; Beckstead 2002; Dyrskjot ; Haldar ; Andreopoulos ). Although clustering has not been used widely to date in health systems research, a few studies have shown that hierarchical cluster analysis (HCA), one form of clustering, can be a powerful tool for comparison of entities and for highlighting inequalities (Lakhani ; Ruger and Kim 2006; Day ; Ottevaere ).

Research on HSPA has mostly taken place in high-income countries (HICs). Yet the marked difference in contexts between HICs and L/MICs means that it is not always possible and/or desirable to copy HSPA approaches and frameworks from one context to the other (Kruk and Freedman 2008; Smith ; Tashobya ). This study is part of a broader research programme that is utilizing the Uganda DLT as a case study, with the aim of contributing to the evidence pool on HSPA in L/MICs and contributing to future developments in district HSPA in Uganda and similar countries. The study reported on here focuses on the presentation and analysis of district health system performance data. The objectives of this study are 2-fold: to explore the use of cluster analysis for the presentation and analysis of district performance data; and to compare the use of league tables and cluster analysis for presentation and analysis of district health systems performance data. The study is expected to provide a fresh perspective to analysis and presentation of district HSPA data in Uganda and highlight possible areas for further study on this under researched subject. Other aspects of this research that include the critique of other characteristics of the DLT and its interaction with the Uganda health system context are being studied and will be reported elsewhere.

Data and methods

A combination of document review and quantitative data collection was used in undertaking this study. The document review focused on the development and reviews of the DLT; on the management of the data including processes and products of data analysis and presentation and on the use of DLT findings for decision making. Documents reviewed include the NHP and HSSPs; AHSPRs, the HSSP mid-term review reports (MTRs) and the aide memoires of Sector Reviews over the period 2003–13. The study benefitted from input from the first author who has worked at the MoH in Uganda for the last two decades and was part of the team that developed and coordinated initial implementation of the Uganda DLT. The knowledge and insights of the author were utilized to identify sources of data/information. Quantitative data on district health systems were accessed from MoH Resource Centre databases for the same period. The end point of quantitative data collection was determined by the beginning of a new sector strategic plan in 2010 that introduced some changes in requirements for reporting and for the DLT.

Statistical analysis using SPSS version 20 and HCA was used to classify the Ugandan districts into clusters for each year over the period 2003–10. The same indicators and data used to compute the composite index in the DLT were used to enable comparisons between the two methods (see Table 1). Because some data had differing scales and magnitudes, the data utilized were standardized to ensure that each variable was compared at the same level in the distance measure. Equal weighting was carried out. This was contrary to the practice in the computation of the DLT. The choice of equal weighting by the researchers was made given the rationale behind the weighting in the DLT was never explicitly indicated, and the widely differing views in the literature on weighting generally (World Health Organization 2003; Lauer ). We used Ward’s method in our HCA. It is the most commonly used clustering methodology because of its capacity to produce clusters of roughly the same size. The squared Euclidean distance is the recommended measure of similarity for Ward’s method and was used to determine how ‘close’ the cases were to each other (Kaufman and Rousseeuw 1990; Romesburg 2004). At each step, the within-cluster sum of squares is minimized over all clusters obtainable by merging two clusters from the previous generation. This approach to HCA has been used by other researchers in health systems research and is appropriate for the relatively few data points in this study (Lakhani ; Berkhin 2006; Ottevaere ).

In initial runs for the years 2007–2011, Kampala district was an outlier, constituting a cluster in itself until the last grouping, and was thus removed from subsequent analysis. A total of 55 districts were included in the analysis for the period 2003–05, 68 for 2006 and 79 for the period 2007–10. Clusters were formed through an agglomerative process, beginning with the combination of the two districts with the highest similarity (lowest Euclidean distance) into a cluster and then these would be combined with the next district/cluster with the lowest distance to either of the two districts. This process would continue until increasingly dissimilar districts were clustered together and then finally all the districts formed one cluster. At the point where seven clusters were formed, a demarcation was made by the researchers. This judgement was made by the researchers given their understanding of the context. It was determined that less than seven clusters grouped together too many districts, whereas more than seven clusters were considered unwieldy for decision making (Romesburg 2004; Berkhin 2006). Cluster variable averages provide a set of values describing the cluster, rather than the individual district.

Results

HCA of Uganda district health system performance data

Clustering was carried out each year over the period 2003–10. Reference in the article is mostly made to the years 2007–10, as data from this period is comparable: the same districts and the same indicators were reported on. This is not the case for previous years due to the split of districts in 2005 and 2006, and the change in indicators in the DLT in 2005.

The fiscal year 2007/08 reported on in the 2008 DLT was used to demonstrate clustering because it was considered that district operations and reporting under the DLT would have reached an appreciable level of stability following the introduction of new indicators in 2005 and the split of districts in 2006. Figure 1 presents an output of the clustering process for 2008 as a dendrogramme, showing how the 79 districts were successively grouped into clusters until they formed a single cluster. A line was drawn through the dendrogramme to illustrate the point at which seven clusters were determined. As more dissimilar clusters are formed, the link lines can be seen to get longer reflecting the higher magnitude of the between cluster distances. Table 2 is a summary of the 2008 clustering output showing cluster variable values. In 2008, cluster A included three districts that had performed very well on outpatient department (OPD attendance), DPT3 (diphtheria, pertussis and tetanus vaccine third dose in infants), HIV control activities and proportion of disbursed funds expended; cluster B (19 districts) good performers on OPD, DPT3, proportion of disbursed funds expended and HMIS timeliness; cluster C (10 districts) good performance on proportion of disbursed funds expended and expenditure against the medicines budget; cluster D (16 districts) good performance on HIV control activities, proportion of disbursed funds expended and HMIS timeliness; cluster E (17 districts) good performance on OPD and HMIS timeliness and poor performance on proportion of disbursed funds expended; cluster F (13 districts) good performance on DPT3 and poor performance on proportion of disbursed funds expended and finally cluster G (1 district) good performance on proportion of disbursed funds expended and poor performance on DPT3, OPD, IPT2 (second dose of intermittent presumptive treatment of malaria in pregnant women), deliveries in health facilities, expenditure against medicines budget and HMIS timeliness. In 2008, a similar rating was given to all the districts for fiscal decentralization strategy (FDS), and therefore this indicator did not influence clustering. The FDS indicator represented the proportion of funds allocated to the district health system from other sectors locally. Good performance on individual variables in this case was taken to mean performance >80% (or 0.8 per capita), poor performance as <20% (or 0.2 per capita) and moderate performance between the two extremes. The clusters are determined by a combination of the identity and magnitude of performance of each of the variables. The clusters have been labelled in declining order of overall performance from A to G, using a traffic light approach (see Table 2) and classified as good, moderate and poorly performing. For example, cluster A in 2008 showed good performance on four variables and did not perform poorly on any variable, whereas cluster G performed well on one variable and poorly on six variables. This classification has been used here to structure the exploration of the clustering methodology and support the comparison between the DLT and clustering, and is not regarded by the authors as an essential aspect of presentation of clustering results.

Figure 1.

Dendrogramme showing clusters for 2008

Table 2.

Clusters for 2008 showing cluster variable average values

The characteristics of the clusters varied from year to year. Cluster characteristics for the individual years for the period 2007–10 are summarized in Table 3. In line with the classification earlier, over the years, clusters A and B can be seen to be usually good performers; clusters C to E as moderate performers and clusters F and G as poor performers. The movement of districts between clusters was studied by following a few districts, as shown in Table 3. The districts selected for this were of differing characteristics: Bushenyi, in the southwest, established, secure; Apac, mid-north, established, post-conflict; Oyam, mid-north, new—since 2007, split off Apac, post-conflict and Nakapiripirit, northeast, new since 2000, conflict prone, nomadic community. Bushenyi moved between clusters B and D—clusters of good-to-moderate performance; Apac and Oyam between C and F—clusters of moderate-to-poor performance and Nakapiripirit between clusters F and G—poorly performing clusters. This indicates that the districts tend to move within a certain (performance) limited range; none of the four districts moved from one extreme to another (good to poor or vice versa), although Oyam a new district is noted to have showed steady movement, with an improving trend over the 4 years.

Table 3.

Cluster characteristics and movement of four districts between clusters for the period 2007–10

The movement of districts between clusters provides us with some insight into specific cluster characteristics. For example, Bushenyi district is in clusters which generally perform well, and consistently perform well on OPD, often in combination with good performance on DPT3 and proportion of disbursed funds expended. Conversely, Nakapiririt tends to be in clusters with poor performance and particular poor performance on deliveries in health facilities and household latrine coverage. Apac and Oyam showed movement between poor and moderately performing clusters that do not show consistent (good or poor) performance on any particular variables. The characteristics and membership of clusters in a given year are determined by the specific performance on the different variables in that year. The specific cluster the district is in is determined by its performance on a combination of indicators and how other districts perform on the different indicators for that given year.

Comparing the DLT and HCA results

DLT rankings for the years 2003–10 were plotted against clusters derived from the same data for each year. The relationship between DLT rank and cluster (see Figure 2) indicates moderate coherence between the two methods. This is not surprising given that indicators included in the DLT composite index were the same as those in the clustering, with the difference that in the latter equal weighting was done and Kampala district was excluded. The variation along the y axis indicating the spread of DLT ranking on each cluster differed across clusters and over the years. The clusters A and G tended to show least variation suggesting more specific characteristics (extreme good and poor performers, respectively; marked performance on certain variables). This was noted earlier and illustrated in Table 3. There are exceptions in 2006 and 2008. In 2006, >40% (28/69) districts did not provide information on proportion of funds disbursed that were expended. This is likely associated with the ‘abnormal’ picture seen with, for example 24 districts in cluster A compared with <10 in the other years.

Figure 2.

Comparing clusters with DLT rank 2003–10. Y axis shows rank as determined by the DLT composite index; X axis shows the clusters, with A being the best performing and G the worst performing cluster

In 2008, Abim district ranked 49th on the DLT but is in cluster A. Primary data show that Abim had very high performance on DPT3 and OPD visits and high performance on HIV control activities and very low performance on household latrine coverage. Cluster A in 2008 was characterized by high performance on OPD, DPT3 and HIV control, thus, the presence of Abim in this cluster. The very low performance on household latrine coverage which gave Abim a mid-table position on the DLT was ‘neutralized’ by the district’s good performance on the other variables. As the number of districts increases, there is more spread within the clusters, indicating relatively increasing heterogeneity within clusters as the number of clusters stayed fixed at seven.

Discussion

The DLT was a breakthrough in HSPA in Uganda, as for the first time it was possible to document the performance of each individual district against several indicators. This has provided a rich data set, covering a range of indicators for the whole country over the period 2003—to date, which can be accessed by a number of stakeholders. The DLT made it possible to have system-wide discussions at both national and district levels—rather than just focusing on specific programme performance. This raised the ‘visibility’ of district health system issues in sector reports and reviews (Ministry of Health 2003, 2008c). The ranking provided a single index for each district providing its position in relation to other districts and to the national average. This made it easy to grasp for national and district level managers—including political, administrative and technical managers—and increased their interest in sector performance dialogue (Ministry of Health 2006b, 2008a). The utilization of district data especially from the (routine) HMIS provided motivation for improvements in data management.

Certain DLT objectives were met, notably the comparison of performance between districts and to a lesser extent the identification of reasons for the variations in performance and the implementation of corrective measures. The DLT data and ranking has been used to support decision making at various levels and by various stakeholders (Ministry of Health 2003, 2006a). Some health system stakeholders sought to understand the factors behind observed district performance; and some used the information to make changes in the health system through planning, resource allocation and management of health services (Ministry of Health 2008b, 2009). The central level has used the DLT information for the purpose of planning, district supervision and mentoring. Some district managers have utilized the information in the development of district operational and strategic plans, and in improving service delivery—for instance, improving prioritization of health inputs and streamlining management processes like procurement of medicines and recruitment of health workers (Kotido District Local Government 2004; Ministry of Health 2005, 2006b, 2009, 2012; Moroto District Local Government 2005). Development partners have utilized DLT information in the formulation of new programmes, and in the supervision and evaluation of existing ones. Civil Society/Advocacy organizations have used DLT information for supervision and assessment of accountability in the sector (Ministry of Health 2008a; Tashobya ; Danish Ministry of Foreign Affairs 2012). District health system managers recognized the importance of data management as this had major implications for individual districts reported performance and DLT ranking. This translated into overall improvement in data management as measured by HMIS timeliness and completeness (Ministry of Health 2008b,c). Other positive aspects of the DLT were perceived transparency, explicitness and objectivity. These have been noted as advantages of league tables by authors elsewhere (Davies and Lampel 1998; Adab ; Freeman 2002; Gysels ).

However, there were also negative aspects. The enthusiasm with which the DLT was initially received by many was later tinged with disappointment and frustration for some. A common complaint was that the DLT was not fair, as it did not take into consideration district peculiarities including health system resources, geography, demography, level of development and ethnic variations (Komakech 2005; Ministry of Health 2008b,c). Some managers, especially those of districts ranked at/near the bottom, found the DLT embarrassing and demotivating. This is despite the MoH’s indicated position of the DLT not intended to ‘name and shame’ (Ministry of Health 2003). This demotivation was marked in cases where district health system managers perceived that key factors behind the observed performance were outside their locus of control as highlighted by the following examples. The districts in the Karamoja region consistently show poor performance on the DLT, especially for the indicators deliveries in health facilities and household latrine coverage. The majority of the people who live in Karamoja is nomads who have maintained strong cultural practices, some of which have a strong negative bearing on health-seeking behaviour. Kampala district made it to the top of the DLT in 2008 and retained this position until 2011. This was deemed to be due to the high concentration of health facilities and an educated and relatively wealthy population. Subsequently, the MoH removed Kampala from the league table (Ministry of Health 2008c, 2012). The DLT rank is a highly summarized index that can be misleading given the complexity of health systems. This has sometimes resulted in conflict and apportioning of blame between health systems stakeholders (Ministry of Health 2008c, 2009).

It has previously been stated that the DLT facilitated improvements in data management. On the other hand, there have been concerns about the quality of the data being used to make the critical judgements of the DLT. It is suspected that the desire to be seen to perform well may have contributed to data manipulation by some district managers (Driwale 2005; Komakech 2005; Ministry of Health 2008c). An HMIS data validation exercise carried out by the MoH in 2008 noted divergence between some health facility and district databases, and between some district databases and data submitted to the national level on the other (Ministry of Health 2008b,c). Many health system stakeholders have been critical of the fact that DLT information does not seem to be a major consideration while making key sector decisions including resource allocation. Public budgets to districts have been maintained largely on historical/incremental basis, without any major effort to rectify some of the observed inequities in the last five or more years (Komakech 2005; Ministry of Health 2008b; District Health Officers’ Association 2012). Districts were initially classified according to DLT rank as top 10 (doing well should be emulated), bottom 10 (need to improve) and the rest in the middle. With the increase in the number of districts this was increased to top 15 and bottom 15, which still left 50 districts in the middle. This classification though left a lot to be desired in terms of facilitating decision making. What was it supposed to mean if you were in the middle 50 districts year after year?

It is likely that the criticisms and challenges above contributed to the less than expected utilization of the information provided by the DLT to support decision making. The DLT did not fulfill the expectation of yielding better insight into the factors explaining performance of districts, in identifying the most appropriate corrective measures, and in facilitating learning from best practices. The expected increased ownership of the analysis by the district managers was only achieved in a limited manner. The criticisms of the Uganda DLT are similar to those raised elsewhere on league tables and other approaches that use performance indicators to rank entities. These include the comparison of entities that are not really comparable making it unfair; the deceptively simple and at times misleading summary indices given complexity and varying contexts in health systems; the ‘naming and shaming’ thereby demotivating those that are shown to have performed poorly; the need for high quality data; the creation of perverse incentives resulting in gaming and misrepresentation of information and the opinion that such tables are a ritual and not used to support decision making (Nutley and Smith 1998; Freeman 2002; Mannion 2002; Gysels ; Marshall ). League tables are used with the understanding that there is no known norm or standard; but also that the entities that you are comparing are comparable. This characteristic though is not necessarily true or applicable in most circumstances (Nutley and Smith 1998; Adab ). Such criticisms were raised, for example, against the ranking of WHO member countries in 2000, and for the UK NHS performance assessment approach (Navarro 2000, 2001; Adab ; Freeman 2002; Rixom 2002).

In this article, we explore the use of HCA as a possible alternative or additional approach to the use of the league table for sub-national (district) HSPA. HCA reorganizes many data points into manageable groupings in the form of clusters based on agreed variables. This can provide policymakers and managers at the different levels of the health system with groups of similar or comparable districts from the perspective of their performance or other chosen perspective. This serves two purposes—it reduces the many data points into fewer entities (clusters) and groups districts that according to the data have marked similarities. In this study, the process and output variables of the DLT are utilized to determine ‘performance clusters’. This approach explicitly begins with the observed performance against a set of agreed indicators and works backwards to initiate the process of trying to understand the factors behind the performance. This is referred to as ‘…un-supervised learning of a hidden data concept…’ (Berkhin 2006). The characteristics defining the different clusters provide the beginning point to trying to understand or explain some of the factors behind the observed performance of the districts.

For example, the districts in cluster A are those with particularly high OPD and DPT3 rates, and include in some years those that have also performed well on HIV control and on the proportion of disbursed funds that were expended. Some of the questions that may be raised in a bid to understand the factors behind the observed performance are the following. What are the other characteristics common to these districts? What might this tell us of the likely reasons for this observed performance? Is it because of exceptional management of health services—to a large extent under district control? Or is it good health facility coverage, which is a health resources issue largely under the control of the MoH and development partner programmes? Or is it because of good health-seeking behaviour of the population? Or because these districts share a border with countries that have relatively poor access to health services leading to influx of foreigners seeking services—which is a contextual issue and beyond the immediate influence of the district and the MoH? At the other extreme, cluster G in 2010 is characterized by very low household latrine coverage and low proportion of deliveries in health facilities. This combination is likely to be related to strong cultural beliefs and practices that may not respond in the short term to activities by the district or even the MoH. The moderate clusters do not show consistent performance (good or bad) on any of the indicators. What is likely to be behind these inconsistencies—is it poor data management or uninspired leadership or other as yet unknown reason?

The Uganda health system like most other health systems is complex and dynamic. Decentralization has provided for varying mandates at different levels, with implications for HSPA. The concern by the MoH and national level partners including some donors is for a bird’s-eye view across the country looking at efficiency, equity and accountability, in line with the DLT objectives of comparing district performance. On the other hand, the districts and other organizations operating at the service delivery level require information to support management decisions in line with the DLT objectives of determining the factors behind good or bad performance, increasing local government ownership, implementing corrective measures and learning from best practices. Freeman (2002) refers to the contrasting expectations of the different stakeholders as summative, emphasizing accountability and assurance from the perspective of the national level especially the MoH, and formative, focusing on improvement and learning from the perspective of the district/operational level.

In addition to the previously indicated criticisms, the DLT ranking emphasizes the summative, external control approach. The league table approach is prescriptive; a judgement against some expectation and as such better suited to foreclose a discussion (Freeman 2002). These characteristics of the DLT do not make it a good tool for supporting decision making in a complex, dynamic system as noted for the Uganda health system. Such a tool also requires very high quality data that is not yet available in the Uganda health system. We argue that from this exploratory study we can show that HCA has potential to counter some of the challenges of the DLT. HCA groups districts into clusters of comparable entities, given the variables used in the analysis. Further analysis can be done with emphasis on the characteristics that define the individual cluster, rather than the relative performance of the different clusters. This is likely to be perceived by the districts as a fair approach to HSPA, which is a major point of consideration in view of the many complaints against the DLT in regard to this issue. For the districts, the clustering provides peer groups for comparison and an impetus for benchmarking and friendly competition, and may reduce incentives to manipulate data. For the MoH, it becomes possible to group districts into clusters of comparable entities that provide the big picture of district variation in performance. The clustering is a compromise between providing the detailed data of all the districts and variables, and the ultra-summarized single figure of a district rank on the DLT. In most instances, clustering can be done on the basis of readily available data as it does not require data of high precision (given application of data) (Freeman 2002; Gibberd 2005).

HCA as we have shown can facilitate the raising of a number of questions with regard to the observed clusters given cluster membership and variable characteristics. These questions provide the starting point for understanding factors behind the observed performance. This approach is more descriptive than prescriptive; the compiled data on the process and output variables and the results of cluster analysis are considered only partial and provisional truth. This provisional truth is best utilized and interpreted by looking at the specific context and other (local) data that may be considered necessary including qualitative data. Data collection and management of change are happening at the same or close levels. Such an approach has been noted to have a higher potential to facilitate positive change (Davies and Lampel 1998, Freeman 2002). The second round of analysis and presentation of findings moves the process forward towards understanding the factors behind the observed performance at the individual district, including teasing out the effects of management from other factors. In this way then, HCA has the potential to encourage a more formative, learning approach. The principle of comparing performance across the country can remain with the use of HCA (especially when used in combination with DLT ranking), but results are presented and can be interpreted in a different way—one that contributes to understanding the reasons behind variation in performance. This information can be used in negotiations with different stakeholders and in influencing decision making about resource allocation, supportive supervision, scaling up of technical programmes and improving management processes.

The likely challenges to implementation of HCA that need to be considered include apparent opacity, given the specific programming that is required to do the analysis, and the requirement of a certain capacity of managers both at national and district levels to carry out cluster analysis and interpret the results in combination with other data pieces to support decision making. The stakeholders need to appreciate that cluster analysis is not a statistical magic bullet but rather an approach that can be utilized as part of an enquiring process around district HSPA. Some of these potential challenges can be minimized by a participatory and evidence-based process for HSPA development or adjustment (Tashobya ).

A set of six attributes for a ‘good’ HSPA framework have been proposed that include the following: a participatory and evidence-based development process; clarity of the health system conceptual model; an explicit and appropriate relationship to the political, social, economic and health system context; clear articulation of the framework including its purpose and indicators; appropriate institutional framework for HSPA; elaboration of the mechanism through which information produced is expected to be used in decision making and the adaptability of the framework over time and changing contexts (Tashobya ). Although the different attributes of an HSPA as articulated earlier are all important, this study focused on the area of analysis and presentation of the findings, an aspect of the attribute on mechanism for change, to facilitate in-depth analysis and maintain a manageable scope for the discussion. The researchers though appreciate the importance of the other attributes and will utilize other studies in the broader research programme to explore some of the other aspects. For example, a review of the development and implementation of the DLT and the effect of the Uganda health system context has been considered in depth in another study.

This study had the objective of exploring the use of cluster analysis for the analysis and presentation of district performance data and comparing it with league tables. The study has raised a number of questions, and many more can be asked given the picture presented by the clusters. For example the fact that the cluster characteristics change over the years, as does the cluster membership, points at a dynamic situation. For the purpose of aiding comparison between DLT and HCA, this study limited itself to the use of indicators and data available and used in the DLTs over the 8-year period 2003–10. Although indicators on district health resources like staff in post, health facility coverage and public health financing are included in the DLT, data on these indicators have not been collected at all (e.g. for health financing) or not collected on a regular basis (staff in post). In addition, consideration of the questions raised can lead to the identification of new variables currently not captured in the DLT including quantitative and qualitative data on districts characteristics. For further exploitation of the potential of HCA for data analysis and presentation, it is proposed that as part of future research agenda for Uganda district HSPA, these data requirements should be determined, the data collected and further analysis carried out. This will provide answers to some of the questions raised earlier.

Limitations of the study

The study faced a number of limitations. The creation of new districts over the period of study made it difficult to analyse trends for the whole 8-year period. Similarly, the changes in indicators over the study period, including those included in the DLT composite index, made it difficult to make comparisons beyond the period 2007–10. The quality of the data was another limitation, with a number of districts having tendered no data or having data of questionable quality on some of the indicators. Although input variables are included in the DLT, data on human resources, health infrastructure and health financing were not consistently collected over the study period.

Conclusion

This study has shown that HCA has potential to be useful as a tool for analysing and presenting subnational performance data that could be utilized to overcome some of the challenges of league tables. Clustering categorizes many data points into a few groups of similar or comparable entities, presenting data in a format that is likely to be considered more manageable and useful for policymakers and health system managers. The formation of clusters of comparable units and the emphasis on cluster characteristics rather than the comparative performance of clusters is likely to be perceived as fair by district health system managers. The cluster characteristics can be used to initiate the process of identifying explanatory factors behind the performance pattern to aid learning, further analysis and decision making. This learning approach is in contrast with the summative and command and control approach of the league table. The methodology can be useful in dynamic contexts like in the case of changing number of districts, as HCA can organize large data sets into smaller and more manageable groups. HCA can be used in combination with other analytical and data presentation approaches like the DLT if so desired. Further research focusing on collecting and utilizing data on district characteristics (e.g. socioeconomic and demographic profiles, health-care system resources) would answer some of the questions raised by this exploratory study and further provide information on the usefulness of HCA in district HSPA. Countries sharing similarities with the context of the Uganda health system may consider exploring the use of HCA for subnational HSPA.