HIV/AIDS Monitoring and Evaluation Systems Strengthening for the National AIDS Program in Albania.

The Institute of Public Health (IPH) is the Principal Recipient (PR) of the Global Fund for AIDS, TB and Malaria Round 5 HIV/AIDS and TB grants. The GF HIV Grant is highly targeted, aimed at filling key gaps in the national response to HIV/AIDS. Albania is considered a low HIV prevalence country based on the number of reported cases. The work presented in this paper is carried out as part of the Monitoring and evaluation framework development in respect to the National AIDS strategy and Global Fund agreement. The performance framework consists of a range of indicators agreed by the PR (IPH in Albania), and specific targets to be met on an annual base. The PR must, therefore, be able to measure its performance against this framework in order to demonstrate that it is meeting its targets. To assist this process, the GFATM requires PRs to undertake an assessment of their capacity to monitor and evaluate the program, and put in place measures to strengthen any identified weaknesses. The Global Fund's M&E Systems Strengthening Tool, MESST, was used. The MESST is conducted via a participatory workshop involving the PR, SRs,(Subrecipients) some representatives of the Country Coordinating Mechanism (CCM), the donor community and other stakeholders or implementing partners relevant to the exercise. The proposed output for this exercise is an M&E Action Plan for the PR. A 2-day MESST workshop was held in Tirana, Albania. Participants at the workshop assessed the M&E capacity of the national HIV program, and its implementers, the IPH and the sub recipients (SRs), by looking at three elements of the M&E system. These are: The Monitoring and Evaluation Plan, The ability of collate and analyze data at the IPH, The reporting systems in the health facilities, the community and the laboratories, The result was identification of a range of strengths and weaknesses, and from these, the development of a list of key actions to address the weaknesses identified.

BACKGROUND

The Institute of Public Health (IPH) is the Principal Recipient (PR) of the Global Fund for AIDS, TB and Malaria Round 5 HIV/AIDS and TB grants. The GF HIV Grant is highly targeted, aimed at filling key gaps in the national response to HIV/AIDS. Albania is considered a low HIV prevalence country based on the number of reported cases (1-8).

There has, however, been a change in the epidemiological situation recently – of the 555 HIV/AIDS diagnosed and reported cases by November 2012, 513 were detected after 2000. Between 1993 and 2000, the average number of reported HIV cases was 6-7 per annum. This number rose to 10-20 per year in 2001-2003 and 30 per year in 2004-2006. Since 2007 Albania has registered over 40 cases per year and in the two last years this figure is over 70.

Through different studies performed (Bio-BSS, Youth Risky behaviors), a change in behavioral patterns has also been perceived, with increases in risky behaviors, particularly among vulnerable sub-populations like injecting drug users (IDUs), MSM and those who have sex with multiple and concurrent partners.

In 2003, the Government of Albania issued its National Strategy for Prevention and Control of HIV/AIDS 2004-2010, which was revised in 2008, and upgraded for the period 2008-2014. The focus of the strategy is to keep Albania a lowprevalence country. The focus of the grant is, thus, to maintain low prevalence in the country by targeting the vulnerable groups and providing care and treatment to PLWHAs (People living with HIV/AIDS).

The program funded by the GF has three components – prevention, care and treatment, and coordination of an evidencebased approach to the disease. For the first, the program provides specific activities for 4 sub-population groups – IDUs (Intravenous drug users), CSWs (commercial sex workers), the Roma and MSM (man that have sex with man), as well as condom distribution and counseling and testing. In order to achieve the second objective, it provides ARVs to the PLWHAs. The third component aims to improve identification and monitoring of vulnerable populations in order to pursue an evidence-based approach to the disease. Thus, while the national strategy is more comprehensive, all the GF grant components have been aligned with it.

As part of its Phase 2 renewal application, the PR has to demonstrate the existence of a well-functioning M&E system. A crucial component of any grant agreement between the GFATM and the Principal Recipient is the Grant Performance Framework. GFATM disbursements are based upon performance. The performance framework consists of a range of indicators agreed by the PR (IPH in Albania), and specific targets to be met on an annual base. The PR must, therefore, be able to measure its performance against this framework in order to demonstrate that it is meeting its targets. To assist this process, the GFATM requires PRs to undertake an assessment of their capacity to monitor and evaluate the program, and put in place measures to strengthen any identified weaknesses.

National AIDS Program (NAP) at IPH, with the technical support of the USAID-funded Grant Management Solution project, conducted an M&E self-assessment. The purpose of self-assessment was to identify strengths and weaknesses in the HIV M&E system, including the M&E Plan, the data management capacity of the PR and the data reporting system for the grant. The Global Fund’s M&E Systems Strengthening Tool, MESST, was used.

STUDY DESIGN

The Monitoring and Evaluation Systems Strengthening Tool

National governments and donors are working to fight many diseases, including HIV/AIDS and to make improvements in a number of health areas. National programs and associated projects are setting ambitious goals and objectives, the achievements of which are measured through M&E indicators. Accurately measuring the success of these initiatives and improving Program management is therefore predicated on strong M&E systems that produce quality data. The M&E Systems Strengthening Tool (MESST) is designed as a generic tool to assess the data collection, reporting and management systems to measure indicators of program and project success. Simply put, this tool addresses primarily the M&E plan and systems that need to be in place to collect and channel data up a system for aggregation into relevant indicators for program management and reporting. The overall objective of the MESST is to help national programs and associated projects improve their M&E and the quality of data.

The MESST was designed as a generic tool to assess monitoring and evaluation (M&E) plans and systems by assessing data collection, reporting, and management systems to measure indicators of program and project success. It addresses the M&E plan and systems that need to be in place to collect and channel data up a system for aggregation into relevant indicators for program management and reporting. The MESST can be used at the national level, within groups of projects, and within individual projects or organizations that are seeking to assess M&E data collection and reporting systems and to implement action plans for strengthening M&E.

The MESST has been designed to:

Assess the M&E plan and capacities of the program’s/project’s implementing capacity.

Evaluate how the M&E activities of programs/projects are linked and integrated within the National M&E system.

Help develop an action plan to strengthen M&E systems.

The developed action plan should help identify M&E gaps and corresponding strengthening measures, guide investments in M&E, ensure that these investments contribute to the strengthening of national systems (avoiding parallel reporting), and improve the quality of programmatic data to enhance planning and program management.

The MESST Tool is based on principles of participatory evaluation and research. The essential assumption that underlies participatory evaluation is that engaging stakeholders in design and implementation of the evaluation, research – or planning process — improves the accuracy and utility of findings, increases stakeholder “buy-in” and “inclusion” – and thereby the likelihood of their using the results, and strengthens the institutional capacity of the stakeholders to design and carry out or manage evaluation processes. The type and degree of participation varies – from engagement of stakeholders throughout the evaluation process to participatory observation, through which the researcher may actually engage fully with the individuals/group of interest.

Following the Checklist is a section titled ACTION PLAN. In this section, respondents should first identify the strengths and weaknesses of the M&E Plan based on answers provided to statements in the Checklist. Then respondents should identify planned strengthening measures and for each of these measures, specify responsibility, timeline, funding source, technical assistance needs and impact on existing work plans/budgets.

Finally, respondents should, based on the answers to the statements in the Checklist, provide an overall rating of the M&E Plan. An “A” rating signifies no gaps, a “B” rating signifies minor gaps and a “C” rating signifies major gaps in the assessed M&E Plan. If the Checklist is reviewed by a “third party,” that entity will also provide its own overall assessment, in light of the answers on the Checklist and its own review and verification comments.

Data Collection

Data collection have been through detailed checklists to assess each of the three elements of the M&E system. The MESST is conducted via a participatory workshop involving the PR, SRs, some representatives of the CCM, the donor community and other stakeholders or implementing partners relevant to the exercise. The proposed output for this exercise is an M&E Action Plan for the PR.

A 2-day MESST workshop was held in Tirana, Albania. Participants at the workshop assessed the M&E capacity of the national HIV program, and its implementers, the IPH and the sub recipients (SRs), by looking at three elements of the M&E system. These are:

The Monitoring and Evaluation Plan

The ability of collate and analyze data at the IPH

The reporting systems in the health facilities, the community and the laboratories

The result was identification of a range of strengths and weaknesses, and from these, the development of a list of key actions to address the weaknesses identified.

FINDINGS AND DISCUSSION

A summary of the weaknesses identified are included as an appendix (Appendix 1). Overall, participants at the workshop assessed that while the IPH does have an M&E Plan, there is incomplete documentation of the M&E system as well as a lack of understanding on the part of the SRs about their roles and responsibilities. Moreover, the M&E system has not been able to generate any baseline data for one component of its grant – namely, commercial sex workers.

Similarly, there is strong commitment among the staff to implement such activities, but they need strong technical guidance to set up the system.

The most significant weaknesses include

The IPH does not have a sufficiently established and documented M&E system. The M&E system for HIV/AIDS needs to be documented and data collection and reporting systems developed. While there is an M&E plan, detailed guidelines for SRs need to be developed. This would include providing feedback to SRs on performance and reporting. The capacity for M&E at IPH needs to be expanded.

There is no baseline data for commercial sex workers, a target population of the grant.

Measurement of quality of services provided is lacking;

There is limited capacity among SRs to collect and report performance information. SRs need training as well as more staff for M&E.

There is a need to develop mechanisms to ensure quality of data–completeness, correctness (double counting, missing data), timeliness–and ensure that the data can be verified;

In response to the full list of identified weaknesses, a number of actions were agreed. Time frame, responsibilities, and a budget for these actions are provided. It is recommended that IPH reviews its M&E Plan every two years to ensure that progress has been made to enhance its ability to monitor progress.

The Figure 1 shows that most of the health facility-based reporting systems for numbers of people reached are in place, even though that are not complete or not applicable in less than 40% of the SR-s. The same view counts also for other indicators: meaning that in most of the cases there are reporting systems in place such as for numbers of commodities distributed, number of individuals trained, number of service points supported, but there is fragmentation, meaning that there are cases that is not applicable or not existing, which leads to lack of standardized system of reporting. The same as above counts for the community based reporting systems, as shown in the Figure 2.

Figure 1

Assessment of health facility-basded reporting system.

Figure 2

Assessment of community-based reporting system.

More inspiring is the view in regard to the system strengthening reporting system with an overall more than two thirds, report that systems are in place and functioning.

There are good possibilities to setup a national M&E plan that can be functional and able to help on decision making, according to the table above, as national strategy and plan for M&E does exist, there are objectives and indicators set, although indicators should be set in all of the programs. The only problem might be the financial support to the M&E programs and also sustainability after the Global fund grant. (Figure 3).

Figure 3

Assessment of system strenghhening reporting system.

The principal weaknesses identified by the authors include:

The IPH does not have a sufficiently established and documented M&E system. The M&E system for HIV/AIDS needs to be documented and data collection and reporting systems developed. While there is an M&E plan, detailed guidelines for SRs need to be developed. This would include providing feedback to SRs on performance and reporting. The capacity for M&E at IPH needs to be expanded. (Figure 4).

Figure 4

Assesment of program/projects M/E plan.

There is no baseline data for commercial sex workers, a target population of the grant.

Measurement of quality of services provided is lacking;

There is limited capacity among SRs to collect and report performance information. SRs need training as well as more staff for M&E.

There is a need to develop mechanisms to ensure quality of data–completeness, correctness (double counting, missing data), timeliness–and ensure that the data can be verified;