Evaluation of pharmaceutical industry-led access programmes: a standardised framework.

Pharmaceutical industry-led access programmes are increasing in number and scope worldwide. We present a new standardised framework for evaluation of these programmes that includes three components: a taxonomy of 11 access programme strategies; a series of logic models, one for each strategy and a set of measurement indicators. The logic models describe pathways of potential programme impact. Concepts relevant across a broad range of strategies were prioritised for inclusion in logic models to ensure consistency and to facilitate synthesis and learning across programmes. Each concept has at least one corresponding measurement indicator with metadata that includes the definition, details on how it should be measured and recommended data sources. The framework establishes a shared language for the collection and reporting of meaningful industry-led access programme information. Broad adoption by programme developers and implementing partners in the for-profit sector and beyond could facilitate shared learning on effective strategies and best practices.

Pharmaceutical industry-led access programmes are increasing in number and scope worldwide, but a lack of evidence on program performance is hindering shared learning and accountability.

Lack of standard publicly available performance assessment tools that focus specifically on industry-led access programmes presents a key barrier to assessing programme achievements and ensuring company accountability for commitments made.

This paper presents a new framework for evaluation of industry-led access programmes.

The framework establishes standards for evaluation that can be applied across programmes and serves as a shared language for the collection and reporting of meaningful programme information.

Companies and non-profit organisations can use the framework to guide the design of their access programmes and evaluation systems for those programmes.

Broad adoption of the framework by programme developers and implementing partners including the for-profit sector could facilitate accountability and promote increased investments in effective strategies.

Introduction

Pharmaceutical industry-led access programmes are increasing in number and scope worldwide, driven by expanding commitments by private pharmaceutical companies.1 These programmes aim to address access barriers using a variety of health system approaches,2 including medicine donation,3 4 supply chain strengthening,5–7 healthcare provider training8 and community outreach.9 Detailed information on industry-led access programmes activities and impacts is often not publicly available,1 making it difficult to assess whether expanding programmes will translate into stronger health systems, increased patient access and improved population health. Lack of standardised reporting also hampers the public’s understanding of programmes and limits potential opportunities to learn about effective approaches and best practices.

More than 20 large biopharmaceutical companies, in partnership with the World Bank and the Union for International Cancer Control (UICC), recently committed to expanding their efforts to improve access for non-communicable diseases (NCD) in low-income and middle-income countries (LMICs) as part of the Access Accelerated initiative.10 11 Access Accelerated programmes go beyond a narrow focus on medicines, and aim to address the full range of health system barriers to access. As part of the initiative, members agreed to measure and publicly report on their programmes, with a focus on their individual and joint contributions to achieving the sustainable development goals (SDGs), in particular Target 3.4 related to addressing NCD.12 13

Private sector measurement of social impact has received increased interest in recent years and more than 100 relevant measurement tools exist.14–16 However, most of these tools lack a transparent methodology and many are proprietary and require payment for use.17 There are no standard publicly available performance assessment tools that focus specifically on access programmes. This presents a key barrier to assessing programme achievements and ensuring company accountability to commitments.18 There is an opportunity and a need to develop a new framework for the evaluation of industry-led access programmes.

In this paper, we present a standardised framework for the evaluation of access programmes that was developed as part of the Access Accelerated initiative, but is not limited to programmes focused on NCDs or programmes led by private pharmaceutical companies. We first describe the principles that guided the development of the framework. We then describe the three main components of the framework: a taxonomy of 11 access programme strategies; a series of logic models, one for each strategy and a set of clearly defined measurement indicators, which include measures of programme performance and achievements. We conclude with a discussion of the framework’s potential uses and future evolution.

Principles of framework development

Developing an evaluation framework requires making value-based judgements, whether explicit or implicit. For example, with respect to content, whether to prioritise medicine access or population health as the primary measure of impact requires deliberation on what should constitute the ultimate goals of programme activities. Programme designers and external stakeholders might reasonably disagree on goals; developing a standardised approach does not require choosing a side in the debate—a flexible approach is often best—but it does require careful consideration of the relevant values.

Four core principles guided the development of the framework, informing decisions on process and content. The process was guided by the principles of independence from industry, methodological rigour and transparency to the public and content decisions were guided by the principle that public health goals are of primary importance. These guiding principles are detailed below.

Independence from industry

The framework was developed with independence from the pharmaceutical industry, which is fundamental to its legitimacy. To ensure a practical approach, we consulted regularly with representatives from Access Accelerated companies, as well as the World Bank and UICC. We made all final decisions on the design of the framework. All legal contracts governing the development of the framework are publicly available.19

Methodological rigour

The framework was designed to facilitate rigorous evaluation by users. The taxonomy of programme strategies was structured to be thorough and precise. The logic models were constructed according to a standard ‘theory of change’ approach to clarify primary pathways through which strategies might achieve impact.20 21 The indicators were selected to balance high construct and content validity with the potential for high-quality data collection.

Transparency to the public

The framework was designed with a primary purpose of facilitating greater transparency in reporting on pharmaceutical industry-led access programmes. Transparency has several benefits, including generating opportunities for learning and potentially strengthening accountability. Standardising indicators facilitates transparency by establishing clear and agreed on definitions and meanings, that is, by ensuring that shared information is intelligible to the public and other audiences.18 22 While designing the framework, we elicited feedback from potential users and public stakeholders to refine the terminology used and strengthen common understanding.

Public health goals

Public health goals were prioritised during the development of framework content, consistent with the pharmaceutical industry’s commitment to the SDGs.12 The WHO has identified the ultimate goals of a health system as population health; financial risk protection and responsiveness (including patient satisfaction).23 As described below, the framework’s impact indicators align with the WHO goals; intermediate access indicators are also included as pathways to these goals.

Framework components

The framework includes three main components: (1) a taxonomy of 11 strategies that describes common approaches used by pharmaceutical industry-led access programmes; (2) a series of logic models—one per strategy—detailing the primary pathways through which programmes might achieve impact and (3) a set of clearly defined indicators for measuring programme activities and achievements.

Taxonomy of programme strategies

The core foundation of the framework is a taxonomy that can be used to categorise access programmes into 11 broad strategies (table 1). The structure of the taxonomy was derived in part from existing access and health system frameworks, including Bigdeli et al,2 WHO’s Equitable Access to Essential Medicines framework,24 and the health reform framework developed by Roberts et al.25 A preliminary list of strategies was tested and refined during a process that included a review of existing industry-led access programmes.26 By design, the taxonomy simplifies the organisation of a tremendous variety of programme approaches. This simplification allows for the application of standardised indicators specific to each strategy.

Table 1

Taxonomy of programme strategies

Type of strategy	Strategy	Definition
Community strategies: strategies with a primary focus on communities and community organisations, with a particular focus on patients	Community awareness and linkage to care	Programmes that provide communities and patients with health-related information on disease prevention and treatment, or improve links between patients and the healthcare system.
System strategies: strategies with a primary focus on aspects of the health system that affect availability and access to medicines.	Health service strengthening	Programme designed to improve the availability, affordability and quality of health services. Common activities include planning, training, infrastructure, technology, management or funding activities.
	Health service delivery	Programmes designed to deliver health services directly to patients. Common activities include screening, diagnosis, treatment and retention.
	Supply chain	Programmes designed to improve medicine supply chains, to improve availability and lower costs. Common activities include planning, training, infrastructure, technology, management or funding activities.
	Financing	Programmes designed to improve health financing systems and reduce catastrophic health expenditure. Common activities include planning, training, infrastructure, technology, management or funding activities.
	Regulation and legislation	Programmes designed to improve and harmonise pharmaceutical regulatory systems, improve government coverage of and access to treatments and/or improve in-country regulatory processes. Common activities include advocacy, training, infrastructure, technology, management or funding activities.
Production strategies: strategies with a primary focus on increasing the production of medicines.	Manufacturing	Programmes designed to build capacity for medicine production. Common activities include planning, training, infrastructure, technology or funding activities.
	Product development research	Programmes designed to support product (eg, drugs, devices) development research through planning, training, infrastructure, technology or funding activities. Relevant research areas include bench science, preclinical and clinical trials and postmarketing monitoring.
	Licensing agreements	Programmes designed to facilitate the manufacture, importation, sale or use of medicines through legally binding relationships including voluntary licensing agreements and technology sharing.
Price strategy: strategies with a primary focus on reducing the price of medicines.	Price scheme	Programmes designed to increase the affordability of medicines for individuals and healthcare systems through a change in the price via subsidies or other means (excluding donations).
	Medicine donation	Programmes designed to increase the availability and/or affordability of medicines through direct donation of medicines, and other healthcare products, to countries or health institutions or non-governmental organisations. Does not include compassionate access programmes.

The 11 strategies are organised in four broad categories: (1) community strategies; (2) health system strategies; (3) medicine production strategies and (4) medicine price strategies. Community strategies focus on communities and patients and aim to increase awareness of symptoms of disease or treatment options, or to link patients to available care. Health system strategies focus on strengthening existing health services, delivering parallel services directly to patients, strengthening medicine and product supply chains, expanding access to healthcare financing, or harmonising regulation and legislation processes. Medicine production strategies focus on increasing production of medicines by increasing manufacturing capacity, strengthening product development research, or easing legal barriers to production through licensing agreements. Finally, medicine pricing strategies aim to improve affordability through donations or price schemes. Most access programmes pursue more than one strategy, and the framework is designed to allow for this.

Logic models

Each of the 11 strategies has a corresponding logic model that describes the primary pathways of potential programme impact. The logic models were structured in accordance with standard formats, such as those recommended by the Centers for Disease Control and Prevention20 and the US Agency for International Development.21 Flowing from left to right, each model includes five columns: inputs, activities, outputs, outcomes (short-term and long-term) and impacts (see figure 1 for an example; the full set of logic models is provided in the online supplementary file). Connexions across columns indicate hypothetical if-then statements, for example, if programme outputs are distributed to intended beneficiaries then short-term outcomes will be achieved. Boxes in each column indicate concepts along the causal pathways from programme implementation to impact; each concept is operationalised with a corresponding measurement indicator as described in the next section.

Figure 1

Example logic model.

The concepts included in the logic models were not intended to be exhaustive; the diversity of programme approaches makes this impossible. The intention was rather to identify priority concepts that are likely to be relevant across a broad range of approaches, to ensure a basic level of consistency and to facilitate synthesis and learning across programs. For individual programme, developers are encouraged to use the framework logic models as a starting point, and to add additional concepts as appropriate. Wherever appropriate, concepts represented in the logic models were repeated across the 11-strategy series. Most repeated concepts have been previously identified as important in the literature on access to medicines including availability, appropriate use and accessibility.27 For example, availability of medicines at outlets has been identified as a key dimension of access22 and is included as an outcome in six of the 11 logic models. Input concepts are identical across all logic models, as are impact concepts. Inputs include the value of resources put into the programme and staff time devoted to the programme. Impacts mirror the WHO goals of a health system: population health, financial risk protection and patient satisfaction (a key aspect of responsiveness).23

Indicators

Each concept included in the logic models has at least one corresponding measurement indicator. For example, ‘population exposed’ is a concept included in the community awareness and linkage to care logic model that has a corresponding indicator, ‘population exposed to community communication activities’ with the definition, ‘number of population reached through a community awareness campaign’ (table 2). Each indicator has a table of metadata that includes the definition, details on how it should be measured and recommended data sources. When appropriate, metadata tables also include suggestions for stratification by demographic characteristics, equity dimensions (gender, geographical location and income), health sector (private, public and faith-based) or level of care (primary, secondary and tertiary). The full set of 72 indicators is organised in a data dictionary (included in the online supplementary file).

Table 2

Example indicator metadata

Item	Description
Indicator name	Population exposed to community communication activities
Indicator type	Output
Strategies that use indicator	Community awareness and linkage to care
Definition	Number of population reached through a community awareness campaign
Method of measurement	Counting of participants that attend campaign meetings or reached by media messages disseminated Calculation:Number of people/participants in the target audience segment participated/attended the community awareness campaign recorded in a given period of time
Recommended disaggregation	Disease, intervention type, target audience
Frequency of reporting	Annually unless otherwise stated
Recommended data source	Routine programme dataNon-routine programme data (eg, target audience survey)Community-based awareness events
Further info	https://www.knightfoundation.org/media/uploads/publication_pdfs/Impact-a-guide-to-Evaluating_Community_Info_Projects.pdf page 9–11

Many indicators correspond to multiple logic models, allowing for potential comparison and synthesis across programmes that employ different strategies. For example, ‘availability of medicines at outlets’ is a common indicator for supply chain strengthening, medicine donation and price scheme strategies. Indicators were selected based on several criteria, including high construct and criterion validity; feasibility of collecting high-quality data at low cost and existence of validated data collection instruments.28 For example, ‘percentage of outlets with medicine in stock’ was selected as an indicator for measuring medicine availability because its high construct and criterion validity have been shown, and because the feasibility of high quality data collection at relatively low cost has been demonstrated.29 For each indicator, we recommend that programme data be collected and reported in a manner that allows for disaggregation by relevant subgroups (eg, socioeconomic status, gender) to assess equity.

Using the framework

We developed a standardised framework for evaluation of pharmaceutical industry-led access programmes, guided by the principles of independence, rigour and transparency, and placing primary importance on public health goals. The framework establishes a shared language for the collection and reporting of meaningful programme information.

Our approach is distinct from existing social performance tools used by the pharmaceutical industry, particularly the Access to Medicine Index.30 The Index includes commercial activities in addition to social programmes in its metrics, and measures performance at the company-level rather than the programme level. Our approach is similar to a framework recently developed for evaluating alcohol industry-led social programme, which also includes a series of logic models and a set of indicators.31

We aim to facilitate shared learning and improved programme performance, and we anticipate several audiences for this work. First, the framework can be used by companies and non-profit organisations to guide the design of their access programmes and evaluation systems for those programmes. We encourage users of the framework to commit to transparency and publish their work in a public forum. We have developed a public platform for transparent reporting on access programmes, the Access Observatory, which can be used by any organisation working in this area.32 Second, the framework can be used as a conceptual model by researchers and policymakers who aim to understand and analyse these programmes.

Finally, local and global stakeholders can incorporate the framework and programme information into their processes for holding companies and implementing organisations accountable. WHO has identified key considerations that should be taken into account when assessing industry-led medicines and health technologies access programmes; among these is whether programmes have clearly defined performance targets.33 In addition, civil society and health systems researchers have called for greater clarity on programme adherence to regulations and guidelines (eg, national treatment guidelines) and on assignment of responsibilities among the various components of the programme partners for implementation activities.34 Our framework can guide countries, civil society organisations and other stakeholders in their dialogue with for-profit companies and non-profit organisations about alignment with national priorities, target setting and reporting. Establishing clear expectations among all stakeholders by defining commonly agreed on indicators and reporting mechanisms is critical for increasing transparency and accountability of for-profit companies and non-profit organisations.

In 2017, 63 pharmaceutical industry-led access programme operating in 103 LMICs applied the framework for the first time. Individual reports for each programme are available on the Access Observatory website,32 and the Access Observatory 2018 Report includes an analysis of the full set of programmes.35 Around one-third of programmes reported indicator data into the Access Observatory in the first year. For many programmes, building the systems needed for data collection and management will be a process that takes time. Looking forward, the framework can play an important role in facilitating rigorous evaluation and transparent reporting on industry-led access programme, but this will require continued engagement on the part of the pharmaceutical industry and global health stakeholders.

The framework has important limitations. First, we focused on capturing the full breadth of strategies that the pharmaceutical industry is using in their efforts to increase access. For practical purposes, to ensure that the framework is usable, capturing this breadth may have limited the depth of the logic models and data dictionary that could be possible. We encourage users of the framework to further refine our tools to best fit their purposes. Second, the framework design was informed by a review of existing industry-led programmes operating in LMICs. Our intention is that the framework is general enough to be applied to non-industry programmes, programmes in high-income countries and programmes focused on any type of health condition. However, more work may be needed to adapt the framework to this type of more general use. Finally, while we set out clear principles at the start of developing the framework, users of the framework may apply their own set of principles, which may be different from ours. We encourage all users to adopt a principle of transparency and clearly state their other principles.

We expect that the framework will evolve in several key areas in the future. First, the taxonomy of strategies will likely expand to include new and innovative programme approaches. Each new strategy will be accompanied by a new logic model and set of indicators. Second, the indicator set will likely change as it becomes clearer which data are most feasible to collect. This may entail dropping some indicators and adding others while maintaining high content validity. Third, while the framework was developed as part of Access Accelerated, the strategies, logic models and indicators are applicable to access programmes more generally, not only those led by industry or focused on NCDs. We anticipate that the components of the framework will evolve as the full diversity of non-industry programmes becomes clearer. Stakeholder commitment and resources are required to further develop the framework.

Our experience suggests that programme teams often have the capacity and resources to collect and report information on input and output indicators, but less often for outcome and impact indicators, which in most instances are more challenging and costlier to collect. As a result, we expect that for many programmes, at least early on, reported data will describe the scope of programme activities but information on programme outcomes and impacts will be limited. Since programme outcome and impact data are critical to determine whether the programme requires changes we expect that programme funders as well as beneficiaries will demand this information. As a result, rigorous impact evaluations would receive a higher priority in the future.

Many existing industry-led access programmes are pursuing similar strategies and pooling resources across programmes may make it easier to conduct resource intensive evaluations.18 In addition, the development of new methodologies that balance rigour with practicality and less costly data collection could produce increased opportunities for efficient evaluation of programme outcomes and impacts.36 The framework creates a shared language that should help in identifying potential synergies and opportunities across programmes.

Conclusions

Improving access to essential medicines is a key global health priority, identified in SDG Target 3.8. The pharmaceutical industry and other stakeholders are increasing their commitments to addressing this issue. Measurement standards will be important for understanding the impacts of these efforts, and for steering resources to strategies that are proven to work. The framework we present in this paper is the first to establish standards for evaluation that can be applied across pharmaceutical industry-led access programme. Other industries have also been recognised as key contributors to the SDGs,37 and our work may provide a roadmap for similar frameworks focused on other goals, for example environmental sustainability. In the coming years, efforts in this area will need to be expanded to increase programme efficiency, sustainability and accountability to contribute to achieving important global development goals.