Evolution of health technology assessment: best practices of the pan-Canadian Oncology Drug Review.

BACKGROUND: In 2007, Canada chose to develop a separate and distinct path for oncology drug health technology assessment (HTA). In 2013, the decision was made to transfer the pan-Canadian Oncology Drug Review (pCODR) to the Canadian Agency for Drugs and Technologies in Health (CADTH), to align the pCODR and CADTH Common Drug Review processes while building on the best practices of both. The objective of this research was to conduct an examination of the best practices established by the pCODR.
METHODS: A qualitative research approach was taken to assess the policies, processes, and practices of the pCODR, based on internationally accepted best practice "principles" in HTA, with a particular focus on stakeholder engagement. Publicly available information regarding the approach of the pCODR was used to gauge the agency's performance against these principles. In addition, stakeholder observations and real-world experiences were gathered through key informant interviews to be inclusive of perspectives from patient advocacy groups, provincial and/or cancer agency decision-makers, community and academic oncologists, industry, expert committee members, and health economists.
RESULTS: This analysis indicated that, through the pCODR, oncology stakeholders have had a voice in and have come to trust the quality and relevance of oncology HTA as a vital tool to ensure the best decisions for Canadians with cancer and their health care system. It could be expected that adoption of the principles and processes of the pCODR would bring a similar level of engagement and trust to other HTA organizations in Canada and elsewhere.
CONCLUSION: The results of this research led to recommendations for improvement and potential extrapolation of these best practices to other HTA organizations worldwide, along with suggestions for continued evolution of the pCODR in conjunction with its integration into the CADTH. It is clear that the transition of the pCODR to CADTH provides an opportunity for practices initiated by the pCODR to become the standard for these newly amalgamated HTA agencies in Canada.

Introduction

The affordability of drugs is a key concern and challenge within health care.1,2 Canada has been a leader in the use of health technology assessment (HTA) to assist in the allocation of resources, particularly drug budgets. Currently, there are two pan-national HTA agencies that evaluate drugs on behalf of participating jurisdictions, ie, the Common Drug Review (CDR) and the pan-Canadian Oncology Drug Review (pCODR). The CDR was initiated in 2003 under the governance of the Canadian Agency for Drugs and Technology in Health (CADTH), and is responsible for outpatient non-oncology drug reviews. A national process exclusive to oncology was initiated in 2007 and formalized as the pCODR in 2010.3 All provincial jurisdictions in Canada participate in both agencies, with the exception of the province of Quebec (which runs a separate, parallel HTA agency). Both organizations evaluate submissions for new drugs and/or new clinical indications based on comparative clinical and cost-effectiveness, and then generate recommendations for funding. Actual funding decisions occur at the regional (provincial) level, since health care spending is a provincial responsibility. The intent of the centralized review processes has been to apply consistent and high quality evaluation methods to generate recommendations, and harmonize funding decisions across jurisdictions to the extent possible.4

The decision to establish a separate, cancer-specific national drug review process was based on various factors, and was influenced by the semi-independent nature of cancer treatment programs in Canada. Early experiences using the CDR process for oral oncology products were not considered to have met the needs of patients and clinicians. A sophisticated oncology review process that relied on a high level of clinician expertise was developed in one province (Ontario) in 2005.

In 2007, the House of Commons Standing Committee on Health recognized four key issues in HTA for cancer drugs:

The need for a distinct process separate from the CDR that addressed oncology-specific needs

The importance of highly credible clinical oncology expertise deeply (and transparently) embedded within all parts of the process

A process that was inclusive of patients

The need for recommendations that focus on patient as well as payer needs, with a clear and transparent rationale for those recommendations.5

The development of the pCODR followed a unique path in Canada, and was composed of three essential elements: a “ground-floor up” approach to inclusiveness of stakeholders, respected leadership with a willingness to take risks, and an opportunity to build from the Ontario process and CDR operations in order to refine best practices in oncology HTA.

After several years of independent operations, the responsibility for the pCODR was transferred to the CADTH effective April 1, 2014. The degree to which the pCODR and CDR will remain separate within CADTH’s governance model is unclear. Administrative transition has been the focus of the first year of integration, alongside stakeholder consultations and a commitment to maintain current practices and procedures “as is” for both organizations during this period.6 The second phase of transition starts in April 2015, and will explore “better alignment of the pCODR and CADTH Common Drug Review (CDR) evaluation criteria, while taking advantage of the best practices of both review processes”.6

Stakeholders are a critical element of the pCODR, having been engaged in its establishment and the development of its guiding principles, and having participated at all stages of drug evaluation since inception. Throughout the transition phase, stakeholders have strongly expressed their desire to maintain the processes, standards, frameworks, and culture of collaboration of pCODR.

The decision by ministers of health to transfer responsibility to the CADTH is an ideal opportunity to review and evaluate the practices of the pCODR with a specific emphasis on stakeholder engagement. Thus, the first objective of this paper is to benchmark the processes of the pCODR in the context of an international framework for best practices in HTA, using stakeholder observations and real-world experiences. The second objective is to identify best practices and potential improvements for adoption by other worldwide HTA organizations, with recommendations for the continued evolution of the pCODR in conjunction with its integration into the CADTH.

Materials and methods

Drummond and members of the International Group for HTA advancement proposed 15 principles by which to assess existing HTA programs or to establish new ones, and these served as the assessment framework for the current investigation.7 Neumann et al8 used these 15 principles to benchmark HTA bodies across the globe; however, their assessment was based solely on information posted on agency websites. This investigation overcomes the limitation of the study by Neumann et al by integrating feedback from current pCODR stakeholders to better understand how the pCODR operationalizes its practices.

The principles proposed by Drummond et al7 are organized into four sections, ie, structure, methods, processes, and use in decision-making (Figure 1). These principles are referred to as “good practices” and are intended to improve access to clinical care that is cost-effective while enhancing the quality, credibility, and acceptability of HTA in resource allocation decisions.

Figure 1

Fifteen HTA principles.

Note: Copyright © Cambridge University Press 2008. Adapted from Drummond MF, Schwartz JS, Jonsson B, et al. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care. 2008;24(3):244–258.7

Abbreviation: HTA, health technology assessment.

The current research intentionally focuses only on the three “process” principles from the framework devised by Drummond et al7, ie, engaging with stakeholders, seeking all available data, and monitoring of implementation. Given its importance to the evolution of the pCODR, this analysis pays particular attention to the “stakeholder engagement” principle. Stakeholders are the individuals, groups, and organizations that have an interest in and are impacted by the HTA recommendations. They include public program decision-makers, health care professionals, consumer groups, patients, and industry.9 Successful stakeholder engagement can achieve many important goals, including high quality assessments that are widely accepted, an improved chance of implementation, enhanced perception of transparency, and increased trust in the process. Drummond et al have recently indicated that, as a minimum, HTA should reconcile the needs of payers, manufacturers, and patients to ensure that relevant recommendations are generated.10

Drummond et al have also proposed specific criteria for measuring HTA organizations against these principles. The criteria for the process principles, including stakeholder engagement, are listed in Table 1. We reviewed these criteria against the actual processes followed by the pCODR to benchmark against these principles, using information publicly available from the pCODR website11–13 and information generated from stakeholder interviews.

Table 1

How does the pCODR meet the process principles

HTA process14	pCODR Process (as described on their website)11–13
Principle 10: Actively engaging all key stakeholder groups
pCODR formally required to engage stakeholders	• Patients• Manufacturers• Clinical experts (disease site experts involved in the clinical assessment)• Health economics experts• Provincial payer representative (provincial advisory group)
Stakeholders involved in scoping of HTA	There is no scoping process at the pCODR. Manufacturers have the opportunity to engage before submission through presubmission meetings that consist of 1–1.5-hour meetings per submission; maximum of one meeting per manufacturer per 6 months, which needs to be formally requested by the manufacturer. There is no comparable process to the NICE scientific advice consultation service that provides guidance for drug development. There is also nothing comparable to the NICE scoping process. The purpose of the pCODR presubmission meeting is to provide an opportunity for the submitter to introduce the new drug and the approach to the economic evaluation, to help the pCODR to plan its workload, and to help provinces plan demand for new products.
Mechanism for identifying relevant stakeholders	A mechanism is in place to form transitional expert panels if the eleven permanent tumor site panels do not cover a submitted drug.
pCODR encourages or requires submission of evidence from stakeholders	A drug manufacturer and/or a provincially recognized clinician-based tumor group may make a submission. As part of the pCODR deliberative framework, when making a funding recommendation, the pERC must consider patient values and consider the potential impact and feasibility of adopting the recommendation. Patient advocacy group and PAG inputs are solicited for each submission to inform those components. Patient advocacy group input is used by reviewer panels and in pERC deliberations. How patient input contributed to the reason for recommendation is published. Input of manufacturers to clarify clinical and economic information and review nondisclosable information during the review process is done through checkpoint meeting.
pCODR allows stakeholders to comment on reports at the draft stage	Manufacturers can review the clinical and economic guidance reports only, to identify factual errors and confidential information just prior to the pERC meeting. All stakeholders can comment on the pERC initial recommendation within 10 working days. The feedback from manufacturers, patients, and the PAG on the initial recommendation and reasons is posted on the pCODR website.
pCODR allows stakeholders to appeal against recommendations/decisions	The submitter and/or patient advocacy group have 10 working days to make a request for reconsideration after publication of the initial recommendation. No new data can be submitted, only arguments. If the submitter, PAG, and patient advocacy groups approve of the initial recommendation, rapid conversion to a final recommendation occurs. If objections are raised by any of these groups, the dossier will be reassessed at the next pERC meeting. The same reviewers will be involved. The opportunity for rapid conversion is lost.
pCODR committees include stakeholder representation	Clinicians and patients are represented on the pERC. Provincial payers are represented by the PAG. All stakeholder groups are represented except the manufacturers.
Principle 11: Actively seeking all available data
Systematic review of clinical evidence includes gray literature and unpublished data	Yes, but the weight given to the evidence will vary according to the source of information. For example, an unpublished clinical trial available only in abstract or poster format will be given less weight than a clinical trial published in a peer-reviewed journal.
pCODR has a process to handle confidential data from the manufacturer	Confidentiality maintained in the published “Recommendation and reasons” and all other published reports (clinical and economic). Confidential pricing respected at the provincial level.Disclosure guidelines provide clear direction on how confidential information will be managed.Note: incremental cost-effectiveness ratios are not considered to be confidential even if associated with confidential price.
Principle 12: Monitoring of HTA findings
pCODR develops an implementation plan for its recommendations	The pERC recommendation includes advice related to implementation in response to issues raised by PAG in the latter’s assessment of adoption feasibility.
pCODR monitors the impact of its recommendations	The pCODR tracks and reports provincial funding decisions and criteria on its website. There are additional steps between pERC’s recommendation and pCODR’s notification to implement and the ultimate provincial funding decision. These include the negotiations between the manufacturer and the PCPA and any provincial decision-making processes that must be followed.

Notes: Copyright © Cambridge University Press 2008. Adapted from Drummond M, Neumann P, Jonsson B, et al. Can we reliably benchmark health technology assessment organizations? Int J Technol Assess Health Care. 2012;28(2):159–165.14 Adapted from: CADTH pan-Canadian Oncology Drug Review. How pCODR works. Process in brief [webpage on the Internet]. Ottawa: CADTH; 2015. Available from: https://www.cadth.ca/pcodr/process-in-brief.11 CADTH pan-Canadian Oncology Drug Review. About pCODR. Advisory Bodies [webpage on the Internet]. Ottawa: CADTH; 2015. Available from: https://www.cadth.ca/pcodr/advisory-bodies.12 CADTH pan-Canadian Oncology Drug Review. Transparency. Find a review [webpage on the Internet]. Ottawa: CADTH; 2015. Available from: https://www.cadth.ca/pcodr/find-a-review.13

Abbreviations: PCPA, pan-Canadian Pharmaceutical Alliance; HTA, health technology assessment; pCODR, pan-Canadian Oncology Drug Review; pERC, pCODR Expert Review Committee; PAG, Provincial Advisory Group; NICE, National Institute for Health and Care Excellence.

Stakeholder input was obtained using an interview guide developed to elicit comprehensive and consistent feedback on implementation of the process principles of the pCODR and their perceived value. A series of eight 1-hour stakeholder interviews were convened from a convenience sample of individuals in the following groups: patient advocacy groups (one), provincial and/or cancer agency decision-makers (two), community oncologists (one), academic oncologists (one), industry (one), expert committee members (one), and health economists (one). All interviewees reviewed and approved the draft manuscript to ensure accurate representation of their views.

Interviews were conducted by telephone with two authors participating in each interview (one to question, both to scribe). Interviews were not recorded but two sets of manual notes were prepared and cross-referenced against each other by both participating authors. Verbal consent was requested and provided for each interview. Consent was also sought for acknowledgement of stakeholder participants. Since not all stakeholders consented to formal acknowledgement, none of the participating stakeholders were named.

Results

The results are presented in two sections, ie, a summary of how the pCODR meets the process principles based on the framework of Drummond et al7 and stakeholder insights related to these principles.

Assessment of the Drummond framework

Table 1 provides an overall summary of the practices of the pCODR framed according to the optimal approaches espoused by the principles of Drummond et al.7 The engagement practices of the pCODR are derived from its governance, evaluation process, and deliberative framework, as outlined in the “Supplementary materials” section.

According to Drummond et al,7 all stakeholders affected should be given meaningful opportunities to constructively engage in the process, well beyond the simple provision of formal written comments during a drug review. “Meaningful” is an important adjective, and is illustrated by the ability of stakeholders to comment on draft reports. This opportunity is commonly provided but may not be meaningful if turnaround timelines are short or if there is no formal response to the issues raised.

Beyond the roles described in Table 1, there are several stages during which stakeholders are directly involved in the pCODR process. First, prior to submission, pCODR meets with the submitter (manufacturer or tumor group) to prepare for the process. Before the review is initiated, input is obtained from the Provincial Advisory Group (PAG) regarding the feasibility of adopting the new drug/indication. Patient advocacy groups are also invited to provide relevant information.

After submission, the clinical and economic guidance panels review their respective portions of the submission. A formal checkpoint meeting occurs during the review process for the reviewers to seek clarity directly from the submitter.

The pCODR Expert Review Committee (pERC)1 receives and reviews all the information available for the file (including unedited input from patients and the PAG, as well as panel reports based on the manufacturer’s submission). Of note is that during the pERC meeting, the clinical and economic panel leads are available to answer questions from any pERC member. The pERC uses all the available information to comprehensively and thoroughly address the four domains of the deliberative framework (Supplementary materials). Members of the pERC are largely active oncologists, in addition to two oncology pharmacists, two health economists, and two patient representatives. Generally, members recruited have supplemental expertise in such domains as ethics, clinical trial design, health care administration, and development of practice guidelines.

An initial recommendation is drafted and posted online, with an invitation for feedback from parties engaged in the process (specifically, the manufacturer, registered patient groups, PAG, and clinicians [if a tumor group submission]). After review of the input, if all parties are in agreement with the initial recommendation, it is converted to a final recommendation (and posted on the website). Otherwise, the submission is returned to the pERC for further consideration based on the feedback received.

To meet the objective of achieving a high level of transparency, the initial and final recommendations, plus all feedback on the initial recommendation, as well as the clinical and economic reviewers’ reports, are publicly posted. In addition, the pCODR tracks and publishes the provincial implementation of its recommendations, although this is not a formal step in the HTA process.

Stakeholder findings

In general, the interviewed stakeholders expressed an overall high level of satisfaction with pCODR engagement, alongside suggestions for improvement. There were several specific domains of pCODR practices that facilitated engagement, and which could be implemented across agencies. These are summarized in Figure 2, and are described below.

Figure 2

Recommendations from stakeholder interviews.

Abbreviations: CADTH, Canadian Agency for Drugs and Technologies in Health; HTA, health technology assessment; pCODR, pan-Canadian Oncology Drug Review; pERC, pCODR Expert Review Committee.

Deliberative framework

Stakeholders acknowledged a range of benefits associated with the use of the deliberative framework11 of the pCODR. For instance, recommendations are written with a consistent approach, which makes it easier to understand any given recommendation in the context of the entire body of recommendations. There are four quadrants in the deliberative framework: clinical benefit, alignment with patient values, cost-effectiveness and feasibility of adoption. The four quadrants ensure comprehensive consideration of input from all stakeholders: patient input, PAG input, clinical value, and economic value. Each quadrant of the recommendation framework explains both the “what” and the “so what” – in other words, a quantitative description of the findings and some qualitative assessment of meaning and relevance for the results. Consideration of implementation issues throughout the review process enhances the likelihood that the recommendation can be and may be implemented within the existing health care system. Consideration of patient issues with the same level of gravitas as the others promotes a sense of trust that the patient voice is not mere tokenism in the process.

Expertise

High-level clinical expertise is deeply embedded in the process through the clinical guidance panels and the pERC members themselves. This promotes the perception that recommendations are only developed after careful evaluation of the evidence by knowledgeable experts in oncology. It also facilitates confidence among all stakeholders that there is a high degree of accuracy and relevance in the recommendations.

Reviewers remain anonymous to the submitter, but a close level of interaction can occur at checkpoint meetings (through chat rooms) to improve understanding of the submission and relevant issues. However, it was reported that the checkpoint meetings could be more efficient if done in a more direct manner, and that efficiency should be weighed against the need to protect the anonymity of the reviewers.

A pragmatic approach is used in balancing conflicts of interest with the need for high levels of expertise. Thus, pERC members can be actively engaged in practice and research and have currency of knowledge and issues, without necessarily being excluded from sharing their insights and expertise in the deliberations (although they may be excused from voting). This is perceived as a superior approach to the selection of strictly conflict-free expert panelists and reviewers, who may lack current clinical context and/or awareness of patient-centered issues or trial design challenges.

Some have suggested that there is perhaps greater opportunity in an HTA body exclusive to a single therapeutic area to have relevant expertise within the recommendation-making body. It should be possible, however, to import high quality expertise into a group as needed for HTA bodies with a broader therapeutic mandate.

Economic expertise is available through the economic guidance panel, via their reviews and their availability during pERC meetings. High-level economic expertise is also reflected in the pERC membership, and it is encouraging to note that additional economic expertise in currently being recruited to the committee.

Input opportunities

Stakeholders in the pCODR do not have the full range of input opportunities suggested by Drummond et al7 (specifically, the opportunity to scope projects). However, they do have repeated opportunities for input prior to the submission (submitter only), at the time of submission (patients, PAG, clinical reviewers, economist reviewers), during the review (submitter only), during the deliberations (patients, clinicians, PAG), and at the time of the initial recommendation (all). Moreover, these opportunities are perceived to be meaningful, with features that ensure due consideration of input.

The degree of transparency with respect to the information and input provided to the pERC is also important. For example, the full uncensored patient input is provided to pERC members for consideration, and patient values are one of the four quadrants considered by the pERC in their discussions. By comparison, the CDR provides its expert panel with edited input from patient groups and edited external clinical expert input (if such input has been solicited, at the discretion of the CDR). As a result, pCODR submitters have clarity on the interpretations of reviewers. Submitters do not, however, have an opportunity to review economic models that have been revised by the economic reviewers, which may generate some misunderstanding or confusion about those revisions.

The pCODR has demonstrated its commitment to enabling patient input at the highest level by the development of a document to guide patient advocacy groups and by conducting training sessions to ensure patients are comfortable with the process and understand how best to provide input. All interviewed stakeholders praised the process for its inclusiveness of patients and the value inherent in such input. One limitation is that advocacy groups with greater than 50% funding from a single source are not permitted to submit input. Typically, the single source may be a manufacturer. Given the dearth of private or public funding sources for patient groups, this can be a challenging requirement. pCODR has encouraged such patient groups to collaborate with larger organizations that have more balanced funding streams to overcome this issue.

Stakeholders are given the opportunity to provide meaningful input to questions of process as well. Stakeholders were invited to provide feedback during establishment of the pCODR, after the initial phase of transition to the CADTH, and at points in between where process changes were proposed. Critically, as has always been the tradition of the pCODR, stakeholder input was made public so that observers could determine how input was or was not incorporated into proposed changes. This provides an ongoing validation for the stakeholders that their input is being “heard” and encourages further interactions.

Transparency and access

Transparency with respect to the process and the recommendation is evident at the pCODR through multiple mechanisms. Recommendations are written to ensure understanding of the decision-making thought process. By posting both the initial and final recommendations, as well as the feedback from all parties, the presence and extent of contribution of the different parties is evident. Posting the clinical and economic reviews at the time of the initial recommendation allows all interested parties to see what evidence was used in the deliberations and how it was used.

Currently, with the exception of confidential prices, any information that directly contributes to the reason for the recommendation must be disclosed in the posted recommendation. The intent is for readers to be able to understand clearly the rationale for the recommendations of the pERC. Where this information is as yet unpublished, manufacturers may choose not to provide it to the pERC to avoid potentially jeopardizing peer-reviewed publication through premature release in the pCODR recommendation. Of note, pCODR has developed mechanisms to allow for the consideration of such information if it is part of a company’s publication plan.11 There continues to be a gap, however, when it comes to information that may be helpful to the pERC but is not amenable to public disclosure.

Unfortunately, the pCODR process does not allow for submission of emerging evidence once a review is underway. The only option is to withdraw the submission and restart the whole process. This would create an additional time delay to access.

Relevance

Overall, stakeholders value the timeliness of the recommendations of the pCODR as an important means to relevance. Timeliness reduces uncertainty, by facilitating access (if it occurs) or removing doubt (if it does not occur). Implementation of recommendations by the provinces is tracked by the pCODR to identify if outputs are informative to drug funding decisions. Members of the PAG are responsible for reporting back to the pCODR on the status of recommendations. This iterative loop serves to reflect the feasibility of pCODR recommendations and the accountability of the provinces to uphold their share of the process, as any delays at the provincial level (or changes from the original pCODR recommendation) are readily observable to all stakeholders.

Discussion

This research examined the specific features and characteristics of the pCODR process in terms of stakeholder engagement, information sourcing, and implementation of recommendations. It is evident that the pCODR complies well with the various audit criteria proposed by Drummond et al.14 The pCODR has achieved its commitment to deliver on a set of core principles established by provincial governments and cancer agencies. At the same time, the pCODR is focused on meeting the needs of patients and the Canadian public.

Canada developed a specific HTA framework for oncology drugs, separate from other therapeutic areas. Relying on highly expert clinical panels, including patient values in its deliberative framework, fostering opportunities for dialog with the manufacturer throughout the review process, and tracking the implementation of its recommendations at the provincial level are key best practices of the pCODR. Alongside these successes, opportunities for improvement were also identified.

The patient input process is one of the pillars contributing to the credibility that the pCODR has with its stakeholders. The process itself was developed in collaboration with oncology patient advocacy groups, and there is ongoing commitment to supporting education of patient groups to improve their input into the process. In a 2012 independent evaluation contracted by the CADTH to assess patient input processes, patient engagement process of the pCODR was judged to be on par with the National Institute for Health and Care Excellence and to be “significantly more evolved … on several dimensions”15 than other Canadian HTA bodies. The National Institute for Health and Care Excellence does offer additional opportunities for patient interaction, including input into scoping exercises and presentations along with observation of expert committee deliberations.

Payer and/or cancer agency engagement is mandated throughout the pCODR process (through the input from PAG), ensuring the practical relevance and feasibility of implementing its recommendations. Payers/cancer agencies consistently remark that recommendations from pCODR are timely, implementable, and reflective of the questions and issues raised during the review process. There have been no instances of supplemental requests for advice regarding pCODR recommendations by payers/agencies, such as those observed with other HTA agencies.16 This indicates that final recommendations are feasible for implementation. However, some provinces take longer than others to process the implementation, occasionally with funding criteria different from those proposed by the pCODR.17

Clinicians are the key drivers in the review of clinical evidence, which is an important differentiating feature of the pCODR process. Beyond the technical and scientific expertise required to interpret cancer-related data and the challenges of cancer treatment trials, understanding the context of the disease, how patients present, expected outcomes, and challenges with currently available treatments are all critical to generating a recommendation relevant to patients, clinicians, and policy-makers. The pCODR practice of making public the names of those clinicians involved in product reviews contributes to process transparency, credibility, and stakeholder trust in the process overall. However, in addition to academic specialists, community oncologists could be sought out and intentionally included in the process (in a manner similar to the deliberate outreach required to ensure patient input).

Manufacturers have expressed that their role is respected and their input is valued at multiple levels with the pCODR. Manufacturers can be dismissed by some agencies with a perception that they lack objectivity and necessarily introduce a self-serving bias in their interactions. In fact, all stakeholders are by definition representative of a unique and non-generalizable viewpoint. A culture of suspicion does not advance the needs of an HTA agency; a culture of respectful collaboration can yield the best opportunity for quality evidence and quality decision-making that meets reasonable expectations for all parties. Potential areas for improvement in the pCODR would be a process to consider emerging new data during the review and providing an early opportunity for dialog with the manufacturer around data needs, gaps, and, more importantly, solutions.

Another area for improvement would be to work more collaboratively with the health economic research community to optimize the quality and utility of economic information needed by the decision-makers. The recent move to enhance the level of health economic expertise at the pERC decision-making table is a positive development in this direction.

It is worthwhile to consider whether the best practices of the pCODR can be maintained and/or extrapolated across therapeutic areas. The focus on a single (albeit broad) clinical specialty permits a high degree of awareness and access within a relatively homogenous and small community (at least, compared with the rest of the health care system). In a sense, pCODR has been an incubator for best practices in HTA in Canada, ie, learning from preceding agencies, offering a fresh start with respected leadership, and fine-tuning within a receptive environment. The challenge will be to maintain best practices in an evolving governance structure, and to extrapolate these practices across other Canadian HTA agencies.

Irrespective of therapeutic area, the best practices of the pCODR are rooted in its collaborative culture. The steering committee guiding the integration of the pCODR within the CADTH has recognized that the specific guiding principles of the pCODR establish its culture and, as such, it has decided that they will be adopted across both agencies. This decision is fundamental to the potential preservation and extension of the best practices initiated by the pCODR.18

In terms of other similar research, Neumann et al8 examined the extent to which the 15 key principles proposed by Drummond et al7 were used by 14 HTA agencies worldwide, including Canada (CADTH), Australia, UK, and Germany. The extent to which HTA organizations supported each of the 15 principles was based on written guidelines indicating that the agency embraces the principle, regardless of whether it actually followed it. The organizations were considered to have implemented the principle if published reports and decisions demonstrated adoption of the specific principle. The study highlighted differences in the uptake of key principles for HTA, with a trend toward global adoption of the general concepts (eg, HTA should be unbiased and transparent), but very little support and execution of the more resource-intense principles (eg, monitoring the implementation of HTA recommendations).

Neumann et al8 recognized the limitations of their evaluation and published it as a commentary. They stated that their work does not constitute a formal benchmarking exercise, which would require additional steps.8 In a benchmarking exercise, more details should be provided about how each principle was defined and what was the range of potential values.14 Researchers in each HTA country could consider developing a local, in-depth assessment of how their own agencies benchmark against universal criteria, such as has been attempted in this paper.

Conclusion

The transition of the pCODR to the CADTH is an opportunity for practices initiated by the pCODR to become the standard for the newly amalgamated HTA agencies in Canada. The strong organizational culture of the pCODR and its practice of genuine engagement reflect a deliberate and unique decision to use collaboration to achieve HTA recommendations that are perceived to be fair, consistent, high quality, and feasible. The processes and framework of the pCODR reflect the value that is placed on integrating knowledge and perspectives across all pCODR stakeholders, ie, patients, clinicians, economists, manufacturers, and payers. Through the pCODR, oncology stakeholders have had a voice and have come to trust in the quality and relevance of oncology HTA as a vital tool in ensuring the best decisions for Canadians with cancer and their health care system. Where possible, adoption of the principles and processes of the pCODR within a broader environment would bring a similar level of engagement and trust to other HTA agencies worldwide.

Supplementary materials

pCODR governance, evaluation, and deliberative framework1–3

Prior to the integration of the pan-Canadian Oncology Drug Review (pCODR) into the Canadian Agency for Drugs and Technologies in Health (CADTH) in April 2014, the pCODR steering committee was its governing body. It provided the original leadership and strategic oversight for the development and management of the pCODR and was accountable to the Conference of Provincial and Territorial Deputy Ministers of Health. With the transition to the CADTH, the steering committee has largely transformed into the pCODR advisory committee (PAC), with accountability to the CADTH board of directors. The PAC includes senior appointees who represent government and cancer agencies across the country.

The pCODR secretariat is made up by an executive director and staff. The executive director is responsible for the leadership, development, and delivery of the pCODR. The pCODR staff are responsible for the administrative tasks of the pCODR process. There is also provincial/territorial oversight at the operational level, in the form of the Provincial Advisory Group (PAG). The PAG is composed of appointed representatives from each of the participating ministries of health and provincial cancer agencies.

The pCODR Expert Review Committee (pERC) is responsible for assessing the clinical value and cost-effectiveness of cancer drugs submitted for reimbursement. Submissions can arise from a drug manufacturer and/or a provincially recognized clinician-based tumor group from the participating provinces or territories. The pERC may also receive submissions or requests for advice from the PAC or the steering committee.

All pERC members (n=16 voting members, including 14 professionals and two patients) have experience and understanding of issues related to cancer diagnosis, treatment, and care. Professional members have a background in medicine, pharmacy, pharmacology, or health economics, and they comply with the conflict of interest and confidentiality requirements and the code of conduct of the pCODR. An alternate patient member is available to ensure that there are always two patient members available to take part in an assessment.

For each review, a clinical guidance panel generates the clinical guidance document that is submitted to the pERC for the deliberative process. There are eleven tumor-site specific clinical guidance panels to ensure that the review of each cancer drug draws from the most important, relevant, and current clinical information. The pCODR can also form transitional expert panels for the duration of a review if the drug submitted is for a tumor not covered by the eleven permanent panels. Each guidance panel consists of three to five cancer specialists, recognized as experts in the specific tumor site.

The economic guidance reports used in pERC deliberations are developed by the pCODR economic guidance panels. The composition of each economic guidance panel is tailored to the requirements of the drug submission under consideration. For each review, pCODR selects from a roster of panel members with expertise in health economics or clinical epidemiology. All panel members must have experience in applied health technology assessment. Panel members are selected by the pCODR executive director and the chair of the pERC.

To ensure consistency and transparency of the review process, pERC follows a well-defined deliberative framework that describes four elements to be considered by pERC to formulate a funding recommendation. These elements are:

Overall clinical benefit (based on effectiveness, safety, burden of illness, and patient need)

Alignment with patient values (based on patient values)

Cost-effectiveness (based on the net cost and efficiency of the drug [plus the companion technology where applicable] compared with alternative drug and non-drug therapies)

Feasibility of adoption into the health system (based on the net budget impact of the new drug on other drug and health system spending, including companion technology and organizational feasibility).

Recommendations describe the drug under review using each of these four elements. Recommendations are provided to the provincial or territorial ministries of health and provincial cancer agencies, along with the reasons why the recommendation has been made. These documents are also available to the public.