Introduction of new technologies and decision making processes: a framework to adapt a Local Health Technology Decision Support Program for other local settings.

PURPOSE: Introducing new health technologies, including medical devices, into a local setting in a safe, effective, and transparent manner is a complex process, involving many disciplines and players within an organization. Decision making should be systematic, consistent, and transparent. It should involve translating and integrating scientific evidence, such as health technology assessment (HTA) reports, with context-sensitive evidence to develop recommendations on whether and under what conditions a new technology will be introduced. However, the development of a program to support such decision making can require considerable time and resources. An alternative is to adapt a preexisting program to the new setting.
MATERIALS AND METHODS: We describe a framework for adapting the Local HTA Decision Support Program, originally developed by the Department of Surgery and Surgical Services (Calgary, AB, Canada), for use by other departments. The framework consists of six steps: 1) development of a program review and adaptation manual, 2) education and readiness assessment of interested departments, 3) evaluation of the program by individual departments, 4) joint evaluation via retreats, 5) synthesis of feedback and program revision, and 6) evaluation of the adaptation process.
RESULTS: Nine departments revised the Local HTA Decision Support Program and expressed strong satisfaction with the adaptation process. Key elements for success were identified.
CONCLUSION: Adaptation of a preexisting program may reduce duplication of effort, save resources, raise the health care providers' awareness of HTA, and foster constructive stakeholder engagement, which enhances the legitimacy of evidence-informed recommendations for introducing new health technologies. We encourage others to use this framework for program adaptation and to report their experiences.

Introduction

In an era of finite resources and ever-increasing medical possibilities, every health care system faces challenges in determining which new health technologies, including medical devices, should be introduced into clinical practice.1 While technology can improve safety2 and have other benefits, it can also bring new risks3–5 and contribute to the increasing cost of care.6,7 Ideally, health organizations should have a systematic process both to gather relevant scientific information about a technology’s safety and effectiveness and to decide whether the technology is suitable for the local setting. However, several studies have concluded that the decision-making process for the adoption of new technologies could be improved at the institutional level.8–10

The International Network of Agencies for Health Technology Assessment defines health technology assessment (HTA) as:

[…] the systematic evaluation of properties, effects, and/or impacts of health care technology. It may address the direct, intended consequences of technologies as well as their indirect, unintended consequences. Its main purpose is to inform technology-related policymaking in health care.11

HTA reports from various international, national, and provincial agencies provide comprehensive, objective, evidence-informed analyses about the safety, clinical effectiveness, cost-effectiveness, and the broader impact of health technologies, including devices, drugs, and procedures. Despite the exponential growth in the numbers and the types of HTA reports over the past decade and their documented impact on health care,12–15 it has been observed that recommendations are not put into practice as often as their envisioned potential.16–18 There are several possible reasons for this. First, local decision makers may be unaware of the wealth of HTA information available to them.19,20 Second, external HTA agencies may not be able to consider operational factors that are critical for local decision-making, such as local needs, financial impact, and the presence of local alternatives, trained personnel, or sufficient resources. Third, health care organizations may lack a systematic process by which to integrate and translate context-free HTA reports with context-sensitive considerations, particularly as decision making on technology adoption at the local level is a complex process, involving many disciplines and players within an organization.21–23 Indeed, it has been shown that a key determinant of successful HTA uptake is a clear, fair, and consistent decision-making process for the approval and introduction of new health technologies.24

To address this issue, the Department of Surgery and Surgical Services in the former Calgary Health Region (Calgary, AB, Canada) developed and implemented a systematic decision-making process to introduce new health technologies called the Local HTA Decision Support Program, which comprises sets of forms and tools.25 The forms gather context-free, scientific evidence about the technology, such as HTA reports, and context-sensitive information about local needs and constraints. The tools provide decision guides based on explicit criteria26 to assist an interdisciplinary advisory committee in translating the evidence to make a recommendation to the surgical executive committee, who then makes the final decision on whether and under what conditions the technology will be used. Over a five-year period, 68 technology requests were reviewed using this program. As well as producing “yes” or “no” decisions on some technologies, the Local HTA Decision Support Program gave other technologies restricted approval, with full approval contingent on satisfying clinical outcomes reporting, training protocol development, or funding.25 Thus, the program provides a strong link between HTA, outcomes measures and research, and improvement in the quality of care.

We have observed considerable interest on the part of health care organizations in developing a similar program to combine the evidence of HTA reports with local operational considerations to produce technology adoption decisions for their own needs. However, our experience has shown that this requires considerable time and resources. An alternative would be to adapt a preexisting program to the needs of the new setting rather than developing one de novo. The present project was launched to develop a framework and tools for adapting the Local HTA Decision Support Program for use by other departments in the Calgary Health Region. The operation of the program itself has been described in detail elsewhere.25 The purpose of this article is to describe the adaptation framework and tools, identify the key elements required for successful participation, and describe the lessons learned to support others who wish to use this framework for adapting our Local HTA Decision Support Program, or a similar one, for their local needs.

Materials and methods

Setting and overview

At the time of this project, the Calgary Health Region was an integrated, primarily urban health authority in Alberta, Canada. It provided services across the continuum of care, from community health services to acute tertiary care, and served about one million people. Its administrative structure comprised 14 major Regional Clinical Departments, which worked within a portfolio structure. Each portfolio was overseen by an executive director (administrator) and a medical director (physician), who worked with the senior management team in the region. Since then, the Calgary Health Region has been restructured as part of a single provincial health services organization (Alberta Health Services). The Department of Surgery and Surgical Services in the Calgary area comprises 286 surgeons in 14 divisions.

Following an extensive HTA education initiative, there was a desire to see the Local HTA Decision Support Program originally developed by the Department of Surgery and Surgical Services adopted by other departments within the Calgary Health Region.19,27 With strong support by senior management, the present project was launched to develop a framework and tools for adapting the Local HTA Decision Support Program for use by other departments in the region.

Adaptation of the Local HTA Decision Support Program involved six steps organized into three main phases: set-up, review and adaptation, and finalization. Each phase included a set of objectives, tasks, and documents (Table 1) modified from the ADAPTE framework.28

Table 1

Overview of the program review and adaptation cycle

Phases	Steps	Tasks and documents
Set-up	1. Develop a Local HTA Decision Support Program: Review and Adaptation Manual and plan project	• Research team develops Local HTA Decision Support Program: Review and Adaptation Manual• Research team and HTA experts hold a half-day planning retreat
	2. Identify and educate candidate departments	• Research team conducts semi-structured interviews and meetings with selected department executives• Research team selects candidate departments using the Assessment of Readiness Tool (Table 3)
Review and adaptation	3. Individual departmental review of the Local HTA Decision Support Program by departments	• Research team holds one-on-one education sessions with appointed HTA leaders• Each department reviews the program and collects feedback from members
	4. Joint departmental review of the Local HTA Decision Support Program via retreats	• Joint departmental retreats held• Feedback collected via the Points to Consider Questionnaire (Table 4)
Finalization	5. Revise the Local HTA Decision Support Program	• Research team compiles and synthesizes feedback• Research team revises the Local HTA Decision Support Program
	6. Evaluate the review and adaptation cycle	• Research team analyzes Project Evaluation Questionnaire

Abbreviation: HTA, health technology assessment.

Set-up phase

The set-up phase consisted of tasks to be completed before the adaptation process. Step 1 was the development of a Local Health Technology Assessment Decision Support Program: Review and Adaptation Manual,29 which contained a description of the Local HTA Decision Support Program, HTA-related reference materials for education purposes (for example, Table 2), a project time line, and tips on program administration and evaluation.

Table 2

Excerpt from the Local Health Technology Assessment Decision Support Program Review and Adaptation Manual

	HTA producers	HTA users
Who?	Large government agencies, universities, not-for-profit companies, some health service delivery organizations	Health service delivery organizations, hospital units, health care providers
Major outcome	Production of an assessment report and recommendationsContext-insensitive	Adoption of a technologyContext-sensitive
What?
Topic selection	Proactive determination of national health needs and technologies with wide potential impact	Driven by local needs
Clinical evidence	Comprehensive synthesis of high-quality primary literature, systematic reviews, clinical trials	Information from HTA reports and scientific literature plus local experience and expert recommendations
Economic evidence	Theoretical cost analysis	Cost analysis reports plus local budget, staff, compatibility, and organizational issues
Societal evidence	Ethical, regulatory	Ethics and regulatory issues plus local access issues, local values, and priorities
Recommendation	Evaluation of the technology	Decision to purchase and implement the technology

Notes: Comparison of HTA producers with HTA users. Copyright © 2007. Reproduced with permission of Alberta Health Services. Local Health Technology Assessment Decision Support Program: Review and Adaptation Manual.29

Abbreviation: HTA, health technology assessment.

Step 2 involved identification, education, and selection of departments ready and willing to review and adapt the program. In addition to Surgery and Surgical Services, we approached ten other departments within the Calgary Health Region – seven clinical departments and three administrative departments. We scheduled semi-structured interviews and meetings with each department’s clinical and administrative heads and their executive committees to introduce the project, gather initial feedback, and gain approval for their participation in the adaptation process. Each department’s readiness for the project was determined using an Assessment of Readiness Tool (Table 3). Outcomes required to proceed included the department’s desire to change its manner of introducing new health technologies and a commitment of resources and personnel, namely the appointment of the local HTA leaders (a physician and administrator team) to oversee the review and adaptation phase. We then met with these leaders to ensure they understood the Local HTA Decision Support Program, the material in the Review and Adaptation Manual,29 and what would be required for the review and adaptation process.

Table 3

Assessment of Readiness Tool

Required meetings	Information presented	Outcomes required to proceed
1. Clinical and administrative heads	• Local HTA Decision Support Program overview• Goals of project• Conditions for participation	• Desire for change• Approval to present to departmental executive committee• Commitment of resources and personnel in principle
2. Executive committee	• Local HTA Decision Support Program overview• Goals of project• Resources required to carry out project	• Desire for change• Approval to proceed• Commitment of resources and personnel
3. HTA physician and administrative leaders	• Local HTA Decision Support Program overview• Program Review and Adaptation Manual	• Commitment to project• Ready to review

Note: The Assessment of Readiness Tool is used to identify departments that are ready and willing to evaluate the Local HTA Decision Support Program.

Abbreviation: HTA, health technology assessment.

Review and adaptation phase

The review and adaptation phase involved independent and joint reviews of the source program by participating departments. For Step 3, each department’s appointed local HTA leaders were charged with conducting an independent review with their membership either in principle by evaluating the structure, forms, and processes of the program or in practice by evaluating actual requests for new technologies from their members. We provided support to these local HTA leaders as needed. Each participating department was asked to provide written feedback using the Points to Consider Questionnaire (Table 4).

Table 4

Points to Consider Questionnaire essential elements

Process rationale and objectives (policy) – why?

• What are the reasons for utilizing a Local HTA Decision Support Program?

• What are the goals and objectives?

Process flow – how?

Prescreening

• When should the local HTA process be used?

• Does the program’s screening guide adequately assist in determining when a local HTA is needed?

Process flow

• How can the process for the introduction of health technology be simplified and made timelier?

Decision making

• How should a decision for technology adoption or purchase be made (ie, consensus, criteria matrix, checklist)?

Coordination and reporting

• How should decisions made at the local level be communicated, integrated, and coordinated with other higher level technology introduction processes within the health care system?

Process structure – who?

Stakeholder involvement

• Who should decide which technologies need an assessment?

Decision makers

• Who (which committees) should approve the adoption and purchase of new technology?

• Who gets the benefit of saving money?

Administrative support

• What type of administrative support is most required to administer and coordinate the program (for example, literature searches, systematic reviews, costing information)?

• Who should provide administrative support?

Education

• What type of educational programs would support the Local HTA Decision Support Program?

Process content – what?

Standardization and collaboration among departments

• Should a common process be developed among departments? Why?

Application form

• Should there be additions or deletions to the application form?

Common nomenclature

• Which terms used in the HTA forms and process should be defined (or redefined) to be better understood by your members?

Conflict of interest

• How should potential conflict of interest be reported and monitored?

Process funding

• How should the Local HTA Decision Support Program be funded?

Abbreviation: HTA, health technology assessment.

Step 4 consisted of a joint review by all participating departments in the form of full-day retreats. The retreat objectives were to learn from each department’s experience in reviewing the program, to share input and ideas for improvement, to explore the need for consistency and the need for flexibility, and to determine future directions and next steps. Retreats were divided into five sessions that followed the outline of the Points to Consider Questionnaire (Table 4) to systematically review the program’s policy, flow, structure, content, and funding.

Participants were asked to share their observations and to make joint recommendations to improve the program. Retreat reports were produced that incorporated structured feedback from participants during working sessions and notes made by the facilitator and research team members.

Finalization phase

The finalization phase involved revision of the Local HTA Decision Support Program and review of the adaptation process itself. For Step 5, feedback was compiled from the semi-structured interviews and meetings held with each of the local HTA leaders from individual departments, written comments provided on the Points to Consider Questionnaire reporting form, and the retreat reports. Data were analyzed using a content analysis-type approach.30 Major themes and subthemes were developed through comparison and categorization of information gathered. This process was completed independently by two members of the research team. The team met to discuss the outcomes and reached consensus in cases where discrepancies arose. The outcomes were then interpreted into meaningful concepts pertaining to the Local HTA Decision Support Program, and the program was revised accordingly.

For Step 6, we developed a Project Evaluation Questionnaire and gave it to all participants at the end of each retreat to evaluate the review and adaptation process used in this project.

Two cycles of the review and adaptation and the finalization phases were performed.

Results

Our initial interviews with the departments revealed dissatisfaction with historical technology introduction processes. Comments provided described how some technologies were introduced, removed, or replaced with alternatives without adequate evidence review or input from clinicians. To give some examples, approval was given to upgrade a combination dissection and electrocautery device. However, the upgraded device was incompatible with existing hand pieces and added a large generator to an already tight space. A promising new minimally invasive device for prostate surgery was purchased. Although HTA reports indicated good evidence for efficacy, the device failed to improve patient outcomes in our local setting, due to the lack of lead time to allow for appropriate training. A sophisticated robotic surgery device was donated by a private foundation. However, no funds were given for its high operating cost, and this unbudgeted expense then fell to the health care system. New endoscopes were purchased, but they required specialized sterilization equipment, which was not available in all operating rooms. Numerous examples were also given of requests for technologies that were based only on information obtained from the manufacturers’ representatives.

Consequently, department administrators and physician leaders wanted a system that would provide greater accountability and broader review for technology adoption. There was also some resistance to change – particularly around the perceived time and work required to run a Local HTA Decision Support Program.

In addition to the Department of Surgery and Surgical Services, we were successful in recruiting eight out of the ten departments originally approached for a total of nine departments: six clinical departments and three administrative departments. All the clinical departments appointed a physician and an administrator as local HTA coleaders, while the administrative departments appointed only an administrator as their HTA leader. The Department of Surgery and Surgical Services already had a functioning physician-administrator team as the HTA coleaders. Of the two departments that declined to participate in the project, one was in the midst of organizational transition after hiring a new department head and the other was not interested in developing its own program as its technology purchasing was overseen by the Department of Surgery and Surgical Services.

Four departments reviewed the Local HTA Decision Support Program in practice (three invited departments as well as Surgery and Surgical Services), and five departments reviewed the program in principle. All participating departments submitted their individual review feedback questionnaire and participated in the joint retreats. In total, we met with 67 individuals during the semi-structured interviews and meetings with each department’s clinical and administrative heads and their executive committees, including physicians, program managers, nurse clinicians, patient care managers, health services directors, health services managers, clinical product specialists, clinical safety leaders, researchers, and financial analysts, as well as the senior administrators. Thirteen individuals participated in the retreat for the first review cycle; 16 participated in the retreat for the second review cycle.

During the joint retreats, a significant amount of time was spent reviewing the “what” section of the Points to Consider Questionnaire (Table 4); that is, what information needs to be gathered to ensure that the department’s Local HTA Committee would have enough information to decide if a technology could be simply approved or if it needed further assessment? This was a contentious issue, because it is difficult to strike a balance between avoiding unnecessary work while still being able to protect patient safety and reduce risk when introducing new technologies. There was good consensus on the “why” section of the questionnaire; that is, the need for a systematic and transparent decision support program for technology review and adoption and a clear policy statement. There was also good consensus on the “how,” “who,” and “funding” sections of the questionnaire, with the general recommendation that the details of program operation should be left to the discretion of individual departments. Similar comments were brought forward by those who reviewed the program in practice as compared with those who reviewed in principle.

To evaluate the review and adaptation process used in this project, the research team reviewed the responses to the Project Evaluation Questionnaire that was given to participants after each of the two retreats. For the May 2007 retreat, 89%–100% of respondents (n=9) agreed or strongly agreed that the retreat objectives were met and 100% agreed or strongly agreed that the desired outcomes were achieved. These outcomes were “the revised Local HTA Decision-Support Program is adaptable to the needs of various clinical departments” and “I am committed to the next steps.” No participant scored “disagree” or “strongly disagree” for any of the objective or outcome statements. For the November 2007 retreat, 80%–100% of respondents (n=10) agreed or strongly agreed that the retreat objectives were met, and 90%–100% agreed or strongly agreed that the desired outcomes were achieved. These outcomes were: “The revised Local HTA Decision Support Program is adequate to ‘go live’ for January 2008 implementation,” and “I am committed to continued participation.” One person disagreed with the first outcome statement, but no other participants scored “disagree” or “strongly disagree” for any of the other objective or outcome statements.

Open-ended comments on the Project Evaluation Questionnaire revealed an overwhelming desire to continue face-to-face meetings. Participants found value in hearing the experiences of others using or reviewing the program, gathering feedback from different perspectives, and working collaboratively to come to consensus. As well, participants affirmed the value in having a fair, standardized, and consistent process for technology adoption across the various specialties within the Calgary Health Region.

Using the above described methodology, two cycles of the review and adaptation and finalization phases were performed with invited departments. As part of another project,26 an additional cycle of review was subsequently carried out to create our currently used Local HTA Decision Support Program. The fully revised program can be accessed at http://www.albertahealthservices.ca/4470.asp.

Discussion

Decision making is a complex process, involving many disciplines and players within an organization when introducing new health technologies, including medical devices.23 Because of cultural and organizational differences, a recommendation for technology adoption in one local setting may not be appropriate in another, even when the scientific evidence, such as that supplied by HTA reports, is the same. Thus, careful consideration of the specific questions relevant to local needs, priorities, legislation, policies, values, norms, as well as human and material resources, is as necessary as acquiring impartial scientific evidence. The Local HTA Decision Support Program developed initially by the Department of Surgery and Surgical Services of the Calgary Health Region is one attempt to provide a model (structure, processes, criteria, and decision tools) for decision making that systematically gathers scientific, operational, and value-based information for consideration when introducing new health technologies. Similar programs exist elsewhere.31–34

Any program that attempts to integrate context-free scientific evidence about a technology with context-sensitive information about the setting in which it will be used must be appropriate and relevant for each local organizational body. However, small rural and community hospitals or health authorities with fewer financial, personnel, and community resources may be poorly positioned for the investment required for de novo decision support program development. Programs developed by large health care systems can be adapted and applied successfully to rural settings, provided that needed tools are provided for the adaptation process.35 This paper provides an adaptation process that can be used in lieu of de novo program development, much like processes for adapting clinical practice guidelines.36,37

The adaptation process described and used in this article involved six steps organized into three main phases: the set-up phase involved the development of a manual and identifying and educating interested departments, the review and adaptation phase involved assisting the departments to conduct an actual review and adaptation of the program, and the finalization phase involved program revision and evaluation of the adaptation process.

During the set-up phase, we found a general dissatisfaction of each department with its traditional decision-making processes when introducing new health technologies. Similar dissatisfaction has been observed by others10 and provides a good opportunity to restructure these processes. To do this, we found that provision of a clearly written document, the Local Health Technology Assessment Decision Support Program: Review and Adaptation Manual, was vital to educate members about the value of HTA and a structured decision support program and to capture interest and buy-in. We also found that many clinicians and managers had never heard of the term “HTA,” therefore having well-written program documents and educational reference materials was vital to ensure their learning and understanding. A key document proved to be the table that explained the distinction between HTA producers and HTA users. We found that understanding this difference was important for understanding the rationale for a Local HTA Decision Support Program. The program does not duplicate the efforts of HTA producers; it complements their work by acting as the receptor to use HTA reports. The Assessment of Readiness Tool was a critical element of the set-up phase. It required that participating departments demonstrate readiness for change and appoint local HTA physician and administrative leaders.

During the review and adaptation phase, the research team provided ongoing assistance to participating departments by means of phone, email, and face-to-face meetings and education. We ensured that we provided a variety of methods to solicit feedback, as recommended by others.38 We found that the Points to Consider Questionnaire (why, how, who, what, funding) was essential to ensure that all important program review questions would be systematically discussed by reviewing departments.

The request that each department appoint an HTA physician leader and an HTA administrative leader was crucial to the success of the project. While physicians were more focused on clinical evidence, training and credentialing issues, and other such clinically relevant topics, their administrative counterparts ensured that infrastructure implications, cost, and organizational impact were reviewed. Similarly, the presence of relevant stakeholders, such as physicians, program managers, nurse clinicians, patient care managers, health services directors, health services managers, clinical product specialists, clinical safety leaders, researchers, financial analysts, and high-level health services administrators in the review and adaptation process can provide critical advantages. First, it ensures that those most likely to use the program have an opportunity to offer feedback and to identify problems before it is finalized. Second, it gives administrators the opportunity to consider the impact on the organization of implementing the program and to begin preparing for its future adoption. Third, the solicitation of practitioner feedback serves as the first wave of dissemination of the proposed program. Fourth, experience elsewhere has shown that when surgeons guide the appraisal of new and emerging surgical technologies, the benefits of an evidence-informed approach in the provision of high-quality patient care are realized.39

The recommendation taken forward to the finalization phase was that the content (what) should be standardized across departments and an interdisciplinary team should review the application, but that administrative details (who and how) should be left to the discretion of each department. This reflects the importance participants saw in having a fair, standardized, and consistent decision support program for technology adoption across specialties within the institution while acknowledging flexibility of operation.

The results of the Project Evaluation Questionnaires highlighted the success of this initiative. Participants demonstrated a strong desire to adopt the decision support program and to keep the review process going by engaging in regular face-to-face group collaboration and interaction. In fact, the review and adaptation and finalization phases can be repeated on a regular scheduled basis in a retreat setting, resulting in a continuous improvement cycle, as well as developing a community of practice, which is known to be effective in sustaining the viability and improvement of programs of interest.40

We identified several obstacles to long-term implementation of the program. First, staff turnover during the course of implementation can result in a continual need to educate and update decision makers. Second, there is yet no process by which any cost savings realized by the adoption of new technologies can be attributed back to the operation of the program. Third, shifting priorities of a health care system (for example, funds diverted to pandemic flu inoculations) may result in the cancellation or delay in the implementation of a recommended new technology. These obstacles are in addition to challenges encountered during the project itself, including finding common meeting times among all stakeholders, some resistance to change (ie, the perceived fear that running a decision support program when introducing new health technologies may be labor intensive), and finding funding for the appointment of HTA leaders and support personnel. Benefits of the project included increased research-mindedness among clinicians, staff, and administrators who participated in the decision support program review and adaptation process, increased awareness of HTA, increased understanding and appreciation of clinicians and administrators in the complexity of evidence-informed technology introduction process, and reduced barriers to implementation of the proposed program as a result of buy-in from the participating stakeholders. These observations will need to be confirmed as others use the decision-making program review and adaptation framework.

Conclusion

In summary, we present a three-phase framework for reviewing and adapting a well-developed Local HTA Decision Support Program for application in different settings. This adaptation process divides a complicated process into a discrete step-by-step approach, which can be repeated regularly to improve quality, and ensures that the adaptation process is systematic, rigorous, and interdisciplinary. We conclude that the adaptation of a preexisting program may reduce duplication of effort, save resources, raise health care providers’ awareness of HTA, and foster constructive stakeholder engagement, which enhances the legitimacy of evidence-informed recommendations for introducing new health technologies. We encourage others to use this framework for decision support program adaptation and report their experiences.