International health IT benchmarking: learning from cross-country comparisons.

OBJECTIVE: To pilot benchmark measures of health information and communication technology (ICT) availability and use to facilitate cross-country learning.
MATERIALS AND METHODS: A prior Organization for Economic Cooperation and Development-led effort involving 30 countries selected and defined functionality-based measures for availability and use of electronic health records, health information exchange, personal health records, and telehealth. In this pilot, an Organization for Economic Cooperation and Development Working Group compiled results for 38 countries for a subset of measures with broad coverage using new and/or adapted country-specific or multinational surveys and other sources from 2012 to 2015. We also synthesized country learnings to inform future benchmarking.
RESULTS: While electronic records are widely used to store and manage patient information at the point of care-all but 2 pilot countries reported use by at least half of primary care physicians; many had rates above 75%-patient information exchange across organizations/settings is less common. Large variations in the availability and use of telehealth and personal health records also exist. DISCUSSION: Pilot participation demonstrated interest in cross-national benchmarking. Using the most comparable measures available to date, it showed substantial diversity in health ICT availability and use in all domains. The project also identified methodological considerations (e.g., structural and health systems issues that can affect measurement) important for future comparisons.
CONCLUSION: While health policies and priorities differ, many nations aim to increase access, quality, and/or efficiency of care through effective ICT use. By identifying variations and describing key contextual factors, benchmarking offers the potential to facilitate cross-national learning and accelerate the progress of individual countries.

Objective

Since 2008, the Organization for Economic Cooperation and Development (OECD) has led an effort to help countries compare information and communication technology (ICT) adoption, use, and impact in the health sector., The ultimate goal is to identify best practices; to raise awareness of barriers and incentives related to health ICT availability and use; and to assist in initiation of strategies to realize associated economic and social benefits, which have the potential to be significant and far-reaching.

In this paper, we report on the first multicountry pilot of 4 prioritized clusters of indicators for comparing health ICT availability and use. We begin with the context, rationale, and history of this effort. We then discuss the approach to developing and testing benchmark measures that would apply across a range of national health systems and approaches to ICT use. We also report on pilot country results, lessons learned with regards to methodological challenges that have possible implications for cross-country comparisons, and planned next steps by OECD and others.

Background and Significance

While health policies and priorities differ across countries, many nations aim to increase the quality and efficiency of care, reduce administrative and operating costs of the health care system, and/or enable new models of health care delivery through effective use of ICT. For example, the 66th World Health Assembly noted in 2013 that:Likewise, a 2010 OECD survey identified 4 objectives for health ICT implementation: (1) increasing the quality and efficiency of care, (2) reducing operating costs of clinical services, (3) reducing administrative costs of the health care system, and (4) enabling new models of health care delivery.

“It is essential to make appropriate use of information and communication technologies in order to improve care, to increase the level of engagement of patients in their own care, as appropriate, to offer quality health services, to support sustainable financing of health care systems, and to promote universal access.”

While success is not guaranteed, a range of studies demonstrate that, under the right conditions, health ICT can contribute to these objectives, driving improvements in timely communication, quality, and efficiency. There is a large body of literature on the experiences of specific organizations and providers in implementing electronic health records (EHRs) and other related applications such as e-prescribing and computerized physician order entry systems. ICTs can also enable new ways of delivering care. For example, telehealth can provide access to advanced services that would not otherwise be available in rural and remote areas., The effective use of electronic records can also facilitate transparency, clinical research, effective public health planning, and the evaluation of health care interventions and their quality at the practice level. At the same time, risks can be introduced if health ICT is not implemented and used appropriately.

As countries develop and implement health ICT strategies, monitoring progress helps to ensure efforts are effective. This can be bolstered by learning from other countries. For instance, by 2012, the World Health Organization, the European Commission, the Commonwealth Fund, and others had published a number of comparative eHealth studies. In that year, the OECD also led a review of approaches to monitoring health ICT in 7 OECD countries and leading international institutions. However, the comprehensiveness of these studies varied, and they used different methodologies. Inconsistent definitions (e.g., what constitutes an EHR differs across countries) or statistical reasons, such as different sampling techniques, also limited the degree to which national and international data were comparable. As a result, the OECD reported that it was difficult to draw conclusions on ICT adoption, use, or impact on care within and across countries from existing information. It was similarly challenging for countries to evaluate the outcomes of policies and to identify practices in other countries from which they could learn. This was the impetus for launching a cross-country benchmarking initiative.

Methods

OECD model survey

In 2012 and 2013, the OECD convened global experts from a range of sectors and disciplines to agree on a priority set of indicators for benchmarking availability and use of ICT in the health sector, as well as approaches to measurement. Four indicator areas were selected: point-of-care EHRs, health information exchange, personal health records, and telehealth. An expert group representing 17 OECD countries (Australia, Belgium, Canada, Denmark, Finland, France, Israel, Italy, Japan, Korea, the Netherlands, Norway, Sweden, Poland, Spain, the United Kingdom, the United States), 4 non-OECD countries (Argentina, Brazil, Egypt, South Africa), the World Health Organization, the European Commission, and the Business Industry Advisory Committee to the OECD then developed a model survey covering these 4 domains.

The model survey’s aim is to support collection of internationally comparable measures on ICT in the health sector. Participants agreed to begin with benchmarking availability and use of these technologies; benchmarking of the impact of ICT on health or other outcomes was out of scope for the initial phase. The model survey consists of a series of self-contained modules that are intended to be flexible and adaptable to a rapidly changing environment. The use of core modules as an add-on to existing national surveys or as standalone surveys allows measurement on an internationally comparable basis. Additional modules and new measures can be added to respond to evolving or country-specific policy needs.

The model questionnaire is structured as shown in Table 1. Part I of the survey is addressed to health care professionals and providers. Part II is addressed to chief information officers and administrators in acute care facilities. A detailed description of the model survey is available in the Draft OECD Guide to Measuring ICTs in the Health Sector. Guidance on definitions and possible international classifications to facilitate the compilation of internationally comparable statistical indicators is provided in the methodology guidance. In total, there are 18 benchmark measures from the health care professional and provider modules and 19 from the acute care modules. Four health ICT benchmarking domains were addressed in the 2 parts of the survey. For example, both the ambulatory and acute care modules included questions on the extent to which health care professionals use electronic systems to store and manage patient health information and data, as well as functionalities that support care delivery.

Table 1.

Structure of the OECD health ICT model survey

Section	Domains of interest	Total no. of measures	# in this paper
PART I: Health care professionals and providers
A	Contextual variables (e.g., basic demographic data about respondents and their practice setting)	4	0
B	Availability and use of electronic records and health information exchange	6	1
C	Availability and use of functionalities that support patient engagement	5	4
D	Availability and use of telecommunications technologies to support health care delivery	3	0
Part II: Chief information officers/IT administrators in acute care
A	Contextual variables (e.g., basic demographic data about respondents and their organization)	5	0
B	Availability and use of electronic records and health information exchange	7	1
C	Availability and use of functionalities that support patient engagement	5	0
D	Availability and use of telecommunications technologies to support health care delivery	2	1

A key decision to promote comparability was to use a functionality-based approach in the model survey. That is, questions focused on the types of clinical and other activities that are supported by electronic systems rather than the availability of specific technologies. This reduces the effects of variations in terminology between countries. For example, while many OECD countries use the terms “electronic medical record” and “electronic health record” interchangeably, in Canada the 2 terms have distinct meanings. In addition, the approach does not require or assume use of a particular technology and therefore should promote consistency over time, even as new technologies are introduced.

Pilot of indicators based on the OECD model survey

OECD invited countries to pilot this functionality-based approach by conducting new standalone survey(s) based on the model survey and/or by extracting and mapping data to some or all of these questions from existing national or multinational surveys or administrative data sources. This occurred in 3 ways, with some countries participating in more than 1 way (see Table 2):

Table 2.

Countries included in health ICT benchmarking piloting process

Conducted new national standalone survey(s) based on model survey	Extracted and mapped data to model survey for pilot/incorporated into existing national data collection efforts	Multinational survey(s) with data related to model survey
Brazil Israel Korea Uruguay	Canada Denmark Finland Germany and Austria (acute care) The Netherlands Switzerland United Kingdom United States (Sweden, Norway, Iceland in the context of Nordic collaboration)12	European Commission surveys: Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, the Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Turkey (acute care survey only), United Kingdom

aSome countries (Denmark, Finland, Germany, Austria, the Netherlands, and the United Kingdom) conducted surveys and/or mapped existing national data sources, as well as participating in multinational surveys. Both results are shown where possible, but country-specific data were considered the primary source for the overall analysis included in this paper (see Figure 1). To understand the impact of some of the methodological issues identified on comparisons, data from national and multinational sources were also compared where both were available for specific countries.

A number of countries participated directly in the benchmarking pilot, drawing on nationally representative surveys or administrative data to derive results for some or all of the OECD model survey questions;

Some of the direct participant countries undertook further data collection and/or benchmarking in cooperation with partner countries. For instance, Nordic countries collaborated on a benchmarking effort that leveraged the OECD model survey work, mapping existing surveys and administrative data sources to the OECD indicators in the Nordic Collaboration. Likewise, since 2013, the ICT Work Group of the Statistical Conference of the Americas of the United Nations Economic Commission for Latin America and the Caribbean built on the Brazilian experience. They developed a module for measuring ICT access and use in the Latin American health care sector. The Pan American Health Organization also supported the initiative. The model survey was approved in 2014 and includes both a model questionnaire and methodological guidelines for measuring access to and use of ICT in the sector; and

The European Commission undertook a pre-pilot of many of the OECD model questions through commissioned surveys of primary care physicians and acute care facilities in 31 and 30 countries, respectively.

(Sweden, Norway, Iceland in the context of Nordic collaboration)

Figure 1.

Proportion of countries with minority, majority, and maturity of adoption of various ICT solutions.

Pilot participants that conducted new national surveys often tailored the model survey and its administration to their local context and policy needs. For example, the health care professional modules were generally administered to general/primary care/family practitioners in ambulatory settings. Not all countries fielded both the ambulatory and acute care surveys. In addition, many countries adapted the model surveys, e.g., by asking only a subset of questions, by adjusting the language of survey questions to take into account national structures/terminology, by adding definitions or interpretive information, and/or by adding questions that were relevant to their national needs but not included in the model survey. For instance, some countries asked respondents about enablers and barriers to health ICT use; others included questions on privacy. Likewise, the European Commission benchmarking exercise used related questionnaires for acute hospitals and general practitioners, adjusting the language of survey questions to take into account national structures/terminology.

Approach to cross-national comparisons

An OECD working group coordinated the pilot process. First, as described above, individual countries and coordinators of multinational benchmarking efforts typically reviewed the applicability of model survey questions in their context, identified currently available related data (where applicable), and decided how to collect new data (if required).

Throughout 2014 and 2015, pilot participants also took part in a series of working group teleconferences to share progress, issues, and results. In preparing for the presentations on national results and experiences, countries usually calculated and highlighted key indicators that mapped to the model surveys. They also shared experiences in designing data collection instruments, as well as feedback on question comprehension, response rates, and other factors. This informed ongoing working group discussions regarding progress and emerging issues.

Following a number of country presentations, the working group agreed on a minimum core set of measures for initial cross-national comparisons, drawing on an analysis of the number of countries able to provide data for measures from the model survey. This set included 5 measures from the ambulatory care survey and 2 measures from the acute care survey (see Table 1). Where results were available both from a country directly and from a multinational source, both are shown where possible, but the data provided by the country were prioritized for the summary analysis in Figure 1. Most of the selected measures focus on availability of certain functions rather than the extent of their use. Pilot country results for each measure were provided by the country lead or the lead of the cross-country effort, and then compiled by the working group (i.e., there was no centralized analysis of survey data). The working group also captured common methodological learnings, challenges, and issues that could affect comparability. Within this context, the European Commission in collaboration with the OECD also organized 2 workshops with member states and stakeholders.

Comparisons and conclusions based on this analysis were validated with member country representatives participating in the working group. It should be noted that results are not adjusted for intercountry differences in practice or hospital size or other similar factors. Other limits to comparability were also identified, as described below. To understand the impact of some of the methodological issues identified on comparisons, data from national and multinational sources were compared where both were available for specific countries. Given these considerations, the working group determined that it would be more meaningful to present indicator results in broad bands (minority adoption: 0–49%, majority adoption: 50–74%, and maturity of adoption: 75%+) than to provide more precise figures. This approach allows the identification of key trends and best practices while recognizing that comparisons across countries at a granular level may not be appropriate.

Results

Results from 38 countries were included in the analysis. The European Commission and 11 countries participated directly in the pilot process (Brazil, Canada, Denmark, Finland, Germany, Israel, the Netherlands, South Korea, Switzerland, the United Kingdom, and the United States). In addition, Sweden, Norway, and Iceland mapped national survey questions against the OECD survey via the Nordic Collaboration.

Provider-centric electronic records at the point of care

As shown in Table 3 (column 1), provider-centric electronic records are widely available at the point of care in almost all countries. All except 2 pilot countries reported at least half of their primary care practitioners used electronic records to store and manage patient health information, for instance. Twenty-nine of 38 countries had adoption rates over 75%, with a few reporting universal use. That said, there are larger differences in the specific data available electronically, what functions digital solutions enable, and how frequently they are used by primary care providers. For instance, Israel found that most clinics could produce a list of patients according to diagnosis, prescriptions, or demographics, but fewer could list patients according to the results of laboratory tests.

Table 3.

Distribution of benchmarking results for 3 health ICT indicators

Country (source)	Proportion of primary care practices that used electronic systems to store and manage patient health information	Proportion of acute care facilities that exchange radiology results/images electronically with outside organizations	Proportion of acute care facilities that have synchronous telehealth capability
Austria (EC)	●	●	○
Belgium (EC)	●	●	○
Brazil (CO)	○	○	○
Bulgaria (EC)	●	○	○
Canada (CO)	●	●	●
Croatia (EC)	●	○	○
Cyprus (EC)	●	○	○
Czech Republic (EC)	●	●	○
Denmark (CO/EC)	●	●	●
Estonia (EC)	●	●	○
Finland (Mixed)	●	●	○ (CO)● (EC)
France (EC)	●	○	○
Germany (CO/EC)	●	○	○
Greece (EC)	○	○	○
Hungary (EC)	●	○	○
Iceland (EC)	●	○	●
Ireland (EC)	●	○	●
Israel (CO)	●		○
Italy (EC)	●	○	○
Latvia (EC)	○	○	○
Lithuania (EC)	○	○	○
Luxembourg (EC)	●	●	○
Malta (EC)	○	●	○
The Netherlands (CO/EC)	●	●	●
Norway (EC)	●	○	○
Poland (EC)	○	○	○
Portugal (EC)	●	○	○
Romania (EC)	●	○	○
Slovakia (EC)	●	○	○
Slovenia (EC)	○	○	○
South Korea (CO)	●	○	○
Spain (EC)	●	○	○
Sweden (EC)	●	●	●
Switzerland (CO/EC)	○	○	○
Turkey (EC)	●
United Kingdom (CO/EC)	●	○	●
United States (CO)	●	○	○
Uruguay (CO)	○	○	○

CO = country, EC = European Commission Survey; = Maturity: 75–100%, = Majority: 50-74%, = Minority: < 50%, blank = data not available.

Health information exchange

As shown in Table 3 (column 2), we found large variation between countries in terms of the proportion of acute care facilities that engage in health information exchange, specifically the percentage that electronically exchange radiology results and/or images with outside organizations. While some countries such as Canada and Finland reported universal or near universal availability of this functionality, in 12 of 36 nations who reported on this indicator, less than half of acute care facilities can exchange radiology results/images with outside organizations.

Use of telehealth

As shown in Table 3 (column 3), pilot results reveal wide cross-national variation in telehealth capacity, specifically the availability of synchronous telehealth (typically videoconferencing) in acute care facilities. While some countries (e.g., Canada) had widespread use of synchronous telehealth, many others had limited or no use. Only 7 countries (Canada, Denmark, Iceland, Ireland, the Netherlands, Sweden, and the United Kingdom) reported availability in at least three-quarters of acute care facilities.

Personal health records or patient access to online services

Results from the 4 benchmark measures related to personal health records or patient access to online services are shown in Table 4. Some countries have achieved broad adoption of these solutions in primary care, including Denmark and the United Kingdom for e-appointment booking and e-requests for prescription renewal/refill. However, in many countries, only a minority of primary care practices have made these functions available to patients. That said, working group members noted that this is an area where there is rapid growth in a number of countries. Accordingly, comparisons between countries may be affected by when surveys were conducted.

Table 4.

Availability of e-functions for patients in primary care practicesa

Country (source)	Access to test results online	e- appointment booking	e-request for prescription renewal/refill	Secure messaging (asynchronous)
Austria (EC)	○	○	○	○
Belgium (EC)	○	○	○	○
Brazil (CO)	○	○		○
Bulgaria (EC)	○	○	○	○
Canada (CO)	○	○	○	○
Croatia (EC)	○	○	○	○
Cyprus (EC)	○	○	○	○
Czech Republic (EC)	○	○	○	○
Denmark (Mixed)	○ (EC)	● (CO/EC)	● (CO/EC)	○ (CO)● (EC)
Estonia (EC)	○	○	○	○
Finland (CO/EC)	○	○	○	○
France (EC)	○	○	○	○
Germany (Mixed)	○ (EC)	○ (CO/EC)	○ (CO/EC)	○ (CO/EC)
Greece (EC)	○	○	○	○
Hungary (EC)	○	○	○	○
Iceland (EC)	○	○	○	○
Ireland (EC)	○	○	○	○
Israel (CO)	●	●	●	○
Italy (EC)	○	○	○	○
Latvia (EC)	○	○	○	○
Lithuania (EC)	○	○	○	○
Luxembourg (EC)	○	○	○	○
Malta (EC)	○	○	○	○
The Netherlands (CO/EC)	○	○	○	○
Norway (Mixed)	○ (EC)	○	○	○ (CO/EC)
Poland (EC)	○	○	○	○
Portugal (EC)	○	○	○	○
Romania (EC)	○	○	○	○
Slovakia (EC)	○	○	○	○
Slovenia (EC)	○	○	○	○
South Korea (CO)	○	○	○	○
Spain (EC)	○	●	○	○
Sweden (EC)	○	○	●	○
Switzerland (CO)	○	○	○	○
Turkey (EC)	○	○	○	○
United Kingdom (Mixed)	○ (EC)	● (CO) ○ (EC)	● (CO) ○ (EC)	○ (EC)
United States (CO)				○
Uruguay (CO)	○	○

aFor this category, the European Commission survey used a different set of questions than the OECD model survey. For example, the OECD survey asks: “Can patients engage in asynchronous/not-real time secure online/electronic communication with a professional about a clinical issue?” The European Commission survey asks: “Does your ICT system allow you to transfer/share/enable/access patient data electronically, permitting you to engage in any of the following? Interact with patients by email about health-related issues.” There is a follow-up question: “Do you use them to…? Interact with patients by email about health-related issues.” The second question was deemed more comparable to the OECD question by the working group and has been used in the table above, except where countries had national data that was more closely aligned with the OECD survey question. In most cases, there was no difference in performance as categorized in this table. The 3 cases where there was a discrepancy are noted in the table. This may be due to timing, question, or other comparability issues, examples of which are outlined below.

CO = country, EC = European Commission Survey; = Maturity: 75–100%, = Majority: 50–74%, = Minority: < 50%, blank = data not available.

Relative adoption progress: summary across benchmarking measures

Analysis of benchmarking data shows that some types of health ICT availability are quite advanced in most countries, but in others, there is wide variation in adoption. Across countries, availability and use tend to be highest for provider-centric electronic records within a particular care setting (see Figure 1). While some countries have broad availability of health information exchange, patient online services, and telehealth, there is more cross-country variability for these measures.

Comparability considerations

The pilot results demonstrate the possibility of collecting cross-national benchmarks and grouping countries according to their availability and use of health ICT using these measures (see, e.g., European country profiles). The functionality-based approach used in the model survey helped to mitigate comparability issues by focusing on capabilities available to health care providers or acute care facilities rather than the technological approaches used to deliver this functionality. The pilot reinforced the utility of this approach, as well as the importance of detailed functionality specification. For instance, the European Commission survey found that 45% of acute care facilities across participating countries reported that they had videoconferencing capabilities, used for e-learning, patient care, and administrative or other purposes. However, only about one-third of these hospitals said that they had telehealth capabilities for patient consultations. This suggests that our approach, which focused on the use of functionalities in the context of patient care, is an important advancement in creating comparable measures.

While the context-specific, functionality-based approach to measurement proved helpful in ensuring valid cross-country benchmarking, important limitations to comparability surfaced during the pilot. This was a key reason behind the decision to report national results in bands (0–49%, 50–74%, and 75%+). At this level of analysis, results for the 7 countries that both provided data from national sources and were included in the European Commission surveys mostly corresponded (see Tables 3 and 4 for discrepancies), whereas more detailed comparisons demonstrated higher variation.

These discrepancies are not unexpected. From the beginning, it was understood that a range of methodological considerations could affect comparability. These include differences in sample frame, sampling methods and size, the mode of data collection/survey administration, response rates, missing data, and weighting approaches. For instance, some countries fielded the whole model survey, while others chose specific questions. These decisions reflected both anticipated respondent burden and domestic policy priorities, decisions that may offer insights to inform future iterations of the OECD model survey. Likewise, in some cases, countries added questions or populations to those in the model survey. For example, Brazil chose to survey nurses as well as physicians.

Achieving high response rates was a challenge for many countries, as is often the case for surveys of health care providers. Respondent burden (e.g., survey length) contributed to lower response rates in several countries. Success factors for higher response rates included active follow-up, visible high-level support for the survey, and emphasizing the contribution of survey results to both national and international benchmarking.

In addition, the experiences of pilot countries highlighted other issues likely to impact comparability but that can be more difficult to identify, such as:

Linguistic differences and framing effects: Many countries adapted the language and/or structure of the model survey to their local context. In some cases, this involved translation of questions into national language(s). In others, it required adaptation of questions to reflect the underlying concept that was intended to be measured. For instance, the Nordic countries identified challenges related to the term “prescription” and its representation in Nordic languages. These types of adaptations may affect comparability across countries, but a failure to take into account local context and likely question interpretations would also affect benchmarking.

Survey administration/data collection: This includes whether the survey was administered in whole or in part, whether a standalone survey was conducted or data were mapped from existing sources, whether fieldwork was conducted on a centralized basis for multiple countries or on a country-by-country basis, and the timing of data collection. The latter may be particularly relevant given that health ICT adoption and use is evolving at different rates in different contexts. Likewise, some countries found challenges in adapting the survey questions to a Web-based data collection model. When many countries adapted particular survey questions, it may suggest a need to clarify the model survey wording and/or to provide guidance on interpretation and measurement.

Mapping from existing data sources: A number of countries, such as Canada, Denmark, Finland, Germany, the Netherlands, the United Kingdom, and the United States, had existing surveys or other types of data collection that aligned with some or all of the indicators included in the OECD Health ICT benchmarking process. Through the pilot process, these countries reviewed alignment with the model survey measures and in some cases made changes to their existing data collection processes to improve that alignment. This approach has the advantage of reducing respondent burden and leveraging existing resources, and may be more sustainable. However, a desire to retain existing question wording, processes for national trending/benchmarking purposes, or how different clinical workflows and policies influence data tracked through system logs may sometimes limit international comparability.

Health ICT architecture: Variations in health ICT architecture, such as health information exchange (HIE) “pull” from secure regional/national databases versus point-to-point “push” delivery via secure messaging, may affect how respondents interpret questions. For instance, the Nordic countries found that rapidly evolving national ICT architecture may affect the interpretation of HIE and personal health record results; functionality available nationally may not yield comparable results to functionality available only between 2 organizations. Likewise, in some countries, online patient access to personal health information occurs via primary care practices; in others, it may be facilitated regionally or nationally. The choice of approach may affect responses regarding whether functionality is available via a particular setting.

Health system structures: Differences in the structure of health systems, such as definitions and organization of primary care and acute care, the distribution and size of organizations offering the services, how health care providers cooperate, and payment systems/incentives that affect health ICT adoption may all affect measurement. For example, within Europe, analysis of variation by organizational setting and health system type clearly hints at ICT adoption being shaped not only by GPs’ individual characteristics and attitudes but also by country-level contextual meso and macro factors. In Brazil, surveys identified significant differences in the uptake of telehealth in public and private acute care facilities, particularly with regards to educational and research uses. For example, the Telemedicine University Network, an initiative of the Ministry of Science, Technology, and Innovation, provides the communication infrastructure for research groups, targeting the improvement and development of new telemedicine projects. Likewise, other organizational factors such as IT strategy, existence of a central IT department, and relationships with IT vendors have been reported to influence adoption. Amarasingham and colleagues found that teaching status, IT budget, and the number of IT staff could also affect uptake, while other studies reported on the influence of system affiliation and location.

Discussion

Lessons learned from benchmarking health ICT use across countries

International comparisons are always challenging given differences in national health systems, cultures, and contexts. There is tremendous variation in how countries have approached health ICT adoption and its maturity, how they organize and deliver health care, their resources, and other factors that affect benchmarking. In addition, there are cultural, linguistic, methodological, and other reasons for differences in the application and interpretation of surveys and indicators. These and other challenges influenced the OECD pilot process, with the health system and cultural considerations being possibly the most challenging to identify and address.

Nevertheless, this process has demonstrated that a voluntary, multicountry effort to collect and benchmark measures of health ICT adoption can deliver insights that inform policy and practice. For example, pilot countries discussed why in some countries more progress had been made in acute care than in primary care while in others the reverse was true, what we could learn from countries that were supporting improved continuity of care through high levels of information exchange, and policy enablers and barriers to improving access to care via telehealth. Likewise, many countries were able to draw conclusions from the results to inform national policy decisions.

In general, countries were more likely to report progress in implementation of health ICT within particular care settings, such as within primary care practices, rather than HIE across organizations/care settings. In part, this may be due to challenges with the compatibility and interoperability of systems and information, a prerequisite for more advanced HIE. Challenges with the usability of HIE systems and data have also been documented. Likewise, organizational divides and policy barriers may affect HIE. That said, focused efforts in some countries, such as the regional exchange of radiology images and/or reports in Canada, may reveal critical success factors for advancing information sharing.

Similarly, adoption of health IT solutions for use by clinicians tended to be higher than adoption of solutions for use by patients, although a number of pilot countries reported that the latter was advancing quickly. In part, this may parallel a general trend towards more person-centered care. It may also reflect the fact that many functions desired by patients (e.g., e-booking or e-prescription renewal) require interaction with health care providers, implying that clinicians need to have IT solutions in place for them to be effective. There are also a range of technical, sociocultural, legal, and other factors that may affect adoption of consumer health IT solutions.

Next steps

The experiences of countries participating in the health ICT benchmarking pilot process parallel those of other model surveys that the OECD has developed. The pilot will inform adaptation or deletion of questions based on field experience. Modules may also be added over time as technologies, usage practices, and policy interests change. New topics are typically considered based on known policy needs and experiences of member countries with interest in those topics. That said, an important consideration is to minimize the number and complexity of questions in recognition of the cost of collecting additional data, both in terms of resources required and respondent burden.

Building on lessons learned during the pilot, a variety of next steps are planned, including:

Further analysis of pilot data and the pilot process with a view to sharing key findings with pilot country participants, OECD forums, and the broader community;

Exploring options for expanding participation in cross-national benchmarking, e.g., via the United Nations Economic Commission for Latin America and the Caribbean, as well as other regional benchmarking opportunities (e.g., the Nordic countries, Germany and Austria);

Researchers choosing to perform in-depth analyses of the differences between countries, such as with a focus on eHealth legislation, engagement of health care providers with national endeavors, influence of the innovative power of organizations and stakeholders, and financial restrictions or incentives (e.g., the Meaningful Use Program in the United States);

Tracking country-level plans to further advance model survey and benchmarking activities; and

Identifying opportunities for advancing the model survey based on feedback from the pilot, country experiences with national data collection, and potential emerging trends and policy priorities (e.g., m-health and work by the Nordic countries who plan to continue their collaboration by developing common health ICT usability and outcome indicators for countries with advanced national ICT infrastructures).

Conclusion

All countries face challenges in modernizing and sustaining high-quality, person-centered health services, and many see effective use of health ICT as central to health care transformation. The stakes are high for citizens, health care providers, and policy-makers. In this context, there is a strong appetite to learn from and leverage the experiences of others. Doing so requires a common understanding of which countries’ experiences may be most instructive, what they have done, and how they made progress.

Given the diversity of health systems, cultures, and language, multinational benchmarking in the health sector is always challenging, but measures that allow for a deeper picture of each country’s status and progress can facilitate cross-national learning. The OECD-led benchmarking pilot has demonstrated the value of this type of work for health ICT, as it has already informed both national policy decisions and is delivering insights from the international comparisons. No one country had the best performance on all measures examined in the pilot; neither was any country behind on all indicators. Thus, every nation has an opportunity to both learn from others and share their leading practices. This offers the potential for broadly accelerating progress.