Building Consensus on the Priority-Setting for National Policies in Health Information Technology: A Delphi Survey.

OBJECTIVE: With growing attention on the healthcare industry as a potential market for big data and artificial intelligence in the Fourth Industrial Revolution, countries around the world are introducing and developing various policies and projects related to health information technology (HIT). To assist prioritizing HIT topics in policy making, this study adopts the Delphi technique to garner expert opinions from various fields of health informatics.
METHODS: Data were collected from November 2019 to February 2020 using the Delphi technique through two rounds of surveys administered via email. The Delphi panel consisted of 16 experts with a high level of experience in health informatics. They were from the Health Information Policy Advisory Committee of the Ministry of Health and Welfare of Korea, and the board of directors of the Korean Society of Medical Informatics. The experts were asked to assess the importance, urgency, and difficulty of HIT topics in three domains: technology, application, and infrastructure.
RESULTS: Of the 40 topic items, a 100% agreement was reached for the importance of 6 items, including 2 items in technology, 1 item in application, and 3 items in infrastructure domains. Especially, Quadrant I of a 2×2 matrix showing high importance and high urgency included 7 items in the technology domain, 2 items in the application domain, and 13 items in the infrastructure domain.
CONCLUSION: Most items with high importance and urgency belonged to the infrastructure domain. The findings indicated that fostering an infrastructural environment should be polices with top priorities of HIT.

I. Introduction

With the increasing burden of healthcare expenditure due to population aging, and the high prevalence of chronic diseases worldwide in recent years, there is a growing emphasis on the importance of disease prevention and lifestyle management [1,2]. This phenomenon has contributed to advancing information technology (IT), and has shifted the paradigm of healthcare from diagnosis and treatment of a disease to prevention and management of the disease, and from hospital-centered care to consumer-centered care [3,4]. In other words, healthcare is transforming to adopt the P4 approach: predictive, preventive, personalized, and participatory healthcare [5].

The IT-based healthcare is being recognized as a solution for improving the quality of health services, increasing patients’ convenience, and extending healthy life expectancy, while lowering the national burden of healthcare costs [6,7]. Furthermore, with its high added value, and potential for creating jobs, the IT-based healthcare is expected to positively contribute to the overall economy, which has motivated countries worldwide to foster this innovation as a new growth engine, and cultivate an environment to achieve sustainable economic growth [3,7].

Since the early 2000s, developed countries have prepared and promoted national-level policies on health information. In 2012, the World Health Organization (WHO) and the International Telecommunications Union (ITU) published the National e-Health Strategy Toolkit to provide guidelines on the vision, action plans, monitoring, and assessment for eHealth, based on which countries have initiated their efforts to develop strategic and comprehensive policies to foster the new industry [8,9]. The European Commission (EC) organized the 15th eHealth Network meeting (eHN) in 2019, and its Multiannual Work Programme published four eHealth strategies to be accomplished between 2018 and 2021 [10]. The four strategies comprise empowering patients, innovative use of health data, enhancing continuity of care, and overcoming implementation challenges. By structuring medical clusters, the German government intervened in the intercorporate network to promote the timely implementation of technologies for health innovation, and sharing of information, which has contributed to advancing Germany’s healthcare technology and boosting profit [7].

In the United States, the Office of the National Coordinator for Health Information Technology (ONC) evaluated health information technology (HIT) accreditation organizations with increased transparency, reliability, and efficiency of the Electronic Health Record (EHR) process. Following the Health Information Technology for Economic and Clinical Health Act (HITECH) enacted in 2009, the government has supported physicians and hospitals in implementing EHR, and had provided incentives to healthcare providers who implemented it [11]. Based on this, the United States developed a large-scale cohort to collect data for precision medicine, and disclosed data through the Blue Button project. Canada also established a nonprofit organization called Health Info-way, and launched the ACCESS 2022 to help people access and use personal health information [12]. As described here, majority of the countries worldwide put efforts to ensure the clarity of regulations to advance the digital healthcare industry and to improve their regulations to enable efficient use of medical and health data acquired in healthcare [3].

South Korea has enormous potential in terms of its IT infrastructure, the National Health Insurance System, top-level workforce, and application of healthcare big data [13]. Korean government is striving to establish legal guidelines on digital healthcare-related technologies and information, as well as a regulatory system that will promote the development of HIT, lay a foundation for research, foster workforce, and expand healthcare services [3]. However, compared to other advanced countries, promotion of the digital healthcare industry is slow due to multiple regulations, disagreement among stakeholders, and ambiguous strategies which may substantially hinder effective investment and application of technologies in healthcare [9], highlighting the need for a discussion and consensus based on health information policies at the national level [3,6].

To promote sustainable policies on HIT, each country needs to devise systematic and complementary strategies tailored to their health status, education, and technology in collaboration with stakeholders, instead of blindly implementing the policy guidelines used by the developed countries [9]. Therefore, it is necessary to identify specific HIT policies based on Korean healthcare information strategies and garner expert opinions to develop a policy roadmap. Consequently, to provide a foundation for developing HIT policies and strategies in Korea, this study grouped topics of health information into technology, application, and infrastructure domains. And the HIT items included in each domain were evaluated using the Delphi technique, based on the importance, urgency, and difficulty.

II. Methods

1. Expert Panel for the Delphi Survey

Panel selection is a critical process in a Delphi study, as the panel experts’ experience and knowledge regarding the topic determines the quality of study findings [14,15]. The research team contacted 17 members of the Health Information Policy Advisory Committee of the Ministry of Health and Welfare of Korea, and 23 members of the board of directors of the Korean Society of Medical Informatics (KOSMI) as the panel for this study. Ludwig [16] recommended a panel size of 15–20, while Gordon [17] recommended 15–35; however, panel size may differ depending on the purpose and design of the study, and the minimum panel size required for the purpose of the study is recommended [18]. In this study, total of 16 experts participated; 5 experts from the Health Information Policy Advisory Committee of the Ministry of Health and Welfare of Korea and 11 from the KOSMI.

2. Survey Questionnaire

The draft of the Delphi questionnaire first developed with 45 items that were derived from review of reports from the WHO-ITU’s National e-Health Strategy Toolkit [9], the EC’s eHealth strategy [10], and papers suggesting the direction of policies related HIT in Korea [1,3,4,6,7,9]. The initial set of items were revised and added by the research team, which finally consisted of 47 items, divided into three domains—technology, application, and infrastructure.

The content validity of the questionnaire was assessed using a 4-point Likert scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = very relevant) by 6 members of the board of directors of the KOSMI, who are experts in the health informatics field. They were also asked to provide comments for any necessary revision for items. Of 47 items, 22 items received the item-content validity index (I-CVI) of less than 0.8. Among 22 items, 12 items were deleted, 9 items were revised, and 1 item was added based on comments and the research team discussion.

The first round of the Delphi questionnaire consisted of 36 items comprising technology (14 items), application (6 items), and infrastructure (16 items) domains. In the second round of the survey, the questionnaire with 40 items was constructed after revising based on the response from the first round survey; 15 items for the technology domain (10 for Development, 3 for Standards, and 2 for Security); 6 items for the application domain (3 for Service Development and 3 for Data Utilization); 19 items for the infrastructure domain (5 for Law/Regulation, 3 for Professional Workforce, 4 for Building an Organizational Foundation, and 7 for Policy and Support). The experts were asked to rate each item for the importance, urgency, and difficulty of HIT policy making with a 4-point Likert scale (1 = strongly disagree, 2= disagree, 3 = agree, and 4 = strongly agree). The importance, urgency, and difficulty were operationally defined as how significant each item is, how fast it needs to be dealt with, and how hard it is to be completed.

3. Delphi Survey

The modified Delphi technique was used in this study that enables researchers to determine the number of survey rounds based on the objective and topic of the study [19]. However, researchers are required to devise strategies to reach a consensus among experts within a shortened number of rounds to save time and cost. To this end, researchers provided a structured questionnaire in the first round of survey to help panelists quickly focus on the research topic, and establish an expert group that helps in facilitating the survey process [18,20]. Therefore, two rounds of survey were conducted and a structured questionnaire was provided in the first round. The research team facilitated the survey and to discuss the composition of the items and survey procedure.

In the first round, the importance, urgency, and difficulty of tasks in technology, application, and infrastructure domains were rated with a structured questionnaire, and open-ended questions were included such that experts could present their opinions about the items. In the second round, a structured questionnaire was also used, which highlighted the revised or added items with individual responses and agreement rate (Number of panelists who chose 3 or 4/Total number of responses) in the first round. Experts were asked to re-rate or confirm their own first-round ratings to 36 items (Supplementary A).

The first and second rounds were administered on December 27, 2019 and February 17, 2020, respectively, via email. A reminder email was sent to the panelists whose responses were delayed. The second round of the survey was conducted on all 16 panelists who participated in the first round of the survey (100% response rate).

4. Data Analysis

Because the Delphi technique utilizes both closed- and open-ended questions, quantitative and qualitative data were collected and analyzed [15]. The responses to the open-ended questions in the first round were reflected in the second questionnaire after discussion with the expert group. The ratings of importance, urgency, and difficulty for each item in the first and second rounds of the Delphi survey were classified into “disagree” (ratings of 1 or 2) and “agree” (ratings of 3 or 4), and the agreement rate was computed by dividing the number of panelists who chose “agree” by the total number of panelists who responded. In a Delphi survey, a 50% or higher agreement is generally regarded as being reached a consensus [15,21], and in this study, it was decided that a consensus was reached when the agreement rate was 70% or higher.

The agreement rates for importance and urgency computed in the second round were analyzed using a 2×2 matrix by placing healthcare information policy items into four quadrants. The priorities of the items in Quadrant I, which have the greatest perceived importance and urgency, were established according to their difficulty level.

5. Ethical Considerations

This study was approved by the Institutional Review Board of Yonsei University (No. Y-2019-0141) where the first author belongs. Information about the purpose and procedure of the study was attached to the email sent to the study participants. The information specified that participation is completely voluntary and that participants have the freedom to withdraw from the study at any time.

III. Results

1. Results from the First Round of the Delphi Survey

Out of the initial 36 items of HIT topics, 29 items (81%) on importance, 23 items (64%) on urgency, and 15 items (42%) on difficulty were reached the consensus (70% or higher agreement). In addition, based on experts’ opinions about the items, 15 out of the 36 items were revised, and 4 new items were added, which made up a total of 40 items for the second round.

2. Results from the Second Round of the Delphi Survey for Each Domain

In the second round, consensus (≥70%) was reached on the importance of 32 out of 40 items. Tables 1–3 present the agreement rate for importance, urgency, and difficulty for each domain, and the quadrants in the importance-urgency matrix analysis. Of the 40 items, a 100% agreement was reached for the importance in 6 items, including 2 items in the technology domain, 1 item in the application domain, and 3 items in the infrastructure domain.

Table 1

Results of the second round of Delphi survey: technology domain (15 items)

Category	Item	Importance (%)	Urgency (%)	Difficulty (%)	Quadranta
1.1 Development	(1.1.1) IoT (Internet of Things)	100	93	75	I
	(1.1.2) AI (artificial intelligence)	94	88	81	I
	(1.1.3) Blockchain	56	53	73	III
	(1.1.4) AR/VR (augmented/virtual reality)	63	44	44	III
	(1.1.5) 3D printing	63	44	56	III
	(1.1.6) Genome sequencing	81	69	81	III
	(1.1.7) PHR (personal health record)	94	100	75	I
	(1.1.8) Mobile or web application	63	63	13	III
	(1.1.9) Cloud	75	56	50	III
	(1.1.10) 5G	75	63	56	III
1.2 Standards	(1.2.1) CDM (common data model)	63	44	31	III
	(1.2.2) Interoperability	94	88	75	I
	(1.2.3) Terminology standards	86	93	64	I
1.3 Security	(1.3.1) Personal information protection	94	88	88	I
	(1.3.2) Technology for information security	100	94	88	I

Quadrant: See Figure 1.

Table 2

Results of the second round of Delphi survey: application domain (6 items)

Category	Item	Importance (%)	Urgency (%)	Difficulty (%)	Quadranta
2.1 Service Development	(2.1.1) Usage of health information	75	69	31	III
	(2.1.2) Telemedicine	69	50	50	III
	(2.1.3) Services adopting technology	81	88	56	II
2.2 Data Utilization	(2.2.1) Data quality	93	100	81	I
	(2.2.2) Data science	100	100	81	I
	(2.2.3) Data platform for precision health	81	75	94	III

Quadrant: See Figure 1.

Table 3

Results of the second round of Delphi survey: infrastructure domain (19 items)

Category	Item	Importance (%)	Urgency (%)	Difficulty (%)	Quadranta
3.1 Law/Regulation	(3.1.1) Approval of HIT and devices	88	94	75	I
	(3.1.2) Certification of HIT and devices	81	94	56	II
	(3.1.3) Personal information protection and ethical management	100	100	69	I
	(3.1.4) Cost and fee systematization of medical services	75	81	81	II
	(3.1.5) Standardization	94	94	67	I
3.2 Professional Workforce	(3.2.1) Monitoring the supply and demand of professionals	88	87	69	I
	(3.2.2) Improving expertise of practitioners	88	93	44	I
	(3.2.3) Developing and implementing formal training curricula	100	100	75	I
3.3 Organizational Foundation	(3.3.1) Establishing a public control tower	88	80	75	I
	(3.3.2) Organizing a stakeholder advisory panel	94	87	69	I
	(3.3.3) Building a channel to reflect opinions from society	94	93	63	I
	(3.3.4) Bolstering the role and independence of professional organizations in HIT	80	81	69	II
3.4 Policy and Support	(3.4.1) Linking R&D of HIT	88	88	50	I
	(3.4.2) Supporting companies with HIT	56	38	25	III
	(3.4.3) Prioritizing investment areas	63	69	38	III
	(3.4.4) Reinforcing education on rights of personal information	80	80	47	II
	(3.4.5) Assessing the appropriateness of HIT	100	94	69	I
	(3.4.6) Managing EMR certification system and process	94	100	56	I
	(3.4.7) Utilization of HIT	88	100	63	I

HIT: health information technology, R&D: research and development, EMR: Electronic Medical Record.

Quadrant: See Figure 1.

As shown in Table 1, in the technology domain, the items with 100% agreement on importance were Internet of Things (1.1.1), and Technology for information security (1.3.2), and an item with 100% agreement on urgency was Personal health record (1.1.7). The following 4 items had the next highest rate of agreement on importance: Artificial intelligence (1.1.2), Personal health record (1.1.7), Interoperability (1.2.2), and Personal information protection (1.3.1).

In the application domain, 100% agreement on importance and urgency was reached for Data science (2.2.2), and 100% agreement on urgency was reached for Data quality (2.2.1) (Table 2). In the infrastructure domain, 100% agreement on importance and urgency was reached for 2 items: Personal information protection and ethical management (3.1.3) in the Law/Regulation category, and Developing and implementing formal training curricula (3.2.3) in the Professional Workforce category (Table 3). The remaining items with 100% agreement on importance or urgency belonged to the Policy and Support category: there was 100% agreement on the importance of Assessing the appropriateness of HIT (3.4.5), and 100% agreement on the urgency of Managing EMR certification system and process (3.4.6), and Utilization of health HIT (3.4.7).

3. Integration of Importance, Urgency, and Difficulty

To examine the priorities of the items for HIT policies, a 2×2 matrix with importance and urgency as the axes was used to classify items with an 83% agreement rate on importance and 80% agreement on urgency (Figure 1). Seven items in the technology domain, 2 items in the application domain, and 13 items in the infrastructure domain were included in Quadrant I. Table 4 lists 22 items for HIT policies included in Quadrant I, which features high importance and urgency in the ranking of their difficulty. None of the items perceived to be important were included in Quadrant IV, which indicates relatively lower urgency, and policies included in Quadrants II and III had low perceived importance, and varying levels of perceived urgency.

Figure 1

Results of the second round of Delphi survey: Importance-Urgency Matrix.

Table 4

Ranks of easiness for items in Quadrant I of the importance-urgency matrix (22 items)

Ranka	Difficulty (%)	Item	Importance (%)	Urgency (%)
1	44	(3.2.2) Improving expertise of practitioners	88	93
2	50	(3.4.1) Linking R&D of HIT	88	88
3	56	(3.4.6) Managing EMR certification system and process	94	100
4	63	(3.3.3) Building a channel to reflect opinions from society	94	93
4	63	(3.4.7) Utilization of health information	88	100
6	64	(1.2.3) Terminology standards	86	93
7	67	(3.1.5) Standardization	94	94
8	69	(3.1.3) Personal information protection and ethical management	100	100
8	69	(3.4.5) Assessing the appropriateness of HIT	100	94
8	69	(3.3.2) Organizing a stakeholder advisory panel	94	87
8	69	(3.2.1) Monitoring the supply and demand of professionals	88	87
12	75	(1.1.1) IoT (Internet of Things)	100	93
12	75	(3.2.3) Developing and implementing formal training curricula	100	100
12	75	(1.1.7) PHR (Personal Health Record)	94	100
12	75	(1.2.2) Interoperability	94	88
12	75	(3.1.1) Approval of HIT and devices	88	94
12	75	(3.3.1) Establishing a public control tower	88	80
18	81	(2.2.2) Data science	100	100
18	81	(1.1.2) AI (Artificial intelligence)	94	88
18	81	(2.2.1) Data quality	93	100
21	88	(1.3.2) Technology for information security	100	94
21	88	(1.3.1) Personal information protection	94	88

R&D: research and development, HIT: health information technology, EMR: Electronic Medical Record.

Sorted in ascending order of difficulty (for the same difficulty, items are sorted in descending order of importance).

The experts stated that among items in Quadrant I, Improving expertise of practitioners (3.2.2) in the Professional Workforce category and Linking R&D of HIT (3.4.1) in the Policy and Support category were relatively less difficult (Table 4). The items with the highest perceived difficulty were Personal information protection (1.3.1) and Technology for information security (1.3.2) in the Security category of the technology domain. Furthermore, as at least 75% agreement was reached on the importance of policies in Quadrant II, which have relatively less perceived importance and high urgency, some of these items could be considered in the determination of policy priorities, such as the Certification of HIT and devices (3.1.2) in the infrastructure domain and Services adopting technology (2.1.3) in the application domain.

IV. Discussion

The findings of this study were derived from health information experts via a Delphi survey to build consensus on the prioritization of HIT topics in the Korean context. Considering feasibility, items in Quadrant I, which featured high perceived importance and urgency with low difficulty, would be the policies that could be promptly implemented. The experts found 22 items to be the most important and urgent, and many of these belonged to the infrastructure domain. The highest agreement was shown on the importance and urgency of personal information protection, and ethical management in the Law/Regulation category of the infrastructure domain. There was also a strong consensus on the importance of technology for information security in the Security category under the technology domain.

In particular, the difficulty of personal information protection, and ethical management in the Law/Regulation category in the infrastructure domain was rated as moderate at 69%. Thus, this topic should be addressed as a priority in health information policy making. Establishment of a relevant legal and institutional environment is essential for the preparation of HIT policies and strategies. The United States and countries in Europe developed and promoted a relevant legal system to collect, exchange, and utilize data while securing accurate health information in order to vitalize service-oriented projects [22,23].

The results also show the importance of training and continuing education programs offered by professional and academic organizations. The education programs aim to improve the expertise of healthcare workforce as well as HIT professionals and practitioners by implementing the formal training curricula for HIT. Training the workforce in healthcare informatics has been emphasized in both developed and developing countries [24-26]. For example, in 2006, the American Medical Informatics Association launched a 10×10 program in partnership with any academic or other organizations with the aim of training 10,000 informaticians by 2010 in the healthcare informatics area [26]. Emerging areas of health informatics are data analytics, data governance, privacy and security, and interoperability [27]. The national HIT strategy should not only focus on establishing the infrastructure, technology, and application from the planning stage, but also on the training and improvement of workforce and national capabilities to operate the healthcare system.

To promote the quality and standardization of the HIT, the ONC has provided for the Certification of HIT, such as EHR and devices [28], and the European Institute for Health Records (EuroRec) also provides quality labeling and certification for EHR [29]. Korea also established the Korea Health Information Service for EMR certification in 2019. Currently, 8 EMR vendors and healthcare organizations have received EMR certifications [30].

This study is significant in that consensus was drawn from leading health information experts in Korea, the members of the Health Information Policy Advisory Committee of the Ministry of Health and Welfare of Korea, and the board of directors of the KOSMI to develop and prioritize HIT policies and strategic planning. Although survey findings were derived from only 16 experts, we were able to reach consensus on the policies after two rounds using the Delphi method. However, the fact that the panel of experts only consisted of 16 experts is a limitation of this survey, and a larger scale study should be conducted to obtain opinions from more experts and the general public to build HIT policies and strategies.

In conclusion, this study shows how to build consensus on the priority-setting for national HIT policies using the Delphi survey with a panel of experts in HIT. According to the main finding of the survey, majority of items with high importance and urgency belonged to the infrastructure domain, indicating that the fostering of an infrastructural environment should be prioritized in national HIT policies. Korea’s strengths lie in its IT infrastructure, medical insurance systems for all citizens, advanced professionals, and utilization of health big data. With the widespread implementation of EMR, picture archiving and communication systems, and relevant advanced IT and software having top level importance worldwide, prioritized items and domains identified in this study should be paid attention and invested for preparing national HIT policies and strategies at present and in the near future.