Establishment of a Quantitative Medical Technology Evaluation System and Indicators within Medical Institutions.

BACKGROUND: The development and application of medical technologies reflect the medical quality and clinical capacity of a hospital. It is also an effective approach in upgrading medical service and core competitiveness among medical institutions. This study aimed to build a quantitative medical technology evaluation system through questionnaire survey within medical institutions to perform an assessment to medical technologies more objectively and accurately, and promote the management of medical quality technologies and ensure the medical safety of various operations among the hospitals.
METHODS: A two-leveled quantitative medical technology evaluation system was built through a two-round questionnaire survey of chosen experts. The Delphi method was applied in identifying the structure of evaluation system and indicators. The judgment of the experts on the indicators was adopted in building the matrix so that the weight coefficient and maximum eigenvalue (λ max), consistency index (CI), and random consistency ratio (CR) could be obtained and collected. The results were verified through consistency tests, and the index weight coefficient of each indicator was conducted and calculated through analytical hierarchy process.
RESULTS: Twenty-six experts of different medical fields were involved in the questionnaire survey, 25 of whom successfully responded to the two-round research. Altogether, 4 primary indicators (safety, effectiveness, innovativeness, and benefits), as well as 13 secondary indicators, were included in the evaluation system. The matrix is built to conduct the λ max, CI, and CR of each expert in the survey, and the index weight coefficients of primary indicators were 0.33, 0.28, 0.27, and 0.12, respectively, and the index weight coefficients of secondary indicators were conducted and calculated accordingly.
CONCLUSIONS: As the two-round questionnaire survey of experts and statistical analysis were performed and credibility of the results was verified through consistency evaluation test, the study established a quantitative medical technology evaluation system model and assessment indicators within medical institutions based on the Delphi method and analytical hierarchy process. Moreover, further verifications, adjustments, and optimizations of the system and indicators will be performed in follow-up studies.

INTRODUCTION

The development and application of medical technologies reflect the medical quality and clinical capacity of a hospital. Meanwhile, the operation of advanced medical technologies is also an effective approach in upgrading medical service and core competitiveness among medical institutions.[1] Therefore, the regulation on medical technologies is a crucial part of medical quality management and hospital management, and it is the primary task for the medical administrative department to ensure the safety and effectiveness of medical technologies in clinical applications.[2] To be specific, the key contents of medical technology management are the evaluation and assessment of the operation procedures and protocols, the technical competency of medical staffs and medical units as well as the external conditions such as the equipment and materials required in the process to determine the safety, effectiveness as well as the adaptability in clinical practice.[3] Thus, the establishment of a comprehensive evaluation system and related indicators of clinical technology are vital in enhancing the management and supervision within medical institutions.[4] According to recent studies, some domestic and foreign medical administrative departments and medical institutions have already established relatively mature admission procedures and management regulations as well as assessment methodology of medical technology.[5] However, researchers and outcomes on the accurate evaluation systems and standards, especially on quantitative assessment systems that can assess to an objective, unified evaluation are still rare, which make it almost impossible for different medical technology for comparison.[6]

On this basis, the current study aimed to build a quantitative medical technology evaluation system through questionnaire survey using the Delphi method and set index weight coefficient to each indicator through analytical hierarchy process within medical institutions. Moreover, it is hoped with the establishment of the evaluation system and further verifications in clinical practice, medical institutions can make a better assessment to clinical technologies, promote the management of medical quality technologies, and ensure the medical safety of various operations among the hospitals.[7]

METHODS

Selection of questionnaire survey experts

According to the principle of the Delphi method, a certain number of experts were selected to participate in the questionnaire survey. In general, the accuracy and constancy of the results are related to the number of experts joining the investigation.[8] Normally, to ensure the credibility and authority of the results, the optimum number of experts investigated was among 15–50.[9] All the experts selected should be involved in the related specialty fields and that are acquainted with the contents of the survey. To ensure the accuracy and objectivity of the results, the professional levels, specialty majors as well as the length of occupations of the experts should be evenly distributed.[10]

Establishment of the evaluation system and two-leveled indicators

According to previous literature reviews and expert consultations, it is generally suggested to build a medical technology evaluation system of the two-leveled indicators.[11] Based on the current reports and studies, practical experiences of medical technology operations in combination with the results of the expert consultation, the research drafted the initial evaluation system of four primary level indicator and 14 secondary indicators. Then, the first round of survey questionnaires was developed according to the system and indexes we drafted and distributed to experts selected. Based on the principle of the Delphi method, experts would be responding to the questionnaire and grade each indicator based on the Likert scale (containing 5 scales, which are “strongly agree”, “agree”, “uncertain”, “disagree”, and “strongly disagree”, with the grade of 5, 4, 3, 2, and 1, respectively, in statistical analysis) by the importance of the subjects according to their expertise and experience.[12] Besides, suggestions of the experts in adjusting the structure of the evaluation system through adding, deleting, or merging indicators would also be faithfully recorded. The results of the first round questionnaires were collected and processed, and the outcomes and suggestions of statistical significance were summed up in rebuilding the second round of survey questionnaires for the same expert team of the same procedures.

Allocation of the index weight

To set the allocation of index weight, analytical hierarchy process was adopted in determining the weight coefficient of each indicator. In the second round of the survey, experts were asked to assign weights to each level of indicators according to their experience and expertise, and the matrix was built in the principle of analytical hierarchy process.[13] In this period, experts were supposed to make comparisons of specific indicators in each judgment matrix, and Saaty 1–9 scale relative materiality table was applied in the grading the importance of the indicators between one and another, the matrix was completed based on the value of relative importance of each subject.[14] Calculate the primary and secondary index weight coefficient by the survey results of the matrix, and the index weight coefficient was used in judging the importance of the indicators. The judgment of the experts on primary level indicators was input into Microsoft Excel 2010 (Microsoft Corporation, Washington, USA) to build the matrix so that the weight coefficient and maximum eigenvalue (λ max), consistency index (CI), and random consistency ratio (CR) were obtained and collected.[15] Consistency test was performed according to the calculated value, as the results of CR <0.1 was considered adaptable, and the average value of the results was calculated to determine the index weight primary level indicators.[16] Then collected the weight coefficient of secondary indicators according to the same principle and with the index weight obtained, calculated the integrated index weight of secondary indicators according to the formula Nij = XiYij (i = 1, 2…; j = 1, 2,…). As Xi refers to the primary level index weight, and Yij is the j level index weight under the i primary level index.[17] All the data and material collected were entered into Excel 2010 software for Microsoft (Microsoft Corporation), and statistical analyses were performed using SPSS version 14.0 (SPSS Inc., Chicago, IL, USA).

RESULTS

General situation of experts and the distribution of questionnaires

In this study, 26 experts of different occupational lengths and professional levels majoring in clinical medicine, hospital management, nursery as well as pharmacy management were invited in the research, as the general situation of the experts are shown in Table 1. According to the initial index system as well as the evaluation indicators we drafted, the first round of survey questionnaires was designed as shown in Supplementary Table 1. In the first round of investigation, 26 questionnaires were distributed and 25 copies were recovered, with a recovery rate of 96.15%. While in the second round, a total of 25 questionnaires were allocated to the 25 experts responded in the last round, and all the questionnaires returned were examined to be effective.

Table 1

General situation of the surveyed experts (n = 26)

Categories	Value, n (%)
Major
Medical management	10 (38.5)
Clinical medicine	11 (42.3)
Nursery	2 (7.7)
Pharmacy	3 (11.5)
Professional levels
Senior level	5 (19.2)
Sub-senior level	9 (34.6)
Medium level	10 (38.5)
Junior level	2 (7.7)
Length of occupations
5–10 years	5 (19.2)
11–20 years	10 (38.5)
21–30 years	8 (30.8)
>30 years	3 (11.5)

Supplementary Table 1

The contents of first round survey questionnaires

Primary level indicators	Your opinion on the selection of primary indicators	Secondary level indicators	Your opinion on the selection of secondary indicators
Safety	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	The establishment of practice guideline and operation regulations	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	Definitions on indications and contraindications of the technology	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	The formulation of risk disposal plans	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	The management on the admission of technical operators	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
Effectiveness	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	Quantity of operated cases	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	The follow-up evaluations of patients	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	Recovery rate	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	Improvement rate	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	The comparison with conventional/similar technologies	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
Innovativeness	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	The origination of innovations	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	Advantages toward conventional/similar technologies	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	Generalization of the technology	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
Benefits	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	Economic benefits	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree
	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree	Social benefits	□Strongly agree □Agree □Uncertain □Disagree □Strongly disagree

Other opinions on the evaluation system and indicators:

Establishment of the evaluation system and indicators

The first round of survey results questionnaires of the 25 survey experts were collected, analyzed and verified through consistency tests. According to the grading of experts, it was suggested that the initial structure of the evaluation system are basically recognized. Meanwhile, modifications on some indicators were made according to suggestions given by the experts [Table 2].

Table 2

The adjusted evaluation system and indicators

Primary level indicators	Secondary level indicators
Safety	Establishment of practice guideline, operation regulations, and risk disposal plans Definitions of indications of diseases Occurrence of contraindications Management on the admission of technical operators
Effectiveness	Quantity of operated cases Follow-up evaluations of patients Recovery rate Improvement rate
Innovativeness	Advantages toward conventional/similar technologies Generalization of the technology Priority of the technology in the professional field
Benefits	Economic benefits Social benefits

Indicators suggested to be added

During the survey, 6 experts out of 25 (24.0%) suggested that the indicator “the priority of the technology in professional field” were of great importance that should be included in the system. Over careful consideration, we decided to include the subject as the secondary indicator under the primary column of “innovativeness”.

Indicators suggested to be combined or separated

Four experts out of 25 (16.0%) proposed the combination of secondary indicators “the establishment of practice guideline and operation regulations” and “the formulation of risk disposal plans” under the primary column of “safety” since they could all be classified as the preoperational systematic requirements of medical technologies. Besides, 3 experts (12.0%) suggested the indicator “definitions on indications and contraindications” be separated in consideration for the aptness of grading. After further discussion, we separated the original indicator into “definitions on indications of diseases” and “occurrence of contraindications” two parts.

Indicators suggested to be deleted

According to the survey results of 3 experts (8.0%), the secondary indicator “the origination of innovations” under the primary column “innovativeness” was thought hard to be quantitatively analyzed and evaluated, and hence that it was suggested to be deleted, and the proposal was accepted by the research group.

With the adjustments made of the evaluation system and the indicators, the second round of survey questionnaires was developed.

The aim of the second round survey was carried out to identify the index weight of each indicator as we distributed the questionnaires to the 25 experts responded in the first round. After the results were collected and analyzed, verification on the consistency of the experts was made, and results showed that all the CR values were below 0.1, suggesting the opinions of experts were relatively stable. Besides, the experts had no revisions on the setting of the indicators in the second round of survey. Thus, the assignments of primary and secondary indicators were analyzed and calculated by building the matrix according to the grading of experts on each individual indicator by Saaty 1–9 scale relative materiality table, as the results of index weight of indicator are calculated as shown in Table 3. According to the results, the maximum index weight of primary indicator was the safety of the medical technology, with the coefficient of 0.33, followed by effectiveness and innovativeness, with the index weight coefficient of 0.28 and 0.27 respectively, benefits of medical technology ranked last in the survey, with the index weight coefficient of 0.12. The index weight of each secondary indicator was conducted according to the formula Nij = XiYij (i = 1, 2…; j = 1, 2,…), and the results are shown in Table 4.

Table 3

Results of experts on index weight of primary indicators

Number	Safety	Effectiveness	Innovativeness	Benefits	CR	CI	λmax
1	0.32	0.24	0.32	0.12	0.04	0.04	7.19
2	0.33	0.27	0.26	0.14	0.01	0.02	7.09
3	0.35	0.25	0.30	0.10	0.03	0.05	7.30
4	0.31	0.35	0.20	0.14	0.02	0.03	7.16
5	0.36	0.25	0.25	0.14	0.03	0.04	7.22
6	0.34	0.24	0.26	0.16	0.04	0.06	7.35
7	0.37	0.23	0.25	0.15	0.01	0.01	7.09
8	0.31	0.32	0.26	0.11	0.03	0.04	7.21
9	0.32	0.30	0.26	0.12	0.02	0.04	7.20
10	0.31	0.32	0.26	0.11	0.03	0.04	7.22
11	0.33	0.29	0.27	0.11	0.03	0.04	7.33
12	0.35	0.25	0.29	0.11	0.04	0.05	7.31
13	0.34	0.26	0.26	0.14	0.03	0.05	7.32
14	0.31	0.27	0.32	0.10	0.02	0.03	7.18
15	0.34	0.26	0.27	0.13	0.02	0.03	7.19
16	0.36	0.26	0.28	0.10	0.03	0.04	7.29
17	0.35	0.31	0.24	0.10	0.02	0.04	7.30
18	0.36	0.27	0.26	0.11	0.04	0.06	7.34
19	0.33	0.33	0.20	0.14	0.03	0.05	7.27
20	0.32	0.31	0.26	0.11	0.02	0.03	7.21
21	0.29	0.28	0.27	0.16	0.01	0.02	7.12
22	0.30	0.28	0.27	0.15	0.03	0.05	7.23
23	0.32	0.26	0.30	0.12	0.02	0.03	7.19
24	0.31	0.27	0.26	0.16	0.03	0.04	7.22
25	0.30	0.29	0.28	0.13	0.03	0.04	7.24
Average value	0.33	0.28	0.27	0.12

CI: Consistency index; CR: Consistency ratio.

Table 4

Results of experts on index weight of secondary indicators

Items of primary indicators	Items of secondary indicators	Average value	Index weight
Safety	Establishment of practice guideline, operation regulations, and risk disposal plans	0.32	0.11
	Definitions on indications of diseases	0.21	0.07
	Occurrence of contraindications	0.19	0.06
	Management on the admission of technical operators	0.28	0.09
Effectiveness	Quantity of operated cases	0.35	0.10
	Follow-up evaluation of patients	0.12	0.03
	Recovery rate	0.25	0.07
	Improvement rate	0.28	0.08
Innovativeness	Advantages toward conventional/similar technologies	0.42	0.12
	Generalization of the technology	0.27	0.07
	Priority of the technology in the professional field	0.31	0.08
Benefits	Economic benefits	0.56	0.07
	Social benefits	0.48	0.05

DISCUSSION

Selection of initial indicators of the system

According to general consensus, the basis and core elements of medical technologies management within medical institutions are consist of safety, effectiveness, innovativeness, and the benefits.[18] In the light of the principle, the study initially set the four basic subjects as the primary indicators of the evaluations system. For the record, since all the medical equipment and material applied in hospitals went through the safety and effectiveness assessment by medical administration departments, the evaluation on safety and effectiveness of medical technologies mainly refers to the operations performed by medical staffs. Similarly, the secondary indicators are drafted according to clinical experience and preliminary research reports, and the acquisition of all the indicators should be linear indexes (such as the secondary indicator “quantity of operated cases” that could be rated by number and “recovery rate” that could be marked by ratio under the column of “effectiveness”) or that could be valued through unified and objective grading standards (such as the secondary indicator “the priority of the technology in professional field” that could be graded by “globally advanced”, “domestic advanced”, “regional advanced”, “horizontal advanced”, “well-developed” as well as “under-developed” and marked by arithmetic sequence number). On this basis, some initial indicators that are hard to be quantitatively estimated (such as “the origination of innovations”) are eliminated or replaced by other indexes.

Application and advantages of the evaluation system and its indicators

The evaluation system we built provided a methodology for medical institutions to analyze the comprehensive adaptability of clinical technologies since all the indicators could be measured quantitatively that made it possible for different medical technologies comparable with one and another.[19] Compared with previous studies, the advantage and innovativeness of the system lie in its accuracy and objectiveness. Besides, the evaluation system along with the indicators enable medical institution to make assessment in medical technologies of all fields (including surgical operations, noninvasive operations, laboratory operations as well as image inspection operations and so on), all classes ( first-, second-, and third-class technologies), and all phases (preadmission and postadmission phase), which provided a standardized, comparable platform in medical technology management for medical institutions.[20] Moreover, the results could be adopted in the assessment of individuals, departments, and institutions of medical fields.

Verification of the evaluation system and adjustment of indicators

To better apply the evaluation system into practical operation, further verification of the indicators should be made in the clinical application. Besides, since the availability and suitability of all the indicators might be changing with time, the reassessment and adjustment of all the indicators should be dynamically made over a certain period.[21]

Study limitations

The accuracy and objectiveness of the evaluation system and the indicators are closely related to the practical situation of medical institutions among different regions.[22] The degree of development, structure, and scale of medical institutions might all have a direct impact on indicators.[13] Since the research was performed within a regional medical institution and all the experts we surveyed are from medical fields in Beijing, considering the different situation in various regions, as well as the limitation of the data and material, the generalization of the results should be more careful in other circumstances.[23]

In conclusions, though a two-round questionnaire survey of experts and statistical analysis, along with the verification on the credibility of the results through consistency evaluation test, the study established a quantitative medical technology evaluation system model and assessment indicators within medical institutions based on the Delphi method and analytical hierarchy process.[24] It is believed that the system could be better applied to clinical practice through continuous revalidation and dynamic adjustment with time. Furthermore, the methodology of the study could be extended to other subjects of hospital evaluation.

Supplementary information is linked to the online version of the paper on the Chinese Medical Journal website.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.