Performance Index for Types of Clinical Research Support Service Providers for Academic Research Organizations in Japan: A Cross-Sectional Survey.

The purpose of this study was to conduct a factual survey to evaluate the type of clinical research support offered by service providers (supporters) in Japanese academic research organizations (AROs). From September to October 2018, we conducted an online questionnaire targeting researchers and supporters of AROs, including individuals supporting research and development (R&D) planning, as well as those involved in study management, biostatistics, coordination, data management, monitoring, and auditing. The number of responses was tabulated for each survey item. For items with written descriptions, we compiled summaries using the inductive regression method of qualitative research. Responses were obtained from 124 researchers, 258 supporters, and 40 AROs. None of the institutions responded that they had a performance index for all types of service providers, whereas 47% of institutions had an index for 1-3 types of service providers, and 40% of institutions had no index. Many institutions responded that they had a performance index for coordinators and data management, but few responded that there was a performance index for individuals engaged in R&D and study management. Furthermore, for all evaluations of AROs and researchers, the level of supporter satisfaction was low at only 20%. There was a discrepancy between the levels of researcher expectations and the actual contribution of R&D in the process of research planning. Our survey revealed that there is currently no performance index for services supporting clinical research. In future studies, we need to examine a performance index that accurately reflects the researcher attitudes revealed in this study.

WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

This is the first study focusing on performance measurement for supporters working in Japanese academic research organizations (AROs).

WHAT QUESTION DID THIS STUDY ADDRESS?

This study addressed whether AROs have performance evaluation indices for professions serving within AROs and what kind of performance evaluation index the ARO supporters and researchers think is appropriate.

WHAT DOES THIS STUDY ADD TO OUR KNOW LEDGE?

Data revealed that there is currently no performance index for services supporting clinical research. Many ARO had a performance index for clinical research coordinator and data management, but few responded that there was a performance index for individuals engaged in research and development (R&D) and study management. The level of supporter satisfaction was low at only 20%. There was a discrepancy between the levels of researcher expectations and the actual contribution of R&D. The data showed researchers evaluate support work well if it contributed to academic achievement, whereas supporters focused on implementation aspects.

HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

This research could promote discussion of the ARO function in Japan, leading to the development of an index to measure the performance of ARO supporters.

Academic research organizations (AROs) are established in medical institutions, such as university hospitals, and have specialized functions to support innovative research originating from academic researchers. Services provided by professionals working at AROs (referred to as supporters hereafter) include research planning, regulatory strategy, pharmaceutical affairs, and intellectual property consulting, as well as specialists in translation, nonclinical assessment, manufacturing, clinical protocol preparation, project management, study management (SM), data management (DM), clinical research associate (CRA), clinical research coordinator (CRC), and secretariat services in the implementation stages.

In Japan, AROs have been installed based on the healthcare policy promotion council and aim to accelerate medical innovation. Effective from 2015, “core clinical research hospitals” registered by the Ministry of Health, Labour, and Welfare of Japan are required to work jointly with AROs to promote high quality multicenter clinical research and translational research from basic science to innovative drug and medical device development. Core clinical research hospitals play a central role in conducting clinical research and investigator‐initiated investigational new drug trials (called “Chiken”) that meet global standards. , , Furthermore, to improve the quality of research and to implement stricter management of conflicts of interest, great importance was placed on AROs to indirectly support the principles of the Clinical Trials Act enacted in 2018.

With respect to translational research, substantial progress has been seen in recent years. Since the system was officially introduced in 2002, over 110 investigator‐initiated trials have been submitted as of 2018. Moreover, since the 2007 launch of the Ministry of Education, Culture, Sports, Science, and Technology’s translational research promotion project, 33 applications for pharmaceuticals, medical equipment, and regenerative medicine technologies have been submitted for regulatory approval. , Compared with the number of academic ideas developed through the US Food and Drug Administration (FDA), showing 252 pharmaceutical products approved between 1998 and 2007 and the 801 products approved since the establishment of the FDA, , there have been few successful cases reported in Japan. Nevertheless, the steady increase in approved products originating from academic ideas demonstrates constant development. In contrast, the function of coordinating high‐quality multicenter research is still developing. Although the establishment of self‐sustainable AROs is recommended by the Ministry of Health, Labour, and Welfare in Japan, adequate billing for ARO service cannot be attained to achieve ARO independence. Data suggests that, in AROs, expenses were charged for 46% of the overall basic fees (80% of core clinical research hospitals). Challenges for AROs, such as the gap in wages between ARO personnel and those in contract research organizations (CROs), which provide similar services, must be considered to ensure experienced personnel. Because there are also serious human resource shortages at their own institutions by comparison with overseas AROs, this reveals that support remains insufficient to fulfill the network function of multicenter research, including international research. , ,

For such AROs to function adequately, long‐term employment of talented personnel and training is needed. However, at present, problems are mounting that have a negative impact on securing human resources, including work environment, wages, career path of workers, and personnel evaluations. , Among these, work evaluation and career path pose serious problems. For instance, some are of the opinion that there are insufficient regular teaching posts in the AROs of core hospitals and that the career path is unclear. It has been pointed out that, as an immediate solution for staffing requirements at core clinical research hospitals, reliance on experience from previous employment at private enterprises would be required. However, there is also the opinion that there is insufficient time allowed for human resource training and continued employment of the younger generation and that the work environment does not inspire motivation. , , In the future, from a long‐term perspective, it is imperative to establish and present an attractive career path to younger employees based on an accurate evaluation. However, with regard to the various ARO services, it is extremely difficult to implement appropriate service assessments based on a validated index. Furthermore, there are no studies that have reported on the evaluation of supporters working in AROs.

We planned the present survey with the aim of assessing the current state of evaluation of services provided by ARO supporters. In the present study, we surveyed the presence or absence of a performance index for evaluating services in each ARO. Our goal was to elucidate better methods of evaluating ARO personnel, as expressed through the perspectives of both supporters and researchers through an opinion poll.

METHODS

This study was a cross‐sectional (survey), qualitative study. The questionnaire was conducted by anonymous completion of open‐ended questions on the Internet (using REDCap). The data collection period was from September to October 2018. In all, 184 institutions belonging to an ARO group in Japan were asked to complete the questionnaire.

Among the institutions that responded, three targets were identified: (1) ARO representatives (ARO); (2) researchers working for an organization in an ARO (researchers); and (3) workers registered with an ARO who support clinical research (supporters). Supporters were subclassified according to the notice pertaining to the Clinical Trials Act, into individuals supporting research and development (R&D), individuals engaged in SM, biostatistics (STAT), CRC, DM, CRA, and auditing. R&D is defined as an individual who sets the overall direction of the research and supports the optimization with an effective research plan or development strategy from a standpoint related to clinical pharmacology, general clinical practice, and clinical research‐related legislation. SMs are those who manage the smooth operation of clinical research based on knowledge and techniques for planned and efficient operational management of clinical research. Survey items are presented in Figure and Table . The items in the questionnaire referred to the background, performance index relevance, and possible factors for performance evaluation of ARO professions.

Figure 1

Study design. ARO, academic research organization; CRA, clinical research associate; CRC, clinical research coordinator; CRO, contract research organization; DDS, doctor of dental surgery; DM, data management; MD, medical doctor; R&D, research and development; SM, study management; STAT, biostatistics.

Background questions included institution attributes, number of physicians, and current state of clinical research support. For ARO supporters and researchers, questions were asked regarding age, sex, national medical qualifications, qualifications relevant to current duties, final degree, number of years and number of cases of clinical research, number of clinical research articles authored, and number of experiences related to clinical trials (including investigator‐initiated clinical trials, principle investigator, responsibility of service, and industry/CRO service). Researchers were also asked about their experience in requesting support.

Of the questions pertaining to performance index relevance, AROs were asked if there are specific measures to evaluate the performance index for each of the seven types of ARO professions and the need for a performance index for each ARO profession. Items related to the performance index policy in AROs, including institute attributes, education, and number of investigator‐initiated trials, were examined. Next, to clarify how this performance evaluation affects ARO professionals in their motivation to work, we asked researchers the importance and necessity for each type of ARO profession to help improve the quality of their research. We also asked about expectations and actual contribution levels regarding each support type. The satisfaction levels of supporters, including evaluation and motivation to perform their duties, were also assessed (see Figure and Table for details).

To determine the appropriate measure of performance, we asked about the possible factors for performance evaluation in ARO professions. Items included authorship, charge, qualification, and satisfaction level compared with CRO. For authorship questions, ARO, researchers, and ARO professionals were asked whether the supporters should be considered co‐authors or co‐researchers. The charge question for ARO included whether or not money is charged for each type of work. Qualification indicated whether researchers consider the job qualification holder an expert. Regarding comparison with CRO, researchers responded with a direct comparison between CRO and ARO in terms of satisfaction levels for their support service. Supporters were also asked for a comparison of CRO in terms of similarity of both duties. Last, ARO researchers and ARO professionals were similarly asked to freely respond to describe the index evaluating the quality of support services (see Figure and Table for details).

The survey items were determined by 5 supporters with 3 or more years of ARO experience. The number of responses corresponding to each survey item, as well as the associated proportions, was tabulated. Written descriptions were analyzed using systematic cause analysis techniques and adopting a qualitative evaluation approach with inductive content analysis. The written descriptions were categorized and checked by two experienced third‐party clinical research supporters. The conceptual systematic cause analysis techniques framework was set in reference to the records of clinical research group meetings and the International Conference on Harmonization (ICH)‐E8.

RESULTS

The background of the ARO, researchers, and supporters who responded is shown in Table and Table , and Table .

Table 1

Characteristics of researchers that provided responses (from 124 institutions)

Information collected	N (%)
Institute attributes
Core hospital	55 (44.4)
Other than core hospital	69 (55.6)
Age
≤ 29 years	0
30–49 years	93 (75.0)
≥ 50 years	31 (25.0)
Sexa
Male	108 (87.1)
Female	12 (9.7)
No. of years of clinical research experience
< 5 years	30 (24.2)
6–9 years	28 (22.6)
10–19 years	48 (38.7)
≥ 20 years	18 (14.5)
No. of clinical trials experience
None	5 (4.0)
< 5	29 (23.4)
5–9 trials	31 (25)
10–29 trials	38 (30.7)
≥ 30 trials	21 (16.9)
No. of experiences related to clinical research
Investigator‐initiated clinical trials (Chiken)
Yes	62 (50.0)
No	62 (50.0)
Principal investigator
None	39 (31.5)
< 5 trials	50 (40.3)
5–9 trials	21 (16.9)
10–29 trials	10 (8.1)
≥ 30 trials	4 (3.2)
No. of articles authored for clinical research participated in
None	26 (21.0)
< 5	52 (41.9)
5–9	20 (16.1)
10–29	18 (45.0)
≥ 30	8 (20.0)
Experience requesting clinical research support
Both	26 (21.0)
ARO only	28 (22.6)
CRO/SMO only	5 (4.0)
None	65 (52.4)

ARO, academic research organization; CRO, contract research organization; SMO, site management organization.

There are some missing data.

Table 2

Characteristics of supporters who provided responses (n = 258)

Support typea	R&D	SM	CRC	DM	Monitoring	Statistical analysis	Auditing	Total
No. of individuals per service type	57	72	66	31	13	13	6	258
	No. (%)	No. (%)	No. (%)	No. (%)	No. (%)	No. (%)	No. (%)
Institute attributes
Core hospital	29 (50.9)	37 (51.3)	43 (65.2)	18 (58.1)	4 (30.8)	7 (53.8)	5 (83.3)	143 (55.4)
Other than core hospital	28 (49.1)	35 (48.6)	23 (34.8)	13 (41.9)	(69.2)	6 (46.2)	1 (16.7)	115 (44.6)
Age
< 29 years	6 (10.5)	3 (4.2)	9 (13.6)	1 (3.2)	0	0	0	19 (7.4)
30–49 years	32 (56.1)	39 (54.2)	43 (65.2)	24 (77.4)	9 (69.2)	11 (84.6)	5 (83.3)	163 (63.2)
≥ 50 years	19 (33.3)	30 (41.7)	14 (21.2)	6 (19.4)	4 (30.8)	2 (15.4)	1 (16.7)	76 (29.5)
Sexa
Male/female	36/19	37/31	8/56	14/16	4/9	9/2	3/3	111/136a
National medical qualificationb
Doctor/dentist	8 (14.0)	17 (23.6)	1 (1.5)	3 (9.7)	1 (7.7)	2 (15.4)	2 (33.3)	34 (13.2)
Pharmacist	22 (38.6)	23 (31.9)	11 (16.7)	3 (9.7)	3 (23.1)	1 (7.7)	2 (23.3)	65 (25.2)
Other	4 (7.0)	10 (13.9)	50 (75.8)	1 (32.2)	4 (30.8)	1 (7.7)	1 (16.7)	80 (31.0)
None	23 (40.4)	22 (30.6)	4 (6.1)	15 (48.4)	5 (38.5)	9 (69.2)	1 (16.7)	79 (30.6)
With final (doctoral) degree	24 (42.1)	31 (43.1)	4 (6.1)	9 (29.0)	3 (23.1)	7 (53.9)	3 (50.0)	81 (31.4)
With qualification to perform servicea	16 (28.1)	31 (43.1)	22 (33.3)	6 (19.4)	2 (15.4)	2 (15.4)	2 (33.3)	81 (31.4)
No. of experiences in clinical research support
< 5	25 (43.9)	18 (25.0)	16 (24.2)	6 (19.4)	3 (23.1)	2 (15.4)	3 (50.0)	73 (28.3)
5–9	10 (17.5)	13 (18.6)	15 (22.7)	9 (29.0)	3 (23.1)	4 (30.8)	0	54 (20.9)
10–29	9 (15.8)	14 (19.4)	8 (12.1)	11 (35.5)	3 (23.1)	0	2 (33.3)	47 (20.9)
≥ 30	13 (22.8)	27 (37.5)	27 (40.0)	5 (16.1)	4 (30.8)	7 (53.9)	1 (16.7)	84 (32.6)
No. of years of clinical research experience
< 5 years	37 (64.9)	29 (40.3)	41 (62.1)	11 (35.5)	6 (46.2)	7 (53.8)	2 (33.3)	133 (51.6)
6–9 years	4 (7.0)	14 (19.4)	10 (15.2)	10 (32.3)	1 (7.7)	1 (7.7)	1 (16.7)	41 (15.9)
10–29 years	11 (19.3)	16 (22.2)	14 (21.2)	7 (22.6)	2 (15.4)	3 (23.1)	2 (33.3)	55 (21.3)
≥ 20 years	5 (8.8)	13 (18.5)	1 (1.5)	3 (9.7)	4 (30.8)	2 (15.4)	1 (16.7)	29 (11.2)
No. of experiences relative to clinical trials
With responsibility of service	18 (31.6)	26 (36.1)	4 (6.1)	13 (41.9)	3 (23.1)	10 (76.9)	6 (100)	80 (31.0)
With drug industry/CRO service	15 (26.3)	23 (31.9)	10 (15.2)	10 (32.3)	3 (23.1)	5 (38.5)	3 (50.0)	69 (26.7)
No. of articles of clinical research supported
Without	41 (71.9)	43 (59.7)	65 (98.5)	24 (77.4)	8 (61.5)	3 (23.1)	3 (50.0)	187 (72.9)
< 10	12 (21.1)	24 (33.3)	1 (1.5)	4 (12.9)	4 (30.8)	6 (46.2)	1 (33.3)	52 (20.2)
10–29	2 (3.5)	4 (5.6)	0	3 (9.9)	1 (7.7)	1 (7.7)	2 (50.0)	13 (5.1)
≥ 30	2 (3.5)	1 (1.4)	0	0	0	3 (23.1)	0	6 (2.3)

CRC, clinical research coordinator; CRO, contract research organization; DM, data management; R&D, research and development; SM, study management.

GCP (good clinical practice) passport, CCRP (SOCRA)/ACRP, PMP, accredited statistics expert, and patent attorney.

There are some missing data.

The following were listed under free descriptions section: clinical‐related expert, society‐accredited qualification (CRC, approved specialist in cancer clinical research.

Complete responses were obtained from 40 of the 184 institutions (21.7%) to which the questionnaire was sent. From these institutions, a total of 124 researchers and 258 supporters responded. The number of each support service type is shown in Table .

Only 28 researchers (22.6%) had experience asking for support from an ARO, whereas 5 researchers (4.0%) had experience asking for support from a CRO, and 65 researchers (52.4%) had no experience asking for support from either. Approximately half of the supporters who responded were from core clinical research hospitals. Few supporters were younger than 29 years old (19 of 258, 7.4%), and most were aged between 30 and 49 years (163 of 258, 63.2%). Eighty supporters (31%) had leadership experience, whereas 26.7% had experience working in private enterprise. More people in STAT, R&D, or SM had experience as managers, whereas few people in CRC had management experience. The number of years of support experience was < 5 years for the majority of supporters (133, 51.6%), and 187 supporters (72.9%) had no experience as authors; notable exceptions include 3 statistics supporters (23.1%) and 2 R&D supporters (3.5%) who had authored > 30 articles.

With regard to the presence or absence of a performance index (Figure ), there were no AROs with evaluations for all seven service types. There were 10 (25%) institutions with a performance index for 1 type and 6 (15%) with performance indices for 2 types. Seventeen institutions did not have evaluation indices for any type of ARO profession. The proportion of institutions without a single performance index tended to be low for core clinical research hospitals, and there was no clear relationship between the number of investigator‐initiated trials support or training program (Figure ). AROs tended to have performance indices for CRC (18, 45%), DM (6, 37.5%), and STAT (11, 30%), but there was no performance index obtained from 38 institutions (95%) for R&D nor from 34 institutions for SM (85%), despite a belief that these are needed (Figure ).

Figure 2

Whether there is any performance index for academic research organization (ARO) professions. (a) Number of institutions with performance indices for research supporters. Answers were categorized by the numbers of professions with a performance index. (b) Number of institutions with and without factors related to performance index: core hospitals, education, and supports experience of investigator‐initiated trials. (c) Number of institutions with a performance index for each ARO profession. ARO professions were divided into seven categories, including research and development (R&D), study management (SM), clinical research coordinator (CRC), data management (DM), clinical research associate (CRA), biostatistics (STAT), and audit. Answers were categorized into “without need,” “without index but with need,” and “with index.”

Figure shows that > 90% of researchers answered “important” and “necessary” for ARO support in each profession. R&D and STAT particularly ranked high in importance. STAT, DM, and CRA ranked relatively high in necessity. Researchers tended to consider the design planning stages particularly important, with statements such as “The hypothesis is set clearly and precisely for the clinical question,” “The design framework is set to determine whether the hypothesis holds true,” and “The design can eliminate any bias from each service type better” (Figure ). Regarding how supporters felt about evaluations, satisfaction in the ARO evaluation and the researcher’s evaluation was indicated by 19% and 22% of supporters, respectively, and satisfaction was greater than dissatisfaction (Figure ). Discrepancies between researcher expectations and actual contributions from AROs were evident in R&D (expectation > contribution) and STAT (contribution > expectation; Figure ).

Figure 3

The importance and necessity of academic research organization (ARO) supports and how the researcher feel their contributions. (a) The importance and the necessity of each ARO supports to improve research quality. Researchers responded to the importance and necessity of each ARO profession with five degrees. ARO professions were divided into seven categories, including research and development (R&D), study management (SM), clinical research coordinator (CRC), data management (DM), clinical research associate (CRA), biostatistics (STAT), and audit. (b) Both supporters and researchers answered the questions regarding which items were important to improve the quality of study; this included 11 items. (c) Expectation and actual contribution of each ARO profession. Researchers responded regarding the levels of expectation and contribution for each of the 11 items to improve study.

Figure 4

The levels of satisfaction for AROs/researcher evaluation and supporter motivation. With regard to evaluations from AROs and evaluations from researchers, data were collected according to a five‐point scale (unsatisfied, somewhat unsatisfied, normal, somewhat satisfied, and satisfied), which was then tabulated according to a three‐point scale (satisfied = somewhat satisfied and satisfied, normal = normal, and unsatisfied = unsatisfied and somewhat unsatisfied). With regard to motivation, data were collected according to a five‐point scale (I want to improve more with this type of service, I am presently satisfied, I am indifferent, I am exhausted, and I want to quit). This was then tabulated according to a three‐point scale (satisfied = I want to improve more with this type of service and I am presently satisfied, normal = I am indifferent, and unsatisfied = I am exhausted and I want to quit. ARO, academic research organization; CRA, clinical research associate; CRC, clinical research coordinator; DM, data management; R&D, research and development; SM, study management; STAT, biostatistics.

Overall, the level of satisfaction in evaluations from both AROs and researchers was ~ 20%. CRA and STAT showed relatively high levels of satisfaction, particularly from researchers. The levels of dissatisfaction among ARO staff were high in R&D and DM. Among supporters, 35.7% answered that their motivation was high, but 24% indicated low motivation. Among all types of service providers, only CRA had motivation levels higher than 50% (62.0%). The motivation among SM, R&D, and STAT was relatively high. Dissatisfaction levels among CRC (30%) and DM (29%) were greater than the satisfaction levels (Figure ).

Regarding items related to performance evaluation, the data showed that researchers considered the contribution of STAT and R&D as co‐authors or co‐researchers (co‐authors, 58.9% for R&D and 72.6% for STAT; Table ). This answer was not contradicted among ARO or ARO professionals, but supporters, with the exception of STAT, tended to show a low percentage of people motivated to be article authors in this survey.

Table 3

Other factors possibly related to the performance index relevance

	Authorship				Cost		Qualification	Differences between CRO and ARO		Which type of performance index is appropriated to evaluate the quality of support services (free description)
	Consider each ARO professions as authors			Consider each ARO professions as co‐researchers	Charged for ARO tasks	Uncharged for ≥ 80% of ARO tasks		Similar	ARO is better than CRO
	ARO (n = 40)	Researcher (n = 124)	Supporter (n = each)	Researcher (n = 124)	ARO (n = 40)		Researcher	Supporter	Researcher/both experienced	ARO	Supporter	Researcher
R&D (n = 57)	28 (70.0)	73 (58.9)	11 (19.2)	61 (49.2)	27 (67.5)	5 (13.5)	76 (61.3)	9 (15.8)	7/15 (46.7)	Research planning (24) Client satisfaction (11), Research achievement (the number of articles, Impact factor, ,paper citation .) (24), Implementation of research support systems (speed, time schedule) (21), Assessment by others (client satisfaction (33), Research planning discontinuation and deviation) (21)	Compatibility (responsiveness, risk management) (16), Implementation of support systems (speed, timing, achievement rate, and cost effectiveness) (66), Research achievement (article publication (29), Ability of ARO professions (17), Service accuracy (15)	Teamwork (communication, cooperativeness, etc.) (13), Level of satisfaction (17), Research planning and design (20), Deviation (4)
SM (n = 72)	15 (37.5)	26 (21.0)	13 (18.1)	26 (21.0)	29 (72.5)	6 (17.6)	54 (43.5)	17 (29.8)	0/7
CRC (n = 66)	4 (10.0)	7 (5.6)	5 (7.7)	18 (14.5)	30 (75.0)	10 (25.6)	80 (64.5)	29 (43.9)	2/8 (25.0)
DM (n = 31)	10 (25.0)	21 (16.9)	7 (22.6)	19 (15.3)	30 (75.0)	1 (3.0)	58 (46.8)	13 (41.9)	3/11 (27.3)
CRA (n = 13)	6 (15.0)	4 (3.2)	1 (7.7)	9 (7.3)	28 (70.0)	8 (28.2)	58 (46.8)	5 (38.4)	1/10 (10.0)
STAT (n = 13)	32 (80.0)	90 (72.6)	6 (46.2)	67 (54.0)	29 (72.5)	3 (8.8)	85 (68.5)	4 (30.8)	4/9 (44.4)
Audit (n = 6)	4 (10.0)	1 (0.8)	2 (33.3)	6 (4.8)	15 (37.5)	3 (20.0)	52 (41.9)	0	0/5

ARO, academic research organization; CRA, clinical research associate; CRC, clinical research coordinator; CRO, contract research organization; DM, data management; R&D, research and development; SM, study management; STAT, biostatistics.

A high number of researchers evaluated supporters holding a service‐related qualification in STAT at 68.5%, CRC at 64.5%, and R&D at 61.3%. The qualifications of ARO referred to for the evaluation of each service type, such as academic society accredited qualifications in CRC and STAT, are presented in Table . However, opinions varied with each type of service.

Our data showed that ~ 60–70% of AROs answered in favor of having charge settings for ARO support across the types of service. However, there are several services that cannot be charged. More than 20% of AROs responded that > 80% of services are uncharged in DM and CRC.

The final questions compared CRO and ARO in each type of profession. Although only a few researchers had experience asking for support for both, R&D and STAT showed a relatively high rate in ARO. Interestingly, this result is compatible in that there is a relatively small number of supporters who believe the work of ARO and CRO are similar in R&D (Table , Table , Table ).

In summary of our survey, we asked researchers, supporters, and AROs what they thought of performance evaluation measures for support services using free description (Table , Table , Table , Table ). AROs commonly indicated the following: research planning (n = 24); client satisfaction (n = 11); research achievement (n = 24), including number of articles; and implementation of research support systems (n = 21), including speed, time, and schedule. Researchers indicated the following: research achievement (n = 29), including article publication; ability of ARO professionals (n = 17); service accuracy (n = 15); teamwork (n = 13), including communication, cooperativeness, etc.; level of satisfaction (n = 17); and research planning and design (n = 20), including deviation (n = 4). Supporters indicated assessment by others (n = 33), including client satisfaction; research planning (n = 21), including discontinuation and deviation; compatibility (responsiveness; n = 16), including risk management; and implementation of support systems (n = 66), including speed, timing, achievement rate, and cost‐ effectiveness.

DISCUSSION

The present study is the first study focusing on performance measurement for supporters working in Japanese AROs. Data revealed that very few AROs have a performance index for each type of service provider. The presence or absence of a performance index differed for each service type; furthermore, there were no institutions with a performance index for all seven service types, but some institutions responded that they had an index for a few services. It was inferred that rather than an institutional problem, the problem is more in the lack of a systematic performance index for ARO supporters overall. Regarding this evaluation from AROs, we found that only 19% of supporters (i.e., the assessed group), were satisfied overall. The level of satisfaction in the evaluation was somewhat high among researchers, with a high consistency among items and service types. However, there tended to be a discrepancy in R&D, suggesting a high level of dissatisfaction from AROs. Interestingly, this was consistent with the lack of performance index in AROs for this type of profession.

This study did not reveal the context of the performance index for each profession. Instead, we asked what researchers, supporters, and AROs thought about the performance index to measure the quality of support work by written description. Researchers tended to place importance on research items, such as articles, social impact, and practical application, whereas supporters and AROs tended to evaluate the practical side of services, such as timing, speed, risk management, and accurate execution. We also found that researchers believed STAT and R&D were primarily types of services expected to improve the quality of research in the research design stage. For R&D, in particular, the level of expectation was considered high when compared with the level of contribution. These inconsistent data in R&D, the profession which researchers consider important for high quality research, raise an issue concerning how research quality should be evaluated and how ARO research quality should be assessed.

To discuss how supporter evaluation is presently performed, we focused on the backgrounds of responding supporters. In the present study, most researchers and supporters were mid‐level in their organizations, between the ages of 30 and 49. This perhaps reflects the rapid development of organizations in recent years; however, few had experience as an ARO. Instead, the percentage of individuals with experience working in the private sector and individuals in charge of services was high at 30%. In particular, for R&D and SM, individuals aged over 50 years accounted for 33.3% and 41.7%, respectively, of the responders, which implies that there were many second‐career personnel assigned because of their high level of management experience. For AROs, which are relatively new organizations, the number of workers believed to hold leadership or managerial positions was relatively high as a result of having built the organization on the immediate strength of experienced personnel. A hypothesis can be drawn that the evaluation method reflects the experience of earlier employment, such as in private enterprise. As to whether the academic results sought by researchers, such as the impact on actual clinical practice in articles and guidelines, are consistent with the results sought by experienced individuals, such as those from enterprise, and support methods for the results sought, as well as the question of how these should be evaluated, all need to be examined carefully.

In that respect, perhaps attention should be given to other factors related to performance evaluation and career paths specific to academia. For example, in academia, support is regarded as intellectual contribution, and supporters are generally deemed co‐researchers and article authors when they satisfy the conditions for authorship outlined in article guidelines. This differs greatly from the service outcomes in CRO, where achievement is related to costs and profits. In particular, private enterprises place importance on profit, and, in that respect, it is argued that free support accounts for a large proportion of support in AROs. For service types involved in the preparatory stage before research funding has been obtained, this could be linked to results in that there is no payment collection for performance. This may also give rise to inconsistency with the reality of STAT, for which researchers judged the contribution to be high, and R&D, which is considered to be important but payment from the researcher is low. Therefore, from the perspective of CROs, it can be inferred that ARO service types cannot be properly evaluated. A recent discussion regarding the difference between the functions of AROs and CROs has been underway overseas. Some are of the opinion that AROs should play a role in strategic planning in close cooperation with academia and that a cooperation model should be built with CROs. Based on our results, we found that researchers value the service execution function in CROs. In terms of access to researchers and scientific perspective, researchers were satisfied with AROs, which is consistent with the conventional goals of establishing AROs (i.e., having high scientific standards that are closer to the researcher, and seeking accompanying support to implement cutting‐edge ideas sprouted from academia).

This leads us to the question of which specific index should be implemented in the evaluation of ARO supporters. First, the duties and characteristics of service types need to be identified. Some types of support work are familiar to researchers, but others are not. In the present results, we found a difference in patterns of thinking about items in which the importance of support was explained in R&D and STAT in the preparatory stage, as well as the researchers’ opinions on authorship compared with other service types. The details of services, such as CRC, are relatively easier for healthcare workers to understand with clear qualifications, and the skill evaluation method has also been studied. In contrast, with regard to service types that are unfamiliar with the definition of the Clinical Trial Act (R&D and SM), the knowledge of researchers may have affected the responses provided. Furthermore, many researchers who responded in the present study had little experience in requesting support. It is possible that results could have been affected by the underlying background (i.e., it was easier for them to understand problems involving planning than those involving implementation). Therefore, as the first step in applying the evaluation viewpoint from researchers to the evaluation of ARO supporters, it is important to clarify the service details.

Next, we probably should reconsider the goals of AROs. Based on the difference with CROs, attempts have been made to discuss the function of an ARO. However, according to our present survey, researchers are satisfied with research support to develop research in which results can be obtained and they therefore place particular importance on planning and design. It has been suggested that support that promotes research with higher scientific standards be evaluated. This may serve as a major direction in the future when examining the evaluation axis of ARO supporters. Based on the evaluation of costs, and the number of cases, etc., examinations should perhaps shift to evaluations focusing on results and scientific standards.

For example, perhaps evaluations should target whether service types involved in the design stage, such as STAT and R&D, offer advice with high scientific standards, or whether research development and logical support has been performed from the perspective of regulatory science. Support for the management of existing evidence, regulatory strategy, and administrative systems, or surveys for target commercial and medical settings may be included. In contrast, with regard to service types engaged in implementation, such as CRC and DM, there was significant discrepancy in the opinions of supporters and researchers. Therefore, the viewpoint that evaluations are performed to improve the quality of research may be the same as the current ARO evaluation method for researchers. Based on the free descriptions in the questionnaire, these service types are more greatly associated with quality management of research and tend to evaluate process control and accuracy rather than outcomes. However, even when basing evaluations on processes and procedures, it is possible that methods to improve actual practice involving the implementation of a protocol with high scientific standards and contribution to the creation of appropriate processes based on new ideas could be applied to the current practice‐based evaluation axis, taking into consideration the aims of the ARO and objectives of the researcher.

In the present study, we examined support evaluation methods based on a thorough opinion poll of researchers, supporters, and AROs. However, the survey did not directly ask about or evaluate the current evaluation methods. To verify the present state revealed by the current survey, we need to extract several current typical evaluation methods and examine them separately. Furthermore, the present survey is only that of current perceptions and does not include the viewpoint of outcomes, such as study results and achievements. Further study is needed to evaluate studies that meet researcher expectations and whether support helped produce those results. Another limitation is that the statutory definition in the Clinical Trials Act was used for the present definition of supporters. Therefore, it is possible that the opinions of supporters engaged in strategic planning (particularly translation, intellectual property‐ related support, and university research administration) are not reflected in the responses. As to whether this definition includes development support responsibility is difficult to interpret based on the regulations for the enforcement of the Clinical Trials Act. In the future, we need to examine opinions of these service types. In conclusion, we found that there are no established methods for performance measurement for each service type in AROs. For the development of ARO performance evaluation, researchers’ demands for improving scientific standards are the key.

Funding

The present study was sponsored by the Japan Agency for Medical Research and Development 2018 project for the evaluation of Academic Research Organizations (Shoji Sanada) Grant Number: 18lk1903001h0002.

Conflicts of Interest

All other authors declared no competing interests for this work.

Author Contributions

K.U. and S.S. wrote the manuscript. K.U., N.U., K.M., Y.N., N.T., and T.O. designed the research. K.A., Y.O., N.Y., S.Y., R.U., and Y.S. performed the research. Y.N., N.T., N.Y., H.N., K.U., and S.S. analyzed the data.

Disclosure

As an Associate Editor of Clinical and Translational Science, Naoto Uemura was not involved in the review or decision process for this paper.

Table S1

Click here for additional data file.

Table S2

Click here for additional data file.

Table S3

Click here for additional data file.

Table S4

Click here for additional data file.

Table S5

Click here for additional data file.

Table S6

Click here for additional data file.

Table S7

Click here for additional data file.

Table S8

Click here for additional data file.