How are trial outcomes prioritised by stakeholders from different regions? Analysis of an international Delphi survey to develop a core outcome set in gastric cancer surgery.

BACKGROUND: International stakeholder participation is important in the development of core outcome sets (COS). Stakeholders from varying regions may value health outcomes differently. Here, we explore how region, health income and participant characteristics influence prioritisation of outcomes during development of a COS for gastric cancer surgery trials (the GASTROS study).
METHODS: 952 participants from 55 countries participating in a Delphi survey during COS development were eligible for inclusion. Recruits were grouped according to region (East or West), country income classification (high and low-to-middle income) and other characteristics (e.g. patients; age, sex, time since surgery, mode of treatment, surgical approach and healthcare professionals; clinical experience). Groups were compared with respect to how they categorised 56 outcomes identified as potentially important to include in the final COS ('consensus in', 'consensus out', 'no consensus'). Outcomes categorised as 'consensus in' or 'consensus out' by all 3 stakeholder groups would be automatically included in or excluded from the COS respectively.
RESULTS: In total, 13 outcomes were categorised 'consensus in' (disease-free survival, disease-specific survival, surgery-related death, recurrence of cancer, completeness of tumour removal, overall quality of life, nutritional effects, all-cause complications, intraoperative complications, anaesthetic complications, anastomotic complications, multiple organ failure, and bleeding), 13 'consensus out' and 31 'no consensus'. There was little variation in prioritisation of outcomes by stakeholders from Eastern or Western countries and high or low-to-middle income countries. There was little variation in outcome prioritisation within either health professional or patient groups.
CONCLUSION: Our study suggests that there is little variation in opinion within stakeholder groups when participant region and other characteristics are considered. This finding may help COS developers when designing their Delphi surveys and recruitment strategies. Further work across other clinical fields is needed before broad recommendations can be made.

1. Introduction

A core outcome set (COS) is an agreed minimum group of critically important outcomes which should be reported by all trials within a research field [1]. The GASTROS study (www.gastrosstudy.org) aims to develop a COS in the field of gastric cancer surgery to promote uniform reporting of important outcomes and facilitate evidence synthesis [2]. This is necessary as there is significant variation and heterogeneity in this field with respect to reporting and measurement of outcomes [3]. Furthermore, the outcomes chosen by researchers to report in surgical trials for gastric cancer often do not reflect the priorities held by patients [4]. For this reason, the GASTROS study has sought consensus between patients and healthcare professionals with respect to outcome selection.

Delphi surveys and consensus meetings are commonly used methodologies in the development of COS [1, 5]. Delphi surveys ask participants deemed by the study group to hold an important perspective (key stakeholders) to prioritise outcomes and achieve consensus. The completed Delphi survey often informs and influences discussions during a subsequent consensus meeting, with the aim of resolving uncertainties regarding prioritisation and ratifying the final composition of the COS. Clear recruitment strategies for Delphi surveys are an important consideration. If recruitment does not result in representative stakeholder groups, there is a risk that the results of the Delphi may not be valid [6]. This is particularly important in international COS where significant regional and cultural differences may influence the results ahead of a consensus meeting and, ultimately, the final COS.

Ensuring stakeholder groups are representative can be a challenging task. There is a need to consider many factors including the incidence of the disease, treatment protocols, international variation in healthcare systems and values and socio-economic issues. In the case of curative surgery for gastric cancer it is known that practice varies worldwide (e.g. how surgery is carried out and the extent of resection) and typically surgeons value different outcomes to patients [4]. For example, due to screening programmes, cancers are generally earlier in the Far East where patients tend to be younger with fewer co-morbidities. There is therefore a need to explore these issues to understand how key stakeholders are selected for survey participation. In the GASTROS study 952 participants were recruited to a Delphi survey (268 were patients, 445 surgeons and 239 nurses) from 55 countries. It was therefore possible to explore how stakeholder characteristics influenced outcome prioritisation.

This study had two main objectives:

To describe the characteristics of Delphi participants and explore their possible influence on the prioritisation of outcomes within stakeholder groups.

To explore how stakeholders from different regions prioritised outcomes.

2. Methods

This was an analysis of registration data supplied by Delphi survey (S1 File) participants as part of the GASTROS study. Both rounds of the survey took place between March and October 2019. Details of the scope, objectives and methodology of the study have been previously described [2-4]. In summary, participants were asked to score outcomes in terms of importance. The results of the Delphi survey informed discussions in a consensus meeting where final recommendations were made regarding which outcomes to include in the COS.

2.1. Stakeholder selection and baseline information

The GASTROS study sought to involve key stakeholders–patients, surgeons, and oncology nurses—to identify a COS for surgical trials in gastric cancer. Our guiding principle has been to promote the ‘patient voice’ as they are the beneficiaries of trials in this field and have all-important ‘lived experience’. The patient voice has previously been shown to be under-represented in COS development [7]. Surgeons provide a clinical perspective and the experience of treating large volumes of patients. Oncology nurses were invited to participate given their central roles as care-givers, patient advocates and core members of the clinical team.

Recruitment was achieved by promoting the study at surgical and nursing congresses, social media and through patient groups and charities. The study website (gastrosstudy.org) allowed stakeholders to register their interest ahead of the Delphi survey. Local recruitment of patient healthcare professionals by members of the international working group was also undertaken. Participation in the Delphi survey was open to all interested stakeholders who fulfilled the following criteria:

Surgeons who had completed their training and routinely treat gastric cancer.

Oncology nurses with a recognised proportion of their role involved in the care and follow-up of gastric cancer patients.

Patients who have undergone surgical resection for gastric cancer with the intention of cure.

There is no sample size requirement for Delphi surveys. To be able to demonstrate the enrolment of a broad and representative range of stakeholders, participants were asked to provide the information listed below:

Patients:

Age

Sex

Surgical approach (laparoscopic or open)

Type of gastrectomy (total or partial)

Modality of treatment (surgery alone or a combination of surgery and chemotherapy or radiotherapy)

Time since surgery

Surgeons:

Experience (number of gastrectomies undertaken)

Nurses:

Experience (years of service)

These datapoints were developed based on information that was likely to be readily known to participants and the expert opinion of the GASTROS study management group (SMG) with respect to important factors that may influence outcomes or perspectives. In the context of patients, different health outcomes, such as complications and survival, may impact their lived experience and ultimately how outcomes are prioritised. Similarly, as clinical experience changes with time, there may be a greater exposure to and therefore appreciation of the impact or importance of longer-term consequences of surgery.

Additionally, all participants were asked to provide their country of residence so that regional differences could be considered. Participants were categorised into ‘Eastern’ or ‘Western’ countries (Fig 1) and ‘high-income’ or ‘low- to medium-income countries’ as defined by the Organisation for Economic Co-operation and Development’s Development Assistance Committee [8]. Eastern countries were defined as those within East Asia, South East Asia, and Eastern Russia, and included China, Japan, South Korea, Thailand, Vietnam, Malaysia, and Singapore [9]. Western countries were defined as those from Western Europe, North America, Australia, and New Zealand [10]. Contrasting between the ‘East’ and ‘West’ is of particular importance to gastric cancer given the differences in incidence, pathology, treatment and outcome. It was hypothesised that these differences in approach and survival may influence how stakeholders in these regions prioritise different health outcomes which could be examined further in this study [11, 12]. Similarly, health priorities may be influenced by resource availability as categorised by country income.

Fig 1

Countries from which participants were recruited.

Eastern countries were defined as those within East Asia, Southeast Asia, and Eastern Russia, and included China, Japan, South Korea, Thailand, Vietnam, Malaysia, and Singapore [9]. Western countries were defined as those from Western Europe, North America, Australia, and New Zealand [10].

2.2 Scoring of outcomes in the Delphi survey and categorisation of outcomes

A list of 56 outcomes identified from previous trials and patient interviews [3, 4] were presented to survey participants who were asked to rate each outcome on a scale of importance (1–3: not important, 4–6: important, 7–9: critically important). Outcomes were organised according to five core areas (mortality/survival; clinical/physiological outcomes; life impact; resource use; adverse events) based on a taxonomy developed for COS development [13]. Patients, surgeons, and nurses group ratings were considered separately to ensure that each group had an equal voice. Participants had the opportunity to suggest further outcomes that they believed had not been presented in round 1. One additional new outcome suggested by participants in round 1 was identified and after consideration by the SMG was presented to participants for scoring in round 2. Therefore, a total of 57 outcomes were presented in round 2 where, for each outcome, participants were shown the scores from each stakeholder group, and given the opportunity to change their rating if they wished.

After two rounds of rating, outcomes were categorised as follows:

To be included in the COS (‘consensus in’)

To be excluded from the COS (‘consensus out’)

‘No consensus’ reached i.e. no decision reached as to whether the outcome should be included in of excluded from the COS.

Criteria for categorising outcomes was set a priori by the SMG and based on established COS methodology [1]. If an outcome was rated 7–9 (critically important) by 70% or more of a stakeholder group and 1–3 (not important) by no more than 15% of the group, then the consensus amongst that group was that the outcome should be included in the COS. If an outcome was rated 7–9 (critically important) by less than 50% of the group, the consensus amongst that group was for the outcome to be excluded from the COS. Unanimous agreement amongst all three stakeholder groups was required for inclusion in, or exclusion from, the COS. Any other combination resulted in the outcome being placed in the ‘no consensus’ category and was discussed at a pre-planned consensus meeting to finalise the COS.

2.3 Data analysis and interpretation

In round 1, participants completing 50% or more of the Delphi survey were included in the round 1 analysis and invited to participate in round 2. Likewise, participants completing 50% or more of the survey in round 2 were included in the round 2 analysis. For the purpose of this present analysis, participants were placed into ‘sub-groups’ according to the registration data they submitted (e.g. patient treatment type, surgeon experience etc) to examine the differences in outcome scoring. The following analyses were performed after 2 rounds of ratings:

The proportion of participants scoring each outcome as ‘critically important’ (score 7–9). This analysis approach was chosen as these figures were presented in the consensus meeting discussing results from the Delphi survey.

The consensus opinion of each sub-group with respect to whether the outcome should be ‘included’ in the COS, ‘excluded’ from the COS or whether ‘no consensus’ could be reached. These categorisations were compared against the overall ‘in’, ‘out’ and ‘no consensus’ categorisations by each stakeholder group (patients, surgeons and nurses) which was presented to the consensus meeting participants.

Participants not providing demographic data during registration were excluded from the sub-group analyses. When exploring differences in prioritisation, particular focus was placed on outcomes that were categorised as ‘consensus in’ by one sub-group and ‘consensus out’ by another.

To examine the possible influence of attrition bias between rounds, the characteristics of stakeholders participating in both rounds were compared to those who only completed round 1. A descriptive analysis was undertaken, and the Chi squared test applied (using SPSS—IBM Corp. Released 2019. IBM SPSS Statistics for Windows, Version 26.0. Armonk, NY: IBM Corp) to examine for statistically significant differences at the 0.05 level.

2.4 Ethical approval

The study was given ethical approval by the North West—Greater Manchester East Research Ethics Committee (18/NW/0347) and governance approvals by Manchester University Hospitals NHS Foundation Trust. All participants were provided with a participant information booklet. Informed written consent was obtained during the online registration process for participants in the Delphi survey.

3. Results

3.1 Overview

The characteristics of participants included in the analysis and attrition rates are summarised in Table 1. After 2 rounds of voting, agreement was reached amongst all three stakeholder groups to include 13 outcomes (disease-free survival, disease-specific survival, surgery-related death, recurrence of cancer, completeness of tumour removal, overall quality of life, nutritional effects, all-cause complications, intraoperative complications, anaesthetic complications, anastomotic complications, multiple organ failure, and bleeding) into the COS. A further 13 outcomes were excluded from the COS (endocrine complications, fatigue, surgical stress response, post-operative psychosis, insomnia, impact on sexual function, ability to eat socially, ability to interact socially, impact on physical appearance, impact on spirituality or faith, wound size, cost and destination on discharge), leaving 31 ‘no consensus’ outcomes for discussion at the consensus meeting.

Table 1

Demographic characteristics of participants included in analysis of round 1 and 2 scores.

Stakeholder Group	Variable	Sub-Group	Total	Completed round 1 only (%)*	Completed both rounds (%)*	p value
Patients	All	-	268	84	184
	Age	<60		38 (45)	77 (42)	0.69
		> = 60		46 (55)	107 (58)
	Sex	Male		52 (62)	101 (55)	0.345
		Female		32 (38)	83 (45)
	Region	West		53 (62)	113 (74)	0.461
		East		23 (38)	39 (26)
	Country income	HIC		53 (63)	113 (61)	0.792
		LMIC		31 (37)	71 (39)
	Years since surgery	<1 year		15 (19)	30 (17)	0.656
		1 to 3 years		34 (44)	68 (39)
		>3 years		29 (37)	75 (43)
	Surgical approach	Open		70 (83)	145 (78)	0.850
		MIS		14 (17)	31 (22)
	Type of surgery	Total		40 (49)	78 (44)	0.503
		Partial		42 (51)	98 (56)
	Treatment Modality	Surgery alone		28 (34)	69 (39)	0.495
		Multimodal therapy		54 (66)	110 (61)
Surgeons	All	-	445	102	343
	Region	West		33 (38)	174 (61)	0.000
		East		53 (62)	109 (39)
	Country income	HIC		45 (44)	201 (59)	0.010
		LMIC		57 (56)	142 (41)
	Surgeon experience	<50		21 (29)	70 (23)	0.45
		50–199		20 (27)	103 (34)
		>200		32 (44)	127 (43)
Nurses	All	-	239	104	135
	Region	West		22 (35)	40 (40)	0.100
		East		57 (65)	61 (60)
	Country income	HIC		24 (23)	46 (34)	0.064
		LMIC		80 (77)	89 (66)
	Nurse experience	0–5 years		59 (57)	59 (45)	0.056
		>5 years		44 (43)	73 (55)

HIC = high income country, LMIC = low- to middle-income country; MIS = minimally invasive surgery.

*All percentages refer to the proportion of participants from each sub-group completing either round 1 or both rounds.

3.2 Prioritisation of outcomes within stakeholder groups (subgroup analysis)

Baseline characteristics reported by stakeholders during the registration process were examined to understand whether these influenced how outcomes were prioritised.

3.2.1 Patient prioritisation of outcomes

A summary of outcomes categorised for ‘inclusion’ into the COS by at least one patient sub-group is presented in Table 2. Thirty outcomes were categorised for inclusion in the COS by at least one subgroup. Four outcomes were simultaneously categorised both for ‘inclusion’ and ‘exclusion’ by different subgroups. None of the outcomes categorised for inclusion by all stakeholder groups were voted ‘consensus out’ by any patient sub-group. Seven outcomes were categorised for inclusion in the COS by all patient subgroups.

Table 2

Outcomes categorised for inclusion in the COS by at least one subgroup of patients.

	Overall	Region**		Country Income		Age in years		Sex		Years since surgery			Surgical approach		Type of gastrectomy		Treatment Modality
	All patients	West	East	HIC	LMIC	<60	> = 60	M	F	< 1	1–3	>3	Open	MIS	Total	Partial	Surgery	Multi-modal
Outcome	n = 184	n = 113	n = 39	n = 113	n = 71	n = 77	n = 107	n = 101	n = 83	n = 30	n = 68	n = 75	n = 145	n = 31	n = 78	n = 98	n = 69	n = 110
Outcome Area: Mortality/Survival
1. Disease-free survival*	85.4	87.0	76.3	87.0	82.9	86.7	84.5	83.0	88.5	86.2	89.4	86.1	85.9	85.7	84.9	85.6	80.3	87.9
2. Dying from stomach cancer*	86.4	88.7	74.4	88.7	82.9	85.5	87.0	88.8	83.3	86.2	89.2	85.9	88.4	80.0	85.3	87.1	81.8	88.6
3. Dying from any cause	66.7	65.0	77.8	65.0	69.1	72.6	62.2	59.6	75.3	77.8	63.6	66.2	67.2	63.0	65.3	66.3	71.4	62.1
4. Surgery-related death*	84.0	86.9	72.2	86.9	79.4	76.7	89.2	80.2	88.6	81.5	85.1	88.7	86.2	73.3	82.7	84.9	79.7	86.8
Outcome Area: Clinical/physiological outcomes
7. Anastomotic complications*	76.7	80.0	74.4	80.0	71.8	74.3	78.4	75.0	78.8	82.8	77.6	74.3	76.1	74.2	69.3	80.9	74.6	76.9
8. Gastro-intestinal functional problems	72.8	85.3	71.8	85.3	53.5	69.7	75.0	66.0	81.3	75.9	63.2	80.6	70.2	77.4	70.1	72.6	67.7	74.5
9. Bowel Complications	71.8	80.0	76.9	80.0	59.2	65.8	76.2	66.0	79.0	75.9	64.7	76.7	69.7	74.2	67.5	72.9	68.2	72.7
12. Multiple organ failure*	86.4	87.9	86.5	87.9	84.1	87.8	85.3	86.7	85.9	86.2	86.4	85.7	87.8	79.3	83.8	87.2	80.3	89.5
16. Hepatic Complications	62.4	65.0	73.7	65.0	57.1	71.6	54.5	52.1	73.4	78.6	52.3	60.9	61.0	62.1	60.8	59.3	67.2	56.4
	All patients	West	East	HIC	LMIC	<60	> = 60	M	F	< 1	1–3	>3	Open	MIS	Total	Partial	Surgery	Multi-modal
17. Pancreatic Complications	70.3	75.5	73.7	75.5	61.4	74.0	66.7	63.8	76.9	82.1	57.8	72.5	68.9	69.0	67.1	69.2	68.7	69.0
18. Abdominal Collection	71.5	72.3	82.1	72.3	70.4	71.6	71.4	65.3	79.2	75.9	70.8	67.2	69.2	77.4	63.9	75.0	73.1	69.0
20. Nutritional Effects*	73.8	77.7	69.2	77.7	66.2	75.3	71.7	69.0	78.3	73.3	72.1	75.7	72.9	71.0	75.6	70.1	69.1	74.5
21. Recurrence of Cancer*	92.2	95.4	84.6	95.4	85.9	88.0	94.3	92.0	91.3	93.1	88.1	95.9	91.5	90.3	88.0	93.8	88.1	93.5
22. Renal complications	70.0	80.0	65.8	80.0	54.3	66.2	71.7	67.0	72.4	82.1	53.8	80.3	69.2	65.5	64.8	71.4	67.2	71.4
23. Urinary complications	58.1	65.7	57.9	65.7	45.7	54.2	60.0	54.2	61.8	64.3	40.0	70.6	57.8	51.7	50.0	60.9	56.1	59.4
25. Respiratory complications	69.5	67.0	66.7	67.0	73.2	70.3	68.9	70.7	67.9	75.0	73.1	63.4	71.2	56.7	60.8	73.7	68.2	68.9
27. Cerebro-vascular complications	77.6	81.0	68.4	81.0	72.9	68.6	84.0	80.9	73.7	75.0	72.3	84.8	78.0	73.3	69.0	82.4	71.2	82.8
28. Thrombo-embolic complications	76.7	80.4	63.2	80.4	71.4	73.2	79.2	78.9	74.0	71.4	73.8	82.4	79.9	60.0	73.6	77.2	66.7	84.2
29. Bleeding*	72.3	67.6	76.9	67.6	78.9	77.5	68.6	67.0	78.9	75.0	72.7	69.1	73.3	66.7	70.0	71.6	72.7	70.6
Outcome Area: Life impact
30. Ability to undertake physical activities	60.4	65.8	56.4	65.8	50.7	56.6	62.3	63.0	56.1	51.7	55.9	64.9	56.6	71.0	50.6	63.9	55.2	61.8
36. Impact on mental health	58.8	61.3	48.7	61.3	56.3	57.9	60.4	61.0	57.3	55.2	58.8	64.9	55.9	71.0	63.6	54.6	52.2	63.6
	All patients	West	East	HIC	LMIC	<60	> = 60	M	F	< 1	1–3	>3	Open	MIS	Total	Partial	Surgery	Multi-modal
40. Overall quality of life*	74.0	79.1	56.4	79.1	66.2	72.4	75.2	74.0	74.1	72.4	77.9	74.0	72.5	80.6	77.9	69.8	64.2	81.7
42. Ability to complete treatment pathway.	79.8	83.2	69.2	83.2	74.6	81.1	78.8	81.8	77.2	79.3	83.6	78.9	77.7	83.9	81.3	76.8	71.2	85.0
43. Completeness of tumour removal*	92.8	95.5	87.2	95.5	88.7	90.9	94.2	93.9	91.5	93.3	91.2	97.2	91.5	96.8	92.2	92.7	88.2	95.4
44. Conversion to open surgery	51.2	53.6	81.6	53.6	47.7	52.2	50.5	43.0	60.5	73.3	31.7	58.1	48.0	62.1	48.5	50.6	59.1	42.9
Outcome Area: Resource use
53. Duration of stay in an intensive care ward	64.1	54.4	62.9	54.4	77.6	59.2	66.7	60.6	67.1	57.7	71.2	56.7	65.7	46.4	64.3	59.8	62.5	63.4
Outcome Area: Adverse events
54. Adverse drug reaction	67.0	72.2	59.0	72.2	59.2	64.5	68.9	64.3	70.4	51.7	64.7	77.5	66.0	66.7	66.7	64.6	67.6	66.0
55. All-cause complications*	75.8	76.6	71.8	76.6	74.6	76.0	75.7	70.7	82.3	67.9	79.1	77.8	77.0	71.0	77.0	72.9	77.6	73.6
56. Intra-operative complications*	80.6	82.9	79.5	82.9	77.1	76.7	83.3	77.3	84.6	79.3	77.3	84.3	80.9	80.6	76.7	81.9	80.3	79.8
57. Anaesthetic complications*	74.9	78.1	66.7	78.1	70.0	74.0	75.5	71.4	79.2	55.2	75.8	81.4	77.2	61.3	74.0	73.4	73.8	74.3

Values are the percentage of participants voting the outcome as critically important (score 7–9).

Green = for inclusion, Yellow = no consensus, Red = for exclusion. HIC = high income country, LMIC = low- to middle-income country; MIS = minimally invasive surgery.

*Denotes outcomes are those which were included in the final list of outcomes for automatic inclusion in the COS.

**Participants not from either Western or Eastern countries were excluded from this analysis.

3.2.2 Surgeon prioritisation of outcomes

Table 3 summarises and compares outcomes categorised for inclusion into the COS by at least one surgeon sub-group. Twenty-one outcomes were categorised for inclusion by at least one subgroup. No outcomes were simultaneously categorised both for ‘inclusion’ and ‘exclusion’ by different subgroups. Twelve outcomes were categorised by all surgeon subgroups for inclusion.

Table 3

Outcomes categorised for inclusion in the COS by at least one subgroup of surgeons.

	Overall	Region**		Country Income		Cases performed
	All surgeons	West	East	HIC	LMIC	<50	50–199	>200
Outcome	n = 343	n = 174	n = 109	n = 201	n = 142	n = 70	n = 103	n = 127
Outcome Area: Mortality/Survival
1. Disease-free survival*	97.7	97.7	98.1	98.0	97.2	95.7	99.0	97.6
2. Dying from stomach cancer*	96.5	97.7	95.4	96.0	97.2	95.7	95.1	96.9
4. Surgery-related death*	96.8	96.6	99.1	97.5	95.8	94.3	96.1	98.4
Outcome Area: Clinical/physiological outcomes
7. Anastomotic complications*	95.3	95.4	95.4	96.0	95.1	95.7	94.2	96.1
8. Gastro-intestinal functional problems	74.9	75.3	70.6	75.1	76.1	82.9	76.7	67.7
12. Multiple organ failure*	81.3	81.0	78.9	81.1	80.9	75.7	83.5	81.7
18. Abdominal Collection	73.4	75.1	67.0	74.5	73.2	71.4	69.9	78.7
20. Nutritional Effects*	72.8	74.6	66.1	73.5	73.9	77.1	75.7	69.3
21. Recurrence of Cancer*	97.7	99.4	95.4	99.0	96.5	97.1	100.0	97.6
25. Respiratory complications	66.5	70.1	59.6	70.6	62.0	65.7	67.0	70.1
28. Thrombo-embolic complications	64.1	63.2	60.6	63.2	65.5	61.4	59.2	70.9
29. Bleeding*	87.5	84.5	95.4	86.1	90.1	81.4	85.4	92.9
Outcome Area: Life impact
30. Ability to undertake physical activities	66.4	71.8	59.6	69.7	63.4	65.7	70.9	66.9
40. Overall quality of life*	86.5	93.1	75.9	90.0	82.3	91.4	87.4	85.7
42. Ability to complete treatment pathway.	78.6	86.2	61.1	82.6	73.6	87.0	74.8	75.4
43. Completeness of tumour removal*	97.4	98.3	97.2	98.5	95.7	91.4	99.0	99.2
Outcome Area: Resource use
49. Readmission to hospital	78.9	78.7	82.4	78.6	80.9	80.0	81.6	81.0
51. Need for an additional intervention.	75.4	82.8	59.3	81.6	66.7	78.6	78.6	71.4
Outcome Area: Adverse events
55. All-cause complications*	81.2	81.5	84.3	83.0	79.4	81.4	76.7	88.1
56. Intra-operative complications*	91.5	88.4	93.5	89.5	93.6	91.4	92.2	92.9
57. Anaesthetic complications*	70.5	70.3	71.0	70.4	70.7	68.6	66.0	75.2

Values are the percentage of participants voting the outcome as critically important (score 7–9).

Green = for inclusion, Yellow = no consensus. HIC = high income country, LMIC = low- to middle-income country;

*Denotes outcomes are those which were included in the final list of outcomes for automatic inclusion in the COS.

**Participants not from either Western or Eastern countries were excluded from this analysis.

3.2.3 Nurse prioritisation of outcomes

Table 4 summarises and compares the outcomes categorised for inclusion by at least one nurse sub-group. Twenty-two outcomes were categorised for inclusion by at least one subgroup. Five outcomes were simultaneously categorised both for ‘inclusion’ and ‘exclusion’ by different subgroups. None of the outcomes categorised for automatic inclusion by all stakeholder groups were voted ‘consensus out’ by any nurse sub-group. Ten outcomes were categorised by all nurses’ subgroups for inclusion.

Table 4

Outcomes categorised for inclusion in the COS by at least one subgroup of nurses.

	Overall	Region**		Country Income		Experience in years
	All nurses	West	East	HIC	LMIC	0–5 years	>5
Outcome	n = 135	n = 40	n = 61	n = 46	n = 89	n = 59	n = 73
Outcome Area: Mortality/Survival
1. Disease-free survival*	85.1	92.5	85.2	93.5	80.9	81.4	89.0
2. Dying from stomach cancer*	80.0	90.0	72.1	91.3	74.2	74.6	83.6
3. Dying from any cause	63.4	64.1	70.5	64.4	65.2	58.6	71.2
4. Surgery-related death	77.6	95.0	65.6	93.5	69.3	72.9	81.9
Outcome Area: Clinical/physiological outcomes
7. Anastomotic complications*	84.4	97.5	82.0	97.8	76.4	79.7	89.0
8. Gastro-intestinal functional problems	69.6	90.0	65.6	89.1	57.3	59.3	75.3
12. Multiple organ failure*	79.9	82.5	78.3	84.8	78.4	83.1	79.2
13. Pain	59.3	85.0	59.0	87.0	44.9	49.2	65.8
18. Abdominal Collection	65.9	65.0	67.2	69.6	61.8	49.2	76.7
19. Other infections	61.2	55.0	70.0	58.7	61.4	54.2	65.3
20. Nutritional Effects*	74.8	87.5	77.0	87.0	66.3	69.5	76.7
21. Recurrence of Cancer*	88.0	97.5	86.9	97.8	82.8	84.5	90.3
26. Wound complications	67.4	62.5	73.8	63.0	67.4	67.8	64.4
29. Bleeding*	80.7	72.5	85.2	76.1	82.0	79.7	80.8
Outcome Area: Life impact
30. Ability to undertake physical activities	56.3	72.5	54.1	73.9	46.1	54.2	56.2
36. Impact on mental health	54.5	70.0	48.3	71.7	44.3	54.2	52.8
40. Overall quality of life*	70.4	90.0	67.2	89.1	59.6	61.0	76.7
42. Ability to complete treatment pathway.	65.9	77.5	60.7	78.3	58.4	54.2	75.3
43. Completeness of tumour removal*	87.3	100.0	86.9	97.8	82.0	83.1	91.8
Outcome Area: Resource use
49. Readmission to hospital	69.9	77.5	68.3	78.3	62.1	60.3	73.6
51. Need for an additional intervention.	56.7	75.0	48.3	76.1	45.5	44.1	63.9
52. Need for pain relief	68.4	72.5	72.9	73.9	63.2	57.6	74.6
Outcome Area: Adverse events
55. All-cause complications*	77.9	77.5	77.2	80.4	75.3	70.2	83.1
56. Intra-operative complications*	85.4	90.0	91.1	91.3	83.3	85.7	87.3
57. Anaesthetic complications*	78.0	80.0	77.8	80.4	76.5	70.9	84.1

Values are the percentage of participants voting the outcome as critically important (score 7–9). Green = for inclusion, Red = for exclusion, Yellow = no consensus. HIC = high income country, LMIC = low- to middle-income country;

*Denotes outcomes are those which were included in the final list of outcomes for automatic inclusion in the COS.

**Participants not from either Western or Eastern countries were excluded from this analysis.

3.3 Impact of regional variation on prioritisation of outcomes

Table 5 details the final categorisation of outcomes in the Delphi survey as agreed by all stakeholder groups. This is compared to outcome categorisation lists based on the region (East versus West) and country income differences (HIC versus LMIC). Consensus agreement to include 8 and exclude 7 outcomes was reached across all regional sub-groups. No outcomes were simultaneously categorised as ‘consensus in’ and ‘consensus out’ across different regional sub-groups.

Table 5

Regional differences in consensus on outcomes voted for inclusion or exclusion from the COS by at least 1 subgroup.

	Final list of outcomes as agreed by all stakeholder groups	Region**		Country Consensus income
Consensus outcome		West (n = 327)	East (n = 209)	HIC (n = 360)	LMIC (n = 302)
Outcome Area: Mortality/Survival
1. Disease-free survival*	Consensus in	Consensus in	Consensus in	Consensus in	Consensus in
2. Dying from stomach cancer*	Consensus in	Consensus in	Consensus in	Consensus in	Consensus in
4. Surgery-related death	Consensus in	Consensus in	No consensus	Consensus in	No consensus
Outcome Area: Clinical/physiological outcomes
6. Endocrine complications	Consensus out	Consensus out	No consensus	Consensus out	Consensus out
7. Anastomotic complications*	Consensus in	Consensus in	Consensus in	Consensus in	Consensus in
8. Gastro-Intestinal functional problems	No consensus	Consensus in	No consensus	Consensus in	No consensus
11. Fatigue	Consensus out	Consensus out	Consensus out	Consensus out	Consensus out
12. Multiple organ failure*	Consensus in	Consensus in	Consensus in	Consensus in	Consensus in
14. Surgical Stress Response	Consensus out	Consensus out	No consensus	Consensus out	No consensus
15. Gallbladder complications	No consensus	No consensus	No consensus	No consensus	Consensus out
20. Nutritional Effects	Consensus in	Consensus in	No consensus	Consensus in	No consensus
21. Recurrence of Cancer*	Consensus in	Consensus in	Consensus in	Consensus in	Consensus in
23. Urinary complications	No consensus	No consensus	No consensus	No consensus	Consensus out
24. Post-operative psychosis	Consensus out	Consensus out	Consensus out	Consensus out	Consensus out
29. Bleeding	Consensus in	No consensus	Consensus in	No consensus	Consensus in
31. Insomnia	Consensus out	Consensus out	No consensus	Consensus out	Consensus out
Outcome Area: Life impact
32. Impact on sexual function	Consensus out	Consensus out	Consensus out	Consensus out	Consensus out
33. Ability to eat socially	Consensus out	No consensus	Consensus out	No consensus	Consensus out
34. Ability to Interact socially	Consensus out	No consensus	Consensus out	No consensus	Consensus out
35. Impact of surgery on social and work roles	No consensus	No consensus	Consensus out	No consensus	Consensus out
36. Impact on mental health	No consensus	No consensus	Consensus out	No consensus	No consensus
37. Impact on Physical Appearance	Consensus out	Consensus out	Consensus out	Consensus out	Consensus out
39. Impact on spirituality or faith	Consensus out	Consensus out	Consensus out	Consensus out	Consensus out
40. Overall quality of life	Consensus in	Consensus in	No consensus	Consensus in	No consensus
41. Impact on perception of physical health	No consensus	No consensus	No consensus	No consensus	Consensus out
42. Ability to complete treatment pathway.	No consensus	Consensus in	No consensus	Consensus in	No consensus
43. Completeness of tumour removal*	Consensus in	Consensus in	Consensus in	Consensus in	Consensus in
45. Duration of surgery	No consensus	Consensus out	No consensus	Consensus out	No consensus
46. Wound size	Consensus out	Consensus out	Consensus out	Consensus out	Consensus out
Outcome Area: Resource use
47. Cost	Consensus out	Consensus out	No consensus	Consensus out	No consensus
50. Destination on Discharge	Consensus out	Consensus out	Consensus out	Consensus out	Consensus out
Outcome Area: Adverse events
55. All-cause complications*	Consensus in	Consensus in	Consensus in	Consensus in	Consensus in
56. Intra-operative complications*	Consensus in	Consensus in	Consensus in	Consensus in	Consensus in
57. Anaesthetic complications	Consensus in	Consensus in	No consensus	Consensus in	Consensus in

Green = for inclusion, Red = for exclusion, Yellow = no consensus. HIC = high income country, LMIC = low- to middle-income country;

*Denotes outcome was categorised as for ‘inclusion’ in COS by all subgroups.

**Participants not from either Western or Eastern countries were excluded from this analysis.

4. Discussion

The GASTROS study (www.gastrosstudy.org) is the first to bring together healthcare professionals and patients with the purpose of identifying outcomes to include in a COS for surgical trials in gastric cancer. The multi-language survey recruited a broad spectrum of stakeholders with different personal and professional experiences from over 50 countries across 6 continents. We aimed to examine whether certain stakeholder characteristics influenced how outcomes were prioritised and whether there were regional influences also. Our analysis from nearly 1000 survey participants suggested that little variation within the stakeholder groups exists. Similarly, when all stakeholders were categorised according to region or country income, significant differences were not identified. These are important findings which should serve to reassure researchers and patients that the resulting COS has sought and considered international opinion which is widely representative. Furthermore, these findings suggest that priorities within stakeholder groups and across regions are more aligned than may have been previously thought.

4.1 Planning recruitment to Delphi surveys

Few studies have previously examined factors which influence how stakeholders prioritise outcomes in the field of COS development. The BRAVO study explored this in the field of breast cancer reconstruction and found that priorities varied within patient and healthcare professional groups [6]. This led them to recommend careful participant selection for Delphi surveys by COS developers. These same differences, however, were not identified in our study. The BRAVO study’s healthcare professional stakeholder group was more heterogenous than the groups in this study (breast surgeons, plastic surgeons, nurses and psychologists grouped together) and so these differences may be expected. Furthermore, reconstructive breast surgery is a complex area which covers many different types of procedures. This may also account for the significant variation in outcome prioritisation by patients which was not mirrored in the GASTROS study. In comparison, gastric cancer surgery generally comprises of either a partial or total gastrectomy both of which can result in similar short and long-term problems which may explain why priorities were more aligned within stakeholder groups. Similarly, a COS study in the field of bariatric surgery identified significant variation in outcome prioritisation amongst healthcare professionals [14]. Again, healthcare professionals in this study were heterogenous, which supports our strategy to separate surgeons and nurses into different stakeholder groups.

Achieving the ‘correct balance’ of representative stakeholders is an important consideration during the design phase. For example, knowledge of the patient demographic and which types of interventions are prevalent within that group, will enable researchers to recruit an appropriate number of stakeholders with those characteristics. As this is the first study to specifically examine regional variations amongst stakeholders in COS development, it is unknown whether these findings necessary mirror those from other COS studies. Further examination of previously undertaken Delphi surveys is required, and future surveys will need to ensure that similar baseline characteristics are recorded. This is relatively straightforward information to capture and can be supplied quickly and easily by survey participants during the registration process.

With respect to the GASTROS study, the importance of seeking international agreement on core outcomes was identified at the conception stage and subsequently influenced the design of the prioritisation exercise. Our strategy for addressing the significant challenges associated with international involvement included 1) an international working group with regional collaborators, 2) translating surveys and 3) seeking the support of relevant patient and professional groups. Transparent reporting of methodological approaches adopted during COS development are of paramount importance. Ultimately, a COS will only achieve its stated goals if researchers use it. And whilst there are likely several factors which influence the uptake of COS, ensuring researchers have the confidence that the COS is relevant to them and has been developed through a methodologically robust process are likely to be important factors which influence uptake and dissemination [15].

There are challenges in deciding how to sample participants for a Delphi study. Epidemiological studies, registries and audits provide descriptive regional or national information [16-18]. However, in the case of gastric cancer, these resources are not always complete or available. Consequently, the study team widened the promotion and enrolment into the Delphi to capture as many patients as possible. In our study, we demonstrated that there was not significant variation in outcome prioritisation within stakeholder sub-groups with respect to the characteristics that we examined. Consequently, whilst over 1000 participants were enrolled, it may not have been necessary to recruit such large numbers. This will likely guide our recruitment strategy during future planned stages of work when reviewing the COS and identifying outcome measurement instruments. Our experience may also help guide other COS developers as they consider the number of participants to recruit to their Delphi surveys. However, given some of our findings differed from those in the field of breast surgery reconstruction and bariatric surgery, more work is needed before broad recommendations can be made.

4.2 Variations within stakeholder groups

When regional variations across the three stakeholder groups were compared, the greatest differences in prioritisation were observed amongst nurses. For example, in four outcomes (pain, ability to undertake physical exercise, impact on mental health, need for additional intervention) different subgroups of nurses categorised them as ‘consensus in’ and ‘consensus out’. These outcomes seemed less important in LMIC and HIC settings within the nurse group. Understanding the reason for this is likely to be complex. It may be argued that this is simply because nurses are reflecting the importance that patients from these cultures or regions place on these outcomes as similar trends were seen amongst patients. Limited resource in LMIC settings which may affect follow-up may also play a role in understanding how important longer-term problems are in these regions. Further exploration using qualitative research methods may help understand these differences further.

In examining the differences between patient sub-groups, one would expect to see some differences given the number of characteristics that were examined. Despite this, only two outcomes (urinary complications and conversion to open surgery) were simultaneously categorised as ‘consensus in’ and ‘consensus out’ by different sub-groups. This finding suggests that despite the many possible influences on patient experience following gastric cancer surgery, there is not a significant variation in how health related outcomes are prioritised in this group. Surgeons had the greatest concordance with respect to outcome prioritisation. Overall, the observed differences in outcome prioritisation were small within each stakeholder group reassuring researchers using the COS that it is based on the views of a representative cohort of patients and healthcare professionals.

4.3 Impact of regional variations on outcomes automatically included in COS

The aim of a COS is to identify outcomes which are critically important across all stakeholder groups participating in the process. In the case of the GASTROS study, an outcome would only be automatically included in the COS if patients, surgeons, and nurses each categorise it ‘consensus in’. Ultimately, it is not possible to confidently assess how regional differences may have affected the final categorisation of outcomes which informed the consensus meeting. Participants in round 2 were shown the scores of all stakeholder groups from round 1 before being asked to change their score if they wish. To assess regional differences, Western participants, for example, in round 2 would have needed to see only Western stakeholder group scores from round 1. Furthermore, there are a number of other confounding factors which influence why participants change scores between rounds (see below) further making an analysis of regional impacts challenging.

Despite this, some assessments could be made. No outcomes categorised for automatic inclusion by all three stakeholder groups were categorised for automatic exclusion by a regional sub-group. And no outcomes categorised for automatic exclusion from the COS by all three stakeholder groups were categorised for automatic inclusion by a regional sub-group. This suggests that the regional differences in approach to management or patient outcome may not significantly influence how stakeholders prioritise outcomes

There were two outcomes (gastrointestinal functional problems and ability to complete treatment pathway) which were categorised for automatic inclusion by stakeholders from the West and HIC that were not included in the final list presented to the consensus meeting. Furthermore, some outcomes (surgery-related death, nutritional outcomes, bleeding, overall quality of life, anaesthetic complications) did not reach consensus for automatic inclusion by regional sub-groups yet were automatically included when the overall views of stakeholders were considered. This may bring some to the conclusion that different COS should be developed for different regions as some researchers may be collecting outcomes that were not deemed critically important in their region. However, researchers should be cognisant of the fact that their trials are internationally relevant and vitally important to the larger picture where evidence synthesis is concerned. From a different perspective, some researchers may feel aggrieved if outcomes which are critically important in their region are not eventually included in the COS. It is important to emphasise that COS are minimum reporting guidelines and that researchers are encouraged to report additional outcomes that they believe are important.

4.4 Dissemination of results

From the study’s inception, the management team understood the importance of regularly reporting findings to encourage participation from all stakeholder groups. Regular reporting also aimed to increase the study’s exposure and highlight its importance to minimising research waste in future trial design. Finally, uptake of the COS requires researchers and funders to have knowledge of its existence. A clear dissemination policy was set out a priori and included scientific publications, presentations at medical and nursing congresses as well as lay summaries delivered to patient participants through patient groups and social media. The success of this policy to this point has been reflected by nearly 1000 participants being recruited to the Delphi survey. Continued efforts will be required to ensure that the COS is used, and researchers and patients benefit from it.

4.5 Strengths and limitations

Strengths of this study include that it is novel and that was able to recruit a large number of participants from many countries. However, there are some limitations which should be acknowledged. This was an analysis which was not powered to make definitive conclusions about relationships between sub-groups and how outcomes were rated. Therefore, the results should be viewed in this context. Furthermore, the sub-groups examined in this paper were chosen by members of the study team based on their extensive experience in the field of gastric cancer and their understanding of factors which may impact on stakeholder experience, perceptions and subsequently how outcomes may be prioritised. It is possible that other unexplored characteristics impact on how stakeholders prioritise outcomes. In addition, this study did not explore how different characteristics interact with one another to impact on outcome prioritisation (e.g. years since surgery and type of gastrectomy). Doing so would create results which would remove the focus from regional differences and would be difficult to interpret. Furthermore, there were significantly fewer patients from Eastern countries enrolled compared to their Western counterparts. This may have influenced how outcomes were categorised ahead of the consensus meeting. However, due to the interplay of other factors described above, reaching a definite conclusion about the degree of this possible limitation is difficult. This is an area that may benefit from further exploration using qualitative research methods.

Delphi surveys are an established method of reaching consensus in the design of COS [1]. They give participants the opportunity to reflect on their ratings from previous rounds before giving a final score. Only after this opportunity should all scores be analysed, and outcomes categorised ahead of the consensus meeting. During the process of rating outcomes in round 2 of the survey, participants are shown the results from each separate stakeholder group in round 1. The topic of why participants change their scores between rounds is an interesting one which has been examined elsewhere [19]. Through our previous analysis we identified that the reasons for changing scores provided by stakeholders were varied, including having the time to reflect on the question being asked, changing their minds on the importance, impact or usefulness of the outcome in question, and changes in personal experience of the outcome. In fact, the influence of other stakeholder ratings as a reason for significantly changing a score in round 2 was cited by only a minority of healthcare professionals and patients. Another factor which may influence scores between rounds is attrition. Our strategy to keep this as low as possible, alongside other approaches to facilitate international participation in Delphi surveys for COS is a topic which will be described separately. Whilst overall attrition was 30%, the group this affected the most were nurses with nearly 45% attrition. However, the characteristics of those completing both rounds were not significantly different to those only completing round 1. Likewise, a statistically significant difference was identified in the characteristics of surgeons completing both rounds who were predominantly Western and from HIC compared to the balance of surgeons completing round 1. It could be argued therefore that retaining a greater number of Eastern and LMIC surgeons may have led to slightly different survey results. However, whilst statistically significant, this difference is unlikely to be clinically significant given that the number of surgeons not participating in round 2 was relatively small.

One may consider that, given the multimodal nature of treatment for gastric cancer, a COS would be more relevant if it incorporated all therapies (e.g. chemotherapy, radiotherapy, and endotherapy) and that the views of oncologists should also be sought. However, at the time that GASTROS was conceived, there were 24 ongoing surgical trials planning to recruit 11 000 patients for whom non-surgical-related outcomes would not be applicable or relevant. Other considerations such as the resource and time required to recruit participants from other stakeholder groups were also important, hence why a pragmatic decision was made to limit stakeholder groups to those chosen in this study.

Our methodology for COS development was based on an established approach as described by the COMET handbook [1] and COS developers. This aims to seek consensus based on identifying a long list of potentially important outcomes from a systematic review and patient interviews, followed by a Delphi survey, concluded by a consensus meeting. Whilst this approach is well-studied and has been adopted by many, our experience indicates that the process, particularly when seeking broad international opinion, can be both time and resource intensive. Some groups have already begun to explore whether COS development can be streamlined [20]. It should therefore be acknowledged that COS methodology is a developing field for which a single ‘optimal’ approach does not necessarily exist. Examination of differing methodological adaptations should therefore form an important part of future COS development studies and appropriate funding should be made available to support this.

4.6 Conclusion

The GASTROS Delphi survey recruited a broad spectrum of international stakeholders of patients, nurses and surgeons to produce a list of outcomes which should be included or excluded from a COS and others which required further discussion at a consensus meeting. Consensus across these groups was achieved to include 13 outcomes into the COS which will be discussed further at a final consensus meeting (disease-free survival, disease-specific survival, surgery-related death, recurrence of cancer, completeness of tumour removal, overall quality of life, nutritional effects, all-cause complications, intraoperative complications, anaesthetic complications, anastomotic complications, multiple organ failure, and bleeding). Whilst some regional differences were highlighted, there was little variation within stakeholder groups and between regions with respect to how outcomes were prioritised. This may reassure COS users that the adopted methodology was robust and that the views captured during its development were representative. COS developers should carefully consider the characteristics of Delphi survey participants when planning their recruitment strategy. These variables should be explored further to examine the generalisability of this study’s findings.

Full version of the Delphi survey which was translated into target language.

(DOCX)

Click here for additional data file.

30 Jun 2021

PONE-D-20-40628

Exploring the impact of regional variation on outcome prioritisation in core outcome set development: a case study in the field of gastric cancer surgery

PLOS ONE

Dear Dr. Alkhaffaf,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please revise the presentation of the study findings. Some guidance is reported below.

Please submit your revised manuscript by Aug 14 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Sandro Pasquali, M.D., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (if provided):

Overall the manuscript is very interesting as it investigates an underappreciated topic in surgical research. In this regard, authors should consider that most of readers are not familiar with the study topic and need some more guidance through the manuscript.

This is particularly important as the manuscript is well written, study hypothesis, aims and methodology are clear and conclusions robust. Also, the Discussion section stated “Ultimately, a COS will only achieve its stated goals if researchers use it.”thus supporting the dissemination of the results of this study.

Overall, I would request authors to improve their study presentation in order to make it more readable.

Title is difficult to get and there are repetitions. I would consider a simpler title such as “Development of a core outcome set for clinical trials in gastric cancer surgery: the GASTROS study”.

Abstract is not very informative of study findings. The result section should include the outcomes to be prioritized and report overall rates, otherwise it is very difficult to have an idea of what the manuscript analysed.

Result section is hard to get. For instance, outcomes should be grouped according to their main area, for instance: prognosis, post-operative morbidity, quality of life, long-term morbidity. This may make it easier for Authors to provide a more narrative reporting of their result section, which currently is just a reminder to tables, which is hard to follow for readers. Again , please consider taking the reader through the study results rather than ask him/her to look at tables.

Clearly authors want to reach surgeons and oncologists with this paper. How they plan to reach nurses? What about patients? It would be interesting to have authors’ strategy reported in the discussion.

A study limitation is the lack of medical oncologists and radiation oncologists. Can you please explain reasons for considering only surgeons for a disease that is treated multidisciplinary? This issue should be discussed.

In the study conclusion it should be reported that three stakeholders (surgeons, nurses and patients) were considered. There should at last be some examples of included outcomes, for instance those with the highest agreement.

I would appreciate some comment on the generalizability of these findings to other surgical oncology disease. Authors comment that gastric cancer has extensive differences compared to breast cancer. However these findings may be valuable, at least to some extents, for GI cancers. Please comment.

There is a typo in the first line of 4.4 – “is was”.

Journal Requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

3. Thank you for including your ethics statement:  "The study was given ethical approval by the North West - Greater Manchester East Research Ethics Committee (18/NW/0347) and governance approvals by Manchester University Hospitals NHS Foundation Trust.".

Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified (1) whether consent was informed and (2) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed).

Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

For additional information about PLOS ONE ethical requirements for human subjects research, please refer to http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research.

4. Please include additional information regarding the survey or questionnaire used in the study and ensure that you have provided sufficient details that others could replicate the analyses. For instance, if you developed a questionnaire as part of this study and it is not under a copyright more restrictive than CC-BY, please include a copy, in both the original language and English, as Supporting Information.

5. In your Methods section, please provide additional information about the participant recruitment method and the demographic details of your participants. Please ensure you have provided sufficient details to replicate the analyses such as:

- the recruitment date range (month and year)

- a description of any inclusion/exclusion criteria that were applied to participant recruitment

- a table of relevant demographic details

- a statement as to whether your sample can be considered representative of a larger population

- a description of how participants were recruited.

6. Please list the name and version of any software package used for statistical analysis, alongside any relevant references. For more information on PLOS ONE's expectations for statistical reporting, please see https://journals.plos.org/plosone/s/submission-guidelines.#loc-statistical-reporting.

7. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

8. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

9.One of the noted authors is a group or consortium GASTROS International Working Group . In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript. Please also indicate clearly a lead author for this group along with a contact email address.

10. We note that Figure 1 in your submission contain map image which may be copyrighted. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For these reasons, we cannot publish previously copyrighted maps or satellite images created using proprietary data, such as Google software (Google Maps, Street View, and Earth). For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright.

We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission:

a. You may seek permission from the original copyright holder of Figure 1 to publish the content specifically under the CC BY 4.0 license.

We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:

“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.”

Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission.

In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

b. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

The following resources for replacing copyrighted map figures may be helpful:

USGS National Map Viewer (public domain): http://viewer.nationalmap.gov/viewer/

The Gateway to Astronaut Photography of Earth (public domain): http://eol.jsc.nasa.gov/sseop/clickmap/

Maps at the CIA (public domain): https://www.cia.gov/library/publications/the-world-factbook/index.html and https://www.cia.gov/library/publications/cia-maps-publications/index.html

NASA Earth Observatory (public domain): http://earthobservatory.nasa.gov/

Landsat: http://landsat.visibleearth.nasa.gov/

USGS EROS (Earth Resources Observatory and Science (EROS) Center) (public domain): http://eros.usgs.gov/#

Natural Earth (public domain): http://www.naturalearthdata.com/

11. Please include a copy of Table 1 which you refer to in your text on page 18.

12. We note you have included a table to which you do not refer in the text of your manuscript. Please ensure that you refer to Table 31 and 32 in your text; if accepted, production will need this reference to link the reader to the Table.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is an interesting paper . I admire your efforts at recruitment for this Delphi.

In answer to the journal’s specific questions:

• Further data/evidence of analysis is offered in response to personal request.

• The manuscript is parsimonious, and well written and presented.

• It offers the tabular analysis I would expect to see to support the body of the paper, and the conclusions drawn seem sound in relation to the data presented. The sample sizes (you do point this out) are too small to make definitive conclusions, and there are many confounding factors (you also point this out) that may play a role in the variation you report.

As a reader:

I was initially not persuaded that subgroup analysis would be pragmatic in this context, but I inevitably felt drawn in by the tables and percentages, and was struck by Table 5 and how the results hang together here. You do state the key point that a COS is intended to allow synthesis, and to be useful in international trials – reminding the reader that the heart of COS methodology is the search for consensus rather than divergence. So the intention to analyse subgroups did raise some questions, but I think I’m persuaded that this was worth exploring (it made me think, always a good sign). Generally speaking, I felt that the paper was carefully worded so as not to stray into too much speculation.

Issues that came to mind as I read through (these are very general):

Section 2.3 Data analysis and interpretation - perhaps I'm misinterpreting this, but the first sentence suggests that participants included in the analysis did not complete the whole Delphi survey - in fact, some of them only completed 50% of the survey? It seemed to me that including surveys that were only partially filled in meant that you are not comparing like-with-like across the different outcomes, but I may be missing something here.

I would have appreciated a sentence or two summarising the concrete differences in pathology, treatment and outcome between regions (rather than the statement that there are differences), so that I could do some independent thinking around this. I don’t see why this paper shouldn’t flag up some potential issues around the consensus approach (rather than just reassuring us!), given that the same methodology for COS development is used across a variety of subject areas which are very different in shape. The COS development process is influenced by the nature of the condition being explored, whether you are trying to tie down outcomes for a single intervention or a range of interventions for the same condition, and the breadth of stakeholder roles involved (which is inevitably affected by the two previous issues). I wanted just a little more information about the background so that I could think about these issues without having to look them up in a separate paper.

It’s a benefit to this analysis to have a low number of stakeholder groups, and it’s a given that a study steering group should carefully think about which role perspectives (besides patients) they should include as respondents in a Delphi survey. I felt that the call to consider more finely grained characteristics of potential respondents when you plan recruitment was perhaps optimistic, as this is partly resource-dependent, and capacity in research institutions varies greatly. Not all COS projects would have funding to access demographic information about patients in order to guide recruitment (and adding demographic questions to the survey could be a burden and increase attrition). I’m not saying this isn’t a good idea for recruitment – just in an ideal world. I wonder whether the paper should touch on capacity issues and make some clear calls for investment in this kind of approach for COS studies that are likely to benefit from it, which would in turn feed back to funders of COS projects?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

2 Nov 2021

Response to Reviewer:

1. Section 2.3 Data analysis and interpretation - perhaps I'm misinterpreting this, but the first sentence suggests that participants included in the analysis did not complete the whole Delphi survey - in fact, some of them only completed 50% of the survey? It seemed to me that including surveys that were only partially filled in meant that you are not comparing like-with-like across the different outcomes, but I may be missing something here.

The study management group considered whether to include partially completed surveys. The argument against excluding partially completed surveys related to the possibility that valid responses would be excluded. Ultimately, we settled on including surveys which had been at least 50% completed as it was probable that these participants had understood the purpose of the study. Furthermore, some participants who had not fully completed the first Delphi went on to complete the second survey which supported our choice. The number of partially completed studies was small (3%) and most partially completed surveys had prioritised closer to 100% of outcomes rather than the 50% minimum response threshold that was set.

2. I would have appreciated a sentence or two summarising the concrete differences in pathology, treatment and outcome between regions (rather than the statement that there are differences), so that I could do some independent thinking around this.

As requested, specific examples of regional differences have now been added to the introduction.

3. I don’t see why this paper shouldn’t flag up some potential issues around the consensus approach (rather than just reassuring us!), given that the same methodology for COS development is used across a variety of subject areas which are very different in shape. The COS development process is influenced by the nature of the condition being explored, whether you are trying to tie down outcomes for a single intervention or a range of interventions for the same condition, and the breadth of stakeholder roles involved (which is inevitably affected by the two previous issues). I wanted just a little more information about the background so that I could think about these issues without having to look them up in a separate paper.

A section which discusses the merits and limitations of using this methodological approach has now been added to the last paragraph before the conclusion.

4. It’s a benefit to this analysis to have a low number of stakeholder groups, and it’s a given that a study steering group should carefully think about which role perspectives (besides patients) they should include as respondents in a Delphi survey. I felt that the call to consider more finely grained characteristics of potential respondents when you plan recruitment was perhaps optimistic, as this is partly resource-dependent, and capacity in research institutions varies greatly. Not all COS projects would have funding to access demographic information about patients in order to guide recruitment (and adding demographic questions to the survey could be a burden and increase attrition). I’m not saying this isn’t a good idea for recruitment – just in an ideal world. I wonder whether the paper should touch on capacity issues and make some clear calls for investment in this kind of approach for COS studies that are likely to benefit from it, which would in turn feed back to funders of COS projects?

The reviewer has made an extremely important point about limited resources often available during COS during studies. However, the demographic information that was gathered at the start of the Delphi survey was relatively basic, was self-reported by participants and did not take long to complete. The concern regarding the burden of supplying demographic details did not materialise during our study which achieved a retention rate of 70% between round 1 and round 2. This has been added to the section 4.1 in the discussion. A call and justification to ensure appropriate funding for COS studies has been incorporated in the discussion.

Responses to Editor’s Comments

1. Title is difficult to get and there are repetitions. I would consider a simpler title such as “Development of a core outcome set for clinical trials in gastric cancer surgery: the GASTROS study”.

This manuscript aims specifically to explore how outcomes are prioritised by stakeholders in different regions. If outcomes are valued differently, then achieving a balance when recruiting participants for the consensus process becomes more important in reducing bias. The study is not the description of the final COS which has been published elsewhere (https://academic.oup.com/bjs/advance-article/doi/10.1093/bjs/znab192/6308782?searchresult=1). As requested, the title has been simplified to reflect this:

‘How are trial outcomes prioritised by stakeholders from different regions? Analysis of an international Delphi survey to develop a core outcome set in gastric cancer surgery.’

2. Abstract is not very informative of study findings. The result section should include the outcomes to be prioritized and report overall rates, otherwise it is very difficult to have an idea of what the manuscript analysed.

There were 56 outcomes to prioritise. This has now been added to the methods section. The outcomes on which consensus was reached to include in the COS have been added to the abstract. For brevity, those which were excluded or where no consensus was reached have not been added to the abstract but can still be found in the main manuscript.

3. Result section is hard to get. For instance, outcomes should be grouped according to their main area, for instance: prognosis, post-operative morbidity, quality of life, long-term morbidity. This may make it easier for Authors to provide a more narrative reporting of their result section, which currently is just a reminder to tables, which is hard to follow for readers. Again , please consider taking the reader through the study results rather than ask him/her to look at tables.

As requested, the results section has been expanded and a narrative description of the results has now been incorporated. A taxonomy used to organise outcomes for Delphi participants has been cited and tables now reflect these ‘core outcome areas’ as recommended by the reviewer.

4. Clearly authors want to reach surgeons and oncologists with this paper. How they plan to reach nurses? What about patients? It would be interesting to have authors’ strategy reported in the discussion.

A statement regarding our dissemination policy has now been added to the discussion section. Updating all stakeholder groups involved in this study is certainly an important consideration which was set out a priori.

5. A study limitation is the lack of medical oncologists and radiation oncologists. Can you please explain reasons for considering only surgeons for a disease that is treated multidisciplinary? This issue should be discussed.

The scope of the study and the reasons why other healthcare providers were not invited to participate in the study has now been added to the limitations section of the discussion. The justification for this has been included in the methods section.

6. In the study conclusion it should be reported that three stakeholders (surgeons, nurses and patients) were considered. There should at last be some examples of included outcomes, for instance those with the highest agreement.

The conclusion has been modified to reflect the recommendation to explicitly describe the stakeholders invited to participate in the study. The outcomes where consensus agreement was reached amongst all stakeholders has also been included.

7. I would appreciate some comment on the generalizability of these findings to other surgical oncology disease. Authors comment that gastric cancer has extensive differences compared to breast cancer. However these findings may be valuable, at least to some extents, for GI cancers. Please comment.

As this is the first time that a group has examined regional variation as a possible variable on how outcomes are prioritised, it is difficult to be able to comment on whether the findings from this study will be applicable to other COS studies – surgical oncology or otherwise. We certainly believe that this requires some further study and we have added a section into the discussion which encourages other developers to collect this type of baseline data during the registration period of the survey.

The differences that we discuss between gastric and breast cancer relate to the way in which stakeholder groups were assembled during the respective COS, as well as the available treatments. Explicit examples have been given. Further work is already underway to identify commonalities and differences between core outcome sets in surgical oncology; this is an extensive project which would be challenging to summarise or include in the present discussion.

8. There is a typo in the first line of 4.4 – “is was”.

This mistake has been corrected.

Submitted filename: Response to reviewer comments 011121.docx

Click here for additional data file.

15 Dec 2021

How are trial outcomes prioritised by stakeholders from different regions? Analysis of an international Delphi survey to develop a core outcome set in gastric cancer surgery.

PONE-D-20-40628R1

Dear Dr. Alkhaffaf,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Reviewer and editor's comments have been addressed and manuscript improved accordingly.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Sandro Pasquali, M.D., Ph.D.

Academic Editor

PLOS ONE

22 Dec 2021

PONE-D-20-40628R1

How are trial outcomes prioritised by stakeholders from different regions? Analysis of an international Delphi survey to develop a core outcome set in gastric cancer surgery.

Dear Dr. Alkhaffaf:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Sandro Pasquali

Academic Editor

PLOS ONE