Determinants of abstract acceptance for the Digestive Diseases Week--a cross sectional study.

BACKGROUND: The Digestive Diseases Week (DDW) is the major meeting for presentation of research in gastroenterology. The acceptance of an abstract for presentation at this meeting is the most important determinant of subsequent full publication. We wished to examine the determinants of abstract acceptance for this meeting.
METHODS: A cross-sectional study was performed, based on abstracts submitted to the DDW. All 17,205 abstracts submitted from 1992 to 1995 were reviewed for acceptance, country of origin and research type (controlled clinical trials (CCT), other clinical research (OCR), basic science (BSS)). A random sub-sample (n = 1,000) was further evaluated for formal abstract quality, statistical significance of study results and sample size.
RESULTS: 326 CCT, 455 OCR and 219 BSS abstracts were evaluated in detail. Abstracts from N/W Europe (OR 0.4, 95% CI 0.3-0.6), S/E Europe (OR 0.4, 95% CI 0.2-0.6) and non-Western countries (OR 0.3, 95% CI 0.2-0.5) were less likely to be accepted than North-American contributions when controlling for research type. In addition, the OR for the acceptance for studies with negative results as compared to those with positive results was 0.4 (95% CI 0.3-0.7). A high abstract quality score was also weakly associated with acceptance rates (OR 1.4, 95% CI 1.0-2.0).
CONCLUSIONS: North-American contributions and reports with statistically positive results have higher acceptance rates at the AGA. Formal abstract quality was also predictive for acceptance.

Background

The annual Digestive Disease Week (DDW) is an important opportunity for the presentation of research in gastroenterology, attracting investigators and clinicians from around the world. Every year, several thousand abstracts are submitted to this meeting which is jointly organized by the American Gastroenterological Association (AGA), the American Association for the Study of Liver Diseases (AASDL), the American Society for Gastrointestinal Endoscopy (ASGE) and the Surgical Society for the Alimentary Tract (SSAT). Only about one half of these submitted abstracts will eventually be published as full papers, a rate comparable to other medical meetings (A Timmer, RJ Hilsden, J Cole, D Hailey and LR Sutherland, unpublished data, 2001) [1-4].

From surveys on unpublished research there is evidence that the failure to publish is, in the majority of cases, due to non-submission rather than to manuscript rejection [4,5]. Several studies have shown that abstract acceptance at a scientific meeting significantly influences the decision to write up and submit a complete manuscript for publication [4,6-8]. However, given the importance of meeting reports in scientific communication, the determinants of abstract acceptance are understudied [9]. While expert review processes aim at selecting abstracts based on the potential scientific impact of a contribution, other factors which are not easily identified during the usual review process also play a role.

A factor of particular concern is the potential for preferential acceptance of studies with statistically significant results as compared to "negative" results. Positive outcome bias in the publication process ("publication bias"), has been acknowledged as a problem primarily in the context of meta-analyses of published clinical trials [10,11]. However, this phenomenon is suspected to apply to any type of research, and may be even more prominent in observational or in less rigorously controlled interventional designs [5].

Other potential determinants of abstract acceptance other than the scientific impact include the origin of the abstract, the research type, and the formal abstract quality, including the quality of the language. Therefore, given the influence of abstract acceptance on subsequent publication and the lack of research on this issue, the objective of this study was to identify the determinants of abstract acceptance for the DDW.

Materials and methods

All abstracts submitted to the DDW between 1992 to 1995 (inclusive), including those rejected for presentation at the meeting, were screened for research type, country of origin and acceptance for presentation. These abstracts are available in print from the annual abstract volume of Gastroenterology. Country of origin was based on the affiliation of the first author as reported in the abstract. Research type was categorized as controlled clinical trial (CCT), other clinical research (OCR) or basic science study (BSS). CCT accounted for about 5% of all submissions, and comprised all clinical trials where a controlled parallel or cross-over design was used. In other words, besides Phase III clinical trials, controlled Phase II and Phase IV studies were included in this group [12]. OCR reports comprised therapeutic/diagnostic studies (i.e. studies other than CCT, examining the effects of therapeutic or diagnostic procedures), epidemiologic research (i.e. studies assessing frequencies of disease states or risk factors for their occurrence), and physiologic studies in humans, i.e. studies devoted to the elucidation of disease processes in men, healthy or diseased. BSS were defined as any study, performed in a laboratory setting, where the unit of analysis was not the intact human (animal studies, and studies in biological material). Abstracts that did not report empirical data or that could not be classified as clinical research or basic science studies were excluded. A random sample of 1,000 abstracts was selected based on computer-generated random numbers. Since we considered positive outcome bias and aspects of formal quality to be particularly relevant in CCT, we used a disproportionately stratified sampling procedure based on research type to increase the proportion of CCT in the sample (n = 400). The examination of abstract characteristics in BSS had a more exploratory character. Therefore, a smaller proportion of BSS was sampled (n = 200, OCR: n = 400). The sampling was also stratified by year of submission.

All sample abstracts were evaluated in random order by a single observer (AT) who was blinded to abstract authors, affiliations and acceptance status. The research type (CCT, OCR, or BSS) was reviewed, and revised where necessary. To detect any systematic errors during the abstract assessment, of every 100 consecutively evaluated abstracts, 10% were evaluated a second time by the same investigator following an interval of 4–6 weeks (test-retest reliability). An additional 10% were evaluated by a second reviewer (RJH), and the two ratings were tested for inter rater agreement. Items recorded were study design, sample size, statistical significance of study results and formal abstract quality.

Formal abstract quality was defined as a combination of the completeness of reporting and the internal validity of the study. Because no measurement instrument was available that would enable a uniform approach across the large variety of research types, designs and topics, a rating system was developed based on previously validated quality scoring instruments for full papers [13,14] with modifications based on recommendations for the reporting of structured abstracts [15,16]. In this rating system, items such as the definition of a research objective, the reporting and appropriateness of the statistical methods, the appropriateness of the sample size, and (where applicable) subject selection criteria, use of control groups, use of randomization, blinding, and control of confounding are combined into a single summary score. The instrument is described in more detail in a separate manuscript which has been submitted for publication. Quality of language was assessed separately by native speakers of North American English currently working in the field of gastroenterology. It was graded as "none or minor errors" (such as typographical errors) or "major errors" (such as grammatical errors or inappropriate vocabulary).

Sample size was recorded as the number of subjects per treatment arm or study group. The distributions of sample size and formal abstract quality score were assessed separately for the three different research types. Within each type, the variables were categorized into tertiles.

Large sample size was defined as a sample size within the highest tertile. Similarly, high formal abstract quality was defined by a formal quality score within the highest tertile of summary scores.

Statistical significance of the results was assumed if p < 0.05 with respect to the main outcome, if the 95% confidence interval (95% CI) excluded the reference value, or if the report stated that statistical significance was achieved. Results were considered negative when statistical significance was not achieved. Results were considered "equivocal" when statements on statistical significance were absent or when multiple results with mixed significance were reported.

Abstract productivity of a country was defined as the number of submitted abstracts per 1,000 physicians in the country, based on published statistical information on population size and physician density [17]. Countries were grouped after data exploration, depending on similarities in the distribution of research type, acceptance rates and geographical region.

Statistical analysis

For the assessment of test-retest and inter rater reliability of the rating system, intra class correlation coefficients (ICCC) were calculated [18].

The sample size was calculated to allow for separate analyses by research type (controlled clinical trials, other clinical research, basic science research), to detect a minimum relevant OR of 2.0 with a power of > 80%. Multiple logistic regression was used to calculate adjusted odds ratios for abstract acceptance. Chi squared tests and Spearman's correlation coefficient were used in the explorative analysis. Statistical significance was assumed on a 95% level of confidence. Publication rates are presented stratified by research type, statistical significance of the abstract results, and region of origin.

Determinants of abstract acceptance were estimated using multiple logistic regression analysis, including only analytical studies (studies including a control group or before-after comparisons). To account for the disproportionate sampling, analyses were performed stratified by research type (separate models for each group). In a combined model, effects of interaction were examined by the inclusion of interaction terms. All data were entered as categorical variables. Model fit was examined by the Hosmer-Lemeshow-test [19].

Results

Description of abstracts and acceptance rates, all abstracts

During the four year period, 17,205 abstracts were submitted. Contributions were divided approximately evenly between clinical and basic science research (Table 1). Only 5.4% of all abstracts were CCT. The percentage of unclassifiable studies was below 1%.

Table 1

Distribution and crude acceptance by country of origin, all abstracts

Country	Number submitted	Research type			Number accepted (%)
		CCT	OCR	BSS
1. USA	7,406	3.3%	39.8%	56.1%	5377 (72.6)
2. Japan	1,636	0.9%	26.4%	64.9%	762 (46.6)
3. Germany	1,308	7.6%	49.8%	42.3%	771 (58.9)
4. UK	1,161	8.2%	58.4%	33.1%	697 (60.0)
5. Italy	994	7.7%	74.8%	17.0%	437 (44.0)
6. France	944	9.9%	61.3%	28.5%	522 (55.3)
7. Canada	765	4.8%	26.4%	67.8%	530 (69.3)
8. Netherlands	443	7.7%	63.0%	28.9%	265 (59.8)
9. Australia	317	6.6%	51.7%	41.0%	213 (67.2)
10. Belgium	254	3.5%	62.6 %	32.7%	150(59.1)
11. Spain	245	8.6%	55.9 %	35.5%	128 (52.2)
12. Switzerland	207	14.5%	58.0 %	27.1%	140 (67.6)
13. Ireland	184	4.3%	67.4 %	28.3%	90 (48.9)
14. Sweden	173	11.6%	29.5 %	58.4%	102 (59.0)
15.-65. othera	1168	10.4%	54.6 %	34.2%	520 (44.5)
Total	17,205	5.4%	46.7%	47.4%	10,704(62.2)

aAll countries submitting < 150 abstracts/4 years

Abstracts were submitted from 65 different countries. However, the top 10 contributors accounted for 90% of all submitted abstracts. Thirty nine percent of the abstracts originated in the US, followed by Japan (10%) and Germany (8%). In 503 abstracts (2.9%), authors from more than one country were reported. Taking into account population size and physician density, Irish investigators were the most productive, with an average of 9.0 submissions per 103 physicians per year (2nd Canada, 3.1/103, 3rd UK, 3.1/103).

The overall acceptance rate was 62.2% and varied by research type (CCT 66.9%, OCR 56.3%, BSS 70.5%). International contributions were more often accepted than those involving only one country (73.8% vs. 61.9%). Crude acceptance rates for the top 10 contributor countries by country of first author are presented in Table 1. There were no substantial changes in acceptance rates by country when the analysis was based on the country of the last author as compared to the country of the first author.

Description of sample abstracts

After revision of the research type, overall, 326 CCT, 455 OCR and 219 BSS were included in the random sample and evaluated in more detail. The evaluation procedure was found to be consistent over time (test-retest reliability, ICCC 0.85) and between the two raters (inter rater reliability, ICCC 0.69). The abstract characteristics are shown in Table 2. OCR studies were evenly divided between therapeutic or diagnostic studies (T&D, n = 159), epidemiology (epid., n = 135) and human physiology (physiol., n = 161). Within these subgroups, there was no difference in acceptance rates or in the percentage of negative studies (overall 12%). It is of note that only 4% of BSS reported negative results (CCT: 30%). Negative results were associated with lower sample size in CCT (p = 0.04) but not in OCR or BSS/animal studies (p = 0.6 and p = 0.8). There were no differences in the distribution of sample size categories based on region when stratified by research type (p = 0.3). Major language errors occurred in 7% of the abstracts (US/Canada 2%, N/W-Europe 9%, S/E-Europe 12%, other 11%). Comprehensibility was compromised by language errors in < 1% of the abstracts.

Table 2

Distribution and baseline characteristics of sample abstracts

	CCT	OCR	BSS	Total
Total	326	455	219	1,000
analytical design	326(100%)	331(73%)	174(79%)	831 (83%)
descriptive design	-	124 (27%)	45 (21%)	169(17%)
Country region
US/Canada	88 (27%)	181 (40%)	121 (55%)	390 (39%)
NW-Europe/Austr.	138(42%)	135 (30%)	44 (20%)	317(32%)
SE-Europe	52(16%)	62 (14%)	9 (4%)	123 (13%)
other	48(15%)	77(17%)	45 (21%)	170 (17%)
Language
no or minor errors	301 (92%)	416(91%)	210(96%)	927 (93%)
major errors	25 (8%)	39 (8%)	9 (4%)	73 (7%)
Median sample size/group
subtype 1	parallel: 30	T & D: 22	animal: 6
subtype 2	cross over: 10	epid.: 38	other: n/a
subtype 3		physiol: 10
Quality score
median	0.66	0.52	0.51
IQR	0.62–0.70	0.44–0.58	0.44–0.57
Stat. significancea
positive	140 (43%)	149 (45%)	65 (37%)	354 (42%)
negative	99 (30%)	41 (12%)	7 (4%)	147(18%)
equivocal	87 (27%)	141 (43%)	102 (59%)	330 (40%)

aanalytical studies only

Predictors of abstract acceptance, sample abstracts

The region of origin was associated with acceptance rates for all research types. The statistical significance and the formal quality were significantly associated with abstract acceptance in CCT and OCR but not in BSS (Table 3). The strong association between abstract origin and acceptance rates was not explained by differences in formal quality (Table 4). Abstracts containing major errors were on crude analysis less likely to be accepted than abstracts with no or minor errors (52.1% vs. 64.2%, p = 0.06). However, when stratified by region, this effect disappeared (p = 0.7). Furthermore, the association between origin and acceptance rates was sustained when abstracts with major language errors were excluded from the analysis.

Table 3

Acceptance rates by region, statistical significance and formal quality

	CCT	OCR	BSS
Total number accepted (%)a	227 (69.6)	214(63.7)	101 (58.0)
Statistical significance
negative	58 (58.6)	25(61.0)	3 (42.9)
positive	108(77.1)	108 (72.5)	37 (56.9)
equivocal	61 (70.1)	81 (55.5)	61 (59.8)
Country region
US/Canada	70 (79.5)	97 (75.8)	67 (70.5)
NW-Europe/Australia	90 (65.2)	60(56.1)	18(52.9)
SE-Europe	38(73.1)	22 (50.0)	4 (44.4)
other	29 (60.4)	35(61.4)	12(33.3)
Quality score
lowest tertile	69(61.1)	47(51.1)	31 (57.4)
median	71 (67.0)	77 (65.3)	35 (62.5)
highest tertile	87(81.3)	90(71.4)	35 (54.7)

aacceptance rates may differ from those reported above due to the exclusion of descriptive studies

Table 4

Acceptance rates by region, stratified by formal quality

	low score		medium score		high score
	N	(%)	N	(%)	N	(%)
US/Canada	91	62 (68.1)	113	86 (76.1)	107	86 (80.4)
NW-Europe/Australia	78	43 (55.1)	93	54 (58.1)	108	71 (65.7)
SE-Europe	33	17 (51.5)	36	20 (55.6)	36	27 (75.0)
other	57	25 (43.9)	38	23 (60.5)	46	28 (60.9)

Logistic regression analysis for the sample abstracts confirmed country region of origin, statistical significance of the results and study type as significant predictors of abstract acceptance (Table 5). Contributions from North America were more likely to be accepted when compared with submissions from Europe or other countries, and this effect was independent of the inclusion of other variables. Reports with negative results were less likely to be accepted as compared to studies with positive results, and BSS were less likely to be accepted than clinical research. As to be suspected from the stratified analysis, language errors were not associated with reduced chances for acceptance after controlling for other variables. No effect was found for one vs. two or more countries (national vs. international studies), and for sample size. Interaction between research type and statistical significance of the study results or origin could not be demonstrated by the inclusion of product terms.

Table 5

Predictors of abstract acceptance (n = 831a, adjusted for year of submission)

	n	OR	95% CI
Research type
CCT	326	1
OCR	331	0.6	0.5–0.9
BSS	174	0.4	0.3–0.7
Statistical significance
positive	350	1
negative	147	0.5	0.3–0.8
equivocal	334	0.7	0.5–0.9
Abstract quality score
lowest or middle tertile	532	1
highest tertile	299	1.4	1.0–2.0
Country group
USA/Canada	305	1
N/W Europe, Australia	291	0.4	0.3–0.6
S/E Europe	102	0.4	0.2–0.6
other countries	143	0.3	0.2–0.5

Model χ2 62.1 (df8), p < .0001; Hosmer and Lemeshow Test:c2 9.3 (df8), p = 0.2 aexcluded from the analysis: all descriptive studies (n = 169)

Estimates from a model restricted to CCT were of similar strength as those of the overall model, with the statistical significance and formal abstract quality being significantly associated with abstract acceptance. In addition, the use of parallel controls as opposed to cross over designs contributed significantly to the prediction of acceptance. Differences based on the country of origin were apparent; however, confidence intervals included the reference value.

For OCR, country of origin, statistical significance of the study results and formal abstract quality were significantly associated with acceptance. In BSS, the gradient based on the origin of the abstract was particularly strong as compared to the other sub-models. Animal studies had lower acceptance rates than studies in biological material.

The inclusion of formal quality as a continuous score rather than a dichotomized variable increased the precision of the models. Due to the narrower confidence intervals in this approach, country of origin was a significant predictor of abstract acceptance in all submodels. Otherwise, no substantially different results were found. The point estimates were similar as compared to the models using only categorical variables.

Discussion

This study identified two possible areas of bias in the process of abstract selection for the annual DDW: a bias in favor of North-American contributions, and a bias based on the statistical significance of the results.

Bias based on national origin has been reported in various aspects of the dissemination and appreciation of research, e.g. with respect to citation-publication ratios, journals' peer review processes, inclusion of trials into meta-analyses and even the distribution of Nobel prizes [20-27]. In our study, the association between the origin of the abstract and the chance for presentation at the meeting was strong and consistent across research types and formal quality categories, although there is an impression that the association is more pronounced in BSS as opposed to CCT. The lack of statistical significance for country of origin in the model restricted to CCT is most likely due to a type II error. The variable became significant when formal quality was entered as a continuous variable, increasing the precision of the estimates. The proportion of abstracts with major language errors was too low to have a significant impact on acceptance rates. The finding of country bias is in contrast to the results of a similar study which has recently been published in Gastrointestinal Endoscopy by Eloubeidi et al [28]. In this study, the determinants of publication and abstract acceptance were examined based on abstracts submitted to the American Society for Gastrointestinal Endoscopy. No effect was found for country of origin (US vs. non US) with respect to abstract acceptance for the meeting. However, these abstracts represented a very limited area of gastroenterology and comprised almost exclusively clinical research. Also, almost all of the non US submissions were of North-West European origin, while at the DDW, Japan was the second most frequent country of origin. Of note, in the study by Eloubeidi, non US submissions were followed by full publications in a significantly higher proportion as compared to abstracts from the United States. Possibly, there are European centers which are particularly strong in endoscopy research. The low subsequent full publication rate in the study by Eloubeidi (25%) is in contrast to publication rates published for other medical specialities (most often around 50% [3]) and may indicate that the results are not representative of more general meetings.

A limitation of our study was the inability to take into account originality and relevance which can only be assessed by expert reviewers. However, experts cannot be unaware to the origin of the abstracts, even if, other than in the DDW review process, attempts for blinding were made. It has been shown that abstract reviewers were able to correctly guess the responsible research lab in about 40% of submissions in blinded review [29,30]. Also, inter-rater reliability has been shown to be poor among expert reviewers [9,31]. Bias based on the identity, the origin, or the affiliation of an author can thus never be completely excluded in expert peer review. On the other hand, the scientific impact of a contribution can not be consistently gauged over a heterogeneous set of research. We would suggest the two approaches (expert review vs. formal assessment) are complementary.

We found evidence for publication bias, i.e. for preferential acceptance of abstracts with positive results, which was demonstrated for clinical trials as well as for "other clinical research". As well, the very low proportion of negative outcome reports in BSS (4%), may represent another form of positive outcome bias at the pre-submission stage.

Publication bias is particularly important and well demonstrated in meta-analyses of clinical trials or of studies on causality [32-34]. The effects of positive outcome bias in the selection procedures of scientific meetings, a phenomenon described before for other societies [4,35], may be more insidious, and possibly no less damaging, e.g. in view of the role of abstract acceptance as possibly the most important predictor of subsequent publication [4,6-8]. Potential reasons for lack of statistical significance may be inappropriate sample size or premature reporting of ongoing studies – both should rightly decrease the chances for acceptance. On the other hand, even if sufficient power (of ≥ 0.80) and a truly present effect was assumed for all studies, at least 20% of the reports should be expected to show negative results. Given the very low medians for sample size (Table 2), it is the more surprising how low the proportion of submitted negative studies was, even in CCT. There seems to be, in addition to the bias shown for abstract acceptance, a significant preference for statistically significant results influencing abstract submission.

A more reassuring finding in our study was the importance of formal abstract quality for abstract acceptance for the DDW. This is in accordance with the findings of other investigators who came to the conclusion that how one writes is at least as important as what one writes, or that attention to form reflects attention to content [36]. Possibly, formal quality is most important, as well as most easily assessed in CCT. The rating system we used is less responsive in BSS due to the lower number of applicable items. Formal abstract quality could in fact not be shown to be associated with abstract acceptance in this group. On the other hand, frequent omissions included basic, essential items such as the definition of a research objective or the number of animals studied. The lack of association with abstract acceptance indicates that formal requirements are considered to be less important in this type of research. Therefore, insufficient control for confounding by formal quality is unlikely to explain the preferential acceptance by country of origin which was particularly strong in BSS. Another potential limitation of our study is the inability to adequately control for sample size, as this can not be directly compared across different designs and research questions. Lastly, one needs to acknowledge that we examined associations in an observational design. Causal inference must, therefore, be drawn with caution. An experimental design such as presenting sets of identical abstracts modifying only certain variables, such as the identity of the author, or the statistical significance of the study results, would be helpful to further substantiate the findings. Such an experiment has for example been performed to test the peer review process in the educational sciences, and clear evidence for confirmatory bias (i.e. the tendency to prefer experiences which confirm the ideas of the reviewer) was found [37].

In conclusion, in the DDW abstract review process there is a remarkable preference for abstracts of North-American origin, and for abstracts presenting statistically significant results. More research is needed to further elucidate the underlying mechanisms.

Abbreviations

BSS basic science studies

CCT controlled clinical trial

DDW Digestive Diseases Week

epid. epidemiology

N/W North and West

OCR other clinical research

physiol. physiology

S/E South and East

T&D therapy and diagnostics

Competing interests

None declared.

Pre-publication history

The pre-publication history for this paper can be accessed here: