Development and evaluation of a quality score for abstracts.

BACKGROUND: The evaluation of abstracts for scientific meetings has been shown to suffer from poor inter observer reliability. A measure was developed to assess the formal quality of abstract submissions in a standardized way.
METHODS: Item selection was based on scoring systems for full reports, taking into account published guidelines for structured abstracts. Interrater agreement was examined using a random sample of submissions to the American Gastroenterological Association, stratified for research type (n = 100, 1992-1995). For construct validity, the association of formal quality with acceptance for presentation was examined. A questionnaire to expert reviewers evaluated sensibility items, such as ease of use and comprehensiveness.
RESULTS: The index comprised 19 items. The summary quality scores showed good interrater agreement (intra class coefficient 0.60 - 0.81). Good abstract quality was associated with abstract acceptance for presentation at the meeting. The instrument was found to be acceptable by expert reviewers.
CONCLUSION: A quality index was developed for the evaluation of scientific meeting abstracts which was shown to be reliable, valid and useful.

Background

The presentation of research at scientific meetings is an essential part of scientific communication. The choice of which research will be presented at a meeting is usually based on peer-review of abstracts submitted by interested investigators. Only about 50% of the research projects initially submitted as conference abstracts will eventually be published as full articles in peer-reviewed journals and full publication may not occur for several years [1,2]. As a result, a published abstract from a scientific meeting is often the only permanent source of information available on the methodology and results of a research project.

When writing an abstract, investigators are limited on the amount of detail on the study's methodology and results that they can include. This may partly explain the poor observer-agreement and several kinds of bias that have been demonstrated in the abstract selection procedures for various medical specialty meetings [3-9]. However, there is evidence that the quality of the presentation of information in an abstract is associated with the scientific quality of the research [10]. Therefore, standardized methods for assessing formal abstract quality may help improve the reliability of abstract selection and, if used as a checklist, result in more informative and useful abstracts.

Quality checklists and scoring instruments have been published for use with full reports of clinical research [11,12]. No such score is available for the evaluation of abstracts. Also, there is little information addressing the reporting of the methodology and results in basic science research. The objective of this study was to develop a reliable instrument to assess the quality of meeting abstracts that would be applicable to a wide variety of research types, including both clinical and basic science.

Methods

Material

We used abstracts submitted to the 1992 to 1995 annual meetings of the American Gastroenterological Association (AGA) to develop and test a quality scoring instrument for abstracts. Each year several thousands of abstracts on all aspects of research in gastroenterology are submitted to this meeting. All submitted abstracts were published in a yearly supplement to the journal Gastroenterology whether or not they were accepted for presentation at the meeting. Abstracts selected for oral or poster presentation were marked, however, the mode of presentation was not specified. Abstracts were submitted on a standard form that limited the length to between 200 to 350 words, depending on font size and inclusion of graphs. A structured format (Background & Aims, Methods, Results, Conclusion) was not mandatory but was commonly used.

For the purpose of this study, we divided the abstracts into different research types. Basic science studies (BSS) comprise all studies, done in a laboratory setting, where the unit of analysis was not an intact human. Controlled clinical trials (CCT) included prospective studies on the effects of diagnostic tests or therapeutic interventions, using parallel or cross over controls. All other studies using humans were categorized as "other clinical research" (OCR) and included human physiology experiments, epidemiological studies and uncontrolled therapeutic studies.

A large sample (n = 1000) was selected for use in a follow up study on publication bias using computer generated random numbers based on a database containing all abstracts submitted (17 205). This sampling was stratified by research type to increase the proportion of RCT's. For the test development and testing, a random subsample (n = 100) was collected from this sample. The eventual sample included 42 CCT's, 39 OCR, and 19 BSS. The abstracts were assessed in a prespecified order.

Item generation and reduction

Formal abstract quality was conceptualized as the combination of the methodological strength of the research as well as the clarity of the presentation. The methodological strength of the research included those methods used to improve the internal validity of the study as reported in the abstract.

Item selection and instrument structure were based on previously published instruments used to assess the quality of full manuscripts in therapeutic trials and epidemiological research [11,13-15]. A measure developed by Cho and Bero was particularly influential because of its applicability to different types of clinical research [16]. We could not identify any work on quality scoring in basic science reports.

When adapting this scale for use with abstracts, we took into account published guidelines for the composition of structured abstracts for original research for publication in biomedical journals [17,18]. Similar guidelines for meeting abstracts and basic science reports had not been published. For content validity, the list of items was discussed face to face and repeatedly with one researcher each in lab based medicine (J. Wallace, Calgary), health care research (RJ Hilsden), and clinical trials (LR Sutherland). The instrument was further modified following a pilot study of 80 abstracts that were scored by two raters and subsequently discussed in detail. Items were simplified or dropped if consensus decisions were difficult or if sufficient information was absent in the majority of abstracts, unless the item was considered essential for the evaluation of study validity by at least one of the reviewers. Input on the final instrument was sought from researchers in the three fields from across Canada by sending them the instrument and a questionnaire on the appropriateness of the items included (Table 2). No further items were suggested to be added or dropped by those surveyed.

Table 2

Sensibility ratings

	Basic Scientists (range)	Clinical & Health Care Researchers
Number of reviewers	4	11
1. Wide applicability	3.5 (1–5)	5.7 (3–7)
2. Use in basic science	2.5 (1–5)	5.4 (4–7)
3. Use in clinical research	4.5 (1–6)	5.8 (4–7)
4. Clarity and simplicity	3.3 (1–5)	5.7 (5–7)
5. Time and effort needed	2.3 (1–5)	5.6 (5–7)
6. Adequate instructions	3.8 (1–5)	5.8 (5–7)
7. Information available from abstract	4.0 (3–5)	4.6 (4–5)
8. Need for subjective decisions	4.8 (3–6)	5.4 (4–7)
9. No redundancy	4.3 (3–5)	6.3 (5–7)
10. Comprehensiveness	5.5 (2–7)	5.9 (3–7)
11. Response options	6.0 (4–7)	5.9 (3–7)
12. Discriminative power	5.8 (5–7)	5.7 (5–7)
13. Measures methodology	5.5 (4–7)	5.7 (4–7)
14. Measures reporting	5.3 (3–7)	5.9 (5–7)
Mean rating	4.3 (3.3–5.7)	5.7 (4.9–6.5)

Ratings on a scale from 1 (unacceptable) to 7 (fully met). Mean > 5: item acceptable

The resulting instrument allowed for the calculation of a summary score with a range from 0 to 1. Summary scores are reported as mean scores with standard errors of the mean (SEM).

Interrater and test-retest reliability

We tested interrater reliability on 100 abstracts scored by two raters with training in gastroenterology and epidemiology (AT and RJH). Test-retest reliability was determined through the duplicate scoring of 85 abstracts by the same observer (AT) with an interval of four to six weeks. The raters were blinded to name and institution of the authors and to whether the abstract had been selected for presentation. For each item, the kappa coefficient was used to assess the degree of agreement beyond what might be expected by chance [19]. A kappa score in the range of 0.41 to 0.60 was considered to represent moderate agreement, and a kappa score in the range of 0.61 to 0.80 was considered substantial agreement [20]. Agreement between summary scores was assessed using the intra class correlation coefficient [21]. This measure is the ratio of between abstract variance to overall variance. A ratio (Ri) close to 1 is found when the variance is almost exclusively due to differences between the abstracts, while low ratios represent a strong influence of random error and/or observer variance on the overall variance. There is no universally accepted cut off of Ri for good agreement, as it is appreciated that this depends on the number of subjects studied and the context in which the decisions based on the instrument will be made [21]. For this study, moderate agreement was defined as Ri > 0.6 and good agreement as Ri > 0.8.

Construct validity

To examine construct validity, we hypothesized that higher abstract quality scores should correlate with abstract acceptance by the AGA. To test this hypothesis, the mean quality scores of rejected and accepted abstracts were compared using Student t-tests for independent samples.

Sensibility

The sensibility of an instrument, as defined by Feinstein, comprises different aspects of usability, such as comprehensibility, ease of use, face and content validity, and suitability for the purpose at hand [22]. A sensibility questionnaire was adapted from Oxman [23,24] to assess the usefulness of the newly developed score (see appendix). This questionnaire was sent to 24 Canadian researchers in gastroenterology and epidemiology. The researchers rated each item based on a seven-point scale (1= unacceptable, 7 = criterion fully met). We considered a mean rating of >5 to indicate general satisfaction with and acceptability of an item.

Results

The review of the literature and expert opinion resulted in the generation of 24 quality assessment items pertaining to the study purpose, methods, reporting of results and conclusions. Following the pilot-study, the instrument was reduced to 19 items. Two items on research methods were discarded because they were never applicable to the abstracts evaluated (1. whether random selection from the study population was performed; 2. if yes, which method of random selection was used). A priori sample size considerations were also never reported. Instead, for use in abstracts, the reporting of posteriori sample size considerations or power calculations for negative results was considered sufficient. Two pairs of items were combined into single items. First, "control for confounding by the study design" and "control for confounding by the analysis" were combined into "confounding controlled for?". Second, because it was too difficult to discern between the inclusion/exclusion criteria and the baseline characteristics of the subjects, it was considered sufficient if either were included in the abstract. Although the method of randomization in randomized trials was never described, an item on this was retained because it was considered essential for the assessment of study quality. Following the pilot testing, several items were reworded for simplification and clarity. A detailed manual was written that provided the definitions used for each item (manual available from the authors). Using the instrument and manual, the average time required to score an abstract was 3:45 minutes (range 2:30 to 6:00 minutes). The final 19-item instrument is shown in the appendix.

Reliability

For the 100 abstracts independently assessed by two raters, quality scores ranged from 0.28 to 0.95, and were approximately normally distributed. As shown in table 1, the inter-observer agreement was highest for controlled clinical trials and lowest for basic science abstracts. Individual item analysis showed substantial agreement for the description of random allocation, investigator and subject blinding, appropriateness of sample size, reporting of attrition, and reporting of statistical tests. Moderate agreement was achieved for the appropriateness of the controls and the appropriateness of the statistical methods. The agreement was low for the remaining items (description of study objective, description and appropriateness of the design, method of subject selection, definition of outcome measure, control for confounding, exact p-values or confidence intervals, reporting of results, and conclusions supported by the results).

Table 1

Interrater reliability.

Type of research	n	mean summary score observer 1 (SEM)	mean summary score observer 2 (SEM)	intra class correlation coefficient (Ri)
Clinical trials only	42	.63 (.02)	.63 (.02)	.81
Other clinical research	39	.57 (.02)	.59 (.02)	.67
Basic science	19	.50 (.02)	.61 (.03)	.60

SEM = standard error of the mean

Test-retest reliability was good independent of the research type, with identical mean scores for first and second evaluations (0.57) and an intra-class coefficient of 0.85. Individual items also had good test-retest reliability (range of kappa coefficients, 0.54 to 0.85).

Construct Validity

In total, 67 of the 100 abstracts were accepted for presentation at the AGA meeting. These abstracts had significantly higher quality scores compared with the rejected abstracts (0.61 vs. 0.54, p = 0.001). The quality scores were higher for accepted abstracts compared with rejected abstracts for all research types, but this was only statistically significant for clinical trials (n = 42, 0.67 vs. 0.52, p < 0.001).

Sensibility ratings were available from 15 reviewers (survey response rate 62.5%). As shown in Table 2, clinical and health care researchers found the instrument useful and acceptable for the purpose described. The reviewers felt that both the research methodology and quality of the report were adequately assessed by our instrument (Table 2). In general, basic scientists found the instrument less useful, largely because of the time and effort required. Both groups of reviewers considered the applicability to basic science abstracts to be a problem. The issue of the limited amount of information available from abstracts was also raised.

Discussion

We have developed a reliable instrument for assessing the quality of an abstract submitted to a scientific meeting. The instrument rates both the quality of the reporting and the quality of the research methods. While a variety of scales are available for the assessment of full reports of clinical trials, abstract evaluation has usually involved informal ad hoc scales or checklists [4,18,25,26] or has been restricted to the quality of reporting only [18,27,28]. Furthermore, most abstract evaluation studies were based on summaries of published research.

The limited amount of information available from abstracts was the greatest challenge in the development of this instrument. The methodological quality of research can only be assessed to the extent that pertinent information is available in the report. There is controversy over how well the quality of research is reflected in a short report [25,29,30]. Furthermore, the quality of the report and the quality of the research are often intertwined. The experts in our sensibility survey felt that our instrument adequately assessed both the research methodology and the quality of the report. However, we appreciate that the lack of a clear distinction between these two concepts of quality in our instrument may be unsatisfactory to some users, as may the restriction to formal aspects of scientific quality. Our instrument avoids any judgment of content, such as, originality, ethics or scientific relevance. It is recommended that these factors be assessed separately, as suggested by other authors [16,28,31]. However, there is evidence that the formal quality of a research report reflects the content or overall quality of the research [28,32]. This view is supported by the association of higher quality scores with abstract acceptance by the AGA.

A few important items seem to be missing in our instrument. These are characteristics which were never reported in any of the abstracts, although they have been shown to be essential to the validity of study results. Most prominently, this concerns concealment of allocation in RCT's. Possibly, the importance of this item was not appreciated by abstract authors because the paper by Schulz, demonstrating the bias introduced by open allocation was only published in 1995, i.e. after conclusion of our abstract submission period studied [33].

Generally, the use of summary scores should be viewed with caution. In the context of meta-analysis, the individual assessment of the various methodological aspects has been shown to be more informative with respect to their effect on effect size. The excellent meta-analysis by Jüni et al. on quality scoring of clinical trials was not published when we developed our instrument [12]. In contrast, the use of a summary score rather than components is more helpful if a range of diverse designs and topics is evaluated, as the number of shared components is rare.

One of the objections raised in the sensibility survey was the time and effort needed to complete a checklist of 19 items. It is appreciated that this list may seem daunting, especially as previous instruments on full reports were restricted to as few as three items. However, instruments such as the checklist used by Chalmers, the Delphi list, or the short instrument by Jadad were developed to assess full reports of randomized clinical trials in the context of meta-analysis [13,14,34]. This entails the possibility (in some cases: necessity) to tailor the instrument to the study subject at hand. In contrast, it was our aim to develop an instrument that would be applicable to a wider variety of research types and subjects. Furthermore, even for randomized controlled trials, the use of a three item score was shown to be not useful in the evaluation of abstract quality in a follow up study on scientific abstracts by I. Chalmers [26]. This may be due to insufficient information provided in abstracts. A more sensitive instrument is needed for abstracts. Also, other items are required for the evaluation of other types of research, such as adequate control for confounding in observational studies. Consequently, depending on the study design, only a limited number of items are applicable for each individual abstract. The average time of less than 4 minutes per abstract seemed a reasonable effort. However, this time does not include an evaluation of the scientific background of the study question nor the appreciation of the importance of its conclusions for the scientific community.

Problems were encountered when we tried to tailor the instrument to basic science research reports. In fact, the results of both the reliability testing and the sensibility survey indicate that the instrument may be less suitable for this type of research. We are not aware of any literature on the formal assessment of quality in the design and reporting of basic science research. Therefore, some of the problems encountered were due to a lack of generally agreed upon criteria for the evaluation of this type of research. However, the main difficulty when reading basic science abstracts related to the omission of basic information such as the formulation of the research objective and clarification of the research design. Lower quality scores, an increased need for subjective judgments because of limited information, and lack of scale acceptance by basic science reviewers may be interrelated and may reflect, in general, a lesser interest in structured reporting in basic science abstracts.

The score was subsequently successfully used in a study examining the determinants of abstract acceptance and publication in gastroenterological research [35,36]. High abstract quality scores were associated with higher chances for acceptance for presentation at a meeting, and were also associated with higher impact factors, if abstracts were followed by full publications.

Conclusions

In summary, we have developed a reliable, valid and applicable instrument for the evaluation of the quality of scientific abstracts. While most useful in the clinical research setting, particularly for clinical trials, limitations may apply for its use in basic science. It may also be helpful as a checklist for the preparation of scientific abstracts or serve as an instrument to compare abstract quality between meetings or over time. Other possible applications include the adjunct use in abstract peer review. More work is needed to improve the instrument for use in basic science research and to assess the applicability to other medical specialties. Also, further research should test the hypothesis that this instrument will reduce bias in the selection of abstracts for presentation.

Competing interests

None declared.

Authors' contributions

A. Timmer and RJ Hilsden developed the instrument, evaluated abstracts and jointly wrote the manuscript. The statistical analyses were performed by A. Timmer. LR Sutherland conceived and supervised the study, and contributed to the instrument modifications and to the writing of the manuscript.

Tables and Appendix

Tables and appendices are added at the end of the manuscript as follows:

Table 1: Interrater reliability

Table 2: Sensibility ratings

Appendix A: Quality scoring instrument (see additional file 1)

Appendix B: Sensibility questionnaire for reviewer survey (see additional file 2)

Pre-publication history

The pre-publication history for this paper can be accessed here:

Additional file 1

Quality scoring instrument.

Click here for file

Additional file 2

Quality scoring instrument.

Click here for file