Assessment of Adherence of Diagnostic Accuracy Studies Published in Radiology Journals to STARD Statement Indexed in Web of Science, PubMed & Scopus in 2015.

RATIONALE AND
OBJECTIVE: The objective of this study is to evaluate the methodological adherence of diagnostic accuracy studies published in radiology journals, which were indexed in different databases with the STARD standard guide 2015.
MATERIALS AND METHODS: The different databases were searched in order to find suitable journals. Among 84 English radiology journals, 31 journal were selected randomly. In order to find the articles, the same search fields and search terms were used. All the items of STARD checklist 2015 were considered to take in to account in assessment of the adherence of the articles to the standard. Total STARD score for each article was calculated by summing the number of reported items.
RESULTS: 151 articles from 31 journals were evaluated to check the adherence of their structure to STARD standard. Based on the results the articles had the most adherence with the STARD standard in material and method part the item of participants, discussion section, and title or abstract. On the contrary, most of the articles were not adhere to other information which are new items in STARD 2015. Among radiology diagnostic accuracy articles only one article (0.66%) had a registration number and 10 (6.62%) articles had a link to full study protocol. More than 60% of articles adhered to the ethics (69.54%) and source of support (63.58%).
CONCLUSIONS: The radiology diagnostic accuracy studies were adhered to 69.45% STARD items, which shows an improvement in reporting the diagnostic accuracy articles in comparison to previous studies.

Introduction

A diagnostic accuracy study is a research which prepares a proof on how well a test or a special procedure accurately recognizes or precludes diseases. This type of study helps clinicians and healthcare suppliers regard the choice of a suitable management method for each patient [1]. A comprehensive and accurate reporting of diagnostic accuracy studies is essential to permit readers to evaluate any possible bias in the study and to assess the reproducibility of the results. Accordingly, in order to compare different diagnostic methods, there should be a standard procedure for perform the study and report the results so as to avoid exaggerated conclusion and make the results generalizable [1].

A group of scientists established STARD (Standards for Reporting Diagnostic Accuracy) statement to improve the quality of reporting these studies. At first, STARD statement consisted of a checklist of 25 items and a flow diagram in which authors can ensure that all relevant information is present [2]. This standard covers the title, abstract, keywords (Item 1), introduction (Item 2), methods (Items 3-13), results (Items14-24) and discussion (Item 25). In the method section, the required information regarding the participants (Items 3- 6), test methods (Items 7-11), and statistical methods (Items 12-13) were described in detail. The required information under result section regarding participants (Items 14-16), test results (Items 17-20) and estimates (Items 21-24) were precisely explained.

This checklist was reviewed in 2015, and its items increased to 30 [3]. This revision is intended to simplify the application, understanding and distribution of the checklist. The new document [3] comprises an explanation of the definition, rationale and optimal usage of each element on the checklist, as well as a brief summary of the existing evidence on bias and applicability. The new standard also covers the title (Item 1), abstract (Item 2), introduction (Items 3-4), methods (Items 5-18), results (Items19-25) and discussion (Items 26-27); it also requires new information (Items 28-30). In the method section, the required information about the study design (Item 5), participants (Items 6-9), test methods (Items 10- 13) and analyses (Items 14-18) are described in detail. Even some items are split into 2 items such as item 10 (requiring the information about the index and reference tests), item 12 (the cutoffs of reference and index test) and item 13 (the availability of reference and index test). The required information in result section about participants (Items 19-22) and test results (Items 23-25) were precisely explained. Item 21, which is about the distribution of the disease and the distribution of alternative diagnosis is split into 2 parts.

Previously, some studies evaluated the effectivity of STARD statement and how well the diagnostic accuracy articles adhered to this standard. Some studies assessed the papers published in specific fields such as ophthalmology, gastrointestinal disease, obstetric and gynecology, musculoskeletal disorders and hepatology. Most of these articles reported a non-desirable adherence of articles to the standard leading to fewer effects of this standard on improving the diagnostic accuracy reports [4-18].

To the best of our knowledge, there are no studies on the adherence of radiology and medical imaging articles to the STARD statement, although there are lots of radiology diagnostic accuracy studies.

The objective of this study is to evaluate the adherence of diagnostic accuracy studies published in 2015 in the English language radiology journals which were indexed in the Web of Science, PubMed and Scopus with STARD standard guide 2015.

Material and Methods

This was a cross-sectional study that evaluated 151 diagnostic accuracy articles published in radiology journals indexed in the Web of Science, PubMed and Scopus databases. The sample size was determined based on 131 articles using formula 1 [19]:

Eq. 1

Where, n= sample size, = confidence interval, P= estimated proportion, d= desired precision.

Sampling

The sampling was done as follows:

1. Firstly, the Web of Science, Scopus and PubMed databases were searched in order to find journals that contain diagnostic accuracy articles. Accordingly, to find such journals, an advanced search was performed. In the website of the web of science, the search fields of title, document type and the topic were used. The keywords and search fields were used in different databases. The keywords were selected in the way to narrow down the search field and to obtain possible results too (Table 1). Additional filters such as subject area and document type were also used after obtaining the results. This search led to collecting 84 radiology journals which contained diagnostic accuracy articles.

Table 1

Search fields and search terms used in different databases

Database	Search fields
	Title	Abstract	Keyword/ Mesh	Source title	Topic
Web of Science	Diagnosis OR diagnostic OR accuracy OR sensitivity OR specificity OR efficacy OR Compare	---------	---------	-------	Radiology and nuclear medicine
Scopus	Diagnosis OR diagnostic OR accuracy OR sensitivity OR specificity OR efficacy OR Compare	Diagnosis OR diagnostic OR accuracy OR sensitivity OR specificity OR efficacy OR Compare	Diagnosis OR diagnostic OR accuracy OR sensitivity OR specificity OR efficacy OR Compare	Radiology OR imaging OR ultrasound	Radiology and nuclear medicine
PubMed	Diagnosis AND diagnostic AND accuracy AND sensitivity AND specificity AND efficacy	---------	---------	--------

2. In order to randomize the obtained list of journals, an online randomizer was used (http://www.randomizelist.com).

3. In order to select 31 English Radiology Journals out of 84 journals, an online random number generator was used (https://www.random.org). Since some Journals did not contain enough number of diagnostic accuracy articles, one more journal was selected randomly so that the overall number of selected journals hit 31.

4. In order to find the articles from each journal, the same search fields and search terms as “ Table 1" were used; the only difference in this step was that the given journal was selected to limit the search to a specific journal. Totally, 275 articles were selected (Figure 1).

Figure1

Flow chart showing the search results and limitation the results.

5. Among selected articles, 151 articles were randomly selected to evaluate the adherence of their structures to STARD statements (Figure 1).

Assessment of STARD Checklist

Although the latest version of STARD checklist [3] was introduced in 2015, since there was not much difference between this and earlier version (just 3 new items were added) and moreover because of its better categorization of different items (splitting some previous items into 2 parts), it was used to take into account in assessment the adherence of the articles published in 2015 to this standard.

It is also assessed that if the word of “STARD” standard was pointed out in the studies or not? The reported items were scored 1 and not reported ones were scored 0. The items that were considered in each section were shown in below tables. Items 1 and 18 were split into 2 sections. Items 1a and 1b were about the presence of the diagnostic accuracy terms in title and abstract respectively. Item 18a and 18b were scored the presence of pre-specifying the sample size and how the samples were collected respectively. We also scored the journals which introduce the STARD statement in instructions for authors. The items of STARD statement that assessed in this study was illustrated in Table 2, Table 3, Table 4, Table 5.

Table 2

The items which were assessed in selected diagnostic accuracy articles to check the adherence of title and abstract section to the STARD. It also assessed if the STARD standard was pointed out in the studies or not.

STARD	TITLE OR ABSTRACT
the word "STARD" used in the study	1a. Title	1b. Abstract	2. Structured Abstract	3. background	4. study objective

Table 3

The items which were assessed in selected diagnostic accuracy articles to check the adherence of method section to the STARD. Item #18 was broken into two sections: 1. If the sample size was pre-specified. 2. How the samples were chosen?

Method	study design	5. data collection
	participants	6. including/excluding criteria
		7. how eligible participants were identified
		8. setting, location, and dates
		9. patient sampling
	Test methods	10a. Index test, in sufficient detail to allow replication
		10b. reference or gold standard
		11. Rationale for choosing the reference standard
		12a. Definition of and rationale for test positivity cut-offs of index test
		12b. Definition of and rationale for test positivity cut-offs the reference standard
		13a. clinical information and reference standard results in performers/readers of the index test
		13b. clinical information and index test results were available to the assessors of the reference standard)
	Analysis	14. Methods for estimating or comparing measures of diagnostic accuracy
		15. How indeterminate index test or reference standard results were handled
		16. How missing data on the index test and reference standard were handled
		17. Any analyses of variability in diagnostic accuracy, distinguishing pre-specified from exploratory
		18. Intended sample size pre-specified
		How the sample size was chosen

Table 4

The items which were assessed in selected diagnostic accuracy articles to check the adherence of result section to the STARD

Results	Participants	19. Flow of participants, using a diagram
		20. Baseline demographic and clinical characteristics of participants
		21a. Distribution of severity of disease in those with the target condition
		21b. Distribution of alternative diagnoses in those without the target condition
		22. Time interval and any clinical interventions between index test and reference standard
	Test results	23. Cross tabulation of the index test results (or their distribution) by the results of the reference standard
		24a. Estimates of diagnostic accuracy (accuracy, area under curve or ROC)
		24b. Estimates of diagnostic accuracy and their precision (such as 95% confidence intervals, SD, Standard error)
		25. Any adverse events from performing the index test or the reference standard

Table 5

The items which were assessed in selected diagnostic accuracy articles to check the adherence of discussion section and other information to the STARD

DISCUSSION	26. Study limitations, including sources of potential bias, statistical uncertainty, and generalizability
	27. Implications for practice, including the intended use and clinical role of the index test
OTHER INFORMATION	28. Registration number and name of registry
	29. Where the full study protocol can be accessed
	30. Sources of funding and other support; role of funders (conflict of interest)

Searching the Text

In order to accelerate the searching inside the text of articles, some keywords were used. For instance, in order to assess the items of title and abstract the keywords such as “diagnostic accuracy”, “sensitivity and specificity”, “efficacy” were used. The related text was found and read to evaluate the item. If the item could not suitably assess by this method the whole text of article was read. The applied keywords for each case was shown in Table 6.

Table 6

The applied keywords for each section of the articles

Section head	Subsection	Item number	Keywords
TITLE OR ABSTRACT	title or abstract	1a	“diagnostic accuracy”, “sensitivity and specificity”, “efficacy”
		1b
		2
		3
		4
Method
	study design	5	"retrospective", "prospective"
	participants	6	"include", "exclude"
		7	"inclusion criteria", "exclusion criteria"
		9	"random", Consecutive"
	Test methods	10b	"gold standard", "reference standard", "standard"
		12a	cut-off
		12b	cut-off
		13a	"blind", "mask", "unaware", "knowledge", "observer", "reader"
		13b	blind, "mask", "unaware", "knowledge", "observer", "reader"
		14	"confidence interval"
		15	"false", "negative", "positive", "predictive value"
	Analysis	16	"miss"
		17	"subgroup"
20		"years", "male", "female"
Results	Participants	21a	"stage", "grade", "level"
		21b	"stage", "grade", "level"
		22	"interval", "after"
	Test results	24a	"ROC", "AUC"
		24b	"confidence interval", "standard deviation", "standard error"
	Discussion	27	"limitation"

Ethical Considerations

Because there was no need to work with patients or animals, it was not necessary to get the approval of ethics committee. But the confidentiality of the authors and journals were considered.

Statistical Analysis

Microsoft Excel 2007 was used to analyze the data. The reported items were scored 1 and not reported ones were scored 0.

Results

Randomized Sampling of Journals

The characteristics of journals which were chosen randomly are shown in Table 7. The information of H-index and the country was obtained from SJR website. Moreover, it was assessed that if STARD standard was introduced in instruction for authors or not. Besides, the number and percentage of journals from each country and continent were assessed.

Table 7

The characteristics of selected journals in terms of being indexed in different databases (y=yes and n=no), H-index, country

Journals	Scopus	PUBMED	Web of Science	introduction of STARD in instruction for authors	H-index	Country
Academic Radiology	y	n	y	n	75	United States
Acta radiologica	y	y	y	n	57	United Kingdom
American Journal of Neuroradiology	y	y	y	y	141	United States
American journal of roentgenology	y	y	y	y	159	United States
Annals of Nuclear Medicine	y	y	y	n	45	Japan
BMC Medical Imaging	y	y	y	n	22	United Kingdom
British journal of radiology	y	y	y	n	86	United Kingdom
Clinical Radiology	y	n	y	y	71	United Kingdom
Dentomaxillofacial Radiology	n	y	y	n	54	United Kingdom
Egyptian Journal of Radiology and Nuclear Medicine	y	n	n	n	4	Netherlands
Emergency Radiology	y	y	y	n	31	United States
European Journal of Nuclear Medicine and Molecular Imaging	y	n	y	n	106	Germany
European Journal of Radiology	y	n	y	n	86	Ireland
European Radiology	y	n	y	n	118	Germany
IEEE Transactions on Medical Imaging	y	n	y	n	165	United States
Indian Journal of Radiology and Imaging	y	y	n	n	14	India
International Journal of Radiation Oncology Biology Physics	y	y	y	n	198	United States
Investigative radiology	y	y	y	n	89	United States
JACC: Cardiovascular Imaging	y	y	y	n	71	United States
Journal of Computer Assisted Tomography	y	y	y	n	80	United States
Journal of Medical Imaging and Radiation Oncology	y	y	y	y	30	Australia
Journal of Ultrasound in Medicine	y	y	n	n	67	United States
Korean Journal of Radiology	y	y	y	y	38	South Korea
Nuclear Medicine Communication	y	y	y	n	59	United States
Pediatric Radiology	y	n	y	n	66	Germany
polish journal of radiology	y	y	n	n	7	Poland
Practical radiation oncology	y	y	n	n	13	Netherlands
radiologica medica	y	y	y	n	34	Italy
Radiology	y	y	y	y	240	United States
Skeletal Radiology	y	y	y	y	68	Germany
Ultrasound in Obstetrics and Gynecology	y	y	y	y	106	United Kingdom
Total	30	23	26	8

The number and percentage of journals which were randomly selected in terms of the country were shown in Table 8.

Table 8

The number and percentage of journals which were randomly selected in terms of the country

Continent	Country	Number/ Country	%	Number/ Continent	%
America	United States	12	38.71	12	38.71
Europe	United Kingdom	6	19.35	14	45.16
	Germany	4	12.90
	Netherlands	1	3.23
	Italy	1	3.23
	Ireland	1	3.23
	Poland	1	3.23
Asia	Japan	1	3.23	3	9.68
	Korea	1	3.23
	India	1	3.23
Australia	Australia	1	3.23	1	3.23
Africa	Egypt	1	3.23	1	3.23
Total		31	100	31	100

The number of primarily selected articles and the number of diagnostic accuracy articles which were evaluated has been indicated in Table 9.

Table 9

The number of primarily selected articles and the number of diagnostic accuracy articles

Journals	Number of selected article	Number of diagnostic accuracy article
Academic Radiology	15	5
acta radiologica	6	6
American Journal of Neuroradiology	14	12
American journal of roentgenology	10	10
Annals of Nuclear Medicine	3	3
BMC Medical Imaging	5	2
British journal of radiology	21	3
Clinical Radiology	12	5
Dentomaxillofacial Radiology	9	3
Egyptian Journal of Radiology and Nuclear Medicine	14	5
Emergency Radiology	3	1
European Journal of Nuclear Medicine and Molecular Imaging	9	8
European Journal of Radiology	14	14
European Radiology	8	6
IEEE Transactions on Medical Imaging	3	1
Indian Journal of Radiology and Imaging	10	5
International Journal of Radiation Oncology Biology Physics	11	3
Investigative radiology	16	2
JACC: Cardiovascular Imaging	3	3
Journal of Computer Assisted Tomography	4	4
Journal of Medical Imaging and Radiation Oncology	8	7
Journal of Ultrasound in Medicine	6	5
Korean Journal of Radiology	10	2
Nuclear Medicine Communication	7	7
Pediatric Radiology	5	3
polish journal of radiology	7	6
Practical radiation oncology	7	1
radiologica medica	6	3
Radiology	11	8
Skeletal Radiology	12	2
Ultrasound in Obstetrics and Gynecology	6	6
Total	275	151

Adherence of Articles to STARD Statement

The adherence of 151 radiology diagnostic accuracy studies to STARD 2015 are shown in Table 10.

Table 10

The number and percentage of articles that adhere to different parts of STARD standard

section	sub group	Items	Number	%	Mean	SD
STARD in text		the word "STARD" used in the study	1	0.66	-----	-----
TITLE OR ABSTRACT	title or abstracts	1a. Title	149	98.68	97.88	2.06
		1b. Abstract	151	100
		2. Structured Abstract	150	99.34
		3. background	145	96.03
		4. study objective	144	95.36
Method	study design	5. data collection	129	85.43	-----	-----
	participants	6. including / excluding criteria	144	95.36	96.68	2.42
		7. how eligible	150	99.34
		8. location and dates	142	94.04
		9. patient sampling	148	98.01
	Test methods	10a. Index test	151	100	77.77	13.70
		10b. reference or gold standard	133	88.08
		11. Reference standard rationality	127	84.11
		12a. test positivity cut-offs of the index test	116	76.82
		12b. test positivity cut-offs the reference standard	96	63.58
		13a. Blind to the index test	99	65.56
		13b. Blind to the reference standard	100	66.23
	Analysis	14. Methods for estimating diagnostic accuracy	138	91.39	55.74	38.36
		15. How indeterminate index test or reference standard results were handled	119	78.81
		16. How missing data were handled	56	37.09
		17. Any analyses for distinguishing pre-specified from exploratory	28	18.54
		18a. pre-specified sample size	16	10.6
		18b. How the sample size was chosen	148	98.01
Results	Participants	19. Flow of participants	88	58.28	55.36	28.99
		20. Demographics of participants	146	96.69
		21a. Distribution of severity of disease in those with the target condition	93	61.59
		21b. Distribution of alternative diagnoses in those without the target condition	26	17.22
		22. Time interval between index test and reference standard	65	43.05
	Test results	23. Cross tabulation of the index test results	122	80.79	66.23	38.74
		24a. Estimates of diagnostic accuracy	145	96.03
		24b. Estimates of precision the diagnostic accuracy	119	78.81
		25. Any adverse events	14	9.27
DISCUSSION		26. Study limitations	130	86.09	93.05	9.84
		27. Implications for practice	151	100
OTHER INFORMATION		28. Registration number and name of registry	1	0.66	35.10	36.49
		29. Where the full study protocol can be accessed	10	6.62
		30. Sources of funding and other support; role of funders	96	63.58
		ethics	105	69.54

151 articles (out of 275) from 31 journals (out of 84) were evaluated to check the adherence of their structure to STARD standard.

According to the results radiology articles adhered to 69.45% of the STARD items. Based on these results, the articles had the most adherence to the STARD standard in title or abstract (97.88%), material and method part, the item of participants (96.68%) and discussion section (93.05%). This means that most diagnostic accuracy articles describe the data collection methods, inclusion and exclusion criteria, location, method and date of data sampling. Most of these articles also described the study limitations (86.09%).

On the contrary, most articles (64.9%) did not adhere to “other information” which is a new item in STARD 2015. This item is intended to make the authors add some information such as registration number, a link to the location where the full study protocol can be accessed, sources of any support and ethics.

Among radiology diagnostic accuracy articles only one article (0.66%) had a registration number and 10 (6.62%) articles had links to the full study protocol. One of the reasons of this low adherence is that these items were not applicable to radiology diagnostic accuracy articles. More than 60% of articles adhered to the ethics (69.54%) and source of support (63.58%) since these items were applicable to these articles.

Other items such as test method (77.77%), test results (66.23) and analysis (55.74) were adhered by more than 50% of the articles.

Reporting adverse events (9.27%), pre-specified sample size (10.59%), reporting the distribution of alternative diagnoses in those without target condition (17.21%) were other items with low adherence to STARD checklist. However, abstracts and definitions of index tests were the items that completely adhered to the STARD checklist. All of the evaluated articles were completely applicable in clinical practice.

14.57% did not determine the study design in the article; however, 85.43% of studies specified the study design as either retrospective or prospective. Among the latter, 54.30% were retrospective and 31.13% were prospective studies, respectively.

If an article adhered to each STARD item, the score of 1 and if not the score of 0 were assigned to it. The total scores were normalized to 100. The mean score of all journals of each country and each continent were calculated and shown in Figure 2.

Figure2

Up) the score of journals (/100) from each country. Down) the mean score of journals in each continent.

Discussion

Diagnostic accuracy studies play a major role in promoting medical science and evolution of diagnostic methods. STARD statement was developed to help the standardization of these reports. Since then, it was always the question that how much these statements improved the reporting of these articles and how much the diagnostic accuracy studies adhered to this reporting format.

This study was intended to evaluate the adherence of diagnostic accuracy studies published in the radiology journals indexed in the Web of Science, PubMed and Scopus with STARD standard guide 2015.

As it was noted before, radiology articles were adhered to 69.45% of the STARD items. Considering that STARD statement was used since 2003 till now (12 years), it is expected to observe this adherence.

Previous studies, from 2005 to 2017, also pointed that the STARD statement was not followed properly in reporting the diagnostic accuracy studies [7,10,12-14,20].

In 2005, N. Smidt et al. assessed dozens of English journals with high impact factors (4 or more). This study revealed that even in journals with high impact factor, the quality of diagnostic accuracy articles was less than normal [4]. According to Figure 2, there was not any significant difference between the score of American, European, African, Australian or Asian journals. Based on the results, the mean scores of the journals with H indexes more than 100 (67.34/100) and the journals with H indexes less than 100 (68.08/100) were not different. Even the scores of lower H indexes obtained a non-significant more scores. This implies that high h- index journals or the journals located in advanced countries have the same quality as low H-index or the journals in developed countries.

In a study in 2006 by Roposch et al., the adherence of the diagnostic accuracy studies regarding developmental dysplasia of the hip was evaluated. They concluded that most of the articles reported just 40% of the STARD statement [8]. In the same year, Smidt et al. evaluated the published article in 12 medical medicine journals in the years 2000 and 2004. The results of their study revealed that the studies published in 2000 reported less than 50% of the STARD items, but the studies published in 2004 reported a little more than 50% of STARD items [6]. Again in the same year in another study by Coppus et al. which was performed on the journals of fertility and sterility, the quality of diagnostic accuracy studies published in 1999 (before the application of STARD guideline) and the papers published in 2004 (after the application of STARD guideline) were compared. The results revealed that less than half of the diagnostic accuracy articles published in both years adhered to 50% or more items of the STARD statement [7]. Table 10 of our study shows that on average the articles adhered to 69.45% of the STARD statement, which shows an almost 40% improvement in reporting the diagnostic accuracy studies at 2015 in comparison to 2006.

In 2007, Smidt et al. published a letter to editor in order to report that the results of their study indicated only a limited number of journals which published diagnostic accuracy studies to introduce STARD statement in the instructions for authors section, and even those that do refer to STARD use varying language in their instructions. This inconsistency can be a reason for the inadherence of these studies with the standard [11]. As Smidt et al. declared in 2007, the results of this study also demonstrated that the STARD statement was not introduced to the authors in most journals. Among 31 journals of this study, only 8 journals introduced STARD statement to the authors which show that most of the authors still do not know STARD statement.

Conclusion

The adherence of radiology diagnostic accuracy studies in radiology journals (the English language articles) to STARD was more almost 70%. So that the reporting of these studies improved. In order to even obtain more improvements, STARD statement should be introduced in the instruction section for authors of the journals that publish diagnostic accuracy studies. The authors, reviewers and editors should be trained to implement STARD to report accurately.

The well-known items of STARD obtained more scores, but the new items which were a little unknown to the authors obtained less scores so that it is suggested to introduce these items to the authors and referees of the journals to pay more attention to them.