Clinical reports of pulmonary metastasectomy for colorectal cancer: a citation network analysis.

INTRODUCTION: Pulmonary metastasectomy for colorectal cancer is a commonly performed and well-established practice of ∼50 years standing. However, there have been no controlled studies, randomised or otherwise. We sought to investigate the evidence base that has been used in establishing its status as a standard of care.
METHODS: Among 51 papers used in a recent systematic review and quantitative synthesis, a citation network analysis was performed. A total of 344 publications (the 51 index papers and a further 293 cited in them) constitute the citation network.
RESULTS: The pattern of citation is that of a citation cascade. Specific analyses show the frequent use of historical or landmark papers, which add authority. Papers expressing an opposing viewpoint are rarely cited.
CONCLUSIONS: The citation network for this common and well-established practice provides an example of selective citation. This pattern of citation tends to escalate belief in a clinical practice even when it lacks a high-quality evidence base and may create an impression of more authority than is warranted.

There is a professional consensus regarding the effectiveness of pulmonary metastasectomy for colorectal cancer (PMCRC), and along with metastasectomy for other primary sites, this surgery has become a routine (Internullo ). In the course of a systematic review of pulmonary metastasectomy for this disease, 51 publications (listed in Appendix A) were identified, which contained data on case selection and outcomes (Fiorentino ). These 51 surgical follow-up studies were published from 1971 to 2006; they include a total of 3504 patients who were operated on over a 40-year period starting in the mid 1960s. Five-year survival is reported at a rate of approximately 30–50% and is attributed to the beneficial effects of metastasectomy. The more recent papers in the review cite earlier papers, and in addition, 293 other clinical reports are cited (listed in Appendix B).

Citation network analysis is recommended as a standard tool in evidence-based medicine (EBM) (Haynes ). The reason for this is well illustrated by Greenberg (2009) who in the British Medical Journal showed how authors seeking a basis for their own practice may create a citation cascade ‘resulting in unfounded authority of claims’. New Scientist, concerned with bias in the citation of stem cell research, used citation network analysis to show that US scientists rarely cited non-US authors (Aldhous, 2010). In the course of sifting, reading, and extracting data for analysis, we became aware of the absence of controlled trials regarding metastasectomy, as have others (Pfannschmidt , 2010). These published clinical reports lack data on how such patients would have fared had they not undergone pulmonary metastasectomy. In the course of planning a randomised controlled trial to address the matter (Treasure , 2010) and in line with EBM recommendations, we undertook a citation network analysis of papers included in our review.

Methods

We examined the reference lists of the 51 papers incorporated into a systematic review and quantitative synthesis (SRQS) (Fiorentino ). We first excluded standard references to statistical methods and to books and book chapters. All the remaining citations were individually characterised by two of the authors (TT and FF). We defined 4 categories of particular interest: clinical follow-up studies of PMCRC, reports regarding hepatic metastasectomy, historical references, papers questioning the practice, and 10 other categories, including epidemiology, adjuvant therapies, pathological studies, and surgical techniques.

We then constructed a citation network representing all papers as nodes and citations as links from one node to another (Greenberg, 2009). A custom-designed relational database (MS Access, Microsoft, Redmond, WA, USA) was used to store and query the citation network data. We used Pajek, a freely available special-purpose social network analysis software tool (available from http://pajek.imfm.si/doku.php?id=pajek/), to perform visualisations of the citation networks (de Nooy ).

Results

The 51 follow-up reports included in the SRQS were published as follows: 1 in 1971, 4 in the 1980s, 19 in the 1990s, and 27 between 2000 and 2006. These papers each cited between 10 and 36 references to a total of 1132. After planned exclusions, there were 972 citations to 293 other unique publications. The frequency distribution of citations was markedly skewed with 203 of 344 cited only once. The 5 most referenced papers were cited 31, 30, 27, and 2 × 22 times (McCormack and Martini, 1979; Mansel ; Goya ; McAfee ; Okumura ).

The network of the citations of 51 SRQS papers to 344 references in the network is illustrated in Figure 1 following the method of Greenberg. A cascade such as the one he described can be discerned.

Figure 1

All papers included in the citation network by decade of publication. The 51 index papers are in grey and are listed in Appendix A. The 293 cited papers are listed in Appendix B.

Of the 51 papers in the systematic review, 32 were classified for citation analysis as exclusively concerning and supporting PMCRC (both citing and cited), 19 were otherwise classified (e.g., mixed series of hepatic and pulmonary metastasectomy), and 21 further PMCRC papers were cited by SRQS papers (Figure 2). The citations among these 72 papers are shown in Figure 3 and in addition, the 4 papers questioning practice. Of these, for example, Aberg and Aberg (1997) argued that apparent longer-than-expected survival is a matter of case selection. Two other authors expressed caution attributing survival to surgery (Casciato ; Todd, 1997). These are rarely cited compared with the dense network of citations between the index papers and other supportive PMCRC publications.

Figure 2

Venn diagram of 51 ‘index papers’ from the systematic review and quantitative synthesis (Fiorentino and 53 papers categorised in the citation network as reports of clinical series of pulmonary metastasectomy for colorectal cancer (PMCRC) and their overlap. The total is 72 publications.

Figure 3

This network includes all the 72 papers in Figure 2 showing their numerous citations to each other. On the periphery are the four questioning papers. The numbers refer to papers listed papers in Appendix A and Appendix B.

Discussion

Our search of the literature for evidence regarding PMCRC was prompted in part by a NICE Cancer Services Manual update stating, ‘Surgery for metastases confined to the liver or lung can be curative…for a minority of patients, it can increase five-year survival rates from close to zero to over 30%’ (National Institute for Health and Clinical Excellence (NICE), 2004). We speculated as to the evidence for this statement, but the single reference cited contained no information regarding pulmonary metastasectomy (Stangl ). This is a clear instance of not only selective but also inappropriate citation.

The manner by which ‘history’ is cited is of interest (Pastorino and Treasure, 2010). In our citation analysis, we categorised four papers as historical. As an example, the report by Blalock (1944) in the 1944 New England Journal of Medicine is cited by 14 of 51 of the index papers and is the thirteenth most frequent of 334 cited papers. Typical citations to it are: ‘The role of pulmonary metastasectomy for colorectal carcinoma was first introduced in 1944.’ (Patel ) and ‘Since Blalock reported the first pulmonary resection for colorectal metastases in 1944, lung metastases…have been considered to be cured by resection in selected cases.’(Ike ). In summarising a guest lecture to The Massachusetts Medical Society, Blalock wrote ‘It is only eleven years since the first one stage removal of an entire lung...’. He was informing his colleagues that pneumonectomy is achievable; the fact that it was for a metastasis was coincidental. He tells us nothing about control of colorectal cancer or the eventual outcome beyond recovery from the operation. Furthermore, in current practice, pneumonectomy is considered an inappropriate operation for metastasectomy (Migliore ).

Blalock's account of a pneumonectomy has no relevance to the current practice of pulmonary metastasectomy, and is not even claimed to be a first; hence, why is this such a popular citation? A possible answer is that among surgeons, his name adds authority: he went on to become extremely famous for the Blalock-Taussig shunt to palliate cyanotic heart disease (Blalock and Taussig, 1945). His paper, which would not have been found by a formal search for metastasectomy but is passed on as folklore, gains a significance that Blalock did not intend.

In marked contrast, Aberg's publications are barely mentioned. His paper in 1980 and editorial in 1997 which challenge the effect of pulmonary metastasectomy are cited by only two of the index papers, and yet, unlike Blalock's paper, Aberg's publications have metastasectomy in their title and could not be missed in a literature search. The failure to cite suggests bias against his paper rather than that it was not retrievable. In addition, US citations may have a higher perceived status (Link, 1998). Blalock was from a prestigious American institution, Johns Hopkins, whereas Aberg worked in a less well-known Swedish hospital.

To quote from a remarkable essay on the whole question of citation (MacRoberts and MacRoberts, 1996), ‘The cumulative effect of citing more and more people who similarly agree with the author is to concretize the universality of the knowledge claim.’ Greenberg describes information cascades: authors write their clinical experience, citing similar practices and thus gain affirmation. However, it is like rolling a snowball: it gets bigger and bigger – but it is just more snow.