Methodological quality and synthesis of case series and case reports.

Case reports and case series are uncontrolled study designs known for increased risk of bias but have profoundly influenced the medical literature and continue to advance our knowledge. In this guide, we present a framework for appraisal, synthesis and application of evidence derived from case reports and case series. We propose a tool to evaluate the methodological quality of case reports and case series based on the domains of selection, ascertainment, causality and reporting and provide signalling questions to aid evidence-based practitioners and systematic reviewers in their assessment. We suggest using evidence derived from case reports and case series to inform decision-making when no other higher level of evidence is available. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

In 1904, Dr James Herrick evaluated a 20-year-old patient from Grenada who was studying in Chicago and suffered from anaemia and a multisystem illness. The patient was found to have ‘freakish’ elongated red cells that resembled a crescent or a sickle. Dr Herrick concluded that the red cells were not artefacts because the appearance of the cells was maintained regardless of how the smear slide was prepared. He followed the patient who had subsequently received care from other physicians until 1907 and questioned whether this was syphilis or a parasite from the tropics. Then in 1910, in a published case report, he concluded that this presentation strongly suggested a previously unrecognised change in the composition of the corpuscle itself.1 Sickle cell disease became a diagnosis thereafter.

Case reports and case series have profoundly influenced the medical literature and continue to advance our knowledge in the present time. In 1985, the American Medical Association reprinted 51 papers from its journal that had significantly changed the science and practice of medicine over the past 150 years, and five of these papers were case reports.2 However, concerns about weak inferences and the high likelihood of bias associated with such reports have resulted in minimal attention being devoted to developing frameworks for approaching, appraising, synthesising and applying evidence derived from case reports/series. Nevertheless, such observations remain the bread and butter of learning by pattern recognition and integral to advancing medical knowledge.

Guidance on how to write a case report is available (ie, a reporting guideline). The Case Report (CARE) guidelines3 were developed following a three-phase consensus process and provide a 13-item checklist that can assist researchers in publishing complete and meaningful exposition of medical information. This checklist encourages the explicit presentation of patient information, clinical findings, timeline, diagnostic assessment, therapeutic interventions, follow-up and outcomes.3 Yet, systematic reviewers appraising the evidence for decision-makers require tools to assess the methodological quality (risk of bias assessment) of this evidence.

In this guide, we present a framework to evaluate the methodological quality of case reports/series and synthesise their results, which is particularly important when conducting a systematic review of a body of evidence that consists primarily of uncontrolled clinical observations.

Definitions

In the biomedical published literature, a case report is the description of the clinical course of one individual, which may include particular exposures, symptoms, signs, interventions or outcomes. A case report is the smallest publishable unit in the literature, whereas case series report aggregates individual cases in one publication.4

The median number of patients in articles with ‘case series’ in the title was found to be seven (range 1–6432). The median (range) of the number of cases of articles with ‘case report’ as a publication type was four (1–178).5 Case reports/case series are usually retrospective although can occasionally be prospective, such as the Herrick’s first case report of sickle cell disease.1 Case reports/series can also define their subject by exposure or outcome (analogous to a cohort study and case–control study). Therefore, a specific number of patients, the temporal direction of follow-up or even the definition by case/exposure are not differentiating characteristics of case report/series. One unique feature, however, is that case report/series are uncontrolled (non-comparative) and have a relatively small number of individuals.

If a case series is prospective, differentiating it from a single-arm uncontrolled cohort study becomes difficult. In one clinical practice guideline, it was proposed that studies without internal comparisons can be labelled as case series unless they explicitly report having a protocol before commencement of data collection, a definition of inclusion and exclusion criteria, a standardised follow-up and clear reporting of the number of excluded patients and those lost to follow-up.6

Evaluating methodological quality

Pierson7 provided an approach to evaluate the validity of a case report based on five components: documentation, uniqueness, objectivity, interpretation and educational value, resulting in a score with a maximum of 10 (a score above 5 was suggested indicate a valid case report). This approach, however, was rarely used in subsequent work and seems to conflate methodological quality with other constructs. For case reports of adverse drug reactions, other systems classify an association as definite, probable, possible or doubtful based on leading questions.8 9 These questions are derived from the causality criteria that was established in 1965 by the English epidemiologist Bradford Hills.10 Lastly, we have adapted the Newcastle Ottawa scale11 for cohort and case–control studies by removing items that relate to comparability and adjustment (which are not relevant to non-comparative studies) and retained items that focused on selection, representativeness of cases and ascertainment of outcomes and exposure. This tool was applied in several published systematic reviews with good inter-rater agreement.12–16

Proposed tool

The previous criteria from Pierson,7 Bradford Hills10 and Newcastle Ottawa scale modifications11 converge into eight items that can be categorised into four domains: selection, ascertainment, causality and reporting. The eight items with leading explanatory questions are summarised in table 1.

Table 1

Tool for evaluating the methodological quality of case reports and case series

Domains	Leading explanatory questions
Selection	1. Does the patient(s) represent(s) the whole experience of the investigator (centre) or is the selection method unclear to the extent that other patients with similar presentation may not have been reported?
Ascertainment	2. Was the exposure adequately ascertained? 3. Was the outcome adequately ascertained?
Causality	4. Were other alternative causes that may explain the observation ruled out? 5. Was there a challenge/rechallenge phenomenon? 6. Was there a dose–response effect? 7. Was follow-up long enough for outcomes to occur?
Reporting	8. Is the case(s) described with sufficient details to allow other investigators to replicate the research or to allow practitioners make inferences related to their own practice?

Questions 4, 5 and 6 are mostly relevant to cases of adverse drug events.

For example, a study that explicitly describes all the cases who have presented to a medical centre over a certain period of time would satisfy the selection domain. In contrast, a study that reports on several individuals with unclear selection approach leaves the reader with uncertainty to whether this is the whole experience of the researchers and suggests possible selection bias. For the domain of ascertainment, self-report (of the exposure or the outcome) is less reliable than ascertainment using administrative and billing codes, which in turn is less reliable than clinical records. For the domain of causality, we would have stronger inference in a case report of an adverse drug reaction that has resolved with cessation of the drug and reoccurred after reintroduction of the drug. Lastly, for the domain of reporting, a case report that is described with sufficient details may allow readers to apply the evidence derived from the report in their practice. On the other hand, an inadequately reported case will likely be unhelpful in the course of clinical care.

We suggest using this tool in systematic reviews of case reports/series. One option to summarise the results of this tool is to sum the scores of the eight binary responses into an aggregate score. A better option is not to use an aggregate score because numeric representation of methodological quality may not be appropriate when one or two questions are deemed most critical to the validity of a report (compared with other questions). Therefore, we suggest making an overall judgement about methodological quality based on the questions deemed most critical in the specific clinical scenario.

Synthesis of case reports/series

A single patient case report does not allow the estimation of an effect size and would only provide descriptive or narrative results. Case series of more than one patient may allow narrative or quantitative synthesis.

Narrative synthesis

A systematic review of the cases with the rare syndrome of lipodystrophy was able to suggest core and supportive clinical features and narratively summarised data on available treatment approaches.17 Another systematic review of 172 cases of the infrequently encountered glycogenic hepatopathy was able to characterise for the first time patterns of liver enzymes and hepatic injury in this disease.18

Quantitative synthesis

Quantitative analysis of non-comparative series does not produce relative association measures such as ORs or relative risks but can provide estimates of prevalence or event rates in the form of a proportion (with associated precision). Proportions can be pooled using fixed or random effects models by means of the various available meta-analysis software. For example, a meta-analysis of case series of patients presenting with aortic transection showed that mortality was significantly lower in patients who underwent endovascular repair, followed by open repair and non-operative management (9%, 19% and 46%, respectively, P<0.01).19

A common challenge, however, occurs when proportions are too large or too small (close to 0 or to 1). In this situation, the variance of the proportion becomes very small leading to an inappropriately large weight in meta-analysis. One way to overcome this challenge is to transform prevalence to a variable that is not constrained to the 0–1 range and has approximately normal distribution, conduct the meta-analysis and then transform the estimate back to a proportion.20 This is done using logit transformation or using the Freeman-Tukey double arcsine transformation,21 with the latter being often preferred.20

Another type of quantitative analysis that may be utilised is regression. A meta-analysis of 47 published cases of hypocalcaemia and cardiac dysfunction used univariate linear regression analysis to demonstrate that both QT interval and left ventricular ejection fraction were significantly correlated with corrected total serum calcium level.22 Meta-regression, which is a regression in which the unit of analysis is a study, not a patient, can also be used to synthesise case series and control for study-level confounders. A meta-regression analysis of uncontrolled series of patients with uveal melanoma treated with proton beam therapy has shown that this treatment was associated with better outcomes than brachytherapy.23 It is very important, however, to recognise that meta-regression results can be severely affected by ecological bias.

From evidence to decision

Several authors have described various important reasons to publish case reports/series (table 2).7 24 25

Table 2

Role of case reports/series in the medical literature

Roles	Examples
Describe a new phenotype or genotype of disease	The first case report of sickle cell disease.1
Recognise a known or common manifestation of a rare disease	Liver cirrhosis as a result of Sitosterolaemia.33
Recognise a rare manifestation of a known or common disease	Secretory diarrhoea and hypokalaemia in colonic pseudo-obstruction.15
Describe a new pathogen (microbe, virus or environmental exposure)	Discovery of AIDS was an observation of a patient with immunodeficiency-related diseases who otherwise had no reason to be immunodeficient.34
Describe unknown adverse effect of an existing drug	Reye syndrome and aspirin in children.35 Thalidomide and malformation of the limbs in pregnant women.36
Describe a novel treatment for a known condition	Colchicine for the treatment of familial Mediterranean fever.37
Elucidate mechanisms of disease	Functional imaging of the brain during auditory hallucinations.38
To remind or educate	Case presentations in clinicopathological conferences for postgraduate education.
Quality improvement	‘Lesson of the week’ published in the British Medical Journal (do not make the same mistake as I did).

It is paramount to recognise that a systematic review and meta-analysis of case reports/series should not be placed at the top of the hierarchy in a pyramid that depicts validity.26 The certainty of evidence derived from a meta-analysis is contingent on the design of included studies, their risk of bias, as well as other factors such as imprecision, indirectness, inconsistency and likelihood of publication bias.27 Commonly, certainty in evidence derived from case series/reports will be very low. Nevertheless, inferences from such reports can be used for decision-making. In the example of case series of aortic transection showing lower mortality with endovascular repair, a guideline recommendation was made stating ‘We suggest that endovascular repair be performed preferentially over open surgical repair or non-operative management’. This was graded as a weak recommendation based on low certainty evidence.28 The strength of this recommendation acknowledged that the recommendation might not universally apply to everyone and that variability in decision-making was expected. The certainty in evidence rating of this recommendation implied that future research would likely yield different results that may change the recommendation.28

The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach clearly separates the certainty of evidence from the strength of recommendation. This separation allows decision-making based on lower levels of evidence. For example, despite low certainty evidence (derived from case series) regarding the association between aspirin and Reye’s syndrome in febrile children, a strong recommendation for using acetaminophen over aspirin is possible.29 GRADE literature also describes five paradigmatic situations in which a strong recommendation can be made based on low quality evidence.30 One of which is when the condition is life threatening. An example of which would be using hyperbaric oxygen therapy for purpura fulminans, which is only based on case reports.31

Discussion

Guideline developers and decision-makers often struggle when dealing with case reports/case series. On occasions, they ignore such evidence and focus the scope of guidelines on areas with higher quality evidence. Sometimes they label recommendations based on case reports as expert opinion.32 We propose an approach to evaluate the methodological quality of case reports/series based on the domains of selection, ascertainment, causality and reporting and provide signalling questions to aid evidence-based practitioners and systematic reviewers in their assessment. We suggest the incorporation of case reports/series in decision-making based on the GRADE approach when no other higher level of evidence is available.

In this guide, we have made the case for publishing case reports/series and proposed synthesis of their results in systematic reviews to facilitate using this evidence in decision-making. We have proposed a tool that can be used to evaluate the methodological quality in systematic reviews that examine case reports and case series.