People who live in ivory towers shouldn't throw stones: A refutation of Curley et al.

Curley et al. [1] recently published a paper entitled “An inconvenient truth: More rigorous and ecologically valid research is needed to properly understand cognitive bias in forensic decisions.” I take serious issue with this paper. Below, I argue that the paper repeatedly misrepresents the extant research on cognitive bias, echoes a common but problematic misunderstanding of bias, and understates the substantial progress that has resulted from ongoing collaborative efforts to combat bias.

Rigor and ecological validity

Cooper and Meterko [2] systematically reviewed 29 studies of cognitive bias effects on judgments of forensic science evidence. Curley et al. criticized these studies on four methodological grounds—namely, the potential for participant reactivity, unknown generalizability, the potential for incomparable groups, and small sample size. For the reasons explained below, I believe that each of their criticisms is dramatically overstated—and in some cases outright misleading.

Reactivity

Naturally, research participants should be blind to a study’s purpose, else they may behave unnaturally and thus render the data uninterpretable [3]. Curley et al. state that the studies reviewed by Cooper and Meterko “consistently failed to blind participants to the purposes of their research.” That statement is simply untrue. In Cooper and Meterko’s own words, only “one of these [studies] was critically limited by the failure of blinding procedures” (p. 37). (Notably, the authors of this one study also explicitly recognized this as a limitation of their study; see Ref. [4]; p. 580.) In other words, 28 of the 29 studies did not fail to use appropriate blinding procedures, and to call this a “consistent failure” is extremely misleading.

Furthermore, Curley et al. fail to mention that several of these studies went to great lengths to incorporate their measurements into examiners’ routine casework, such that examiners were not even aware that they were part of a study (e.g., Ref. [[5], [6], [7]]). This is no small feat, given that it requires the cooperation of laboratory managers who are both willing to assist with data collection and sympathetic to the problem of cognitive bias—which many are not [8]. Though they are logistically challenging to conduct, extant studies of this nature outnumber the one study in which reactivity was a critical flaw.

Generalizability

Curley et al. also lament the “failure to report demographic materials, [which] makes it difficult for researchers to generalize the results,” and they cite Kukucka and Kassin [9] as an example of a study that “[did] not report the age, gender, and/or any other demographic information about their participants, except that they were undergraduate students in psychology.”

These claims are patently false. First, the participants in Kukucka and Kassin’s two experiments were not undergraduate students; they were adults recruited via social media (Study 1) or Amazon Mechanical Turk (Study 2). Second, the paper did report demographic information for both samples, including age, gender, race, and education level (see p. 260 and p. 263). It is disappointing that Curley et al. did not read this paper carefully before levying an unfounded criticism in a public forum. (It is also unfortunate that the reviewers did not pick up on these, and other, false claims.)

It is certainly preferable that researchers thoroughly describe their sample—and when the participants are practitioners, certain information is especially critical to report (e.g., years of experience). However, this may be difficult or impossible in studies where data were collected covertly (see above). Regardless, I hardly view the failure to report demographic information as a fatal flaw, as there is simply no evidence that cognitive bias is moderated by a person’s age, sex, or race. Rather, the literature suggests that cognitive bias is an innate feature of human psychology that does not discriminate on the basis of these characteristics [10].

Random assignment

As every scientist knows, participants must be randomly assigned to conditions in order to establish causality; in theory, this practice creates two groups that are, on average, identical apart from the experimental manipulation. Curley et al. argue that “many of the reviewed studies did not address differences between experimental and control groups” and these studies may have therefore compared groups that were inherently incomparable. In so doing, they noticeably overstate Cooper and Meterko’s conclusion that only one of the 29 studies showed “critical deficiencies due to the lack of comparability between groups” (p. 37)—and importantly, this one study was an archival study [11], where random assignment is not even possible.

Moreover, Curley et al. neglect to mention that many of these studies utilized a repeated-measures design (e.g. Refs. [5,6,12,13]), where there were not separate experimental and control conditions, but rather the same individuals provided multiple judgments, such that the control and experimental conditions consisted of the same individuals and were therefore comparable by definition.

Sample size

Of the 29 studies reviewed by Cooper and Meterko, 21 studies used practitioners as participants. Curley et al. bemoan the generally small sample size of these studies, noting that the average study featured 36.85 practitioners, and 13 of the 21 studies featured fewer than 25 practitioners. (As an aside, the latter statement—while true—is noticeably cherry-picked, given that three other studies had sample sizes of 23, 24, and 24.) It is true that the earliest demonstrations of cognitive bias in fingerprint examiners relied on small samples (e.g., Ref. [6]). However, these were totally innovative and groundbreaking studies at the time—a necessary, even if imperfect, first effort that has stimulated an ever-growing wave of research on this topic.

Indeed, a closer look at these practitioner studies reveals that their sample sizes have steadily increased over time. As shown in Fig. 1, publication date is a significant positive predictor of sample size for the 19 practitioner studies published since 2006, B = 6.97, t = 2.54, p = .021. Thus, while the average sample size of practitioner studies remains modest, researchers have made considerable progress in this regard—presumably in line with increased attention to this issue and practitioners’ increased willingness to support and/or conduct such studies.

Fig. 1

Publication date (2006-present) predicts sample size of forensic practitioner studies (N = 19).

Conceptualizing bias in terms of accuracy misses the point

Methodological oversights notwithstanding, Curley et al.’s paper also reveals a fundamental—but not uncommon—misunderstanding of why cognitive bias is detrimental. The authors assert that researchers have “assumed that cognitive biases in forensic decision making reduce the accuracy of the decisions,” and they posit that “cognitive biases may actually improve accuracy.” The former half of this argument is not true; many researchers have acknowledged this possibility, explaining, for example, that “contextual influences can unwittingly lead forensic examiners to the right decision” [14]; p. 114; see also [[15], [16], [17]].

The latter half of this argument is a talking point that forensic examiners have used to argue against context management procedures (e.g. Ref. [18,19]),—and it is deeply and fundamentally flawed. The point is not whether task-irrelevant information increases or decreases examiners’ accuracy; the point is that knowledge of task-irrelevant information destroys the independent probative value of forensic examiners’ judgments, even if their judgment is accurate. That is to say, if a forensic examiner uses task-irrelevant information to form a conclusion (irrespective of accuracy), their conclusion necessarily hinges on their evaluation of extraneous information that goes beyond their area of expertise, which in turn misleads the court about the rationale for their conclusion.

To illustrate, Curley et al. describe a study by Stevenage and Bennett (2019) in which fingerprint examiners were informed of DNA test results prior to comparing two fingerprints. They found that examiners were less accurate when the DNA evidence implied the opposite of the ground truth of the fingerprints (i.e., that they did or did not match), but the DNA evidence improved accuracy when it matched the ground truth of the fingerprints (e.g., when the DNA implicated the suspect, accuracy decreased if the fingerprints did not match, but increased if they did match). Curley et al. seemingly interpret this as a benefit of cognitive bias, but in actuality, any effect of task-irrelevant information on an examiner’s judgment—whether “positive or negative”—is problematic. For instance, a person with zero training or experience in latent fingerprint identification could likely achieve a respectable accuracy rate if they always knew of DNA results before comparing two fingerprints and simply followed whichever conclusion the DNA suggested—but it would be absurd to call this person a fingerprint expert or to ascribe any independent value to their opinion.

In other words, cognitive bias can lead examiners to erroneous judgments, but it can also lead them to accurate judgments that create the illusion of corroboration (see, e.g. Ref. [14,20,21]), and the latter are likewise problematic. In turn, jurors—who are heavily persuaded by myriad forms of forensic science evidence (e.g. Refs. [22,23]),—are likely to assume that the examiner’s judgment is independent when it is in fact not, and therefore give bias-tainted forensic evidence more weight than it deserves.

Collaboration is Key—but it’s hardly new

Curley et al. conclude their paper with a call for collaboration between decision scientists and forensic scientists, adding that “decision scientists … have developed methodological tools to conduct research that can provide valid answers to questions regarding cognitive bias.” Of course that is true, but historically the problem has not been that psychologists are unwilling or unqualified to conduct such research; it has been the logistical difficulty of conducting such research [15] combined with forensic examiners’ resistance to recognizing bias as a cause for concern [8]—and the misleading criticisms levied by Curley et al. may ironically fuel that resistance.

Thankfully, the past decade has seen tremendous progress in terms of acknowledging and combating the scourge of bias. Several government reports have drawn attention to the issue (e.g. Ref. [[24], [25], [26]]), and numerous laboratories have described their efforts to revise their protocols so as to minimize the risk of bias (e.g., Ref. [[27], [28], [29], [30]]). In 2014, the U.S. National Institute of Standards and Technology also established the OSAC for Forensic Science—a collaboration between forensic examiners, psychologists, lawyers, and statisticians—to develop scientifically-informed best practices for a variety of forensic disciplines. Psychologists have served in this organization and worked to facilitate productive dialogue and build bridges between these stakeholders. Finally, and perhaps most importantly, ISO 17020 and 17025—the international accreditation standards for forensic laboratories—now include explicit requirements of impartiality and freedom from undue influence.

Conclusion

By its nature, science progresses slowly and gradually. Fifteen years ago, the issue of cognitive bias was virtually unknown within the forensic science community. Then, a smattering of groundbreaking, albeit small and imperfect, studies first opened our eyes to a phenomenon with far-reaching implications for the administration of justice [31]. Since that time, research on this topic has proliferated and improved, government agencies have drawn attention to the problem and dedicated resources to addressing it, and forensic laboratories have increasingly adopted context management procedures that increase confidence in the value of their conclusions.

Surely there remain many unanswered questions and much work left to be done, and Curley et al. are welcome to join the ongoing pursuit of these goals. However, they must understand the inherent complexity of doing research—especially field research—on this topic. Rather than deriding the existing literature and ignoring the progress that has already been made, the question is how to integrate various studies, each with its own contributions and limitations, to further our common goal of strengthening forensic decision-making.

Declaration of Competing Interests

The author has no competing interests to declare.