Robert Boyle on the importance of reporting and replicating experiments.

Boyle’s research programme: building theory using empirical research

Although Robert Boyle was a proponent of the theory that matter was made up of small indivisible particles called corpuscles, he was wary of those who tried to understand the world by adhering to theoretical frameworks – ‘systems’. Boyle delivers this message in several works. For instance, he writes:He doesn’t forbid such theories, but he cautions his readers:Boyle’s writings, although full of archaic language, repetition and digressions, give us detailed accounts so that we can understand his approach to experimentation. Many aspects of modern scientific work are reflections of these accounts. Boyle is very aware of contingent influences in a system, and he tries to control variables in experimental design and address them in discussion. He makes careful measurements but notes that obsessional precision is not always required. Reports of his experiments are accompanied by possible explanations for the findings, but with competing explanations and influences discussed.

[What] I wish for, as to systems, is this, that men, in the first place, would forbear to establish any theory, till they have consulted with … a considerable number of experiments, in proportion to the comprehensiveness of the theory to be erected on them.[1] (p. 194)

And, in the next place, I would have such kind of superstructures looked upon only as temporary ones; which though they may be preferred before any others, as being the least imperfect, or, if you please, the best in their kind that we yet have, yet are they not entirely to be acquiesced in, as absolutely perfect, or uncapable of improving alterations.[2] (p. xviii)

Boyle’s research approach was to set out heads of enquiry after ‘a general survey of the subject’ (p. xvxiv).[2] He refers to these as ‘primary titles’. Then, ‘by reading, conference, meditation, and the experiments suggested by the heads of enquiry of the first class’, he proceeds to form a set of second titles. Sometimes a third set will be needed. This process forms the beginning of a ‘natural history’ of the subject.

Boyle is far-seeing about the progressive and provisional nature of science:Boyle presents accounts of his experiments in painstaking detail. Indeed, he often apologises for this, explaining that he does it because he wants others to be able to replicate his experiments and to add new discoveries. Indeed, he sees scientific progress as a collective endeavour. He wants to encourage his readers to pursue experimental work because:While clearly keen to promote a ‘Common-wealth of Learning’, Boyle was also quite sensitive about his work being cited without credit, and he complains about this in many places. In The Sceptical Chymist, for example, he calls for the sources of other work to be made clear:

For, even after all this is done, the history will, in all probability, be only begun, not compleated; the nature of things, and the industry of skilful men being so fertile, that the history will, doubtless, be encreased, corrected, and improved from time to time; but never, I fear, in many ages, arrive at absolute perfection.

the Common-wealth of Learning would lose too many useful Observations and Experiments, and the History of Nature would make too slow a Progress, if it were presum’d, that none but Geometers and Mechanitians should imploy themselves about writing any part of that History.

… rather for each Experiment they alledge name the Author or Authors, upon whose credit they relate it; For, by this means they would secure themselves from the suspition of falshood (to which the other Practice Exposes them) and they would Leave the Reader to Judge of what is fit for him to Believe of what is Deliver’d, whilst they employ not their own great names to Countenance doubtfull Relations; and they will also do Justice to the Inventors or Publishers of true Experiments, as well as upon the Obtruders of false ones.[3] (p. A3)

Boyle’s disapproval of biased underreporting of research

Boyle chastises those who withhold the results of empirical investigations because they judge them to be inconsistent with then current theories – biased under-reporting of research.[4-6] In Certain Physiological Essays, written at distant times, and on several occasions, Boyle attacked authors who adopt what he refers to as a ‘Systematical’ way of writing – writers who try to persuade themselves and their readers that they have a complete understanding of the way the world is organised.[7] Boyle suggests this approach is prejudicial not only to their readers but also to their own reputation because it does not distinguish clearly those of their own ideas that might advance knowledge from those of others. Boyle goes on:Boyle believed that unsuccessful experiments should be reported. Indeed, he describes leaks, explosions and other apparatus failures with the same care as he describes successful experiments. In New Experiments Physico-Mechanical[8] addressed to his nephew Viscount Dungarvan he writes:

But the worst Inconvenience of all is yet to be mentioned, and that is, That whilst this vanity of thinking men oblig’d to write either Systems or Nothing, is in request, many excellent Notion or Experiments are by sober and modest men suppressed, because such Persons being forbidden by their Judgment and Integrity to teach more than they understand, or assert more than they can prove, are likewise forbidden by custom to publish their thoughts and observations, unless they be numerous enough to swell into a System. And indeed it may be doubted whether the Systematical Writers have not kept the world from much more useful Composures than they have presented it with. For there are very few men, if any at all, in the world, that are enriched with a competent stock of Experiments and Observations to make out clearly and solidly, I say not all the Phaenomena of Nature, but all those that belong to Chymistry, Anatomy, or any such considerable subordinate doctrine of Physiology. And those very men that are diligent and judicious enough to study prosperously any of those parts of Physiology, are obliged to spend so much time in the accurate prosecution of that, and are wont to be thereby made so wary, and so thorowly acquainted with the Difficulty of Physiological Investigations, that they will be least of all forward to write Systems. (pp. 4–5)

To these Experiments concerning Fire we added another, which though it succeeded not, may perhaps without impertinency be recorded: partly, because that (as we have in another Treatise amply declar’d) it is usefull to recite what Experiments miscarry as well as succeed. And partly also, because it is very possible that what we endeavoured in vain, may be performed by Your Lordship, or some other Vertuoso that shall have stancher vessels than we had, and more Sunny days than the present Winter allows us. (p. 49)

And this unsuccessfulness whereto our Experiment is liable, being such, that by all our watchfulness and trials, we could never reduce it to any certain Rules or Observations; since in all constitutions of the Weather, times of the Day, &c. It will sometimes answer, and sometimes disappoint our expectations; We are much discourag’d from venturing to frame an Hypothesis to give an account of it: which if the Experiment did constantly succeed, might the more hopefully be attempted; by the help of the following Phaenomena laid together: some of them produc’d upon trials purposely made to examine the validity of the conjectures, other trials had suggested. (p. 155)

Boyle’s observations on the importance of replicating experiments

Just as biased under-reporting of research remains a problem four centuries after Robert Boyle expressed his concerns about it, so also does replication of research. Boyle repeats experiments himself and encourages others to repeat them as well.Boyle, of course, did not have to grapple with issues such as the Journal Impact Factor or Open Access payments, but some of the topics he covered are remarkably contemporary. For instance, he anticipated discussion at a symposium organised by the Academy of Medical Sciences, the Biotechnology and Biological Sciences Research Council, the Medical Research Council and the Wellcome Trust on Reproducibility and Reliability of Biomedical Research. One conclusion was that it was vital to foster a ‘no blame’ culture when study findings are not replicated.[10]

I have divers times in cases, where the Experiments seem’d like to be thought strange, or to be distrusted, set down several Trials of the same thing, that they might mutually support and confirm one another.[9]

Boyle would also have been interested in the views of Jason Mitchell, John L. Loeb Associate Professor of the Social Sciences at Harvard University, who created a stir in 2016 by publishing a piece entitled ‘On the emptiness of failed replications’ here.[a] It was initially thought to be a parody, but it seems to have been a sincere attempt at defending the thesis that ‘unsuccessful experiments have no meaningful scientific value’. Furthermore, according to Mitchell, ‘Whether they mean to or not, authors and editors of failed replications are publicly impugning the scientific integrity of their colleagues’. One of us (DB) has taken issue with this standpoint:[11] we should not assume that a failure to replicate a result is due to fraud or malpractice, but rather should encourage replication attempts as a means of establishing which results are reproducible.

Boyle has ‘Two Essays on the Unsuccessfulness of Experiments’ in a collection of papers entitled ‘Certain Physiological Essays and other Tracts written at distant times, and on several occasions’.[12] These were published in two editions, Boyle noting in an ‘Advertisement’ under the title of the second edition: ‘The author of these discourses had enlarged them … with divers observations and experiments, but that he has made use of them already in other papers belonging to his Sceptical or doubting Naturalist’.

In the ‘Two Essays on the Unsuccessfulness of Experiments’ Boyle discusses (at inordinate length!) the problems that arise when an attempt to replicate an experiment is not successful. He starts by noting that such unsuccessful experiments are not uncommon:He is interested in exploring the reasons for such failure. His first explanation seems equivalent to one with which people using statistical analyses are all too familiar – a chance false-positive result:

… in the serious and effectual prosecution of Experimental Philosophy, I must add one discouragement more, which will perhaps as much surprize you as dishearten you; and it is, That besides that you will find … many of the Experiments publish’d by Authors, or related to you by the persons you converse with, false or unsuccessful, … you will meet with several Observations and Experiments, which though communicated for true by Candid Authors or undistrusted Eye-witnesses, or perhaps recommended to you by your own experience, may upon further tryal disappoint your expectation, either not at all succeeding constantly, or at least varying much from what you expected. (opening passage) (pp. 204–205)

And that if you should have the luck to make an Experiment once, without being able to perform the same thing again, you might be apt to look upon such disappointments as the effects of an unfriendliness in Nature or Fortune to your particular attempts, as proceed but from a secret contingency incident to some experiments, by whomsoever they be tryed. (p. 205)

And he urges readers not to be discouraged – replication failures happen to everyone!

… though some of your Experiments should not always prove constant, you have divers Partners in that infelicity, who have not been discouraged by it. (p. 205)

Boyle identifies various possible systematic reasons for such failure: impurities in ingredients, problems with the skill of the experimenter, or variation in the specific context in which the experiments were conducted. He even, implicitly, addresses statistical power by referring to the need for many observations to distinguish what is general from individual variation.Because of such uncertainties, Boyle emphasises the need for replication, and the dangers of building complex theory on the basis of a single experiment:We’re sure there are some modern scientists who must be thinking their lives may have been made much easier if they had heeded this advice. But perhaps the most relevant to the modern world, where there is such concern about the consequences of failure to replicate, are Boyle’s comments on the reputational impact of publishing irreproducible results:

… the great variety in the number, magnitude, position, figure, &c. of the parts taken notice of by Anatomical Writers in their dissections of that one Subject the humane body, about which many errors would have been delivered by Anatomists, if the frequency of dissections had not enabled them to discern betwixt those things that are generally and uniformly found in dissected bodies, and those which are but rarely, and (if I may so speak) through some wantonness or other deviation of Nature, to be met with. (p. 220)

… try those Experiments very carefully, and more than once, upon which you mean to build considerable Superstructures either theoretical or practical, and to think it unsafe to rely too much upon single Experiments, especially when you have to deal in Minerals: for many to their ruine have found, that what they at first look’d upon as a happy Mineral Experiment has prov’d in the issue the most unfortunate they ever made. (p. 224)

… if an Author that is wont to deliver things upon his own knowledge, and shews himself careful not to be deceived, and unwilling to deceive his Readers, shall deliver anything as having try’d or seen it, which yet agrees not with our tryals of it; I think it but a piece of Equity, becoming both a Christian and a Philosopher, to think (unless we have some manifest reason to the contrary) that he set down his Experiment or Observation as he made it, though for some latent reason it does not constantly hold; and that therefore though his Experiment be not to be rely’d upon, yet his sincerity is not to be rejected. Nay, if the Author be such an one as has intentionally and really deserved well of Mankind, for my part I can be so grateful to him, as not only to forbear to distrust his Veracity, as if he had not done or seen what he says he did or saw, but to forbear to reject his Experiments, till I have tryed whether or no by some change of Circumstances they may not be brought to succeed. (p. 224)

In conclusion

Joseph Agassi’s 2013 book The Very Idea of Modern Science: Francis Bacon and Robert Boyle discusses Boyle’s philosophy, his agreement with and dissent from Bacon and the way he single-handedly trained a crowd of poorly educated English aristocrats and rendered them into an army of able amateur researchers.[13]

Robert Boyle’s thinking as revealed in his writings in the 17th century leave no doubt that he represents a key milestone in the evolution of scientific thinking and the beginning of The Enlightenment. He cautioned against trying to understand the world by adhering to theoretical frameworks (‘systems’). Instead, he commends starting an investigation by conducting ‘a general survey of the subject’ before moving to meticulously designed and reported experiments.

He chastises those who fail to report experiments yielding results that are inconsistent with current theories. He stresses that the results of experiments should be seen as ‘provisional’: he repeats experiments himself and encourages others to repeat them as well. If these replications fail to confirm the results of previous studies, he urges consideration of why this may have been so. And all of this guidance reinforces the need for science to be seen as a collective endeavour.