Homeopathic Arnica from Boiron and post-operative bleeding in mastectomized women in Milan: Statistical flaws and bias to be addressed.

Image 1.

In a recent paper, Sorrentino et al., reported that a homeopathic preparation of an extract from Arnica montana L, known as Arnica 1000 K, significantly reduced the amount of post-operative blood and seroma (improving bleeding) in female subjects who underwent an unilateral total mastectomy. This double blind randomized controlled trial versus placebo was performed throughout two years-long period, from 2012 to 2014, and about 53 patients (26 cases and 27 controls) with an age range 20–75 years and hospitalized for unilateral total mastectomy, were enrolled. In their study, the authors stated that patients underwent also a standardized protocol of treatment for surgical procedures including an antithrombotic prophylaxis with 4000 UI heparin 12 h before surgery. Patients who followed a randomization blinded protocol received a 30% v/v ethanol preparation of Arnica 1000 K (five drops sublingual) or a placebo (i.e. simply 30% v/v ethanol) from 1 day before surgery to the fourth day following surgery, which was performed by the same surgical team. Those patients treated with Arnica 1000 K and collected in the group who excluded subjects who violated the protocol (PP group), showed less blood leakage and seroma (serum from surgical drainages) release from post-operatory drainages, reporting a better bleeding process.

Arnica 1000 K is a korsakovian homeopathic dilution, made into 30% v/v ethanol, which according to the authors should not contain any toxic compound, as the starting material is very highly diluted. Starting from a 0.04% sesquiterpene lactones (SLs) amount, which following previous reports might be about 1.0 mM SLs, Arnica 1000 K is absolutely far away from a chemical ponderal dosage, as the dilution is made by emptying completely the vial with the previous content, refilling it with only 30% v/v ethanol, shaking strongly, emptying it again, refilling with the same solvent, shaking, emptying and so following this iterative process 1000 times, reaching the last step in a refilling with the alcoholic solvent. The method is practically similar to the one used by Boiron, which prepared Arnica 1000 K and placebos for the study, for its Oscillococcinum 200K.

In this perspective it is rather difficult to achieve a molar mass evaluation of SLs in the 30% ethanol solvent, as at least theoretically each emptying procedure might leave less than 10% of the total volume, according to the authors and our calculations should reach a value that exceeds the Avogadro-Loschmidt's threshold already following only 7–8 passages, thus leading to the consideration that Arnica 1000 K does not contain any SL molecule but simply solvent. Actually, if considering that Arnica 1000 K is practically a control (placebo) from a chemical point of view, any resulting value at p < 0.05 through a Student's t-test, might lead to false positives if variances in the distribution are biased and inhomogeneously dispersed. The authors did not succeeded in retrieving statistically significant differences with the exception of a single p < 0.03 in the third treatment model in the PP group. The statistical re-calculation of the data is shown in Table 1. In the Arnica ITT group (26 subjects) means followed a normal distribution (Anderson–Darling test p = 0.394, Shapiro–Wilk test p = 0.307, Cramer–von Mises test p = 0.430), as like as placebos (Anderson–Darling test p = 0.136, Shapiro–Wilk test p = 0.134, Cramer–von Mises test p = 0.137) and apparently their two sided paired t-test gave a value of p = 0.04331, with t = −2.34989. Yet, a Grubb's test for potential outliers, if applied, would turn this significance to a p value = 0.07422.

Table 1

Statistics reappraisal of data reported in Ref 1.

Test	Data	Statistics	Samples	p
Kolmogorov–Smirnov	ITT group	0.4Mean 1 = 62.194; Mean 2 = 60.418Diff = 1.776	Variability	0.87302
	PP group	0.6Mean 1 = 41.886; Mean 2 = 54.092Diff = −12.206		0.35714
	ITT group	0.2Mean 1 = 77.424; Mean 2 = 79.602Diff = −2.378	Mean	1
	PP group	0.4Mean 1 = 68.82; Mean 2 = 87.562Diff = −18.742		0.87302
	ITT + PP	0.4Mean 1 = 52.04; Mean 2 = 57.255Diff = −5.215	Variability	0.41752
		0.3Mean 1 = 73.122; Mean 2 = 83.682Diff = −10.56	Mean	0.78693
Wilcoxon-Mann Whitney (for data <30)	ITT + PP	18Variable 1 = 18.55758; Variable 2 = 18.28493	Variability	0.375
	ITT + PP	7Variable 1 = 38.34219; Variable 2 = 48.05253	Mean	0.03711
	ITT + PP (1)	7Variability 1 = 30.79461; Variability 2 = 37.26469		0.07422

(1) If considering Grubbb's test resulting false outlier p > 0.3.

Bold numbers are significant values at p < 0.05.

Distribution of patients in the ITT group appeared biased. As a matter of fact, the authors did not retrieve any positive effect of Arnica 1000 K on blood and serum amount in drainages in the intention-to-treat (ITT) population. They reported differences only in the per-protocol (PP) population and exclusively if considering the model (number 3) including type of treatment, the amount of blood and/or seroma collected on the day of surgery, and patient's weight. Nevertheless, weight distribution in the placebo (a) and Arnica (b) group, respectively, in the ITT population, was rather different, as the delta values (most probably IC95) for weight were 80 kg (a) and 28 kg (b). Distributions, which have comparable medians (a = 60.40; b = 62.90) and means (a = 64.19; b = 66.05), should suggest that, taking into account these differences, homoscedasticity was most probably different in the two groups, resulting in a possible bias in the evaluation of Arnica effect because of the existence of size effects and due to the distribution of patients weights and vascular pressure in the placebo respect to the Arnica group. Bartlett's test confirmed this inhomogeneous variance distribution between cases (Arnica treated) and controls (placebos) (p < 0.0001, χ2 = 113.03173). Difference in the variances within the ITT group is a possible confounder.

This fundamental bias might be generated from the less powerful t-test in a relatively small sized sample. Actually, the authors used Student's t-test for their statistics. While in a frequentist statistical perspective a t-test might be exceptionally adopted for samples <30 subjects (as the number used for either cases or controls in the reported study), though with caution for the possible occurrence Type II errors, the use of small sample sizes, from a Bayesian point of view, should result in inflated effect sizes and false positives.5, 6, 7, 8 A very simple reason shedding a light on this assessment may be addressed. An investigator, particularly if performing a double blinded experimental setting, cannot know whether an H0 null hypothesis is true or false. The probability that a value is true or false could be estimated from the existing literature on the topic, but, at least for homeopathy and Arnica 1000 K, many technical and cultural prejudices lead to the utmost consideration that the dilution “cannot” work (for its chemical absurdity) and that whatsoever is considered as a “positive” signal, would be potentially considered as the confirmation of the excellent quality of the experimental setting, rather than a true effect, the reliability of which is exclusively due to the experimental setting and performance. Therefore the statistical power of the test is particularly stringent in giving the best positive predictive value. If we suppose a 50% of the null hypothesis as true and 50% as false, with D = 1, if the sample size is small, the statistical power is low. With the lowest value of 2 (sample size), the probability for a true positive is 4.7% and for a false positive is 2.5%, i.e. the probability that this statistics results in giving a significant finding as a true effect is only 65%, while for a sample size of 100, the positive predictive value is 95%, so suggesting that for a sample <30, a statistically significant result is clearly more likely to be a false positive.9, 10, 11

Therefore, in this circumstance, the positive value in the PP population in the model 3 of treatment with Arnica 1000 K, might be biased and in this sense should need reproducibility. The authors concluded that Arnica 1000 K gave noticeable differences, though small, and attributed this failure to the extremely high dilution of the Arnica extract in the Arnica korsakovian preparation. This conclusion might apply to any Arnica dilution exceeding the Avogadro-Loschmidt threshold, yet.12, 13

In conclusion, the paper by Sorrentino et al., should need some reappraisal, in order to assess the experimental reproducibility by checking and revising possible effects from confounders and bias.

Conflict of interest

The authors state they have no conflicts of interest.