Bias and misleading concepts in an Arnica research study. Comments to improve experimental Homeopathy.

Basic experimental models in Homeopathy are of major interest because they could get insightful data about the ability of high dilutions to work in a biological system. Due to the extreme difficulty in the highlighting any possible effect and trusting its reliability, methods should be particularly stringent and highly standardized. Confounders, handling process, pre-analytical errors, misleading statistics and misinterpretations may lead to experimental biases. This article tries to elucidate those factors causing bias, taking into account some recent reported evidence in the field.

Background

Marzotto et al. reported that a plant extract from Arnica montana L. from Boiron Laboratoires (Lyon, France) in 30% v/v EtOH/water contained 36.0 mg/100 ml of sesquiterpene lactones, namely 1.05 × 10−3 M. The 1:100 preparation named 1c, was used as the starting solution for a series of further 1:100 dilutions in 30% v/v EtOH/water, which showed an effect on the expression of some extracellular matrix genes when tested on IL-4 polarized THP-1 cells [1]. The paper is particularly interesting but raised fundamental concerns about the experimental setting in basic Homeopathy, which is the objective of this article.

First, in order to calculate the molarity of sesquiterpene lactones in the alcoholic preparation, the authors referred to Staneva et al. who identified at least eight components in an Arnica extract related to helenalin and dihydrohelenalin by 1H NMR spectroscopy and assumed an average molar mass for dihydrohelenalin-derived compounds of 340.41 [2]. The calculation evaluated by Marzotto et al. which does not rely on any reported chromatographic data, would be an approximation to the estimation done by Staneva et al. with 1H NMR. Staneva et al. reported possible errors in the quantitative analysis performed by using only the average molar mass, particularly for compounds such as methacryloyl-helenalin and assessing that the molar mass calculated by summing the molar weights of single lactones, particularly for isobutyryl-helenalin, 6-O-(2-methylbutyryl)-helenalin, 2-methylbutyryl dihydrohelenalin, which cannot be separately evaluated, was lower [2]. The molar estimation calculated by Marzotto et al. in Arnica 1c should refer to the main sesquiterpene lactones present in A. montana (erroneously reported as Arnica m.), i.e. helenalin and 11α,13-dihydrohelenalin esters, giving the reported theoretical molarity [1], [2]. The A. montana 2c made in 30% v/v EtOH/water should therefore contain 51.43 mM EtOH and 10.5 nM sesquiterpene lactones. If considering helenalin and 11α,13-dihydrohelenalin as the major compounds from A. montana in the extract, the authors showed an effect using doses at least three orders of magnitude lower, than those ones previously reported as effective on in vitro immune cells [1], [3], [4], [5]. If true, this interesting result raises the conundrum of the activity associated with further dilutions, e.g. Arnica 5c, as this preparation should be made by 51.43 mM EtOH and 0.0001 fM sesquiterpene lactones, with a ratio EtOH/lactones = 5 × 1014 to 1, a circumstance for which it is very difficult to exclude the molar activity of ethanol with respect to the negligible one of lactones. The same UV–VIS performed by the authors on Arnica 1c shows clearly solely the UV absorbance of ethanol, at 205 nm for an A1cm > 1.0 at its lowest εM value [1]. Therefore, the molar fraction of the chemical components in an A. montana L. extract, would suggest that ethanol is the only chemical bioactive species aside from water, which should be present in the centesimal dilutions.

Ethanol as a confounder and controls performance

The role of ethanol should be better highlighted even though it is difficult to believe that the dilutions may work due to its existence. Ethanol was used in several experimental papers using homeopathic dilutions [6], [7], [8], raising comments elsewhere [9], [10], [11]. Verma et al. reported recently that ethanol is able to induce the release of nanosized membrane extracellular vesicles able to induce macrophage activation [12]. There is no doubt that ethanol has a chemical activity in those systems where the molar mass of the active principle is absolutely negligible [10], [13], [14]. However, the most frequent criticism to this comment is that ethanol is present both in controls and in herbal dilutions (cases) and hence, this solvent could not be considered a statistical confounding [15]. If ethanol is present, at the same concentration, both in dilutions and in controls, its confounding effect should be negligible or even null. However, this is true only if both controls and cases are processed in a double blind fashion and are prepared with the same procedure and handling in a high stringency condition. Pre-analytical biases may occur in this case. Batch-derived biases were even reported for gene microarray, particularly in pooling the RNA samples [16], [17] and therefore, any difference in the handling, storage and treatment of the ethanol batches of dilutions may interfere and affect the reliability of the results. We must admit that, from a chemical point of view and based on the issues addressed above, a control 30% EtOH/water is perfectly similar to, e.g. an Arnica 15c into 30% ETOH/water, because both systems are practically made by only ethanol and water mixed together, due to the negligible or even null amount of sesquiterpene lactones (SLs) in the 15 [1]. Therefore, researchers are most probably comparing two controls with each other, a “control” (A) vs a “dilution” (B). Furthermore, if B undergoes 0.22 μm filtration and A not, as reported [1], if B comes from a batch stored for 12 months while A is a fresh preparation, if B comes from the serial dilutions of previous 4–5 iterative dilutions while A is made from only the previous dilution, and if a blind setting is not considered, differences in the batch of chemical systems that are practically controls at all, may generate bias in the statistics of the outputs and misunderstanding in the conclusive remarks on the reported evidence. Confounders are therefore occurring in the chemical activity of the solvent (ethanol) and in control biases.

A statistical evaluation can shed a light on possible biases

Controls should have a highly homogeneous distribution of their inner variance. Previous comments on data variability in experimental Homeopathy showed that even the distribution of standard error of mean (SEM) may lead to statistical significance, due to alpha error in a H0 null hypothesis [9].

If controls have no homoscedasticity in their data distribution, then a control may bear a “biological” effect due to the existence of a chemical confounder. The herbal dilution with an EtOH/active principle ratio > 1010 to 1, is practically a control, where the only chemical active species is the alcohol, as SLs are negligible or practically lacking [1]. Ethanol has a specific activity on gene expression and on differentially expressed genes (DEGs) and may cause bias in the estimation of p values, particularly if performed with an approach, such as Friedman test, which has the very low power of the sign test [14], [18]. In this case, effects can be related to ethanol as the main confounder of the dilutions [1], [10], [15].

To give a possible example of this issue, in a recent paper, statistics was performed using a Friedman sign test, which is less powerful than other non-parametric rank tests, such as the Wilcoxon–Mann Withney test [1], [18]. This evidence resembles previous reported data, with RT-PCR [6]. According to the authors, any dilution in 30% v/v EtOH/water was able to change DEG patterns, with p < 0.05 in a Wilcoxon test [1]. An evaluation performed on data from Supplementary Tables in the paper [1], using a non-parametric Wilcoxon–Mann Withney rank test, gave the results showed in Table 1. The simple matching of any dilution, from 3c to 15c, vs the control mean (averaging 5 separate experiments) using the RPKM values, gave the following results (bold character = non-significant, i.e. p > 0.05 outputs).

Table 1

Wilcoxon–Mann Whitney test of A. montana effects on IL-4 treated THP-1 gene expression (RPKM).a

Sample	Statistics	Test 1	Test 2	W-value	Mean difference	Sum of POS ranks	Sum of NEG ranks	Z-value	Kolmogorov–Smirnov (P)	p value (2-tailed)
20	Wilcoxon–U-Mann Whitney	1 CTRL	Pooled 3c	34	686.74	176	34	−2.6506	P = 0.98314	0.00804
		2 CTRL		55	505.79	155	55	−1.8666	P = n.s.	0.06148
		3 CTRL		45	637.04	165	45	-2–24	P = 0.98314	0.0251
		4 CTRL		12	672.29	198	12	−3.4719	P = 0.98314	0.00052
		5 CTRL		33	640.64	177	33	−2.688	P = n.s.	0.00714
20	Wilcoxon–U-Mann Whitney	1 CTRL	Pooled 5c	35	688.14	175	35	−2.6133	P = n.s.	0.00906
		2 CTRL		43	507.19	43	167	−2.3146	P = n.s.	0.02088
		3 CTRL		51	638.44	159	51	−2.016	P = n.s.	0.04338
		4 CTRL		15	673.69	195	15	-3–3599	P = 0.98314	0.00078
		5 CTRL		36	642.04	174	36	−2.576	P = n.s.	0.00988
20	Wilcoxon–U-Mann Whitney	1 CTRL	Pooled 9c	31	689.82	179	31	−2.7626	P = n.s.	0.00578
		2 CTRL		36	508.87	174	36	−2.576	P = n.s.	0.00988
		3 CTRL		48	640.12	162	48	−2.128	P = n.s.	0.03318
		4 CTRL		14	675.37	196	14	−3.3973	P = 0.98314	0.00068
		5 CTRL		37	643.72	173	37	−2.5386	P = n.s.	0.01108
20	Wilcoxon–U-Mann Whitney	1 CTRL	Pooled 15c	38	687.89	172	38	−2.5013	P = n.s.	0.001242
		2 CTRL		0	506.94	0	210	−3.9199	P = n.s.	0
		3 CTRL		62	638.19	148	62	−1.6053	P = n.s.	0.1074
		4 CTRL		11	673.44	199	11	−3.5093	P = n.s.	0.00044
		5 CTRL		40	641.79	170	40	−2.4266	P = n.s.	0.0151

Cluster 01 – Controls. [1 vs 2] p = 0.00026; [2 vs 3] p = 0.00116; [3 vs 4] p = 0.00068; [4 vs 5] p = 0.01016; [1 vs 3] p = 0.0151; [1 vs 4] p = 0.05238; [1 vs 5] p = 0.07346; [2 vs 4] p = 0.00005; [2 vs5] p = 0.10044; [3 vs 5] p = 0.24604, Bold letter: biased or critical values. About 70% control matches are biased.

[A. montana 2c] p = 0.01108; [A. montana 3c] p = 0.01242; [A. montana 5c] p = 0.0226; [A. montana 9c] p = 0.01684; [A. montana 15c] p = 0.0477

[A. montana 2c] p = 0.35238; [A. montana 3c] p = 0.09102; [A. montana 5c] p = 0.37346; [A. montana 9c] p = 0.22628; [A. montana 15c] p = 0.65994, (bold character p > 0.05, i.e. not significant), assessing therefore a circumstance that can be retrieved also from Table 1, Table 2, where dilutions are matched to controls of each single experiment. Data suggest that the variance distribution within the control series is not homogeneously dispersed and give possible biases in the interpretation of the presumptive working of homeopathic dilutions. To ascertain control homoscedasticity, a Bartlett's test should be accomplished. The Bartlett's test on the control distribution showed that this variability was highly significant (p < 0.0001, χ2 = 409.19452). The overall RPKM evaluation of the signed rank comparison between all averaged controls and means for each tested dilution, gave the following statistics:

Table 2

Wilcoxon–Mann Whitney test of A. montana effects on IL-4 treated THP-1 gene expression (RPKM).a

Sample	Statistics	Test 1	Test 2	W-value	Mean difference	Sum of POS ranks	Sum of NEG ranks	Z-value	Kolmogorov–Smirnov (P)	P value (2-tailed)
20	Wilcoxon–U-Mann Whitney	1 CTRL	Pooled 3c	72	−27.47	72	118	−0.9296	P = n.s.	0.35238
		2 CTRL		239	18.55	239	257	−0.1764	P = 0.97184	0.85716
		3 CTRL		90	−24.47	90	100	−0.2012	P = n.s.	0.84148
		4 CTRL		75	−25.9	75	115	−0.8048	P = n.s.	0.42372
		5 CTRL		74	−27.34	74	116	−0.8451	P = n.s.	0.39532
20	Wilcoxon–U-Mann Whitney	1 CTRL	Pooled 5c	75	−29.73	75	115	−0.8048	P = n.s.	0.42372
		2 CTRL		40	−31.79	40	150	−2.2133	P = 0.97808	0.0271
		3 CTRL		86	−26.73	86	104	−0.3622	P = n.s.	0.71884
		4 CTRL		74	−28.16	74	116	−0.8451	P = n.s.	0.39532
		5 CTRL		76	−29.6	76	114	−0.7614	P = n.s.	0.44726
20	Wilcoxon–U-Mann Whitney	1 CTRL	Pooled 9c	91	−22.81	91	99	−0.161	P = n.s.	0.87288
		2 CTRL		53	−26.26	53	118	−1.4154	P = 0.97184	0.1556
		3 CTRL		90	−19.81	90	100	−0.2012	P = n.s.	0.84148
		4 CTRL		81	−21.24	81	109	−0.5634	P = n.s.	0.57548
		5 CTRL		71	−22.68	71	119	−0.9658	P = n.s.	0.33204
20	Wilcoxon–U-Mann Whitney	1 CTRL	Pooled 15c	85	−23,71	105	85	−0.4024	P = n.s.	0.68916
		2 CTRL		50	−25.77	50	140	−1.8109	P = 0.97908	0.0703
		3 CTRL		83.5	−20.71	83.5	106.5	−0.4628	P = n.s.	0.64552
		4 CTRL		84	−22.14	84	106	−0.4427	P = 0.97908	0.65994
		5 CTRL		80	−23.58	80	110	−0.6036	P = n.s.	0.5485

Cluster 02 – Controls. [1 vs 2] p = 0.25848; [2 vs 3] p = 0.14706; [3 vs 4] p = 0.90448; [4 vs 5] p = 0.27572; [1 vs 3] p = 0.68916; [1 vs 4] p = 0.63122; [1 vs 5] p = 0.4965; [2 vs 4] p = 0.29372; [2 vs5] p = 0.68916; [3 vs 5] p = 0.0703, Bold letter: biased or critical values. No control match is biased. Bartlett's tests on controls p = 0 χ2 = 409.19452.

[A. montana 2c] p = 0.13622; [A. montana 3c] p = 0.23404; [A. montana 5c] p = 0.21498; [A. montana 9c] p = 0.21499; [A. montana 15c] p = 0.17702, which should suggest the existence of a possible bias in the distribution used to evaluate the dilution activity on THP-1 cells, as these comparisons would indicate the complete absence of effects on the gene expression of macrophages by A. montana alcoholic extracts. This evidence seems to contradict the conclusive remark forwarded by the authors about the activity of Arnica [1]. Goodness of fit test, performed with a Shapiro–Wilk test and a Lilliefors-van Soers test assessed that any distribution was non parametric. The number of outliers in a Rosner's extreme studentized deviate test (p < 0.00001, ≥10 out of values) was 2.25 higher for controls than for any test solutions.

Apparently, the authors did not seem to have addressed this concern. A possible reason is the following. The false discovery rate (FDR) approach has been standardized for barcoded cDNA of samples in a RNA-seq library and sequencing [19] and actually sample pooling yet shows many critical aspects, so that the increase of replicate samples has been suggested as the best choice [20]. Particularly, when negligible concentrations of active principle are challenged with an in vitro model of gene expression, a throughput RNA sequencing method should encompass stringent criteria for the statistical evaluation of DEGs In this context, even the concordance of an NGS with a gene microarray in the case of a genome-wide array of differential gene expression, is affected by the treatment effect size, depending by the transcript abundance and the biological complexity of the different modes of action of the tested chemicals, their dosage and ability to interact with genes [21]. Interestingly, ethanol, at the concentration 51.43 mM, i.e. 0.03% v/v into water, is particularly active on a biological system. Ethanol may cause mitochondrial injury [22] and even mitochondrial DNA damage [23] and in doses as low as 50 mM of ethanol is able to cause mitochondria damage, oxidative stress and apoptosis in several cell models [24], [25], [26], [27], as 50 mM ethanol may cause 2.03% apoptosis in cardiomyocytes and 4.32% apoptosis in 24 h treated endothelial cells [25], [26]. In this circumstance, when ethanol is used in test homeopathic dilutions, an AnnexinV/PI or TUNEL test should be performed, besides a cytotoxicity assay [1]. Ethanol, as a possible confounder should be virtually removed by introducing the same amount of ethanol in parallel matched controls. Controls and cases (i.e. tested dilutions) should be treated in a blinded or double blinded fashion, having the same procedural handling and matching the same experimental running [28], [29], [30].

Possible further causes of high dilutions working

While possible biases in statistics due to confounders might generate a misunderstanding in the interpretation of experimental Homeopathy, the question which arises here is that can ethanol help dilutions into water generating nanosized structures. According to some authoritative reports, nanobubbles might be generated upon the replacement of ethanol by water, as an effect of the super-saturation resulting from either the greater solubility of air gases in the alcohol than water or by the exothermal mixing of ethanol into water [31], [32]. Further detailed research has shown that the EtOH/water solvent usually contains large-scale structures within the range of 100 nm, a hallmark that seems to be commonly shared by any daylife used aqueous systems [33]. This can be considered a “mesoscale inhomogeneity” with a long-lived feature and a relatively slow kinetics of formation that can be detected upon mixing the solutes and solvents. The nature of these structures has been recently investigated, also in EtOH/water mixtures, with static and dynamic light scattering, equilibrated with air at 1.0 atm and nanoparticle tracking analysis (NTA) and results did not confirm the hypothesis that these structures are nanobubbles stabilized by the solutes and contaminants adsorbed at the gas/solvent interface [33]. Some authors performed an interesting study on the presence of nanosized structures, by using a NanoSight LM10 (Malvern), equipped with laser at 532 nm and the NanoSight NTA 3.0 analysis software to analyze data [1]. Their graph has a bewildering similarity with the plot frame of the UV–VIS spectra, probably because the authors re-elaborated the NTA outputs with the same elaboration software used for the UV–VIS, yet not reported in the methods section. This cannot warrant for being fully aware of the NTA outputs and give a forthright comment about the interesting data of these nanostructures in Arnica 1c, namely if these nanostructures are really nanosized elements, nanobubbles or a mesoscale inhomogeneity. The plot represented in ref. [1] about NTA was not released by the NanoSight NTA 3.0 analysis software [1], [34]. Actually, those structures might be either micro-nanobubbles (MNBs) or supra-molecular nanostructures, the authors should have better performed an analysis with optical microscopy by employing the total internal reflection fluorescence excitation (TIRF) to assess nanobubbles > 100 nm [35]. Further techniques adapted to detect nanosized structures in the dilutions, such as atomic force microscopy (AFM) or also CryoTEM, should be evaluated, in order to ameliorate the reliability of the reported results [1].

MNBs are micro- or nanoscopic gas filled cavities which should derive from bubbles generated by a vigorous mixing of gases (generally from environmental air) into a liquid, where the mechanical stress should create a wide range of bubble diameters. Most of these air-filled cavities disappear rapidly, as they have a buoyancy that leads them to rise to the surface of the liquid and burst in equilibrium with the atmospheric pressure, according to Stoke's equation and following the nature of particles at low Reynolds number [36], [37]. According to the authors, these nanosized structures in Arnica 1c formed a colloidal heterogeneously polydispersed system, of about 1.83 × 108 particles/ml with a mean dimension of 274 nm and a ζ-potential of −25.54 mV [1]. The negative ζ-potential should suggest they are MNBs [38]. Although the authors did not specify this, probably they caught aliquots from the bulk liquid. While most of nanobubbles are in the interface gas/liquid and in a lesser extent on the solid/liquid interfaces of the container walls, stable bulk MNBs have a half-life from 1.0 s to 100 μsec, particularly in water [39], and probably the structures observed by Marzotto et al. are not MNBs [40]. Moreover, ethanol, according to recent reports, cannot form surface nanobubbles, contrarily to other organic solvents such as formamide [41]. Ethanol/water systems usually contain large-scale structures within the range of 100 nm, in a “mesoscale inhomogeneity” with a long-lived feature and a relatively slow kinetics of formation that can be detected upon mixing the solutes and solvents. Investigation with static and dynamic light scattering, equilibrated with air at 1.0 atm and nanoparticle tracking analysis (NTA) did not confirm the hypothesis that these structures are nanobubbles stabilized by the solutes and contaminants adsorbed at the gas/solvent interface [33]. Furthermore, nanostructures with dimensions higher than 180 nm can be very rarely described as exosomes [42]. When describing the theoretical composition of an A. montana 9c (final working concentration) the calculation should be as follows: a) Ethanol = 5.14 × 10−4 M; b) active principles (sesquiterpene lactones) = 1.05 × 10−22 M; c) supramolecular structures = unknown. In these conditions, the only chemical species appear to be the single ethanol. In this case, a great concern may be raised about the reliability of the reported results.

Conclusion

Experimental setting in Homeopathy is particularly sensitive to bias, due to the very subtle mechanisms underlying the possibility that negligible doses of biochemical active principles cause effects on standardized and well suited in vitro biological models. Particularly for confounders, used also in solvents and for control setting, researchers must pay particular attention to the possibility that their analytical system might be tarnished by gross biases due to these confounders. Even statistics may hide bias due to an incorrect or not properly used statistical method.

In our opinion, in order to ascertain if the reported effects are caused by some chemical activity within the dilutions, the authors might address the bullet points recommended in Table 3.

Table 3

A. Bullet points of issues to be addressed, B. Possible bias in the experimental setting.

Issue	Description
A
Herbal preparation	■ The analytical pattern of the herbal preparation (UV-HLPC, IR-HPLC, NMR, other) must be reported in any paper concerning herbal remedies.
Dilutions	■ Each tested dilution should be evaluated for the presence of chemical molar mass, due to recent evidence and models [43], [44]. ■ Nanosized particles and ζ-potential should be evaluated in each tested dilution.
Nanoparticles	■ Reported results and plots must be produced directly from the NanoSight NTA analysis software upon throughput elaboration of data from the analytical instrument (e.g. NanoSight LM10 (Malvern). ■ An optical microscopy of the nanostructures (TIRF or AFM) should be considered for any dilution tested.
Experimental setting	■ Double Blind fashion of the setting should be accomplished. ■ Batch effects on the microarray geneset evaluation.
Controls	■ 30% ethanol/water dilutions (sham) should be prepared and tested perfectly matched with any herbal dilution.
Statistics	■ False discovery rate (FDR) analysis should be assessed by an improved estimation of p value based on the mixture model [45], [46].
Minor points	■ In taxonomy the name of the genus is punctuated, the species not, e.g. A. montana instead of Arnica m..
B
Dilutions	■ Dilutions stored for at least 12 months may be not similar to fresh prepared controls. ■ Dilutions stored for at least 12 months allow readers to believe that the authors used the same preparations for different experimental studies. If a bias occurred in one dilution, a carry over effect might be created.
Controls	■ Controls (sham dilutions) are not matched with cases because are prepared and treated in a different manner than cases (herbal dilutions) and without a double blind setting.
Statistics and setting	■ Environmental EMF can cause bias in the statistics of microarray study [47]. ■ The use of Friedman test should be carefully addressed. Friedman test should be considered a generalization of the sign test and in this sense it has a modest statistical power of the sign test, both for normal and non normal distributions [18].
Pooling samples/data	■ Bias can be introduced when RNA samples are polled due to insufficient ability to test genesets. Pooling appears to lower the efficiency of the labeling or hybridization steps, causing artefacts, as reported in Kendziorski's experiment [16], ■ Bias on microarray due to batch effects (RNA quality and RNA quantity) [17].

Sources of funding

None.

Conflict of interest

None.