Variability and effect sizes of intracranial current source density estimations during pain: Systematic review, experimental findings, and future perspectives.

Pain arises from the integration of sensory and cognitive processes in the brain, resulting in specific patterns of neural oscillations that can be characterized by measuring electrical brain activity. Current source density (CSD) estimation from low-resolution brain electromagnetic tomography (LORETA) and its standardized (sLORETA) and exact (eLORETA) variants, is a common approach to identify the spatiotemporal dynamics of the brain sources in physiological and pathological pain-related conditions. However, there is no consensus on the magnitude and variability of clinically or experimentally relevant effects for CSD estimations. Here, we systematically examined reports of sample size calculations and effect size estimations in all studies that included the keywords pain, and LORETA, sLORETA, or eLORETA in Scopus and PubMed. We also assessed the reliability of LORETA CSD estimations during non-painful and painful conditions to estimate hypothetical sample sizes for future experiments using CSD estimations. We found that none of the studies included in the systematic review reported sample size calculations, and less than 20% reported measures of central tendency and dispersion, which are necessary to estimate effect sizes. Based on these data and our experimental results, we determined that sample sizes commonly used in pain studies using CSD estimations are suitable to detect medium and large effect sizes in crossover designs and only large effects in parallel designs. These results provide a comprehensive summary of the effect sizes observed using LORETA in pain research, and this information can be used by clinicians and researchers to improve settings and designs of future pain studies.

INTRODUCTION

Pain is a fundamental experience for survival, guiding behavior towards minimizing harm. It arises from the integration of sensory, cognitive, emotional, and motivational processes in the brain (Apkarian, Bushnell, Treede, & Zubieta, 2005). The complex interaction of these processes are the result of specific patterns of neural oscillations that can be directly characterized by measuring the electrical brain activity with magneto‐ and electroencephalography (M/EEG) (Ploner, Sorg, & Gross, 2017). Ultimately, researchers and clinicians want to identify the spatiotemporal dynamics of the brain sources in physiological and pathological pain‐related conditions (Davis & Seminowicz, 2017; Ploner & May, 2018). This could help obtaining robust biomarkers from these measures with the aim of stratifying patients and improving medical diagnosis and treatments (Mouraux & Iannetti, 2018; Tracey, Woolf, & Andrews, 2019).

In this regard, source localization methods are a particular type of spatial filter that provide information about the activity and localization of the neural sources (Michel & Murray, 2012). A popular approach to estimate the brain sources from scalp EEG data is by low‐resolution brain electromagnetic tomography (LORETA) (Babiloni et al., 2017; Pascual‐Marqui et al., 1999). This method and its standardized (sLORETA) and exact (eLORETA) variants have been used to characterize the brain activity through the estimation of intracranial current source density (CSD), and localize sources in pain studies (González‐Roldán, Cifre, Sitges, & Montoya, 2016; Hansen et al., 2017; Lelic, Hansen, Mark, Olesen, & Drewes, 2017; Moont, Crispel, Lev, Pud, & Yarnitsky, 2011; Nir, Sinai, Moont, Harari, & Yarnitsky, 2012; Shao, Shen, Yu, Wilder‐Smith, & Li, 2012). This family of methods have been validated with standard methods for source localization, such as functional magnetic resonance imaging (fMRI) or positron emission tomography (PET) (Cannon, Kerson, & Hampshire, 2011; Esslen, Kochi, Lehmann, & Pascual‐marqui, 2002; Michel & Murray, 2012; Oakes et al., 2004; Pascual‐Marqui, Esslen, Kochi, & Lehmann, 2002). However, most studies employing LORETA emphasize statistically significant differences between areas or experimental conditions, disregarding the variability and effect sizes behind these differences.

In order to constitute a robust biomarker, a measurement must be valid but also reliable, that is, its variability must be sufficiently low across experimental conditions (subjects, sessions, assessors) when experimental or clinical conditions remain unchanged (Downing, 2004). While test–retest reliability of LORETA studies was examined in different settings, for example during resting EEG (Cannon et al., 2012) and superficial painful stimulation (Hansen et al., 2017), the interpretation of the outcomes is usually carried out in oversimplified, binary terms, that is, the measurement is deemed reliable or not based on a comparison with a fixed threshold set using arbitrary criteria, which can be misleading if data and results are not critically assessed (Biurrun Manresa et al., 2014).

In this context, it is clear that there is no consensus in the literature on the magnitude of clinically or experimentally relevant effects for CSD estimations, and what is the magnitude of the variability behind these measures. This information is crucial for properly planning future research designs but is currently unavailable. Thus, the aims of this study were (1) to systematically examine reports of sample size calculations and estimations of effect sizes in studies that used LORETA for pain research; (2) to assess the reliability of LORETA CSD estimations in experimental settings; (3) to estimate hypothetical sample sizes for future experiments using source localization of neural oscillations for the identification of potential biomarkers related to pain.

METHODS

This study is divided in three parts: the first part is a systematic review of existing literature involving CSD estimation using LORETA, focusing on reports of effect sizes and variability in pain research. The second part describes methodological considerations and results from an experiment specifically carried out to determine the reliability of CSD estimations in controlled experimental conditions. Finally, the third part integrates the findings from the previous two to estimate future sample sizes for potential experiments involving these measurements.

Part I: Systematic review of effect sizes and variability of CSD estimates in pain‐related studies

Search strategy

Electronic literature search was performed by the first author (J.M.V.) in SCOPUS and PubMed databases for articles published before July 20th, 2020. The search descriptors in the title, abstract or keywords were “pain” and at least one of the following “LORETA” or “eLORETA” or “sLORETA.”

Inclusion and exclusion criteria

Articles were included in the analysis if they met the following criteria: (1) full articles, written in English and published in peer‐reviewed journals; (2) human pain research studies; (3) included more than five participants (no case studies); (4) included EEG frequency analysis; (5) provided information about central tendency and dispersion of the CSD estimations for the sample.

Data extraction

All studies involving pain research that used LORETA were evaluated to report the type of study (exploratory or confirmatory), the percentage of studies that reported sample size calculations, measures of central tendency and dispersion and, in case of experimental studies, estimation of effect sizes (Noordzij et al., 2010). If the effect size was not reported, it was estimated using the mean and standard deviation of the CSD estimation and the number of participants (assuming a normal distribution after a logarithmic transformation) (Lakens, 2013). If the mean and standard deviation were not reported numerically, they were extracted from the graphics using the software Engauge Digitizer 12.1 (http://digitizer.sourceforge.net).

Part II: Reliability of CSD estimates in controlled experimental conditions

Endpoints

CSD is usually estimated in the θ (4–7.9 Hz), α (8–13.9 Hz), β (14–29.9 Hz), and γ (30–90 Hz) bands, which are commonly reported to be involved during tonic pain‐related neural oscillations (Backonja et al., 1991; Chang, Arendt‐Nielsen, Graven‐Nielsen, Svensson, & Chen, 2001; Dowman, Rissacher, & Schuckers, 2008; Li et al., 2016; Nir et al., 2012; Nir, Sinai, Raz, Sprecher, & Yarnitsky, 2010; Ploner et al., 2017; Schulz et al., 2015; Shao et al., 2012). Furthermore, the CSD for these frequencies is extracted from specific regions of interest (ROIs) in the brain. A survey of the studies from the systematic review revealed that data from quantitative analysis was available for the anterior cingulate cortex (ACC), the anterior insula (AI), and the primary somatosensory cortex (S1) in the θ and α frequency bands. We also included the γ band in the analysis of our own experimental data, since cortical oscillations in these frequency bands have been associated with pain processing (Hauck, Lorenz, & Engel, 2007; Nickel et al., 2017). Moreover, common settings in experimental pain research involve controlled conditions for nociceptive stimulation, as well as baseline or reference recordings of non‐nociceptive stimulation as well. For this experiment, we used a model of sustained deep‐tissue pain (SDTP) elicited using cuff‐pressure algometry, and we also performed sustained deep‐tissue non‐painful stimulation (SDTnP) and recorded resting‐state EEG (REEG) for comparison purposes. Thus, we defined the primary endpoint as the test–retest reliability of CSD estimations in the aforementioned three brain regions and three frequency bands, during SDTP. Secondary endpoints were the test–retest reliability of CSD estimations at the same brain regions in the same frequency bands, for SDTnP and REEG.

Setting

The experiment was carried out at the Department of Health Science and Technology, Aalborg University, Aalborg (Denmark). The study was conducted according to the Declaration of Helsinki and was approved by the ethical committee of Northern Jutland, Denmark (N‐20170047).

Participants

Twenty‐one healthy volunteers participated in the experiment (mean age = 25.0 years, SD = 2.6; 14 females). Inclusion criteria were (1) healthy men and women between 18 and 50 years old, able to speak and understand English. Exclusion criteria were (1) pregnancy or breast‐feeding, (2) previous neurologic, musculoskeletal, or mental illnesses, (3) history of chronic pain or current acute pain, (4) skin allergies, (5) presence of wounds in the forearm, and (6) incapacity to provide informed consent. Furthermore, participants were asked to refrain from any pain medication at least 24 h before the experimental sessions.

EEG recordings

EEG data were recorded using a medical‐grade amplifier (g.HIamp, g.tec‐medical engineering GmbH, Austria) using 64 active electrodes. Before the EEG recording, the head of the participants was measured using the distance between the nasion and inion for appropriate placement of the electrodes according to the international 10/20 system. A flexible EEG cap (g.GAMMA cap2, g.tec‐medical engineering GmbH, Austria) with chinstrap was used to maintain fixed electrode positions. The impedance of the electrodes was kept below 10 kΩ and the sampling rate was set at 1,200 Hz. The AFz electrode served as ground and the left earlobe (A1) served as reference. Recordings for each condition lasted at least 3 min, and conditions were randomized across participants, with a 5‐min pause between conditions.

Computer‐controlled cuff pressure algometry

A computer‐controlled pneumatic pressure algometry (NociTech, Denmark, and Aalborg University, Denmark) was used to deliver sustained deep tissue stimulation. A 10‐cm wide silicone tourniquet cuff (VBM, Germany) was tied around the right forearm at a 3‐cm distance of the cubital fossa and over the extensor carpi radialis brevis muscle belly. Participants were trained to score the perceived stimulation sensation using a computerized visual analogue scale (VAS) where 0 represented no sensation, 10 represented the most intense pain imaginable and 5 indicated the pain threshold. The average cuff pressure that elicited ratings of 3 and 7 in the VAS of three consecutive ramps (rate = 1 kPa/s, interstimulus interval = 5 min, pressure limit = 110 kPa) was used to estimate the pressure of SDTnP and SDTP, respectively.

EEG pre‐processing

Raw EEG data were off‐line pre‐processed using Matlab R2018b (The Mathworks Inc, Natick, MA) and EEGLAB toolbox v15.0b (Delorme & Makeig, 2004). Data were digitally band‐pass‐filtered by applying a Hamming‐windowed sinc finite impulse response filter with an order of 7,920 and cutoff frequencies of 0.1 and 100 Hz. Additionally, a 50‐Hz notch filter was used, and data were resampled to 500 Hz. Channels with low signal‐to‐noise ratio were removed from both recordings of a subject. Filtered data were cleaned by visual inspection to exclude artifacts (non‐cerebral source activities). Furthermore, blinks and muscle activity were removed using independent component analysis (Jung et al., 2000). Removed channels were interpolated afterwards using their neighboring channels. Finally, artifact‐free EEG data were reduced to 120 s for each condition to ensure the same amount of data for all subjects.

Estimation of cortical sources

The exact low‐resolution brain electromagnetic tomography (eLORETA, LORETA‐KEY software v20150415, http://www.uzh.ch/keyinst/loreta.htm) method was used to localize cortical brain activity (Pascual‐Marqui, 2007). This inverse modeling method estimates the current density in the brain volume, thus providing an appraisal of where in the brain the scalp‐recorded EEG is being generated (Pascual‐Marqui, Michel, & Lehmann, 1994). The ROIs consisted of the collection of voxels that are within a 15‐mm radius sphere around the seed point (Canuet et al., 2012). The coordinates of the seed point of the ROIs are in Table 1 (Julious, 2004). In particular, the left ACC and S1 regions were selected because they are contralateral to the stimulus, whereas activity in the right AI was quantified as there is evidence of a preponderance of this side in pain processing (Brooks, Nurmikko, Bimson, Singh, & Roberts, 2002; Craig, 2003; Ostrowsky et al., 2002). It is known that the CSD estimations obtained from LORETA do not show a uniformly normal distribution (Thatcher, North, & Biver, 2005), which is a requirement for the following reliability estimations (Julious, 2004). Hence, a logarithmic transformation was used to obtain a normal distribution.

TABLE 1

Spatial locations of regions of interest considered for the analysis

Region	BA	x	y	z
Left‐anterior cingulate cortex (L‐ACC)	32	−2	32	22
Left‐primary somatosensory cortex (L‐S1)	2	−32	−36	60
Right‐anterior insula (R‐AI)	13	36	12	8

Note: Coordinates are in Talairach space.

Abbreviations: BA, Brodmann area; x, medial‐lateral; y, anterior–posterior; z, superior–inferior.

General methodological aspects

The experiment was designed to minimize the influence of external confounding factors in a test–retest reliability assessment (e.g., population age, time of the day where the experiment was carried out, and interval between sessions), which are known to affect EEG activity and pain sensitivity (Carrier, Land, Buysse, Kupfer, & Monk, 2001; Hjermstad et al., 2011). Therefore, only young adults were included, and the experiment was scheduled at the same time of the day for every volunteer. The experiment was carried out in two different sessions, separated by 7 ± 2 days. Participants were thoroughly familiarized with constant cuff‐pressure stimulation to reduce any effects of arousal or anxiety before data acquisition in the first session. During the experimental sessions, the participants were seated on a comfortable chair in front of a computer screen (HP P17A ProDisplay, Hewlett‐Packard Company, USA) that showed a vertical bar, whose length represented the perceived intensity rating of the VAS, anchored as described in Section 2.2.4. Participants also wore foam earplugs (Earplugs, TaperFit 3M, Minnesota) to mask ambient noise and to minimize any auditory bias introduced by the air compressor. All participants attended the second experimental session at the same time of the day (±1 h of difference regarding the first experimental session), to rule out possible inference of the circadian rhythm in pain perception (Glynn, Lloyd, & Folkard, 1976). Furthermore, to rule out inter‐assessor variation, the same researcher (J.M.V.) collected data from all volunteers.

Data analysis and statistics

Between‐session reliability of cuff pressure and CSD estimations was determined using Bland–Altman (BA) analysis. BA analysis consists of plotting the differences vs. the average of repeated measurements for each subject. Furthermore, the limits of agreement () can be derived, which express the average difference () ± 1.96 times the standard deviation of the differences between repeated measurements (). In this way, the difference between the upper and lower delimit the range where 95% of the differences between two repeated measurements are expected to lie (Bland & Altman, 1986). The 95% confidence intervals (CI) are reported for both and (Bland & Altman, 1999). The between‐session reliability of the stimulus used for the experimental intervention (cuff‐algometry) was calculated for the SDTP and SDTnP as well. The reliability of CSD during SDTP, SDTnP, and REEG was calculated for theta, alpha, and gamma bands and the three ROIs (L‐ACC, L‐S1, and R‐AI). Values are expressed as mean ± SD unless stated otherwise.

Part III: Considerations for future research designs

Sample size estimation

Hypothetical sample size estimation is a valid and complementary approach for assessing reliability (Shieh, 2014). It was calculated for both parallel () and crossover () study designs. indicates the amount of subject needed for parallel studies (in which there are two groups, for example, patients and healthy volunteers), and indicates the amount of subject needed for crossover studies (where the same group of participants is assessed in two different conditions). The estimation of and was calculated considering a type I error level of 5% and 80% power for an independent‐sample or paired‐sample t test, respectively.

Relationship between reliability estimates, sample size, and effect sizes

All the reliability assessment methods used in this study are related to the within‐subject standard deviation (). For a test–retest experiment, the can be calculated as , and the can be reformulated as (Biurrun Manresa et al., 2014). Finally, sample size estimator can also be estimated from these relations, since and , where and are the average measurements of the two groups in a parallel design, or the average values of the two conditions for the same group in a crossover design, and is the correlation among repeated measures ( is the standard deviation measurements of the sample) (Julious, 2004). In this way, is the simple effect size. Additionally, a standardized effect size can be derived using Cohen's for parallel designs or for crossover designs, where and are the standard deviation measurements of the two groups/conditions (Lakens, 2013).

RESULTS

A descriptive flowchart of the selection process for eligible studies can be seen in Figure 1. The systematic review resulted in 100 articles, from which 58 unique studies remained after removing duplicates found in both databases, and 46 studies were included in the analysis after checking for the first three items in the inclusion criteria. Afterwards, the main reason for exclusion was that 29/46 studies performed time‐domain analysis (as opposed to frequency‐domain analysis), for which we nonetheless still present results regarding reporting of estimates of central tendency and dispersion. Table S1 contains the bibliographic information about the studies included in the systematic review and Tables S2 and S3 contain details on the articles with frequency‐ and time‐domain analysis, respectively, including research design and setting, and whether they include effect sizes and variability of CSD estimates.

FIGURE 1

Flow chart for the systematic literature review. CSD, current source density; SD, standard deviations; SE, standard error

Table 2 shows the percentage of studies that reported indexes of central tendency and dispersion for CSD estimates, as well as estimates of effect sizes and sample size calculations. Remarkably, no studies reported sample size calculations, although it should be clarified that 12 studies (26%) were typified as exploratory, in which a sample size calculation is not strictly necessary. After screening the papers that performed a frequency‐domain analysis on the CSD estimates (17/46), we found that 14 of them did not provide information about central tendency or dispersion of the CSD at any ROI, which prevents further estimations of sample size. Thus, the systematic review yielded three publications selected for further analysis in Part III.

TABLE 2

Percentage of pain studies using LORETA that reported sample size calculation and parameters needed for future sample size calculations (n = 46)

Parameter	Reported
Sample size calculation	0 (0%)
Mean CSD at any ROI for any condition/group	15 (33%)
SD or SE of the mean CSD at any ROI for any condition/group	7 (15%)
Cohen's d effect size	13 (28%)

Abbreviations: CSD, current source density; SD, standard deviation; SE, standard error; ROI, region of interest.

Briefly, Ye, Yan, Yao, Lou, and Peng (2019) attempted to identify abnormalities of spontaneous cortical oscillations among patients with somatoform pain disorder, and observed differences in resting‐state alpha oscillations between patients and controls. While statistically significant differences were found in CSD estimations in the parietal region, quantitative values for patients and controls ( and , respectively, in Table 3) were also reported for other areas, such as the central region including S1. Meanwhile, Prinsloo et al. (2019) proposed to use LORETA‐based neurofeedback to attempt to modify brain activity in patients with acute pain from head and neck cancer. Among other quantitative data, they reported CSD estimations in the ACC, S1, and AI at baseline and pain onset ( and , respectively, in Table 3). Finally, Prichep, Shah, Merkin, and Hiesiger (2018) compared CSD estimations of chronic pain patients and age‐ and gender‐matched controls ( and , respectively, in Table 3) in several brain regions in the theta frequency band. A summary of statistical indexes of central tendency and dispersion for these articles can be found in Table 3.

TABLE 3

A summary of empirical findings of information about central tendency and dispersion of the log CSD estimates in pain research using LORETA

Article	Parameters	ACC			S1			AI
		θ	α	γ	θ	α	γ	θ	α	γ
Ye et al.	X_1¯ (SD) X_2¯ (SD)	–	–	–	–	0.30 (1.14) −0.10 (0.76)	–	–	–	–
Prinsloo et al. a	X_1¯ (SD) X_2¯ (SD)	–	1.48 (0.24) 1.57 (0.32)	–	–	1.26 (0.24) 1.05 (0.33)	–	–	1.09 (0.24) 1.29 (0.33)	–
Prichep et al. b	X_1¯ (SD) X_2¯ (SD)	0.67 (1.29) 0.60 (1.23)	–	–	1.38 (1.44) 1.01 (1.59)	–	–	–	0.43 (1.51) 0.57 (1.34)	–

Abbreviations: ACC, anterior cingulate cortex cluster that included this region; AI, anterior insula cluster that included this region; S1, primary somatosensory cortex or cluster that included this region.

Did not report the units of CSD (suspected log10 CSD μA/mm2).

Reported mean CSD and SD log values under z‐transformation.

(SD)

(SD)

(SD)

(SD)

(SD)

(SD)

Cuff pressure algometry

Individual average pressures of SDTP and SDTnP are shown in Figure 2. The average pressures that elicited a perceived intensity of 3 and 7 in the VAS scale were 27.7 ± 9.1 and 63.3 ± 14.0 kPa on session 1 and 24.7 ± 6.7 and 60.3 ± 12.2 kPa on session 2, respectively. A visual inspection of Bland–Altman plots (Figure 3) did not reveal clear signs of heteroscedasticity. Statistically significant differences in cuff pressure between sessions were observed during SDTnP (, , Cohen's ), but not during SDTP (, , Cohen's ).

FIGURE 2

Magnitude of the pressure used to elicit a painful sensation (VAS = 7, SDTP, left) and non‐painful sensation (VAS = 3, SDTnP, right) in both sessions. VAS, visual analog scale; SD, standard deviations; SDTP, sustained deep tissue pain; SDTnP, sustained deep tissue no‐pain; S1, session 1; S2, session 2. The boxplots represent the median (central black mark), the edges of the box the 25th and 75th percentiles. Gray lines next to boxplots represents the mean (gray dot) and standard deviation (gray whiskers)

FIGURE 3

Bland–Altman plots of the magnitude of the pressure used in the cuff in the left: sustained deep tissue pain (SDTP) and right: sustained deep tissue no‐pain (SDTnP) stimulations across days. The dashed line indicates the bias between sessions and the dotted lines the limits of agreement (LoA), calculated as ±1.96 times the standard deviation (SD) of the differences in measurements between sessions. Shaded areas indicate the 95% confidence intervals of the bias and the limits of agreement

LORETA CSD during SDTP, SDTnP, and REEG

For reproducibility purposes, Table S4 includes, for each condition, ROI and frequency band, a summary of the mean, standard deviation, maximum and minimum values for CSD estimations, as well as quantitative data of bias and LoA with their corresponding CIs from the Bland–Altman analysis. Furthermore, Table S5 includes every individual CSD estimation for each subject, condition, ROI and frequency band. Individual CSD differences between sessions and Bland Altman plots are shown in Figures 4 and 5, respectively. Both figures represent conditions (SDTP, SDTnP, and REEG) in each ROI (L‐ACC, L‐S1, and R‐AI) and frequency bands (theta, alpha, and gamma). A visual inspection of Bland–Altman plots (Figure 5) did not reveal clear signs of heteroscedasticity. Systematic bias was only observed between session in the gamma band in the R‐AI during SDTP and in the L‐ACC during SDTnP.

FIGURE 4

Log current source density (CSD) at each region of interest (ROI), for the theta, alpha and gamma band during sustained deep tissue pain (SDTP) and sustained deep tissue no‐pain (SDTnP) and resting (REEG) in session 1 (S1) and session 2 (S2). L‐ACC, left anterior cingulate cortex; L‐S1, left primary somatosensory; R‐AI, right anterior insula. The boxplots represent the median (central black mark), the edges of the box the 25th and 75th percentiles. Lines next to boxplots represent the mean (central dot) and standard deviation (SD) (whiskers)

FIGURE 5

Bland–Altman plots of the difference between session 1 (S1) and session 2 (S2) of the log10 current source density (CSD) for each region of interest, at theta, alpha and gamma band during sustained deep tissue pain (SDTP), sustained deep tissue no‐pain (SDTnP) and resting (REEG). The dashed line indicates the bias between sessions and the dotted lines the limits of agreement (LoA), calculated as ±1.96 times the standard deviation (SD) of the differences in measurements between sessions. Shaded areas indicate the 95% confidence intervals of the bias and the limits of agreement. L‐ACC, left anterior cingulate cortex; L‐S1, left primary somatosensory; R‐AI, right anterior insula

Sample size calculation with data from the systematic review in Part I

Table 4 shows the hypothetical sample sizes that are needed to detect the effect sizes reported for the articles selected in the literature review, with a type I error of 5% and a power of 80%.

TABLE 4

Hypothetical sample size calculation for the effect sizes obtained from the systematic review

Article		ACC			S1			AI
		θ	α	γ	θ	α	γ	θ	α	γ
Ye et al. N1 = 17 N2 = 17	d	–	–	–	–	0.43	–	–	–	–
	Np	–	–	–	–	84	–	–	–	–
Prinsloo et al. N = 12	dz	–	0.36	–	–	0.98	–	–	0.88	–
	Nc	–	112	–	–	8	–	–	8	–
Prichep et al. N1 = 77 N2 = 77	d	0.07	–	–	0.54	–	–	0.18	–	–
	Np	4,245	–	–	56	–	–	450	–	–

Note: Effects size where significant differences were effectively reported are highlighted in bold (p < .05).

Abbreviations: ACC, anterior cingulate cortex cluster that included this region; AI, anterior insula cluster that included this region; , Cohen's standardized effect size; N, total number of participants reported in the study; , number of participants estimated for cross‐over experiment design. , number of participants in each group estimated for parallel‐over experiment design; S1, primary somatosensory cortex or cluster that included this region.

Sample size calculation with data from the experimental findings in Part II

Hypothetical sample sizes are listed in Tables 5, 6, 7 for SDTP, SDTnP, and REEG, respectively. The estimation indicates the minimal number of participants that will be required to correctly reject the null hypothesis of no effect or no difference given a type I error of 5% and a power of 80%. After observing the variability in the experimental data, three different simple effect sizes were proposed, representing absolute differences of ±0.05, ±0.10, and ±0.25 in the log10 CSD estimates between groups (for parallel designs) or conditions (for crossover designs).

TABLE 5

Hypothetical sample size calculation for crossover () and parallel () sustained deep tissue pain (SDTP) experiment design as a function of the effect size ( and )

Effect size (log10(CSD))		L‐ACC			L‐S1			R‐AI
		θ	α	γ	θ	α	γ	θ	α	γ
±0.05	Nc	61	166	156	153	159	138	102	218	242
	dz	0.33	0.20	0.21	0.25	0.21	0.29	0.28	0.17	0.21
	Np	446	409	305	1,129	426	273	384	457	705
	d	0.17	0.18	0.21	0.13	0.18	0.29	0.20	0.17	0.17
±0.10	Nc	15	42	39	38	40	34	25	54	61
	dz	0.66	0.39	0.42	0.50	0.42	0.58	0.56	0.35	0.42
	Np	111	102	76	282	107	68	96	114	176
	d	0.34	0.35	0.43	0.26	0.36	0.58	0.40	0.34	0.35
±0.25	Nc	2	7	6	6	6	6	4	9	10
	dz	1.65	0.98	1.06	1.26	1.04	1.45	1.39	0.87	1.05
	Np	18	16	12	45	17	11	15	18	28
	d	0.86	0.88	1.07	0.65	0.90	1.46	1.01	0.85	0.87

Note: Even if some of the sample sizes are smaller than 10 participants, the minimum suggested sample size for a group should not be less than that (Hopkins, 2000).

Abbreviations: L‐ACC, left anterior cingulate cortex; L‐S1, left primary somatosensory; R‐AI, right anterior insula.

TABLE 6

Hypothetical sample size calculation for crossover () and parallel () sustained deep tissue no‐pain (SDTnP) experiment design as a function of the effect size ( and )

Effect size (log10(CSD))		L‐ACC			L‐S1			R‐AI
		θ	α	γ	θ	α	γ	θ	α	γ
±0.05	Nc	106	160	215	119	83	156	103	255	74
	dz	0.26	0.19	0.24	0.27	0.26	0.20	0.28	0.16	0.38
	Np	454	372	378	1,351	210	258	592	479	152
	d	0.18	0.18	0.25	0.11	0.23	0.22	0.16	0.17	0.37
±0.10	Nc	26	40	54	30	21	39	26	64	18
	dz	0.52	0.39	0.47	0.54	0.52	0.41	0.56	0.33	0.76
	Np	114	93	94	338	52	64	148	120	38
	d	0.36	0.36	0.50	0.23	0.46	0.45	0.33	0.34	0.75
±0.25	Nc	4	6	9	5	3	6	4	10	3
	dz	1.31	0.96	1.18	1.34	1.30	1.02	1.39	0.82	1.90
	Np	18	15	15	54	8	10	24	19	6
	d	0.90	0.89	1.26	0.56	1.16	1.12	0.82	0.85	1.87

Note: Even if some of the sample sizes are smaller than 10 participants, the minimum suggested sample size for a group should not be less than that (Hopkins, 2000).

Abbreviations: L‐ACC, left anterior cingulate cortex; L‐S1, left primary somatosensory; R‐AI, right anterior insula.

TABLE 7

Hypothetical sample size calculation for crossover () and parallel () sustained resting EEG (REEG) experiment design as a function of the effect size ( and )

Effect size (log10(CSD))		L‐ACC			L‐S1			R‐AI
		θ	α	γ	θ	α	γ	θ	α	γ
±0.05	Nc	81	177	99	74	18	253	173	278	165
	dz	0.35	0.29	0.27	0.33	0.66	0.17	0.20	0.19	0.18
	Np	435	475	323	721	74	703	692	812	385
	d	0.21	0.25	0.21	0.15	0.46	0.14	0.14	0.16	0.16
±0.10	Nc	20	44	25	18	4	63	43	70	41
	dz	0.70	0.57	0.54	0.66	1.33	0.34	0.39	0.38	0.35
	Np	109	119	81	180	19	176	173	203	96
	d	0.43	0.49	0.42	0.30	0.91	0.28	0.28	0.31	0.33
±0.25	Nc	3	7	4	3	1	10	7	11	7
	dz	1.75	1.43	1.34	1.65	3.32	0.84	0.98	0.95	0.88
	Np	17	19	13	29	3	28	28	32	15
	d	1.07	1.24	1.05	0.75	2.28	0.71	0.69	0.78	0.82

Note: Even if some of the sample sizes are smaller than 10 participants, the minimum suggested sample size for a group should not be less than that (Hopkins, 2000).

Abbreviations: L‐ACC, left anterior cingulate cortex; L‐S1, left primary somatosensory; R‐AI, right anterior insula.

DISCUSSION

The aim of the current study was to assess the variability and effect sizes of intracranial current source density (CSD) estimations in pain research. The first part of the study consisted of a systematic review that was performed to examine the frequency that human pain studies reported sample size calculation, estimates of effect size and measures of central tendency and dispersion for CSD quantities. The review revealed that none of the 46 included studies reported sample size calculations and only 7 out of 46 reported the parameters needed for future sample size calculations. The second part of the study analyzed the test–retest reliability of CSD estimations during controlled experimental conditions. Here, EEG data collected during resting, non‐painful, and painful cuff‐pressure stimulation from 21 healthy participants were analyzed. Results show that LORETA CSD estimations are equally reliable for within‐session comparisons among all conditions, ROIs and frequencies, although estimations during REEG at S1 in the alpha band showed less between‐session variability than the rest of estimations. Finally, hypothetical sample size calculations were derived using effect sizes observed in the literature as well as effect sizes proposed following the observed variation in the experimental data.

Systematic review of effect sizes and variability of CSD estimates in pain‐related studies

None of the 46 included studies from the systematic review reported sample size calculations. This is in agreement with a previous systematic review in EEG studies, reporting that published articles rarely reported sample size calculations (Larson & Carbine, 2017). Still, this is not an exclusive trend in EEG, since studies using other neuroimaging techniques, such as fMRI, lack in reporting sample size calculation (Guo et al., 2014). Even though there has been a small improvement in this respect in other fields of medicine in recent years (Castellini, Gianola, Bonovas, & Moja, 2016), this is still far from optimal and journals should suggest and implement guidelines for appropriate sample size calculations (Lee & Tse, 2017).

Since the first study using LORETA was published in 1994, there should be plenty of data available to properly describe CSD estimations from different brain regions, as well as the expected differences due to experimental paradigms or clinical conditions. However, less than 20% of the articles included in the systematic review reported the minimum parameters required for future sample size calculations involving CSD estimations. In contrast, another systematic review on scalp EEG found that half of the articles provided information about central tendency and dispersion of EEG scalp potentials (Larson & Carbine, 2017). These parameters are not only useful for planning future studies, but also help in the peer‐review process to determine if the study was well‐planned and adequately powered.

Furthermore, the majority of the articles that did not report these parameters only reported images of voxel‐by‐voxel statistical maps, showing the locations of statistically significant variation of CSD estimations and not the actual effect sizes. Using these reported ‐values or , , or ‐statistics is not enough to calculate sample size for future studies (Sullivan & Feinn, 2012). More importantly, reporting the test statistics does not help in the evaluation of experimentally or clinically relevant effect sizes, which are crucial for the design of future experiments.

Reliability of CSD estimates in controlled experimental conditions

Most reliability studies use a single measurement, often the intraclass correlation (ICC) and arbitrary cutoff thresholds to assess whether a measurement or method is reliable or not. However, a proper analysis of reliability is not as simple or straightforward (Bruton, Conway, & Holgate, 2000). In some cases, high ICC values may not necessarily reflect high reliability and vice versa (Biurrun Manresa et al., 2014). More importantly, ICC is a measure of relative reliability, but most researchers are actually interested in the absolute reliability of a measurement, with the key advantage that it can be interpreted in terms of the original measurement units. Finally, many authors suggest to study reliability in broader terms and not base the conclusion solely in one number as if reliability was a binary construct, that is, considering that “the measure is reliable/good or the measure is not reliably/poor” (Biurrun Manresa et al., 2014; Kottner et al., 2011; Rousson, Gasser, & Seifert, 2002). For this reason, reliability in this study was presented in terms of these experimental conditions and the amount of measurement error that is acceptable given the application.

The reliability of cuff pressure algometry

Controlled experimental conditions require that not only the outcomes are reliable, but also the experimental stimulus. Graphical analysis of Bland–Altman plots showed no clear evidence of heteroscedasticity in the between‐session cuff pressure algometry. In other words, the differences in pressure values do not seem to increase with increasing mean pressure values between sessions. There is a small systematic mean difference between the six sessions for the SDTnP and SDTP that might be related to a learning process, considering that volunteers are naïve to the stimulation in the first session but not in the second. Still, this observed bias in the SDTP is small (3 kPa approx.) compared to the mean (4.7%) and may be irrelevant for pain research (Polianskis, Graven‐Nielsen, & Arendt‐Nielsen, 2002). On the other hand, the bias in the SDTnP compared to the mean (10.8%) is more relevant and should be taken into consideration in between‐session experiments that involve similar cuff pressure stimulation. The specific reason for this effect is not clear. Previous work found systematic bias for the pain threshold between sessions of cuff pressure algometry applied in the forearm (Graven‐Nielsen, Vaegter, Finocchietti, Handberg, & Arendt‐Nielsen, 2015). One possibility is that volunteers become familiar with the stimulation and therefore less anxious about the experiment. Some authors suggest that at least one practice session is necessary to overcome this carry‐over effect (McEvoy, Smith, & Gevins, 2000). Nevertheless, the main measure in this study was the EEG data during constant stimulation, which elicited a sensation of 3 and 7 in the VAS scale in both sessions, and both were achieved in the present study.

The reliability of CSD estimations

Bland Altman plots showed no clear signs of heteroscedasticity in the CSD estimation between sessions. This was expected because CSD data were log‐transformed, and in most cases, log‐transformation will address heteroscedasticity (Schmidt, Germano, & Milani, 2019). Further analysis of the plots revealed that there is no clear sign of systematic bias during SDTP, SDTnP, and REEG (except in the AI during SDTP and the L‐ACC during SDTnP both in the gamma band). This is evident because the CI of the bias did not overlap the zero‐log (CSD) difference between sessions (except in the two aforementioned cases).

Overall findings revealed that the reliability of CSD estimations is largely independent of the selected ROIs. This is evident because the are not different among conditions, ROIs and frequencies. A subsequent analysis exposed higher mean CSD estimations and narrower during REEG in S1 in the alpha band, which leads to a better reliability (Olofsen, Dahan, Borsboom, & Drummond, 2014). There is no clear explanation of why alpha reliability is higher during resting at S1. As many researchers have reported, participants exhibited higher alpha activity in parietal and occipital regions during eyes‐open resting state (Barry & De Blasio, 2017; Chen, Feng, Zhao, Yin, & Wang, 2008; Pitchford & Arnell, 2019; Smith et al., 2020). Therefore, the higher signal‐to‐noise ratio in the alpha band at S1 may be to a large extent responsible for the higher reliability (Elkum & Shoukri, 2008). (Martín‐Buro, Garcés, & Maestú, 2016).

A few studies attempted to assess the reliability of CSD using the LORETA method during different conditions, nonetheless, their results may appear inconsistent (Cannon et al., 2012; Hansen et al., 2017; Segalowitz et al., 2010; Tenke et al., 2017). Whereas some studies report good to excellent reliability (Cannon et al., 2012; Hansen et al., 2017), others reported mixed results (Segalowitz et al., 2010; Tenke et al., 2017) depending on different factors, such as the ROI or the frequency band from which the CSD was estimated. Under this framework, it is not clear from the literature whether the LORETA method is reliable or not, or which are the causes of these inconsistencies. These inconsistences may not be fully related to methodological aspects of the experiments, but also in which manner studies report and interpret reliability. In most cases, researchers judge reliability based on arbitrary thresholds and a make a binary decision (reliable/not reliable) disregarding the width of the 95% CI of the estimations which again depends on the sample size. Furthermore, none of these studies examined heteroscedasticity, which is known to impact relative and absolute reliability (Brehm, Scholtes, Dallmeijer, Twisk, & Harlaar, 2012; Schmidt et al., 2019).

Methodological considerations

Pain is a broad concept, encompassing a large number of subcategories, for example, experimental or clinical, acute or chronic, nociceptive or neuropathic, and tonic or phasic, just to name a few. While our original intention was to focus on experimental tonic pain, the preliminary analysis of the literature revealed a scarcity of quantitative data in this area, revealing one of the large issues addressed in this study. Only by broadening the scope to include all articles in pain research were we able to find reports of effect sizes and variability of CSD estimations, mainly from exploratory studies of clinical pain. A related issue arose with regards to the ROIs from which quantitative data were reported: although it is well‐known that multiple areas in the brain (aside from those specifically mentioned in this study) contribute to pain processing, and this often affects neural oscillations in all frequency bands, measurable outcomes were often not reported in sufficient detail for these variables. In these regard, the purpose of the present article is not to present comprehensive results of CSD estimations from all possible brain areas and frequency bands during different types of pain, but instead to draw attention to current flaws in outcome reporting in the field, and to provide reliable (if limited) data on effect sizes and variability of CSD estimations from the literature and the present experimental findings, in order to contribute to the proper planning of future research designs using these tools.

Although there are other CSD estimation methods, such as those based on minimum norm estimates (Grech et al., 2008), we chose to focus on LORETA for the systematic review and experimental analysis because it was by far the most commonly used. Furthermore, eLORETA was used to estimate the CSD in the L‐ACC, L‐S1, and R‐AI. Even though this technique has no localization bias (Pascual‐Marqui, 2007), the spatial accuracy is highly dependent on the number of electrodes and whether individual MRI data or a template head model is used (Michel et al., 2004; Michel & Brunet, 2019). A whole‐head, dense‐array sampling (e.g., 256 channels), the individual MRI of each volunteer and the co‐registration of the spatial locations of the electrodes will obtain a more accurate source localization. In this study, however, a 64‐channel EEG amplifier and a standard head model were used. This approach reduces the complexity of the experiment and avoids measuring the electrode positions on every volunteer. Nonetheless, variability due to anatomical differences and the real electrode positions were not considered, and this certainly increases spatial variability. Additionally, participants were not presumably consuming any drugs interfering with EEG activity (in accordance with the inclusion criteria), but this cannot be verified with absolute certainty. Finally, the sample in this experiment was not gender balanced. Gender differences have previously been observed in resting state EEG (Shankman et al., 2011) and CSD estimations (Stewart, Bismark, Towers, Coan, & Allen, 2010); whereas reliability is unlikely to be affected by differences, authors are encouraged to take this into consideration for future experimental designs involving effect sizes of CSD estimations.

Considerations for future research designs

Over the last decade, there was an increased concern about the “crisis of unreplicable research.” This crisis is not entirely about replication failures, but also due to the misinterpretation of statistical inferences (Amrhein, Trafimow, & Greenland, 2019). Several difficulties arise when researchers base their conclusions solely on arbitrary thresholds such as . For example, ‐values smaller than 0.05 may arise from random variation, and could be considered similar to from a practical perspective, in which the only difference might be the sample size (Halsey, Curran‐Everett, Vowler, & Drummond, 2015). Thus, a small sample size might negatively impact the quality of the conclusions that can be drawn from an analysis, particularly when nonsignificant results are often obtained. Nonsignificant results are the consequence of (1) absence of real effect of the treatment/condition (true negative) or (2) a real effect that is too small to be detected with the current sample size (false negative, type II error).

However, it is not always adequate to increase the number of participants. Since increasing the number of participants will allow obtaining smaller ‐values even with small effects, it can be problematic in medicine where those small differences can be attributed to the natural variability among participant or to measurement error. To overcome this situation, it is important to have a proper estimation of the effect size of the treatment/experimental condition, compared to the variability of the measurement (i.e., the measurement error). Therefore, researchers suggest that inferences should not be based only on ‐values (Ioannidis, 2019), and the results and their implications should take into consideration the physiological and clinical relevance (Schober, Bossers, & Schwarte, 2018). In this context, the capacity of an experiment to truly identify a relevant effect is highly dependent on adequate sample size.

In this study, hypothetical sample size calculations revealed that the number of participants to sufficiently power the study seems to be independent of the conditions, ROI and frequency band. Following the rule of thumb for the interpretation of Cohen's effect size (Sawilowsky, 2009), results show that a small effect size () requires a large number of participants ( and ), a medium effect size () requires sample size of and and a large effect size () requires a sample of and . It is evident that for small effect sizes, especially in parallel designs, CSD estimations may not be suitable to truly detect a clinically relevant effect. A further revision of the 46 unique papers revealed that, in average, between 20 and 30 subjects were included per study (see Tables S2 and S3). Based on the results, these sample sizes are suitable to detect medium and large effect sizes in crossover studies and only large effects in parallel studies. From here on, it is up to researcher and clinicians to establish if these effect sizes are experimentally or clinically relevant in each particular setting, in order to properly plan their experiments or interventions before they are carried out.

Conclusions

None of the studies included in the systematic review of CSD estimations during pain reported sample size calculations, and less than 20% of them reported absolute measures of central tendency and dispersion, which are necessary to estimate effect sizes. Based on these data and our own experimental results, we determined that sample sizes commonly used in pain studies using CSD estimations are suitable to detect medium and large effect sizes in crossover designs and only large effects in parallel designs. The results presented here provide a comprehensive summary of the effect sizes that can be observed using LORETA in pain research, that can be used by clinicians and researchers to improve settings and designs of future pain studies.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

AUTHOR CONTRIBUTIONS

Juan Manuel Völker: conceptualization, methodology, software, analysis, investigation, writing—original draft, reviewing, and editing. Federico Gabriel Arguissain: conceptualization, methodology, writing—reviewing and editing. Ole Kæseler Andersen: conceptualization, methodology, writing—original draft, reviewing and editing, supervision, funding acquisition. José Biurrun Manresa: conceptualization, methodology, analysis, writing—original draft, reviewing and editing, supervision.

ETHICS STATEMENT

The present study was conducted according to the Declaration of Helsinki and was approved by the ethical committee of Northern Jutland, Denmark (N‐20170047).

Table S1 Articles found in the literature review, after assessment of eligibility (n = 46).

Table S2: Summary of systematic review for the articles with frequency‐domain analysis.

Table S3: Summary of systematic review for the articles with time‐domain analysis.

Table S4: Summary of quantitative data from the experimental procedure and the Bland–Altman analysis for each condition, at each region of interest (ROI) and frequency band.

Table S5: log10 of current source density (CSD) estimations during sustained deep tissue painful stimulation (SDTP), sustained deep tissue non‐painful stimulation (SDTnP) and resting state EEG (REEG), in three regions of interest: left anterior cingulate cortex (L‐ACC), left primary somatosensory cortex (L‐S1) and right anterior insula (R‐AI) at three frequency bands (theta, alpha and gamma).

Click here for additional data file.