The use of random-effects models to identify health care center-related characteristics modifying the effect of antipsychotic drugs.

PURPOSE: A case study was conducted, exploring methods to identify drugs effects modifiers, at a health care center level. PATIENTS AND METHODS: Data were drawn from the Schizophrenia Outpatient Health Outcome cohort, including hierarchical information on 6641 patients, recruited from 899 health care centers from across ten European countries. Center-level characteristics included the following: psychiatrist's gender, age, length of practice experience, practice setting and type, countries' Healthcare System Efficiency score, and psychiatrist density in the country. Mixed multivariable linear regression models were used: 1) to estimate antipsychotic drugs' effectiveness (defined as the association between patients' outcome at 3 months - dependent variable, continuous - and antipsychotic drug initiation at baseline - drug A vs other antipsychotic drug); 2) to estimate the similarity between clustered data (using the intra-cluster correlation coefficient); and 3) to explore antipsychotic drug effects modification by center-related characteristics (using the addition of an interaction term).
RESULTS: About 23% of the variance found for patients' outcome was explained by unmeasured confounding at a center level. Psychiatrists' practice experience was found to be associated with patient outcomes (p=0.04) and modified the relative effect of "drug A" (p<0.001), independent of center- or patient-related characteristics.
CONCLUSION: Mixed models may be useful to explore how center-related characteristics modify drugs' effect estimates, but require numerous assumptions.

Plain language summary

The generalizability of clinical trial results may be compromised by a selection bias toward key effect modifiers, at a patient- or health care setting level. However, very few studies explored how health care center-level characteristics may modify drugs’ effect estimates. In this case study, a mixed linear regression model was used to explore drugs’ effect modifiers, while also estimating the extent of variance in patients’ outcome explained by unmeasured confounding at a center level. Nearly, one-quarter of the variance at a patient level was explained by unmeasured confounding at a center level, emphasizing the importance of health care setting characteristics. The identification of drugs’ effect modifiers at a center level may contribute to fine-tuning trial design and identifying adequate recruiting centers.

Introduction

The pharmacological effect of drugs – referred to as the efficacy” – is best measured using randomized controlled trials (RCTs) conducted under highly controlled conditions. However, it has been acknowledged that RCTs do not always render drugs’ “effectiveness,”1 that is, their effect when delivered in real-world settings, by practicing staff, to broader and more representative populations.2,3 Effectiveness is being increasingly recognized to be of utmost importance for clinical or regulatory decision-making.4–6 Similarly, safety outcomes may be underestimated in RCTs compared to those measured in routine clinical practice.7 The chiasm, or gap, between the “efficacy” of drugs and their “effectiveness,” known as the “efficacy–effectiveness gap,” has been the subject of extensive study over the past decade.8 In particular, studies conducted on the subject aimed at better understanding why, and under which circumstances, RCTs may fail to reflect the impact of drugs on patients treated under routine clinical care and how to overcome this issue. Conceptually, effectiveness may be understood as a result of interactions between the drug’s pharmacological effect and characteristics related to patients (eg, age, gender, comorbidity, and genetics), the genuine use of the drug in routine clinical care (eg, dosage and drug adherence), and the setting of care (eg, access to care and health care cost coverage). From a statistical standpoint, it has been highlighted that population sampling errors occurring in RCTs, known to affect the distribution of important drug effect modifiers, are likely to have an effect upon the investigational drug’s effect estimates.9 Drug effect modifiers are factors (related to the patient, the drug use patterns, or the health care system) for which the effect of the drug is different depending on it being present or not, which is different from confounding. In other words, if an RCT includes a sample of patients who are not representative of the target population “owing to” important drug effect modifiers, then the drug’s effect estimates are likely to be biased. The influence of sampling errors involving drug effect modifiers, upon the risk for an efficacy–effectiveness gap, was evoked in previous publications.10,11 According to Huybrechts et al,12 “the relevant issue is whether the factors that distinguish studied groups from other groups somehow modify the effect in question.” In consequence, it seems important to explore and identify these effect modifiers ahead of designing the trial, so as to define the trial population that would best capture the effect modifiers for the drug under investigation. Of note, these drug effect modifiers may differ from one therapeutic class, or group of drugs, to another.

In the realm of the European public–private GetReal consortium,13 a series of case studies have been performed to explore methods aiming at identifying key effect modifiers of drugs at a patient level, and across several therapeutic areas: diabetes,14 Hodgkin’s lymphoma,15 and schizophrenia.16 These methods used a combination of systematic literature reviews, experts’ interviews, and data analyses (eg, data visualization, subgroup analyses, or regression modeling techniques). However, it is also possible that the effectiveness of drugs differs across health care settings and across countries. Several studies have explored to which extent the “country effect” was taken into account in RCTs, or which methods could be useful to explore this effect.17,18 Taking account of a between-setting or a between-country difference in drugs’ effectiveness raises numerous methodological questions. For instance, how should a health care setting or a country be characterized: as a unit, or as a set of explicit characteristics? Or what in the “country” may be the source of the difference – if any?

The current case study explored the methods to identify the effect modifiers, at a health care system level. Our objectives were to identify health care system characteristics potentially modifying the effect of antipsychotic drugs and to explore the methods to analyze this modification of effect.

Patients and methods

The study was performed using observational data, to identify the effect modifiers of antipsychotic drugs, at a health care system level. While exploring methods to achieve this goal, the objective of this study was neither to measure the relative effectiveness of antipsychotic drugs nor to generate results to be used outside the context of the current case study.

Data source

The European Schizophrenia Outpatient Health Outcome (SOHO) observational cohort dataset was made available to the authors for the purpose of this study. Data were de-identified, and no additional ethics review board or ethics approval was required for the conduct of the current study.

The rationale for using the SOHO database was that this is a wide cohort including observational information on physicians, patients and their condition, antipsychotic drugs prescribed, and patients’ outcome. In short, the SOHO study involved 1100 psychiatrists from across 10 European countries (Germany, Denmark, France, Greece, Ireland, Italy, the Netherlands, Portugal, Spain, and the UK), who recruited 10,218 outpatients with schizophrenia between 2001 and 2002. Participants were at least 18 years of age, required initiation or switch of antipsychotic drugs as per psychiatrists’ clinical judgment, and agreed to participate in the study. All data were collected prospectively by psychiatrists at baseline, 3 months, 6 months, and then every 6 months for 36 months. Further details about the methodologies used can be found elsewhere.19–21 For the current study, only baseline and 3-month data were used.

Data structure

Data were hierarchical: in each country, psychiatrists were required to recruit 10 patients each on average. We defined the “health care system” using the perspective of a patient who is cared for: a) by one psychiatrist (or the health care team in general), b) in a given health care setting (urban/rural, hospital/outpatient, public/private, etc), and c) in a given country where specific health care policies are applied (reimbursement policies, social security, availability in mental health care, etc). The sources of variability in care care (a, b, and c) were regrouped within the so-called center level, as opposed to the variability between patients (patient level). Data were categorized using the two levels of information: 1) “patient-level” data and 2) “center-level” data.

We hypothesized that patients taken care of in the same “center” (ie, same country and by the same psychiatrist) would form a “cluster,” meaning that they would tend to be similar in terms of treatment patterns and symptom evolution.

Variables and measures

Exposure to antipsychotic drugs

Antipsychotic drugs initiated at baseline were informed by physicians: 52.7% were prescribed olanzapine, 32.9% were prescribed another oral second-generation antipsychotic drug, and 6.7% of patients were prescribed a first- generation oral antipsychotic drug; the remaining patients were prescribed depot drug or a combination of different antipsychotic drugs.19,20

For the current study, exposure to antipsychotic drugs was defined as the initiation of (or switch to) an antipsychotic drug at baseline, regardless of how it is actually used afterward.

The brand/generic names of antipsychotic drugs were de-identified to preserve the noncompetitiveness of the GetReal project and replaced by letters (drug A, drug B, etc.). Patients initiating olanzapine at baseline were oversampled as per the study protocol. We excluded these patients from our analyses, first because many psychiatrists recruited only one patient initiating olanzapine (in this case, heterogeneity of patients’ characteristics was set at null for those psychiatrists). Moreover, we suspected that the choice of olanzapine prescription was driven by the study protocol and not by psychiatrists’ preferences. Including patients on olanzapine would have biased the results on a “cluster effect.” After excluding patients initiating olanzapine, our study population consisted of 4627 patients recruited by 899 psychiatrists.

Antipsychotic drugs initiated at baseline were divided into 1) the most frequently initiated drug (hereafter called “drug A”) and 2) “other” antipsychotic drugs (all other antipsychotic drugs – regrouped), so that groups of approximately equal size could be achieved. Exposure to antipsychotic drugs was dichotomous (drug A vs other).

Outcome

The severity of schizophrenia symptoms was measured by psychiatrists at baseline and 3 months using the Clinical Global Impression-Severity (CGI-S) scale,22 a single-item scale that rates global severity of symptoms between 1 (lowest severity) and 7 (highest severity).

For the current study, the outcome of interest for patients was the evolution of schizophrenia symptoms 3 months after the initiation of antipsychotic drugs (baseline), computed as the difference between CGI-S scores at 3 months and baseline (ΔCGI-S, continuous variable).

Potential confounders

All patient-level variables were considered a priori as potential confounders in any observed association between exposure to antipsychotic drugs and patients’ outcome. These variables included the following: patient’s gender, age, housing (independent or not), employment status, duration of illness (time elapsing since the first schizophrenia treatment), substance use disorder (SUD; no structured diagnosis tool was used to diagnose this disorder), number of psychiatric hospitalizations in the previous 6 months, baseline CGI-S score, and patient’s adherence to drug at 3 months as measured by psychiatrists using a single-item scale with four categories: 1) no antipsychotic drug prescribed in the prior 4 weeks; 2) good adherence: “the patient almost always adhered to antipsychotic drugs”; 3) moderate non-adherence: “the patient adhered about half of the time”; and 4) severe non-adherence: “the patient almost never adhered.”

Potential effect modifiers

All the center-level variables listed in the following paragraphs were considered a priori as potential effect modifiers of antipsychotic drugs.

Data collected on the psychiatrist included the following: psychiatrist’s gender, age, length of practice experience, health care practice setting (urban or rural location, as defined by the psychiatrist), and type of practice (private, public, or combined type of practice).

No information on the health care system implemented in each of the 10 European countries participating in this study was directly available in SOHO, except for the country of recruitment. We hypothesized that the performance of a country health care system could modify the effect of antipsychotic drugs. We searched in the literature and in websites of international or European health organizations (eg, World Health Organization [WHO], Organization for Economic Cooperation and Development [OECD], and the European Commission) to identify potentially relevant performance indicators of 1) the health care system in general (ie, not restricted to mental health care) and 2) the mental health care and with data available for the ten countries of interest in SOHO. Two performance indicators were identified and used. First, the Healthcare System Efficiency (HSE) score23 was used as a proxy for the global efficiency of the health care system. This indicator was derived from WHO data on life expectancy at birth (weighted 60%), relative (weighted 30%), and absolute (weighted 10%) cost of health care per capita; higher scores indicated a greater efficiency of the health care system. Second, the number of psychiatrists per 100,000 people, practicing in public or private settings as of 2011 (hereafter called “psychiatrist density”), was used as a proxy for the accessibility to mental health care.24 This indicator was identified in a report from the OECD.25 No other characteristic of countries was used; in particular, the reimbursement policy of each country was not considered because reimbursement policies are rather similar across western European countries. Table 1 summarizes all the variables used in the current study.

Table 1

Structure and origin of data used in the current study

Center level		Patient level
Information collected	Information source	Information collected	Information Source
On the psychiatrist	SOHO database	On the patient	SOHO database
Age		Age
Gender		Gender
Length of practice experience		Employment status
Health care practice setting		Housing conditions
Type of practice (private/public/combined)		Comorbid SUD
		Recent suicide attempt
On the health care system of the country	WHO, OECD	On the illness	SOHO database
HSE score		Duration of illness
		Severity of schizophrenia symptoms (CGI-S score)
Psychiatrist density		Number of hospitalizations in the previous 6 months
		On the treatment	SOHO database
		Antipsychotic drugs initiated at baseline
		Adherence to antipsychotic drugs at 3 months

Abbreviations: CGI-S, Clinical Global Impression-Severity; HSE, Healthcare System Efficiency; OECD, Organization for Economic Cooperation and Development; SOHO, Schizophrenia Outpatient Health Outcome; SUD, substance use disorder; WHO, World Health Organization.

Statistical analyses

Statistical analyses were carried out using the software R 3.2.3.26 Missing data (except for the outcome, CGI-S) were imputed using Multivariate Imputation by Chained Equations.27

Measure of effectiveness

A “core model” for relative effectiveness was built to explore effect modification: patients’ outcome (continuous, normally distributed variable) was computed as a linear function of antipsychotic drug initiation. Using this measure of relative effectiveness, we explored whether center-level variables acted as effect modifiers in the association between antipsychotic drugs and patients’ outcome.

To take account of a cluster effect at a center level, a mixed linear regression model was used, providing an estimate of the association between exposure to antipsychotic drugs and patients’ outcome, with a random intercept being the psychiatrist identifying number.28 The use of a mixed linear regression model provided the opportunity to 1) adjust for potential confounders (as in fixed-effect linear models)29 and 2) measure the relatedness of clustered data using the intra-cluster correlation coefficient (ICC) computed as the ratio between the within-cluster variance and the total variance (within clusters + between clusters).30 Practically, the ICC is the estimation of the unmeasured variance at a psychiatrist level.

Modeling

First, a multivariable mixed linear regression model was built to take account of patient-level potential confounders (as mentioned earlier) and also to adjust center-level variables (all variables listed earlier, except for psychiatrist’s age which was collinear with his/her length of practice experience). In this model, we identified which of the center-level characteristics were associated with patients’ outcome using a level of significance set at α=0.05.

Second, center-level characteristics found to be associated with patients’ outcome were tested against effect modification. For each center-level characteristic associated with patients’ outcome, the multivariable mixed linear regression model was refitted with the addition of an interaction term between the center-level characteristic and the antipsychotic drug. One model was fitted for each center-level characteristic found associated with patients’ outcome, and, virtually, a maximum of six models were to be built. The center-level characteristic was considered to be an effect modifier if the test on the interaction term returned a significant result (α=0.05).

Finally, in case of statistically significant interaction, the model was stratified by two levels of center-related characteristic, to explore how the strength of association between antipsychotic drug initiation and patients’ outcome was modified.

Results

The CGI-S score at baseline or 3 months was missing for 349 patients, who were therefore excluded, leaving 4278 (92.5%) patients for the analyses: at baseline, 1915 (44.8%) initiated drug A, while 2363 (55.2%) initiated “other” antipsychotic drugs (Figure 1).

Figure 1

Flowchart of patients included in the analyses.

Abbreviation: SOHO, Schizophrenia Outpatient Health Outcome.

Table 2 summarizes the characteristics of patients across the two therapeutic groups. Table 3 provides the health care system performance indicators of the 10 participating countries as well as the characteristics of the 899 participating psychiatrists by country of origin.

Table 2

Characteristics of the 4278 participants (at baseline and 3 months) by drug initiation group (drug A vs other antipsychotic drugs)

Baseline characteristics	Drug A (N=1915)	Other antipsychotic drugs (N=2363)
	n (%)	n (%)
Country of inclusion
Denmark	6 (0.3)	7 (0.3)
France	191 (10.0)	170 (7.2)
Germany	363 (19.0)	825 (34.9)
Greece	219 (11.4)	81 (3.4)
Ireland	6 (0.3)	15 (0.6)
Italy	564 (29.5)	763 (32.3)
The Netherlands	32 (1.7)	27 (1.1)
Portugal	26 (1.4)	30 (1.3)
Spain	466 (24.3)	385 (16.3)
UK	42 (2.2)	60 (2.5)
Male gender	1119 (58.4)	1076 (54.5)
Age (years), mean (SD)	40.1 (13.1)	40.4 (12.5)
Independently housed	868 (45.3)	1154 (48.8)
Employed	435 (22.7)	497 (21.0)
Duration of illnessa (years), mean (SD)	11.7 (11.4)	12.1 (10.8)
Alcohol use disorder	61 (3.2)	69 (2.9)
Other SUDb	52 (2.7)	56 (2.4)
Number of hospitalizations in the previous 6 months, mean (SD)	0.61 (1.7)	0.76 (1.9)
CGI-S score, mean (SD)	4.38 (0.99)	4.41 (1.02)
3-month characteristics	n (%)	n (%)
Patient always adheres to antipsychotic drugs	1687 (88.1)	2070 (87.6)
Adheres half of the time	192 (10.0)	223 (9.4)
Never adheres	31 (1.6)	59 (2.50)
No antipsychotic drugs prescribed	5 (0.3)	11 (0.5)
ΔCGI-S21, mean (SD)	−0.71 (0.94)	−0.69 (0.97)

Notes:

Computed as the difference between age of the patient and age at diagnosis

Including cocaine, LSD, cannabis, etc (except nicotine).

Abbreviations: CGI-S, CGI-S, Clinical Global Impression-Severity; SUD, substance use disorder.

Table 3

Characteristics of the 10 participating health care systems and the 899 psychiatrists involved in the recruitment of 4278 patients in SOHO

Characteristics	Overall	Germany	France	Spain	Italy	Greece	The UK	Portugal	The Netherlands	Denmark	Ireland
Health care system performance
HSE scorea	–	51.6	64.6	58.1	76.3	49.9	63.1	47.2	41.1	45.7	NA
Psychiatrist densityb	–	20.9	22.1	10.5	18.5	16.6	19.5	10.8	20.5	16.9	21.1
Psychiatrists	N=899, n (%)	N=291, n (%)	N=185, n (%)	N=146, n (%)	N=129, n (%)	N=49, n (%)	N=43, n (%)	N=23, n (%)	N=21, n (%)	N=6, n (%)	N=6, n (%)
Age (years), mean (SD) (years)	46.3 (6.8)	47.3(5.3)	45.5(6.9)	42.6 (6.4)	50.3 (7.3)	44.3 (6.8)	47.6 (7.1)	41.5 (4.6)	45.3 (6.6)	54.7 (9.7)	45.4 (4.6)
Male gender	670 (74.5%)	221 (75.9%)	126 (68.1%)	103 (70.5%)	116 (89.9%)	35 (71.4%)	36 (83.7%)	7 (30.4%)	18 (85.7%)	3 (50.0%)	5 (83.3%)
Clinical experience, mean (SD) (years)	15.5 (7.8)	14.2 (5.9)	15.4 (7.3)	13.7 (7.3)	23.4 (8.1)	10.2 (7.0)	18.1 (7.5)	10.6 (5.4)	9.8 (6.8)	13.2 (8.6)	17.0 (5.7)
Urban practice (vs rural)	777 (86.4%)	245 (84.2%)	170 (91.9%)	119 (81.5%)	116 (89.9%)	49 (100%)	28 (65.1%)	23 (100%)	19 (90.5%)	6 (100%)	2 (33.3%)
Practice type
Private only	110 (12.2%)	39 (13.4%)	30 (16.2%)	6 (4.1%)	13 (10.1%)	22 (44.9%)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)
Public only	379 (42.2%)	27 (9.3%)	107 (57.8%)	94 (64.4%)	75 (58.1%)	1 (2.0%)	36 (83.7%)	9 (39.1%)	21 (100%)	6 (100%)	3 (50%)
Combined	410 (45.6%)	225 (77.3%)	48 (25.9%)	46 (31.5%)	41 (31.8%)	26 (53.1%)	7 (16.3%)	14 (60.9%)	0 (0)	0 (0)	3 (50%)

Notes:

HSE score estimated from life expectancy and relative and absolute costs of health care per capita – source: Bloomberg Business and Financial Information.23

Expressed/100,000 people – estimation from 2011 (in Spain, this only includes hospital psychiatrists) – source: OECD.25

Abbreviations: HSE, Healthcare System Efficiency; OECD, Organization for Economic Cooperation and Development; SOHO, Schizophrenia Outpatient Health Outcome; NA, not available.

The results of the mixed multivariable linear regression model estimating the association between health care system characteristics and patients’ outcome are detailed in Table 4. The model explained 18% of the overall variance in patients’ outcome (R2=0.18), and the extent of shared variance among patients managed by the same psychiatrist was 23% (ICC=0.23).

Table 4

Association between symptom evolution at 3 months from baseline (ΔCGI-S) and health care system characteristics

	Mixed multivariable linear regression modela
	Regression coefficient (β)	95% CI	p-valuea
Initiation of antipsychotic drug “A” (vs “other”)	−0.043	(−0.096, 0.009)	0.105
Country-related characteristics
Psychiatrist densityb	−0.007	(−0.017, −0.003)	0.187
HSEc	−0.004	(−0.009, −0.001)	0.098
Psychiatrist-related characteristics
Male gender	0.017	(−0.078, 0.112)	0.719
Length of practice experience (years)	0.006	(0.0004, 0.012)	0.037
Urban care setting (vs rural)	0.046	(−0.070, 0.162)	0.439
Private practice alone (reference)	0	–	–
Public practice	−0.020	(−0.148, 0.108)	0.761
Combined practice	−0.024	(−0.146, 0.099)	0.703

Notes:

Model adjusted on patients’ characteristics: age (continuous), gender, employment, time elapsed since first psychotic symptoms (continuous), substance abuse disorder (cannabis, alcohol, and others), severity of symptoms at baseline, number of hospitalizations in the 6 months preceding baseline (continuous), and adherence to the antipsychotic drug initiated at baseline (as measured at 3 months).

Expressed in number of psychiatrists per 100,000 population.

Score calculated with life expectancy (weighted 60%), relative per capita cost of health care (30%), and absolute per capita cost of health care (10%) data. Results of the mixed multivariable linear regression model are provided as regression coefficient (β) and 95% CI

Abbreviations: CGI-S, Clinical Global Impression-Severity; HSE, Healthcare System Efficiency.

No association was found between patients’ outcome and any of the following: psychiatrist’s gender, health care practice setting, type of practice, HSE score, psychiatrist density, and antipsychotic drug initiated at baseline. Length of practice experience was found to be associated with patients’ outcome, with longer experience being associated with less favorable outcomes.

A significant interaction was found between the “length of practice experience” and the “antipsychotic drug” (β=0.009; 95% CI=0.003, 0.015; p<0.001). The multivariable model was then stratified by the length of practice experience (≤15 vs >15 years). The mean change in CGI-S was ΔCGI-S=−0.72 (SD=0.94) in patients managed by practicing psychiatrists with ≤15 years of experience, with a significant association found between patients’ outcome and drug A (vs other antipsychotic drugs) (β=−0.103; 95% CI=−0.180, −0.025; p=0.01). The mean change in CGI-S was ΔCGI-S=−0.68 (SD=0.97) in patients managed by psychiatrists with ≥15 years of experience, and the effect of drug A vs other antipsychotic drugs was not significantly different from 0 (β=0.028; 95% CI=−0.046, 0.101; p=0.46) in this group.

Discussion

In the current study, the large multinational and observational SOHO cohort was used to explore the effect modification of antipsychotic drugs, at a health care system level. The “relative effectiveness of drug A” (as compared to any other antipsychotic drug) was computed as a mean to elaborate on potential effect modifiers.

Methodological considerations

Choice of variables

The patients’ outcome was measured by treating psychiatrists using the CGI-S scale, a tool that is easily implemented in observational studies. Its inter-rater reliability is often considered good provided the physicians are adequately trained.31 Therefore, a classification bias (in this study, also confounding) cannot be excluded, particularly considering that younger psychiatrists might have overestimated the level of symptom improvement in their patients, as compared to more experienced psychiatrists. Moreover, the choice of antipsychotic drugs was left at the discretion of treating psychiatrists, and it is also possible that the modification of effect evidenced in our study reflected the fact that younger psychiatrists overestimated the effect of a switch to drug A, as compared to more experienced psychiatrists.

Regarding the health care system, we hypothesized a priori that patients’ outcome would be different, in different countries characterized by the “performance” of corresponding health care systems. In other words, we made the hypothesis that the performance of the health care system would act as an effect modifier of antipsychotic drugs. The first step in the case study consisted of identifying relevant and reliable “performance” indicators. Two indicators were identified: the density in psychiatrist and the global performance of the health care system. The rationale for using the psychiatrist density was that it could act as a proxy for the accessibility to mental health care.24 Previous studies in Europe and the USA have found an association between lower density of psychiatrists and higher suicide rates in the general population32,33 and in patients with schizophrenia.34 In the current study, no association was found between psychiatrist density across the 10 European countries participating in SOHO and symptom evolution in patients, in the short term. It is possible that no such association exists. It is also possible that the patients present in the database were actively followed-up by the medical staff, hence their recruitment into the cohort study. In this case, accessibility to mental health care was in fact not an issue for patients, and the psychiatrist density was not the relevant proxy to use in this study. For further studies aiming at exploring the impact of accessibility to mental health care at a country level, a possible solution would be to use electronic health care records, hence minimizing the impact of conducting an ad hoc study on patients’ sampling. Regarding the global performance of each country, no association was evidenced between global performance and patients’ outcome, and thus, no test of interaction was performed. Other health care system characteristics could also have been tested, following the results of previous studies which have investigated the association between characteristics of health care system and patients’ outcome. In the UK, higher rates of compulsory admission in mental health were found to be associated with a decrease in hospital bed provision over time35 and burnout of health care teams.36 In addition, higher suicide rates were found to be associated with greater turnover of nonmedical staff.37 In the current study, we did not explore these characteristics because data were lacking. Although only outpatients were recruited in the SOHO cohort, it would have been interesting to describe these aspects of setting of outpatient care (eg, staff turnover, nurse visits, and psychotherapy). For instance, a recent study suggested that following antipsychotic drug switch, schizophrenia symptoms tend to improve to a greater extent in patients who benefit from a psychosocial therapy, compared to patients not benefiting from psychosocial therapy.38

Regarding information on the psychiatrist, all variables were used with no hypothesis made a priori. An association was evidenced between psychiatrists’ length of practice experience and patients’ outcome, suggesting more favorable symptom evolution in patients managed by younger psychiatrists, independent of patient-level confounders. This result is in line with results of studies performed in nonpsychiatric39 and psychiatric settings.40 Choudhry et al39 suggested that physicians who have been in practice for longer possess less factual knowledge and are less likely to adopt new effective therapies or may be less receptive to new recommendations, thus explaining poorer patients’ outcome. Previous work from our study group evidenced that patient-reported satisfaction with care was higher in patients managed by younger psychiatrists, independent of patient-related confounders.41 Although not overlapping, the two findings might indicate that, on average, younger psychiatrists may adopt more up-to-date types of management in patients with schizophrenia (eg, shared decision-making and psychotherapy), owing to the fact that the management of schizophrenia has greatly evolved over the past decade.

First, the use of a large and multinational dataset including several levels of information (information on the health care system of a country, on the care setting for patients, and on the prescriber/psychiatrist) allowed us to explore a “cluster effect” in relation with the psychiatrist recruiting patients. Using a mixed model with a random intercept for psychiatrist identifying number revealed that unmeasured confounding at the psychiatrist level explained nearly one-quarter of the variance found for patients’ outcome. Although we cannot specify what exactly is at stake in the “cluster,” this result emphasizes the importance of the role played by health care centers (eg, the psychiatrist) beyond psychotropic drug prescription.

Second, in what concerns effect modification, our analyses were performed in the realm of linear regression models, and thus, using an additive scale. By using a multiplicative scale (eg, logistic regression), different results may have returned.

Third, although effect (measure) modification and interaction are not superimposable,42 we explored effect modification through stratification after showing statistical interaction (Wald test for interaction) for simplicity reasons. Regarding the length of practice experience, the Wald test for interaction (between the length of practice experience and drug A initiation) was statistically significant. This result cannot be generalized to other settings because the measure of “drug A effectiveness” was only used as a mean to explore effect modification and has no validity in itself. Our case study simply illustrates that not only participants should be representative of real-life patients but also the physicians should be representative of real-life prescribers, to ensure the external validity of study results. In the current study, however, the level of modification was not clinically relevant: the mean level of symptom improvement in patients managed by younger psychiatrists was ΔCGI-S=−0.72 (SD=0.94), with a significant effect of drug A (β=−0.103; p=0.010). In patients managed by older psychiatrists, the mean level of symptom improvement was ΔCGI-S=−0.68 (SD=0.97), with the effect of “drug A” being nonsignificantly different from 0 (β=0.028; p=0.457). This result highlights that statistical significance does not mean clinical relevance. In practice, it is not certain that including a biased sample of psychiatrists – owing to their length of practice experience – would alter the accuracy of antipsychotic drug effect estimates.

Conclusion

It is important for decision makers and clinicians to understand how the results of a trial performed across different health care settings and countries would translate into other health care settings and countries with different health care systems. In routine clinical practice, it is important to emphasize that the clinical outcome in a specific patient does not depend solely on the pharmacological effect of the drug prescribed but also on many other factors, related to the patient and health care setting.

To anticipate and take account of this heterogeneity, clinical trials should include information on the important drug effect modifiers at both patient and health care system levels. However, the current case study illustrates the fact that identifying effect modification at a health care system level is far from being straightforward. Effect modification has to be apprehended in the context of the drug (or class of drugs) being taken into account, the effect being measured (whether relative or comparative effectiveness), the primary outcome (whether continuous or dichotomous outcome), the type of model used, the related assumptions (whether additive or multiplicative risk), and the clinical relevance of findings, to name a few examples. Moreover, the identification of relevant health care system characteristics and their availability in databases are also important issues. Further studies are required to explore other methods and other types of data sources to build on knowledge on the identification of effect modifiers.