Identifying Mental Disorders in Primary Care: Diagnostic Accuracy of the Connected Mind Fast Check (CMFC) Electronic Screen.

Primary care physicians (PCPs) often daily address diagnoses and treatment of mental disorders in their practices. The current study examined the Connected Mind Fast Check (CMFC), a two-tiered electronic screen, assessing six common mental disorders. The eight-item Initial Screen assesses possible symptoms, whereas SAM modules establish provisional diagnoses and areas of clinical concern. With 234 patients from five independent PCP offices, diagnostic accuracy was tested with the SCID-5-RV as the external criterion. Concerningly, many patients were unaware of their current mental disorders and comorbidities. The CMFC Initial Screen evidenced strong sensitivity, identifying with very few missing diagnoses. About two-thirds of provisional SAM diagnoses were confirmed with high specificities. Bipolar Disorder posed the most challenges at both tiers. Importantly, the suicide screen identified all patients with suicide plans and three-fourths with ideation. In general, the CMFC effectively identified provisional diagnoses, impairment, and potential suicidality.

Introduction

Globally, mental disorders continue to increase with an estimated 264 million people functionally impaired by depression alone (James et al., 2018). According to World Health Organization (WHO), most persons with mental disorders, including those in the USA, remain untreated in medical and psychiatric settings (Wang et al., 2007). The National Institute of Mental Health (NIMH, 2020) identified over 46 million adults that were diagnosed with at least one serious mental disorder. Diagnostic severity with its functional impairment must also be considered. For example, despite its low prevalence, 82.9% of persons with bipolar disorders were classified at the highest level of severity (Kessler et al., 2005a). Moreover, lifetime prevalences for four diagnostic categories are troubling: 28.8% for anxiety disorders, 24.8% for impulse control disorders, 20.8% for mood disorders, and 14.6% for substance use disorders (Kessler et al., 2005b).

The effects of mental disorders are far-reaching and marked with life-changing events, such as at greater risk for employment termination (Nelson & Kim, 2011; Olesen et al., 2013), victimization (Swartz & Bhattacharya, 2017), and suicide attempts (Brådvik, 2018). Financially, treatment costs associated with inpatient hospitalizations for serious mental disorders can exceed $5000 per week (Stensland et al., 2012). For patients, both access and cost barriers can substantially undermine the treatment process, that is, if they can afford to seek treatment at all (Andrade et al., 2014; Rowan et al., 2013; Russell, 2010).

Clearly, the prevalence, severity, and effects of mental disorders (e.g., Kroenke et al., 2007) underscore the compelling need to improve clinical methods for screening and assessment. From a multidisciplinary perspective, one valuable approach involves primary care physicians (PCPs), who may integrate mental health screens into their practices (Shedler et al., 2000).

Mental Disorders in Primary Care Settings

The majority of primary care practices specialize in family medicine, followed by general internal medicine and pediatrics (Petterson et al., 2018). In 2016, more than half (54.5%) of 883 million physician office visits in the USA involved PCPs (Rui & Okeyode, 2016) with many visits including mental health issues, such as depression and anxiety (Ashman et al., 2019; Olfson, 2016; Olfson et al., 1995). Primary care providers must often address a wide range of mental health concerns in their practices. More than three decades ago, Regler et al. (1978) insightfully recognized that primary care operated as the de facto mental health service for most patients (see also Kessler, 2005, 2005a, b, c). The need for mental health screening has become even more urgent in the last decade (Faghri et al., 2010; Kroenke & Unutzer, 2017; Weisberg et al., 2007).

This urgency is undoubtedly associated with patient access to care and the complex relationship between mental and physical health. Patients often experience a host of barriers, for example low income or limited insurance coverage, that prevent them from obtaining proper mental health treatment (Kohn et al., 2004; Rowan et al., 2013). Furthermore, research has generated ever-increasing evidence that poor psychological health is inextricably linked to physical illness (Pomerantz et al., 2009; Weiss et al., 2009). Regarding access, primary care settings often serve as the only point of contact for individuals experiencing mental health problems (Ross et al., 2015). Thus, PCPs—regardless of their specialty—play a crucial role in the diagnosis, treatment, and management of mental disorders (Phillips et al., 2011; WHO, 2018; Wittchen et al., 2003).

For physicians, assessing and treating behavioral health disorders may vary by type of diagnosis (Terry & Terry, 2019). For instance, a survey of primary care practices in the USA revealed most physicians felt generally more comfortable screening for depression (78%) and anxiety (70%; Beck et al., 2019). Conversely, more than half described little confidence treating patients with severe mental disorders, with similar concerns for bipolar and substance use disorders (Beck et al., 2019).

Diagnostic Challenges in Primary Care Settings

Primary care settings attempt to be responsive to patients’ expressed needs, but typically lack systematic methods for evaluating these needs, thus resulting in diagnostic errors. Two types of inaccuracies involve missed diagnoses (i.e., undetected diagnoses) and misdiagnoses (i.e., incorrect diagnoses) which remain as enduring problems in primary care settings. Classic US and European studies of major depression (Christensen et al., 2003; Löwe et al., 2004; Tiemens et al., 1999) illustrate this point; diagnosis of depression approximated a coin toss with missed diagnoses (54–60%) and misdiagnoses (48–54%; see Rogers & Shuman, 2005). In the last decade, undetected diagnoses have reached as high as 80% (Fernández et al., 2010). As a further complication, depression is sometimes correctly identified, but manic or hypomanic episodes are not recognized (Ghaemi et al., 1999; Zimmerman et al., 2008), leading to misdiagnoses of depression rather than bipolar disorders (Cerimele et al., 2019). Troublingly, Wittchen et al. (2003) described “diminishing halves” (p. 115) with about 50% being correctly diagnosed and just half of those receiving effective treatment.

Several factors substantially contribute to diagnostic inaccuracies in primary care settings. PCPs are understandably more focused on medical care and may overlook psychological distress, especially when faced with challenging medical conditions such as HIV (Asch et al., 2003), or for patients with multiple physical ailments (Grant et al., 2013; Loeb et al., 2012). Relatedly, time constraints during office visits may necessitate an almost exclusive focus on physical symptoms for patients with serious medical illnesses (Kroenke & Unutzer, 2017).

The intersectionality of mental and medical disorders can further contribute to overlooked diagnoses. For example, some medical conditions (e.g., dementia; Allan et al., 2014) mask depression and other mental disorders. Extensive comorbidity of mental disorders is more the rule than the exception (e.g., Pietrzak et al., 2012), further complicating diagnoses. Moreover, accurate diagnoses are also particularly difficult when patients lack insight and plausibly misattribute the physiological symptoms of mental disorders (e.g., muscle tension in GAD; Combs & Markman, 2014) to solely medical causes. As a final consideration, PCPs may understandably focus on the stated purpose of the visit without the time and resources to delve into other diagnostic matters (Russell, 2010). As discussed in the next section, time-efficient mental health screens may be used to assist PCPs by systematically checking for common mental disorders.

Approaches to Mental Health Assessment in Primary Care

By its very nature, primary care is characterized by effectiveness and efficiency. Toward this end, Spitzer et al. (1994) developed one of the first mental health screens for medical settings: The Primary Care Evaluation of Mental Disorders (PRIME-MD). Although the initial step was efficient (Spitzer et al., 1999), the follow-up interviews imposed substantial time demands on primary care staff. Moreover, its effectiveness varies with the level of severity for the designated mental disorders (Tamburrino et al., 2009).

Very brief screens may also help capture mental disorders and pressing clinical issues, such as suicidal thinking. Several self-report questionnaires, like the Patient Health Questionnaire (PHQ-9; Spitzer et al., 1999), were adapted from the PRIME-MD to balance the importance of standardization with the very real time restrictions in primary care (Kahn et al., 2004). Brief screens can also aid PCPs in monitoring the course and severity of a particular disorder (Kocalevent et al., 2013). Importantly, their implementation has become increasingly more common (e.g., Beck et al., 2019).

A review by Mulvaney-Day et al. (2018) identified 24 mental health screens that primarily focused on anxiety, depression, and substance use. The most popular measures were variants of the PRIME-MD with benefits of being brief and easily accessible (Lakkis & Mahmassani, 2015). However, paper-based screens require individual scoring by primary care staff and are often confined (e.g., the PHQ for depression) to a single diagnosis.

A Two-Tiered Electronic Approach to Mental Health Screening

The Connected Mind Fast Check (CMFC) is an electronic mental health screen developed in 2017 by Lehinger (see https://connectedmind.me/) and his multidisciplinary team including psychologists. Of practical importance, the CMFC is widely eligible for insurance re-imbursement for PCPs. The CMFC assesses six common mental disorders (Beck et al., 2019; Cerimele & Kern, 2017; Wittchen et al., 2003) frequently found in primary care settings: (a) Major Depressive Disorders (MDD), (b) Anxiety Disorders with a primary focus on Generalized Anxiety Disorder (GAD), (c) Bipolar Disorders, (d) Attention-Deficit/Hyperactivity Disorders (ADHD), (e) Somatic Symptom Disorders, and (f) Substance and Alcohol Use Disorders.

Similar to the PRIME-MD, the CMFC has adopted a two-tiered assessment process, which is administered entirely on a tablet. The Initial Screen constitutes the first tier with eight yes/no questions for the six disorders; patients with no affirmative responses do not warrant further assessment. Each disorder has at least on screening item; for the two most prevalent disorders (anxiety and depression), a second item was added to improve its screening accuracy. Any positive response on the Initial Screen immediately triggers the second tier with disorder-specific Standardized Assessment Modules (SAMs). They employ more comprehensive questions for identifying provisional DSM-5 diagnoses. The CMFC utilizes a continuous scoring algorithm for SAM modules. This prevents patients from being asked redundant questions. If responses warrant further consideration, then additional questions are provided. Thus, everything is standardized and consistently applied via the scoring algorithm. As a result, the CMFC has twin goals of accuracy and efficiency. Once completed, the CMFC also generates total scores and an interpretative report directly to the PCP’s assessment portal.

The CMFC has yet to be subjected to any peer-reviewed published research. Specifically, we recently (accessed on 7-9-21) searched for “Connected Mind Fast Check” and “CMFC” in the following data bases: Academic Search Complete, MEDLINE, PsychINFO, and Google Scholar. Although “CMFC” did occur, it was completely unrelated to mental health screens (e.g., Couples, Marriage, and Family Counseling). Moreover, a personal communication with its developer (Christian Lehinger, 7-9-21) failed to produce any unpublished reports. As subsequently described in the Current Study, the next step clearly involves independent research on the effectiveness of the CMFC with primary care patients.

Current Study

The primary objective was to evaluate the classificatory accuracy of the Initial Screen and SAMs with the Structured Clinical Interview for DSM-5 Research Version (SCID-5-RV; First et al., 2018) as the external criteria. Both sensitivity and specificity are important1 and are applied differently to Initial Screen (step 1) and SAMs (step 2). For step 1, sensitivity is highlighted for the Initial Screen to make sure most patients with a specific disorder are administered the appropriate SAM. For step 2, specificity is emphasized to ensure that few patients are provisionally diagnosed with a specific disorder that they do not warrant. The application of these steps serves to improve both sensitivities and specificities. In addition to the provisional disorders, the CMFC suicide screen was also tested for its classificatory accuracy.

As the second objective, the current study evaluated an ANOVA-based discriminant validity of the CMFC provisional disorders. Provisional diagnoses (i.e., present or absent) constituted the independent variables with symptoms of the disorder being the dependent variables. For evidence of discriminant validity, patients with specific CMFC provisional diagnoses were expected to exhibit much greater symptom severity (i.e., large effect sizes) than their non-diagnosed counterparts. Two methodological considerations were implemented to improve the rigor of this research component. First, regarding effect sizes, the common convention is to utilize Cohen’s benchmarks (e.g., 0.50 for medium effect size) for group differences in social science research. However, these benchmarks are far too lax clinical practice; for example a d of 0.50 has a 67% overlap between criterion groups (Cohen, 1988, p. 22). Instead, rigorous clinical standards were applied to Cohen’s ds (see Rogers, 2008, 2018): for moderate (d ≥ 0.75), large (d ≥ 1.25), and very large (d ≥ 1.50) effect sizes. Second, although it is customary to use a single source for external criteria, we strengthened the design by adding a second measure. The SCID-5-RV was complemented with a comprehensive mental health screen, the Psychiatric Diagnostic Screening Questionnaire (PDSQ; Zimmerman & Mattia, 2001), which has 21 items for depression and suicidality alone.

Method

Study Design

The current study complied with the Standards for Reporting Diagnostic Accuracy Studies 2015 (STARD, 2015). The STARD 2015 consists of a 30-item checklist to address methodological rigor and to improve transparency and completeness of reporting data in studies of diagnostic accuracy (Bossuyt et al., 2015). Occasionally, STARD 2015 requires specific wording which is carefully denoted in the text (e.g., “index test” for the measure being validated). The full checklist is included in the supplementary material.

The cross-sectional, prospective design evaluated diagnostic accuracy and comorbidity at five PCP settings. As the index test, the effectiveness of the CMFC Initial Screen and SAMs in identifying common mental disorders was tested using SCID-RV diagnoses as the diagnostic reference standard. As later described, diagnostic efficiency statistics (e.g., sensitivity) were used to assess diagnostic accuracy.

Participants

From an initial group of 258 participants, the final sample was comprised of 234 consecutive patients recruited from five independent PCP offices in the Dallas-Ft. Worth Metroplex. Data from 24 participants were excluded because of missing electronic data due to poor internet connectivity (n = 23) or active psychosis (n = 1; see Fig. 1). With the one noted exception, all other adult patients were able to participate in the study irrespective of medical or psychiatric conditions. There were no other inclusion or exclusion criteria. Scheduled to end in May 2020, in-person data collection were administratively discontinued six weeks early by the university due to Covid-19 restrictions. As a result, the projected number decreased from 300 to 258 participants in total and 234 in final sample This number is generally consistent with other published studies on the diagnostic efficiency of mental health screens (see Vilagut et al., 2016).

Fig. 1

Flow chart of participants and their DSM-5 diagnoses

Primary Measures

Connected Mind Fast Check (CMFC)

The CMFC (personal communication with Christian Lehinger on 2/23/2021) is an electronic mental health screen for six common DSM-5 disorders. As previously described, it is composed of an Initial Screen and SAMs that automatically generate provisional diagnoses and clinical concerns (i.e., “red flags”), such as suicidal thinking. The Initial Screen consists of eight “yes/no” questions. The number of items on SAM modules range from 11 for GAD to 27 for MDD. The total number of SAM items administered are determined by the continuous scoring algorithm. Finally, relevant details (e.g., bereavement for depression) are provided for provisional diagnoses as well as severity scores for each administered SAM. The CMFC can usually be completed within 15 min and is automatically scored at the PCP’s secure assessment portal.

Structured Clinical Interview for DSM-5 Disorders–Research Version (SCID-5-RV)

The SCID-5-RV (First et al., 2018) is a semi-structured interview administered by trained mental health professionals. It is widely considered the most rigorous diagnostic assessment approach (Hersen et al., 2007) with the most recent version (SCID-5-Clinical Version or SCID-5-CV) representing the best external standard for research and publication (Brodey et al., 2018). The SCID-5-RV implemented additional modules and refined the SCID-CV scoring by providing an intermediary rating (i.e., 0 = false/absent, 1 = subthreshold, 2 = threshold/true). The SCID-5-CV, and by extension the SCID-5-RV, has excellent internal consistencies (alphas > 0.80; Shankman et al., 2018), interrater reliability, and validity (Osorio et al., 2019).

Psychiatric Diagnostic Screening Questionnaire (PDSQ)

The PDSQ (Zimmerman & Mattia, 2001) is a 125-item self-report inventory designed to initially evaluate 13 DSM-IV disorders. Responses in a “Yes/No” format evaluate symptoms in the last 2 weeks and 6 months. The PDSQ scales have evidenced moderate to high internal consistency (mean α = 0.82; Zimmerman & Mattia, 1999, 2001), and good concurrent validity with the SCID (Zimmerman & Chelminski, 2006).

Ancillary Measures

Patient Health Questionnaire-9 (PHQ-9)

The PHQ-9 (Spitzer et al., 1999) is a 9-item screen of depressive symptoms experienced in the last two weeks. Severity is rated on a four-point scale (0 = not at all, 1 = several days, 2 = more than half the days, 3 = nearly every day). The PHQ-9 has demonstrated excellent internal consistency (α = 0.86; Kroenke & Spitzer, 2002), and criterion validity for depressed patients (Kroenke et al., 2001).

Patient Health Questionnaire-15 (PHQ-15)

The PHQ-15 (Kroenke et al., 1998) is a 15-item screen for the presence and impact of common physical symptoms. The negative effects of symptoms are rated on a three-point scale (0 = not at all, 1 = bothered a little, 2 = bothered a lot). The PHQ-15 has demonstrated adequate reliability in the general population (Kocalevent et al., 2013). However, it evidenced only modest sensitivity and specificity with somatic disorders (Ravesteijn et al. 2009).

Generalized Anxiety Disorder-7 (GAD-7)

The GAD-7 (Spitzer et al., 2006) is a seven-item screen for generalized anxiety with the same severity ratings as the PHQ-9. The GAD-7 evidenced good reliability and validity for anxiety in both general (Löwe et al., 2004) and clinical (Johnson et al., 2019) settings.

CAGE Adapted to Include Drugs (CAGE-AID)

The CAGE-AID (Brown & Rounds, 1995) is a 4-item screen related to alcohol and drug consumption, answered in a “Yes/No” format. It has moderate sensitivity and specificity (Brown & Rounds, 1995), and excellent concurrent validity (Basu et al., 2016).

Adult ADHD Self-Report Scale (ASRS-v1.1) Part A

The ASRS-v1.1 (Kessler et al., 2005a) Part A consists of a six-item screen of ADHD symptoms. Responses are rated for frequency during the last six months (0 = never, 1 = rarely, 2 = sometimes, 3 = often, 4 = very often). It has demonstrated good internal consistency and concurrent validity (Adler et al., 2012). It also has acceptable sensitivity for identifying possible ADHD cases (Hines et al., 2012).

Mood Disorder Questionnaire (MDQ)

The MDQ (Hirschfeld et al., 2000) is a “Yes/No” 15-item screen designed to assess symptoms consistent with Bipolar Disorder. “Yes” responses to #2 and six or more additional questions represents a “positive” screen, if at least moderate impairment is reported. The MDQ is more effective with Bipolar I Disorder (i.e., depression and mania) than Bipolar II Disorder (i.e., depression and hypomania; Dodd et al., 2009).

Procedure

To facilitate data collection, researchers visited each PCP office to administer the measures over the span of close to one year (April 2019 to March 2020). Later in data collection (July 2019), six traditional single-diagnosis screens were implemented (see Sect. Ancillary Measures) with the potential goal of prescreening patients for single diagnoses. Given the extensive comorbidity, however, this approach was not feasible, and no patients were excluded from the study.

Outpatients attending medical appointments learned about the study via flyers. Initial patient questions were answered by a designated staff person with IRB-approved training. Next, interested patients met with a researcher, typically in the following week. After written informed consent, participants were then individually administered the SCID-5-RV diagnostic sections, followed by the CMFC and PDSQ in a counterbalanced order. As mentioned, single-diagnosis screens were implemented for eight months of data collection. Following data collection, patients were compensated with $60 for their travel, wait, and test time, which was estimated at about 2 h.

STARD 2015 requires the following information be reported. To avoid any criterion contamination, researchers administering the SCID-5-RV interviews (diagnostic reference standard) did not have access to CMFC results (index test) until after data collection were completed. For the timing of data collection, SCID-5-RV and CMFC administrations occurred with no time delay or clinical interventions. For participants, no adverse reactions were experienced that might alter their results during test administration. For data entry, two research assistants were utilized who played no role in data collection. For diagnostic efficiency statistics, data for all participants were classified (i.e., no indeterminate data were excluded).

Data Analytic Plan

For diagnostic efficiency statistics, the minimum number was set at 10 for each corresponding disorder. This benchmark is consistent with past SCID research (Osório et al., 2019). Diagnostic efficiency statistics (Riley et al., 2015; Wong and Lim, 2011) included sensitivity, specificity, positive predictive power (PPP), and negative predictive power (NPP). As noted, sensitivity measured the proportion of patients with a particular disorder correctly classified by the CMFC; conversely, specificity identified the proportion of patients without a particular disorder correctly classified. In addition, PPP measured the likelihood of patients with a positive CMFC finding (Initial Screen or SAM) warranting the corresponding SCID-5-RV diagnosis, whereas NPP estimated the likelihood of patients with a negative CMFC finding not having the diagnosis which is sometimes referred to as “non-cases.”

Results

The final sample consisted of 149 female and 85 male patients with none self-identifying as gender non-binary or transgender. Patients averaged 47.10 years (SD = 16.43), ranging widely from 18 to 82. For self-identified ethnicity, most described themselves as European American (74.7%), followed by African American (11.0%), Hispanic American (9.4%), and Multiracial (4.9%). For marital status, the breakdown was 48.1% married, 30.9% never married, 16.3% divorced, and 4.7% other/unreported. Educational attainment varied considerably: 14.0% had less than high school, 58.8% graduated from high school, and 35.2% went beyond high school to earn at least an associate degree. Concerning past treatment for psychological issues, 42.3% had been treated for a specific mental disorder with a much smaller but appreciable percent (15.8%) being psychiatrically hospitalized.

Diagnoses and Comorbidity

In total, 171 patients (73.1%) warranted one or more of the SCID-5-RV diagnoses with considerable comorbidity being observed. As reported in Table 1, current diagnoses most frequently included GADs (25.6%) and MDDs (20.9%). For those diagnosed, comorbidity averaged 1.82 disorders. Because the study focused on 11 SCID-5-RV disorders, this average should be considered partial comorbidities, likely underestimating actual comorbidities. Interestingly, lifetime diagnoses showed a reverse pattern from current diagnoses, with MDDs (52.1%) outstripping GADs (34.2%).

Table 1

SCID-5-RV diagnoses and their partial comorbidities for primary care patients

SCID-5-RV diagnosis	Current (%)		Lifetime (%)
	Total	Comorbida	Total	Comorbida
Corresponding disorders
Major depressive disorder	20.9	95.3	52.1	83.6
Generalized anxiety disorder	25.6	95.3	34.2	96.5
Substance use disorder	14.1	95.3	14.1	95.3
Bipolar disorders	11.5	98.2	11.5	98.2
Somatic symptom disorder	6.0	98.2	6.0	98.2
Attention deficit/hyperactivity disorder	15.4	95.9	15.4	95.9
Related disorders
Persistent depressive disorder	5.1	98.8	16.7	98.2
Social anxiety disorder	10.3	97.7	10.3	99.5
Panic disorder	12.0	97.7	12.0	97.7
Specific phobia	10.3	96.5	10.3	96.5
Agoraphobia	1.7	100.0	1.7	100.0

aComorbid = Partial Comorbidity (i.e., limited to the 11 assessed disorders)

Patient-reported psychiatric histories revealed striking differences when compared to current SCID-5-RV diagnoses. For example, 42.1% of the currently diagnosed patients did not report ever having a mental disorder. Moreover, 48.5% lacked insight into co-occurring disorders (i.e., reported only one diagnosis). If confirmed in PCP medical records, missed diagnoses appeared to be very common in the treatment of primary care patients.

CMFC Diagnostic Accuracy

Six SCID-5-RV diagnoses were categorized as corresponding disorders (i.e., closely aligned with the CMFC SAMs): MDDs, GADs, Alcohol Use Disorders, Substance Use Disorders, Bipolar Disorders, Somatic Symptom Disorders, and ADHD. The remaining diagnoses were characterized as related disorders.

Initial Screen

The CMFC Initial Screen excluded from further consideration 61 participants who did not respond positively to any of its eight questions. As strong evidence of its effectiveness, 95.1% of those excluded did not warrant any of the six corresponding mental disorders.

As noted, the Initial Screen was intended to retain a high proportion of patients with possible disorders, which was largely achieved via sensitivities (weighted M = 0.89; see Table 2). For Bipolar Disorders, the sensitivity was only in the low moderate range. As a practical alternative, however, a positive screen for MDD could also be used clinically to trigger the Bipolar SAM because the MDD Initial Screen showed excellent sensitivity (0.94). With this simple step, the weighted M for sensitivities increases to 0.92.

Table 2

Diagnostic accuracy of the CMFC initial screen with current SCID-5-RV diagnoses in primary care patients

SCID-5-RV diagnoses	CMFC initial screen
	Prev.	Sens. [95% CI]	Spec. [95% CI]	PPPa	NPPa
Corresponding disorders
Major depressive disorder (MDD)	.21	.94 [.82–.98]	.65 [.57–.72]	.41	.98
Generalized anxiety disorder (GAD)	.26	.93 [.83–.98]	.63 [.56–.72]	.47	.96
Substance use disorder	.14	.80 [.60–.90]	.92 [.87–.95]	.70	.95
Bipolar disorder in MDD patients	.07b	.63 [.36–.84]	.79 [.60–.90]	.42	.89
Somatic symptom disorder	.06	1.00 [.73–1.00]	.78 [.72–.84]	.53	1.00
Attention deficit/hyperactivity disorder	.15	.94 [.80–.99]	.61 [.53–.67]	.38	.98
Related disorders
Persistent depressive disorder	.05	.92 [.60–1.00]	.55 [.48–.62]	.34	.96
Social anxiety disorder	.10	.92 [.72–.99]	.53 [.46–.60]	.33	.96
Panic disorder	.12	.79 [.59–.91]	.52 [.45–.59]	.29	.91
Specific phobia	.10	.68 [.45–.84]	.51 [.44–.57]	.25	.86
SCID-5-RV suicide criteriac
Thoughts of own death	.16	.75 [.54–.96]	.89 [.82–.96]	.62	.93
Suicidal ideation	.10	.75 [.61–.90]	.84 [.76–.92]	.54	.93
Specific suicidal plan	.02	1.00 [.73–1.00]	.80 [.71–.89]	.55	1.00

Prev. prevalence of the disorder, Sens. sensitivity, Spec. specificity, PPP positive predictive power, NPP negative predictive power

aTo standardize comparisons across disorders, PPP and NPP were calculated using 20% base rates

bThis is the overall prevalence of bipolar disorder; for MDD only, the prevalence is .32

cNo patients reported the actual intent to commit suicide on either the SCID-5-RV or CMFC suicide screen

The strong sensitivities are directly linked to high NPPs so that patients scoring below the cut score are almost always correctly classified as not having the corresponding disorders. With NPP ranging from 0.89 to 1.00 (see Table 2), weighted M of 0.97 is very high. In other words, the CMFC Initial Screen correctly classified patients as being without specific mental disorders (i.e., “non-cases”) in 97% of cases. Thus, most without current mental health concerns were quickly screened out easily in less than one minute.

For related disorders, Agoraphobia was intentionally omitted from Table 2 because of its very low prevalence (1.7%). For the remaining disorders, the NPPs yielded high estimates ranging from 0.86 to 0.96 with excellent sensitivities for two disorders (Persistent Depressive Disorders and Social Anxiety Disorders). Panic Disorder only evidenced moderate sensitivity, and the CMFC Initial Screen was least sensitive to specific phobias (0.68).

As an ancillary consideration, diagnostic accuracy of single-disorder screens was also assessed (see Supplementary Table S1). Regarding sensitivity, none of the single screen measures were considered optimal (i.e., ≥ 0.90). The CAGE-AID demonstrated the greatest sensitivity (0.86) and specificity (0.88). In contrast, the MDQ performed the worst with its sensitivity being no greater than chance (0.50), although its specificity was excellent (0.94). With the exception of the ASRS-v1.1 and MDQ, most screens illustrated moderate to strong specificity, and predominantly moderate to moderately strong sensitivity.

SAMs

As noted, SAMs are intended to maintain higher specificities with at least moderate sensitivities for CMFC provisional diagnoses. See Fig. 1 for the diagnostic outcomes for the index test (i.e., SAMs) and the reference standard (i.e., SCID-5-RV diagnosis). This goal was mostly achieved; specificities for corresponding CMFC disorders averaged 0.912 (see Table 3). Sensitivities ranged from low (0.45 for MDD) to high (0.93 for Somatic Symptom Disorders). Overall, approximately two-thirds (66.2%) of CMFC provisional diagnoses were confirmed with SCID-5-RV corresponding diagnoses. Somatic Symptom Disorder and GAD evidenced the strongest CMFC performance with excellent sensitivities (0.93 and 0.90, respectively) and moderately strong specificities (0.86 for both). ADHD and Substance Use Disorder exhibited low-moderate sensitivities; however, their specificities were moderately strong (0.86) and excellent (0.96), respectively. The Bipolar SAM again illustrated lowest sensitivity (0.50), though its specificity was excellent (0.97).

Table 3

Diagnostic accuracy of the CMFC standardized assessment modules with current SCID-5-RV diagnoses in primary care patients

SCID-5-RV diagnoses	CMFC SAMs
	Prev.	Sens. [95% CI]	Spec. [95% CI]	PPPa	NPPa
Corresponding disorders
Major depressive disorder	.21	.45 [.31–.60]	.93 [.88–.96]	.63	.87
Major depressive disorder (preliminary)b	.21	.73 [.54–.87]	.92 [.87–.96]	.70	.93
Generalized anxiety disorder	.26	.73 [.60–.84]	.89 [.83–.93]	.63	.93
Generalized anxiety disorder (preliminary)b	.26	.90 [.76–.96]	.86 [.79–.91]	.62	.97
Substance use disorder	.14	.67 [.48–.81]	.96 [.91–.98]	.79	.92
Bipolar disorder MDD patients	.32	.50 [.26–.75]	.97 [.82–1.00]	.81	.89
Somatic symptom disorder	.06	.93 [.64–1.00]	.86 [.81–.90]	.63	.98
ADHD	.15	.69 [.52–.83]	.86 [.79–.90]	.54	.92
Related disorders
Persistent depressive disorder	.09	.33 [.11–.65]	.93 [.89–.96]	.40	.82
Social anxiety disorder	.10	.42 [.23–.63]	.75 [.68–.80]	.29	.88
Panic disorder	.12	.32 [.28–.66]	.76 [.69–.81]	.32	.85
Specific phobia	.10	.42 [.23–.63]	.75 [.68–.80]	.29	.84

Prev. prevalence of the disorder, Sens. sensitivity, Spec. specificity, PPP positive predictive power, NPP negative predictive power

aTo standardize comparisons across disorders, PPP and NPP were calculated using 20% base rates

bFor future research, preliminary decision rules reduced specificity slightly (i.e., .01 and .02), but potentially increased sensitivity substantially (.28 and .27)

As a preliminary (i.e., not aprioristic) analysis, CMFC decision rules for SAMs were reviewed for the MDD and GAD. As a preliminary result, sensitivities were substantially increased for MDD (from 0.45 to 0.73) and GAD (from 0.73 to 0.90) while only minimally affecting specificity. Obviously, further validation would be required before implementing any changes.

As a strength, the SAM’s diagnostic accuracy appeared to maintain excellent NPPs from the Initial Screen while the PPPs were higher than the Initial Screen. Provisional diagnoses on SAMs classifying the majority of patients ranged from 54% (ADHD) to 81% (Bipolar Disorders in depressed patients). Successful exclusion of outpatients without a corresponding SCID-5-RV disorder was excellent, ranging from 89% (Bipolar Disorders) to 98% (Somatic Symptom Disorder).

In final consideration, SAMs for related disorders were also evaluated via diagnostic efficiency statistics (Table 3). All sensitivities were very low to low (0.18 to 0.42), which was expected because the CMFC does not specifically assess for these disorders, and thus, would not be expected to identify them as provisional diagnoses.

Suicide Screen

Clinical indicators of suicide on the CMFC span from thoughts about death to intended suicide. Fortunately, no patients intended suicide, and very few had a specific plan (see Table 2). Sensitivities ranged from moderate (0.75) for suicidal thoughts to perfect (1.00) for suicidal plans. Of critical importance, no patients with suicide plans were missed.

Differences in Symptom Severity with and without CMFC Provisional Diagnoses

PCPs are interested in whether those provisionally diagnosed have markedly greater symptom severity—thus becoming a treatment priority—than others. Where feasible, differences in symptom severity were tested with both the SCID-5-RV and PDSQ with large and comparable effect sizes providing very strong evidence of discriminant validity. The two most common disorders (see Table 4) clearly provided this evidence: GADs (ds of 1.66 and 1.79) and MDDs (ds of 1.64 and 1.58). ADHD also exhibited an extremely large difference (d = 2.16) in symptom severity (i.e., 66.9% more intense symptoms for those with than without ADHD). More generally, large effect sizes were observed for Somatic Symptom Disorders, Alcohol Use Disorders, and Substance Use Disorders. Despite challenges in diagnostic accuracy, Bipolar Disorders yielded a moderately large (d = 1.02) effect size. Thus, patients provisionally diagnosed with Bipolar Disorders by the CMFC are likely to be in considerable distress.

Table 4

Differences in symptom severity between CMFC preliminary diagnoses (presence and absence) on external measures of symptom severity (SCID-5-RV and PDSQ)

Symptom severity	Measure	CMFC preliminary diagnosis				F	d
		Present		Absent
		M	SD	M	SD
Major depressive disorders	SCID-RV	21.20	4.41	13.56	4.89	74.39	1.64
	PDSQ	11.03	4.43	4.10	4.34	76.50	1.58
Bipolar disorders	SCID-RV	21.95	6.56	16.24	4.47	15.09	1.02
Generalized anxiety disorders	SCID-RV	14.75	3.44	8.76	3.76	121.30	1.66
	PDSQ	7.68	2.51	2.66	3.07	135.10	1.79
ADHD disorders	SCID-RV	38.94	7.85	23.33	6.51	216.56	2.16
Somatic symptom disorders	SCID-RV	7.74	2.84	4.91	1.44	88.18	1.26
	PDSQ	2.41	1.42	0.97	1.18	48.11	1.10
Alcohol use disorders	SCID-RV	16.83	6.10	11.38	1.80	102.90	1.21
	PDSQ	2.13	2.02	0.20	0.70	109.93	1.28
Substance use disorders	SCID-RV	31.16	9.22	22.60	3.02	103.80	1.25
	PDSQ	3.53	2.15	0.31	1.00	186.15	1.92

For all F ratios, p < .001

Discussion

Overlooked Mental Disorders and Comorbidity in Primary Care

Primary care patients appeared remarkably unaware of their diagnoses related to mental health. Regarding psychiatric histories, more than 40% were unmindful of any mental disorders at any time, despite warranting at least one current SCID-5-RV diagnosis. Even those aware almost always lacked knowledge of their own comorbidity. This lack of patient insight sharply constrains the ability to communicate effectively with physicians (Combs & Markman, 2014), and thus receive needed treatment. Based on past and current results, PCPs are cautioned against relying solely on reported mental disorders by primary care patients. Over-reliance on patient reports may grossly underestimate the presence of many treatable conditions, which remain undiagnosed, and thus, untreated (e.g., see Wittchen et al., 2003).

The current study also found high rates of co-occurring disorders in primary care patients. Kessler (2005, 2005a, b, c) reviewed the highly consequential effects of psychological comorbidity in medical settings, including decreased physical health (Jansen et al., 2018), more complex case management (Loeb et al., 2012), and more costly medical services (Lerner et al., 2004). As previously noted, untreated mental disorders are also linked to additional medical illnesses (Bijl et al., 2003; De Hert et al., 2011) and exacerbation of existing medical disorders. Thus, undiagnosed mental disorders and overlooked comorbidity may create serious ripple effects on the overall health of primary care patients. Given the bidirectional relationship between mental and physical health (Glew & Chapman, 2016), further integration of mental health screening and treatment into primary care settings is needed (De Hert et al., 2011; Phillips et al., 2011).

PCPs have multiple options on how they implement mental health screens into their medical practices. There are clear differences in the comparative effectiveness of such screens (e.g., Henkel et al., 2004). Nonetheless, the take-home message is compelling: Any systematic screens for mental health issues are far superior to the absence of such methods. This point appears equally persuasive in light of missed diagnoses and misdiagnoses.

Effectiveness of the CMFC in Primary Care Patients

The current study evaluated the diagnostic accuracy of the CMFC Initial Screen and SAMs in identifying common mental health disorders for primary care patients. As noted, diagnostic efficiency statistics, strong sensitivities are recommended for preliminary screens so that potential cases are not missed, and higher specificities are indicated for more comprehensive measures to decrease misdiagnoses (Ren et al., 2015).

Utility of the CMFC Initial Screen and Suicide Screen

The CMFC Initial Screen worked exceptionally well in accurately identifying patients with specific mental disorders while missing very few “false negatives” (i.e., patients with the disorder not identified by the screen). In other words, the CMFC Initial Screen mostly achieved its intended goal to accurately screen-in patients and markedly reducing missed diagnoses.

In the current study, the CMFC Initial Screen performed comparably to and sometimes stronger than traditional screens (e.g., PHQ-9) used in primary care settings (Mulvaney-Day et al., 2018). For example, the Initial Screen for Somatic Symptom Disorder displayed perfect sensitivity. In contrast, the longer PHQ-15 for somatic symptoms revealed comparatively lower diagnostic efficiency statistics (Ravesteijn et al. 2009; Toussaint et al., 2020; Vroege et al., 2012). Similarly, the CMFC Initial Screen appeared to surpass the PHQ for depression (see meta-analysis by Manea et al., 2015), the GAD-7 for anxiety (Plummer et al., 2016; Zhong et al., 2015), and the ASRS–v1.1 for ADHD (Dunlop et al., 2018; Kessler et al., 2005c). The notable exception on the CMFC Initial Screen is for Bipolar Disorder, although its limitations are generally shared by all screening tools for this disorder (e.g., Miller et al., 2004, 2011). It evidenced more accuracy than the MDQ, the most widely used assessment for Bipolar Disorder in primary care (see Kilbourne et al., 2012).

Beyond sensitivity and specificity, the Initial Screen’s impressive NPPs also have important practical implications. The CMFC was able to successfully screen-out patients without a mental health disorder. For physicians, time is a very real concern during office visits (Kroenke & Unutzer, 2017). In less than one minute, the CMFC can aid PCPs in quickly ruling-out common mental disorders.

Finally, the CMFC suicide screen proved effective at identifying patients with varying degrees of suicidal ideation. The inclusion of suicide risk assessment can be critically important to early and effective management. Regrettably, some very brief screens have omitted suicide items (e.g., PHQ-2; Duewke et al., 2018) with the potential for tragic outcomes. On this point, a review of medical records in the USA revealed that nearly half of suicide decedents sought healthcare services within the month prior to their deaths (Ahmedani et al., 2014). As a related matter, screens may allow primary care patients to acknowledge suicidal thinking, perhaps easing the discomfort of directly disclosing suicidal ideation to healthcare professionals (King et al., 2017).

Provisional Diagnoses with the CMFC SAMs

As previously discussed, the diagnostic efficiency of the SAMs focused on increased specificity (Ren et al., 2015). As expected, the majority of the SAMs performed strongly regarding specificity with substantially improved PPP, meaning the diagnostic accuracy of identifying patients with a particular disorder increased. Moreover, excellent specificities and NPPs indicate the SAMs correctly ruled-out most patients without these mental disorders.

Beyond diagnoses per se, PCPs would benefit from clinical data that documents increased symptom severity frequently associated with psychological distress and impairment. For the CMFC SAMs, comparing data from two different evaluative methods (i.e., structured interview and written questionnaire) produced strong and generally consistent evidence of discriminant validity. On average, effect sizes are very large for both the SCID-5-RV (M d = 1.46) and PDSQ (M d = 1.53). Promisingly, the Bipolar SAM—the most challenging module—still demonstrated moderately large differences in symptom severity between patients with and without this provisional diagnosis. Generally, preliminary diagnoses on the SAMs effectively differentiated between patients experiencing varying level of distress.

Methodological Considerations

As a methodological strength, the current procedures preserved patient confidentiality and provided independent administrations to avoid any criterion contamination. With the sole exception—clearly noted on informed consent—the SCID-5-RV interviewer could break confidentiality for actively suicidal patients. Fortunately, this psychiatric emergency did not occur, and confidentiality was never broken.

A second strength of the current research is its reliance on aprioristic decision rules for CMFC provisional diagnoses. Thus, the resulting classifications are not influenced by current clinical findings. That said, two preliminary scoring rules for MDD and GAD should be rigorously evaluated in future research.

A basic limitation of the current cross-sectional design is that changes in clinical status could not be evaluated. It would be useful to know whether the CMFC would be effective at assessing improvements as well as deteriorations in common mental disorders. This research would allow current and past episodes to be examined separately. Moreover, in patients with stable disorders, a short-term follow-up could assess the stability of CMFC findings via test–retest reliability.

Methodologically, the design could be modified so that all the CMFC SAMs were administered to every participant. This modification would allow comorbidity to be examined more thoroughly on the CMFC. While test items are typically scored in a positive direction (i.e., the presence of a symptom), the absence of certain items may be instructive, which could be tested if all modules were administered. For example, a negative response to an item such as “I wish my life was more interesting and stimulating,” might help to rule-out Bipolar Disorders.

Future CMFC research could also consider its usefulness in other health care and professional settings, such hospitals. When faced with public health crises, such as Covid-19, CMFC online applications may be appropriate and timely. On this particular point, a free online CMFC version was created in response to Covid-19. Additionally, adapted versions could be empirically tested for school settings and institutions for higher education.

Concluding Thoughts

PCPs need efficient and effective tools, such as mental health screens, to accurately inform their busy and often time-constrained practices. The CMFC may play an indispensable role in addressing undiagnosed mental disorders with its automatic scoring and electronic generation of reports. With re-administrations, it may help track changes in symptom severity over the course of treatment. In addition, the CMFC suicide screen is effective at identifying patients with suicide plans. In closing, PCPs need resources such as mental health screens to treat the whole person addressing physical and mental health needs and the often-dynamic interactions between the two.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 14 KB)