Introduction

There is growing support for the use of integrated measurement-based care to capture symptom data in real time so treatment providers can make informed decisions about intervention strategies for mental health problems, such as depression and anxiety, both of which are known to increase suicide. We examined the potential scalability and effectiveness of a mobile health (mHealth) application with integrated behavioral health functions to capture remote patient-reported measurement of suicidal ideation and overall symptoms of depression and anxiety.

Methods

This study was an observational retrospective review of deidentified patient data, including symptoms of suicidal ideation, depression, and anxiety as measured by the Patient Health Questionnaire-9 (PHQ-9) or Generalized Anxiety Disorder 7 (GAD-7) scale, which were administered to and completed by patients on a smartphone or desktop application. After controlling for age, sex, and the presence of moderate versus severe symptoms at baseline, mean scores were analyzed with the Student's t-test.

Results

Of patients who took the PHQ-9 assessment at baseline and 8 weeks later (n = 764), the proportion who endorsed suicidal thinking decreased from 25% to 14.66% (p < 0.001) over 8 weeks. The mean PHQ-9 score was reduced from 14.69 (standard error [SE], 4.09) to 10.50 (SE, 5.94; p < 0.001), and a subset of individuals who continued use and took the PHQ-9 again at 24 weeks (n = 185) had a further decrease to 9.03 (SE, 7.09, p < 0.01). Although 21.62% of this subset still had suicidal thinking, the frequency of suicidal thoughts decreased. Of patients who took the GAD-7 at baseline and 8 weeks (n = 797) the mean score decreased from 14.20 (SE, 3.31) to 10.08 (SE, 5.55; p < 0.001) at 8 weeks and to 7.48 (SE, 6.54; p < 0.001) for a subset (n = 278) who continued use and took a GAD-7 at 24 weeks (n = 278). The sum of subgroup samples is larger than the whole because of instances of comorbid depression and anxiety.

Conclusion

Remote assessments within technology-supported integrated behavioral health care were feasible at scale. Change in symptoms could be observed at the individual and group level in real time, which may allow clinical teams to adjust treatments and improve outcomes. Prospective controlled studies are needed to determine what factors contribute to reductions in symptom severity.

In the United States, 1 in 5 adults suffer from a mental health disorder. The lifetime prevalence of anxiety and depression are 20.6% and 30.1%, respectively, and are both known to increase the likelihood of suicidal thinking and suicide.[17] Suicide is the 10th leading cause of death, steadily increasing in occurrence, with 1.4 million suicide attempts in 2018, one in three of which were completed (488,344 deaths).[810] Despite the high morbidity and mortality of suicide, depression, and anxiety in the United States, approximately 35% of adults with depression and 63% of adults with anxiety do not receive treatment.[11,12] For those who do receive treatment, 54% of those with depression and 62% of those with anxiety do not receive adequate care.[13]

There are internal and external reasons for undertreatment. Among internal factors are the internalized stigma of behavioral health conditions, lack of cultural sensitivity embedded within the intervention,[14,15] the nature of the symptoms in and of themselves (e.g., apathy, hopelessness,[16,17] and fear), and distrust of treatment options, such as medication.[18,19] All these internal factors serve as barriers that reduce an individuals' desire to seek care. External factors include a shortage and uneven distribution of behavioral health care clinicians,[20,21] high out-of-pocket costs,[22,23] and difficulty with paying for medications.[24]

Behavioral health focuses on the connections between how behaviors impact someone's physical and mental health. Behavioral health care clinicians work to counsel and support patients with mental illnesses.[25] Among efforts to better address suicide, depression, and anxiety is the option of integrating behavioral health care into primary care. In the collaborative care model (CoCM),[26] the primary care physician (PCP) is responsible for screening, diagnosis, and care of patients' behavioral health problems. In this model, the PCP receives support from a behavioral health care specialist who is available to treat individuals determined to need a higher level of care, such as providing psychotherapy or cognitive behavioral therapy (CBT). The integration of behavioral health care into primary care in a collaborative manner has proven effective for people of all ages,[2730] and has the potential for markedly improving access to behavioral health care. Improved access can then lead to improved outcomes for anxiety, depression, and suicide.[3133]

An important aspect of CoCM is measurement-based care (MBC), which can be defined as the practice of basing clinical care on client data collected throughout treatment.[34] In MBC, structured patient-centered and validated measures of mental health symptoms, such as the patient health questionnaire-9 (PHQ-9) and generalized anxiety disorders-7 (GAD-7),[3537] are collected at regular intervals and used to adjust care plans as needed to improve client outcomes.[38] Multiple meta-analyses show that MBC significantly improves behavioral health outcomes[3941] resulting in recent calls to scale MBC.[42,43] These calls reflect the reality that despite the strong evidence supporting MBC, this approach is used by fewer than 20% of behavioral health practitioners.

Research shows widespread adoption has been limited by concerns about privacy, compliance with the HIPAA (Healthcare Insurance Portability and Accountability Act), and time to obtain and store paper/pen(cil) measurements at the individual patient/provider level.[42] Limited use has also been attributed to the idea that the most at-risk may not charge their phones or may not be motivated to check in with an online service.[44,45] Although MBC is used in some integrated behavioral health settings, there is little standardization of the delivery of assessments, treatment, and follow-up protocols and limited data on frequency of use.[46]

Digital tools represent one platform to scale solutions and improve accessibility to MBC. Retention rates vary amongst technologies because there are no standardized measures;[47] however, preliminary results from mobile health (mHealth) technology developed by Happify found a retention of 20% based on users completing a second PHQ-9 after 8 weeks.[48] In a mHealth platform developed by the Veterans Affairs, 41.6% of users returned to use the app 1 month after downloading, but only 28.6% after 3 months.[49] Some potential advantages of using smartphone apps for measurement of suicidal ideation, depression, and anxiety symptoms include that ability to utilize ecological momentary assessment (EMA), which is the real-time and real-world measurement of symptoms. The ability for patients to answer surveys at their convenience on their own device in their real-world environment may also decrease the Hawthorne effect (the tendency to behave or answer differently when under observation). A small prospective study demonstrated a strong positive correlation (r = 0.84) between smartphone-administered and traditional paper/pen(cil) PHQ-9 assessment.[50] When data are collected in a technology platform, it may be possible to evaluate data from much larger patient groups and aggregate patient data for the purpose of improving behavioral health care treatment and outcomes at both the individual and population level.

In this study, we sought to determine whether changes in suicidal ideation and symptoms of anxiety and depression could be observed from baseline to 8 and 24 weeks based on use of an mHealth technology platform. We analyzed whether symptoms of suicidality, depression, and anxiety could be effectively captured and if these symptoms changed over time controlling for age, sex, and whether baseline symptoms were moderate or severe. We expected that it would be feasible to collect data consistent with an MBC approach via an mHealth platform.

We conducted an observational retrospective review of deidentified patient records with the aim of identifying whether measurements of suicidal ideation and symptoms of depression and anxiety changed among individuals receiving mHealth-supported integrated MBC. On behalf of the authors, Advarra institutional review board determined that this research was exempt from institutional review board ethical oversight per the Department of Health and Human Services regulation 45 CFR 46.104(d)(4).

The NeuroFlow Platform

In this study, all measurements were collected on the NeuroFlow mHealth platform, which was designed to allow patients to complete data on their own devices. All measurements were securely stored in a HIPAA-compliant database. In the platform, a clinician only accesses their patients' information via a digital dashboard that integrates with the electronic medical record (EMR). The reporting system in the mHealth platform alerts clinicians to patients who are not progressing or who have declining behavioral health. Features of the mobile app include daily self-rating scales for stress, mood, sleep, and pain; mindfulness tools, general health education, and step tracking. The mHealth platform is aligned with the recovery model through the use of trackers and wellness content and by also being complementary to psychiatric care.[51,52] Among all mHealth app usage, 30-day user retention was 70% compared with a typical 30-day retention rate of 3.3% for behavioral health mobile apps.[48]

Data Collection

Individuals' deidentified records were included in this observational retrospective review if they were an adult between 18 and 64, and had also completed at least one mHealth-measured PHQ-9 or GAD-7 with a score of 10 or higher, indicating moderate to severe depression, between May 2018 and May 2020. The presence of suicidal ideation was defined as a score of 1 or more on Q9 of the PHQ-9 and frequency of suicidal ideation was defined as the total score for that question. Individuals' records were included if a second PHQ-9 or GAD-7 score was available approximately 8 weeks later (56 ± 7 days), and the individual had used the mHealth app during at least 4 weeks of that 8-week period. Those who had a third measure at approximately 24 weeks (168 ± 7 days) were included in the 24-week sample if they used the mHealth app during at least 12 of the 24 weeks. A 14-day time window was used because mHealth users are prompted to take a survey 7 days before the due date and may take it until 7 days after.

We gathered age, sex, dates, and scores of PHQ-9 and GAD-7 assessments, and the cumulative number of mHealth-platform activities completed at the time of the final test. Information on socioeconomic factors including income level, health insurance type, and the social constructs of race and ethnicity were not gathered because these data are kept by the provider and were not in the deidentified mHealth records.

Measures

The PHQ-9 is among the briefest and most easily administered rating scales for depression. Patients self-rate the presence of symptoms in the previous 2 weeks on a scale of 0 (not at all) to 4 (nearly every day). Scores range from minimal (0–4), to mild (5–9), moderate (10–14), moderately severe (15–19), and severe depression (20–27).[36] Question 9 (Q9) asks specifically about suicidal ideation and has been established as a predicter risk of suicide attempt and death by suicide.[3,4] The GAD-7 is a 7-item questionnaire with 89% sensitivity and 92% specificity for generalized anxiety disorder. On the GAD-7, each question is rated by the patient on a scale of 0 (not present) to 3 (present almost every day).[36] Clinically significant scores range from 5 to 9 for mild, 10 to 14 for moderate, and 15 to 21 for severe anxiety. The presence of anxiety disorders has been shown to predict increased risk of suicidal attempts.[1,2,53]

Statistical Analysis

The data retrieved were sent to an independent consultant who performed the statistical analysis. The proportion of people with suicidal ideation was calculated as the number of records with a score of 1 or more on Q9 of the PHQ-9 at each time point divided by the total number of records reviewed for that time point. For records in which an individual continued use for 24 weeks and completed a PHQ-9 at baseline, 8 weeks, and 24 weeks, the mean score on Q9 was considered to reflect the frequency of suicidal ideation.

For both the PHQ-9 and GAD-7, first scores and final scores were averaged to arrive at a mean baseline and mean final score on each measure for those who used the mHealth platform for 8 weeks, and the subset who continued use for 24 weeks. For each group, separate means were calculated for the following: (1) those who had moderately severe (PHQ-9 > 15) or severe (PHQ-9 > 20; GAD 7 > 15) depression or anxiety symptoms at baseline; and (2) those who had moderate (PHQ-9 = 10–14; GAD-7 = 10–14) symptoms at baseline. Student's t-tests were used to identify both the presence and quality of a difference between PHQ-9 and GAD-7 scores, as compared with baseline. A Student's t-test was employed given the small sample size. We used a prespecified 2-tailed paired Student's t-test to assess differences between mean baseline and final scores to evaluate for change in either direction. We used a paired test because both scores were available for every individual included in the study, creating a matched sample for the scores being compared.

We also carried out a prespecified analysis to evaluate whether any observed differences were affected by sex, age, or activity level by quartile after calculating the mean baseline and final scores for each sex, 3 different age ranges, and activity quartile. A prespecified 2-tailed unpaired Student's t-test was used to evaluate differences among these samples of different sizes and variations. A post-hoc analysis of activity level by age was conducted by merging and deduplicating all records reviewed for each measure and a 2-tailed unpaired Student's t-test was used to evaluate any differences in mean activity level by age. Results of the change in PHQ-9 and GAD-7 were analyzed was performed using Microsoft Excel on Windows 10. Results identifying age and suicidal ideation were analyzed using open source software: Python 3.7.0 (python.org, Python Software Foundation), Pandas 1.2.1 (pandas.pydata.org), NumPy 1.19.2 (numpy.org), and SciPy 1.6.0 (scipy.org). Figures were created in Matplotlib 3.3.2 (matplotlib.org).

We identified 4040 records in the mHealth platform database with a first-measured PHQ-9 score of 10 or more, suggesting moderate to severe depression, and 2783 records with a first GAD-7 score of 10 or more, suggesting moderate to severe anxiety. Comorbid depression and anxiety were present in approximately 27% of the records (1813/6823). We identified a subset of 1258 records with second scores for at least 1 of 2 measures recorded within 49 to 63 days of the first score (defined as the 8-week timepoint). Of these, 392 records had a third score recorded within 149 to 163 days of the first (defined as the 24-week timepoint). The rate of comorbidity in this subset was 24% (306/1258), similar to that of the overall record pool. There were 764 records with PHQ-9 scores at 8 weeks, and 185 of those had a 24-week score available. There were 797 records with an 8-week GAD-7 score, and 278 of those had a 24-week score available.

Demographics and Platform Activity

The only demographics maintained in the mHealth platform are age and sex (Table 1). All other demographics are kept by the provider/system the mHealth platform supports in providing MBC and were not included in the deidentified data.

Table 1

Demographic characteristics in records reviewed

Demographic characteristics in records reviewed
Demographic characteristics in records reviewed

We evaluated mean activity in the app over the 8-week period by age groups defined by decade of life (Fig. 1). Individuals under 20 years of age did more activities in the mHealth app compared with individuals over age 20 years, although this was statistically significant only for those with anxiety (p < 0.05), and the sample size for this group was small (n = 13). Individuals less than age 20 did more activities in the mHealth platform over the approximate 8 weeks at a mean 225 (SE, 38; p < 0.05 for all comparisons) for both those with anxiety (171 activities, SE, 30, p > 0.21) and for those with depression (p > 0.05).

Figure 1

Activities (mean + standard error) in the mHealth app by age group over 8 weeks.

*: p < 0.05; **: p < 0.04; +: comparison between ages 30 and 39 and 60+; #: comparison between ages 50 and 59 and 60+ groups.

Figure 1

Activities (mean + standard error) in the mHealth app by age group over 8 weeks.

*: p < 0.05; **: p < 0.04; +: comparison between ages 30 and 39 and 60+; #: comparison between ages 50 and 59 and 60+ groups.

People aged 20 to 59 years did a mean 99 to 137 activities (or 12–17/week) with no statistically significant differences seen (p > 0.06 for all comparisons).

Individuals aged 60 or older completed fewer activities (mean of 87 activities), but this reached statistical significance only in those with depression compared with individuals aged 30 to 39 or 50 to 59 with depression (p = 0.03). Mean activity levels for those with anxiety versus depression were similar in people under age 40 (p > 0.16), but after age 40, people with anxiety did significantly fewer activities compared with those with depression (p < 0.04). For those who had scores recorded at 24 weeks, the number of activities remained consistent over time (p > 0.12).

Suicidal Ideation

Change in the number of individuals endorsing or not endorsing suicidal ideation and the proportions with decreased scores on Q9 are shown in Figure 2A. Of the initial 764 records with PHQ-9 scores at 0 and 8 weeks, 25.0% (191/764) had scores > 0 on Q9. After 8 weeks of mHealth supported integrated behavioral health care, 14.7% (112/764; p < 0.001) had scores > 0. Of the subset (n = 185) who continued use for 24 weeks, 27.6% (51/185) had suicidal ideation at 8 weeks that was reduced to 21.7% (40/185; p < 0.05) at 24 weeks. Among 140 records with use for 8 weeks and scores > 0 on Q9 at baseline, the mean score decreased by 0.72 over the 8 weeks (p < 0.001). For the records with 24 weeks of use and > 0 on Q9 at baseline, there was no statistical difference from the larger group at weeks 0 and 8, and scores on Q9 dropped by a similar 0.74 over the first 8 weeks (p < 0.001) drop was similar. From weeks 8 to 24, this group had an increase of 0.27 in the mean score on Q9, although this was not statistically significant (p > 0.05).

Figure 2

(A) With use of the mHealth-supported measurement-based behavioral health care (MBC), the proportion of individuals with suicidal ideation decreased. (B) For those who had suicidal ideation at baseline and used mHealth MBC the frequency of suicidal ideation as measured by mean score on Q9 decreased. For those who had suicidal ideation at baseline and used mHealth MBC for 24 weeks, scores increased but this was not significant.

*: p < 0.05; **: p < 0.001.

Figure 2

(A) With use of the mHealth-supported measurement-based behavioral health care (MBC), the proportion of individuals with suicidal ideation decreased. (B) For those who had suicidal ideation at baseline and used mHealth MBC the frequency of suicidal ideation as measured by mean score on Q9 decreased. For those who had suicidal ideation at baseline and used mHealth MBC for 24 weeks, scores increased but this was not significant.

*: p < 0.05; **: p < 0.001.

Symptoms of Depression and Anxiety

Clinically significant reductions in depression and anxiety symptoms were seen over 8 weeks of mHealth-supported MBC, and improvements continued for those who had mHealth-supported MBC for 24 weeks (Figs. 3 and 4). Mean baseline PHQ-9 and GAD-7 scores were 14.69 (n = 764; standard error [SE], 4.09) and 14.30 (n = 797; SE, 3.31), respectively. At 8 weeks, mean PHQ-9 and GAD-7 scores were reduced to 10.50 (SE, 4.09; p < 0.001) and 10.10 (SE, 5.55; p < 0.001), respectively, just above the cutoff for mild depression or anxiety. For the subsets of people with depression (n = 185) or anxiety (n = 278) who had mHealth-supported MBC for 24 weeks, PHQ-9 and GAD-7 mean scores further decreased to 9.03 (SE, 7.09; p < 0.01) and 7.48 (SE, 6.54; p < 0.001), both indicative of mild depression or anxiety. Clinically significant improvements occurred whether people had severe anxiety (Figure 3B), severe or moderately severe depression (Figure 4B), or moderate depression or anxiety (Figs. 3C and 4C). There were no statistically significant differences among mean scores for males versus females or for different age groups in baseline, 8-week, or 24-week scores (p > 0.10 all comparisons).

Figure 3

With use of mHealth-supported measurement-based behavioral health care, clinically meaningful and statistically significant mean improvement occurred for people with moderate to severe depression symptoms (PHQ-9 ≥10) at baseline (A) and was not dependent on whether depression symptoms were moderately severe to severe (PHQ-9 ≥ 15 [B]) versus moderate (PHQ-9 = 10–14 [C]) at baseline. In this figure, solid circles represent the mean, the dashed line is the best fit linear regression, and the whiskers represent the standard error.

PHQ: patient health questionnaire.

Figure 3

With use of mHealth-supported measurement-based behavioral health care, clinically meaningful and statistically significant mean improvement occurred for people with moderate to severe depression symptoms (PHQ-9 ≥10) at baseline (A) and was not dependent on whether depression symptoms were moderately severe to severe (PHQ-9 ≥ 15 [B]) versus moderate (PHQ-9 = 10–14 [C]) at baseline. In this figure, solid circles represent the mean, the dashed line is the best fit linear regression, and the whiskers represent the standard error.

PHQ: patient health questionnaire.

Figure 4

With use of mHealth-supported measurement-based behavioral health care, clinically meaningful and statistically significant mean improvement occurred for people with moderate to severe anxiety symptoms (GAD-7 ≥10) at baseline (A) and was not dependent on whether anxiety symptoms were moderately severe to severe (GAD-7 ≥15 [B]) versus moderate (GAD-7 = 10–14 [C]) at baseline. In this figure, solid circles represent the mean, the dashed line is the best fit linear regression, and the whiskers represent the standard error.

GAD: Generalized anxiety disorder.

Figure 4

With use of mHealth-supported measurement-based behavioral health care, clinically meaningful and statistically significant mean improvement occurred for people with moderate to severe anxiety symptoms (GAD-7 ≥10) at baseline (A) and was not dependent on whether anxiety symptoms were moderately severe to severe (GAD-7 ≥15 [B]) versus moderate (GAD-7 = 10–14 [C]) at baseline. In this figure, solid circles represent the mean, the dashed line is the best fit linear regression, and the whiskers represent the standard error.

GAD: Generalized anxiety disorder.

Findings from this study indicate that it was possible to collect mental health data via an mHealth technology platform at scale. These results are promising given the importance of capturing symptom data in real time so treatment providers can make informed decisions about intervention strategies. Results also revealed that in patients with severe, moderately severe, or moderate depression or anxiety, clinically significant improvements were observed by using mHealth-supported MBC.

In this retrospective record review, we have shown that measurements can be obtained from large numbers of patients who remain engaged with the mHealth platform for up to 6 months. Of note, these measurements can also be used to observe changes in suicidal ideation and symptoms of depression and anxiety, which is important for implementing a key component of MBC, which is to adjust treatment in response to a patient's symptom improvement or deterioration. In addition, we found that overall symptoms of depression and anxiety decreased with mHealth-supported care, as did the presence and frequency of suicidal ideation as measured by Q9 on the PHQ-9. Frequency of suicidal ideation, however, did not change significantly from weeks 8 to 24, which may reflect the small size of the subgroup that continued use for 24 weeks. However, this lack of improvement could also reflect a higher likelihood that individuals who continue use of the platform past 8 weeks have suicidal ideation that does not, or is slower to, respond to treatment. Prospective studies are needed to further explore this possibility.

Increasing and improving the use of MBC in integrated behavioral health care is proposed as a means of addressing existing treatment gaps that result in close to two thirds of people with depression and anxiety not receiving best-practice care in the United States [29,30,31]. Measuring response to treatment is a critical part of helping individuals with depression and anxiety return to functional levels. Improvement to mild symptoms within 6 months of collaborative care (similar to what was seen in our record review of mHealth-supported MBC), correlates with an odds ratio of 12.3 for return to functional levels.[54] Of note, improvements were seen in both severe and moderate levels of anxiety and depression, suggesting that the benefits of mHealth-supported MBC in collaborative care may be helpful for people who are typically more likely to have treatment-resistant depression and anxiety.[55] Moreover, we saw response to treatment occurring within an 8-week period, supporting that response to treatment occurs more quickly with MBC, which was 5.6 weeks as compared with 11.3 weeks in a randomized clinical trial.[56]

With social distancing in response to the COVID-19 pandemic and a resulting shift to telehealth for most behavioral health care services, the opportunity to have patients remotely self-report suicidal ideation and symptoms of depression and anxiety in real-time, real-world settings provides many advantages. Moreover, recent literature has shown that underserved populations are ready and willing to use mHealth technology for care, as the majority of this population does not have a primary care physician and are more willing to engage with their care in a remote fashion.[57]

This study demonstrated that the mHealth platform has the potential to rapidly scale up the use of MBC within integrated behavioral health by overcoming the barriers of time for measurement, the Hawthorne effect, and concerns about confidentiality and HIPAA compliance. The mHealth platform leverages the recovery model to help and encourage users to stay in control of their lives[58] but also support individuals in the event they do return to a premorbid state. Digital platforms can help engage patients, provide ease of measurement, and enable physicians to receive data without having to devote clinical or staff time to obtaining the measurements. While it may be claimed to be difficult to deliver assessments and measurements outside of the clinic,[59,60] the NeuroFlow mHealth platform was created to lessen the burden on patient and provider. We observed high levels of engagement among mHealth platform users in this study, in which people of all ages used the app more than a dozen times per week.

Limitations

As a retrospective record review of only deidentified information recorded in the mHealth database, this study was not able to determine what factors other than receiving mHealth-supported integrated MBC contributed to self-reported reductions in symptoms. Questions regarding effects of pharmacotherapy and specific neuropsychological modalities within the digitally supported collaborative approach remain to be answered. Information on socioeconomic status, comorbidities, and social determinants that may have affected outcomes also was not available. All retrospective studies are limited by definition as post hoc studies. We worked to minimize these limitations via use of validated measurement scales and independent data review and analysis. However, it should be noted that the NeuroFlow platform was developed with a strong focus on user-centered design and uses engagement tactics including behavioral economics. Future studies should be performed on other platforms that use remote MBC to analyze for any generalizability. Despite these limitations, finding a sizable group of patients, selected only for a specific duration of app use, had clinically significant improvements that improve the odds of remission and recovery supports use of mHealth-supported integrated MBC.

This study lays the foundation for mHealth-supported MBC, showcasing that digital technology platforms can effectively capture data in a manner similar to manual pen-and-paper or in-person through the EMR. Our research supports the use of digital methods to gather measurements that are essential to improved care for depression and anxiety, which will reduce risks for suicide. Changes in symptoms, including suicidal ideation, can be reliably assessed with the mHealth platform. Remote monitoring may offer a cost-effective solution to deliver initial assessment of self-harm and suicide at scale of a large practice. Further research to understand how clinicians use information generated by the mHealth platform in the care they provide is needed, in addition to the patient experience of the mHealth platform. Additional research may consist of understanding how and to what extent crisis resources are used within the app, and how this might improve patient outcomes.

We acknowledge the support of VT4 (Philadelphia, PA), an independent consulting firm, for providing statistical analysis and review of the data and the manuscript in exchange for consulting fees.

1.
Hawgood
J,
De Leo
D.
Anxiety disorders and suicidal behaviour: an update
.
Curr Opin Psychiatry
.
2008
;
21
:
51
64
.
2.
Sareen
J,
Cox
BJ,
Afifi
TO,
et al
Anxiety disorders and risk for suicidal ideation and suicide attempts: a population-based longitudinal study of adults
.
Arch Gen Psychiatry
.
2005
;
62
:
1249
1257
.
3.
Ribeiro
JD,
Huang
X,
Fox
KR,
Franklin
JC.
Depression and hopelessness as risk factors for suicide ideation, attempts and death: meta-analysis of longitudinal studies
.
Br J Psychiatry
.
2018
;
212
:
279
286
.
4.
Rossom
RC,
Coleman
KJ,
Ahmedani
BK,
et al
Suicidal ideation reported on the PHQ9 and risk of suicidal behavior across age groups
.
J Affect Disord
.
2017
;
215
:
77
84
.
5.
DSM-5 Task Force.
Diagnostic and Statistical Manual of Mental Disorders.
5th ed. American Psychiatric Association Publishing;
2013
.
6.
Hasin
DS,
Sarvet
AL,
Meyers
JL,
et al
Epidemiology of adult DSM-5 major depressive disorder and its specifiers in the United States
.
JAMA Psychiatry
.
2018
;
75
:
336
346
.
7.
Any anxiety disorder.
National Institute of Mental Health website.
Accessed May 1,
2021
.
8.
Major depression.
National Institute of Mental Health website.
Accessed May 1,
2021
.
9.
Generalized anxiety disorder.
National Institute of Mental Health website.
Accessed May 1,
2021
.
10.
Murray
CJL.
The state of US Health, 1990-2010
.
JAMA
.
2013
;
310
:
591
.
11.
Generalized anxiety disorder (GAD).
Anxiety and Depression Association of America website.
Accessed May 1,
2021
.
adaa.org/understanding-anxiety/generalized-anxiety-disorder-gad.
12.
Facts & statistics. Anxiety and Depression Association of America website.
Accessed May 1,
2021
.
adaa.org/about-adaa/press-room/facts-statistics
13.
Alonso
J,
Liu
Z,
Evans-Lacko
S,
et al
Treatment gap for anxiety disorders is global: results of the world mental health surveys in 21 countries
.
Depress Anxiety
.
2018
;
35
:
195
208
.
14.
Pescosolido
BA.
Culture, children, and mental health treatment: special section on the National Stigma Study-Children
.
Psychiatr Serv
.
2007
;
58
:
611
612
.
15.
Chatmon
BN.
Males and mental health stigma
.
Am J Mens Health
.
2020
;
14
:
155798832094932
.
16.
Pompili
M,
Rihmer
Z,
Akiskal
H,
et al
Temperaments mediate suicide risk and psychopathology among patients with bipolar disorders
.
Compr Psychiatry
.
2012
;
53
:
280
285
.
17.
Serafini
G,
Gonda
X,
Canepa
G,
et al
Extreme sensory processing patterns show a complex association with depression, and impulsivity, alexithymia, and hopelessness
.
J Affect Disords
.
2017
;
210
:
249
257
.
18.
Gaddis
SM,
Ramirez
D,
Hernandez
EL.
Contextualizing public stigma: endorsed mental health treatment stigma on college and university campuses
.
Soc Sci Med
.
2018
;
197
:
183
191
.
19.
Corrigan
PW,
Mittal
D,
Reaves
CM,
et al
Mental health stigma and primary health care decisions
.
Psychiatry Res
.
2014
;
218
:
35
38
.
20.
Hoge
MA,
Morris
JA,
Stuart
GW,
et al
A national action plan for workforce development in behavioral health
.
Psychiatr Servi
.
2009
;
60
:
883
887
.
21.
Beck
AJ,
Singer
PM,
Buche
J,
et al
Improving data for behavioral health workforce planning: development of a minimum data set
.
Am J Prev Med
.
2018
;
54
(6 Suppl 3)
:
S192
S198
.
22.
Hodgkin
D,
Horgan
CM,
Stewart
MT,
et al
Federal parity and access to behavioral health care in private health plans
.
Psychiatr Serv
.
2018
;
69
:
396
402
.
23.
Thornicroft
G,
Chatterji
S,
Evans-Lacko
S,
et al
Undertreatment of people with major depressive disorder in 21 countries
.
Br J Psychiatry
.
2017
;
210
:
119
124
.
24.
Find support.
National Alliance on Mental Illness website.
Accessed May 1,
2021
.
25.
Centers for Disease Control and Prevention.
Mental health in the United States: prevalence of diagnosis and medication treatment for attention-deficit/hyperactivity disorder–United States, 2003
.
Morbidity and Mortality Weekly Report
.
2005
;
54
:
842
847
.
26.
Barry
CL,
Huskamp
HA.
Moving beyond parity — mental health and addiction care under the ACA
.
N Engl J Med
.
2011
;
365
:
973
975
.
27.
Asarnow
JR,
Rozenman
M,
Wiblin
J,
Zeltzer
L.
Integrated medical-behavioral care compared with usual primary care for child and adolescent behavioral health
.
JAMA Pediatr
.
2015
;
169
:
929
.
28.
Zivin
K,
Miller
BF,
Finke
B,
et al
Behavioral health and the comprehensive primary care (CPC) initiative: findings from the 2014 CPC behavioral health survey.
BMC Health Serv Res.
2017
;
17.
29.
Lauerer
JA,
Marenakos
KG,
Gaffney
K,
et al
Integrating behavioral health in the pediatric medical home
.
J Child Adolesc Psychiatr Nurs
.
2018
;
31
:
39
42
.
30.
Mental health care services by family physicians (position paper).
American Academy of Family Physicians website.
Accessed May 1,
2021
.
31.
Olfson
M.
The rise of primary care physicians in the provision of US mental health care
.
J Health Polit Policy Law
.
2016
;
41
:
559
583
.
32.
Campo
JV,
Geist
R,
Kolko
DJ.
Integration of pediatric behavioral health services in primary care: improving access and outcomes with collaborative care
.
Can J Psychiatry
.
2018
;
63
:
432
438
.
33.
de Saxe Zerden
L,
Lombardi
BM,
Jones
A.
Social workers in integrated health care: Improving care throughout the life course
.
Soc Work Health Care
.
2018
;
58
:
142
149
.
34.
Scott
K,
Lewis
CC.
Using measurement-based care to enhance any treatment
.
Cogn Behav Pract
.
2015
;
22
:
49
59
.
35.
Kroenke
K,
Spitzer
RL,
Williams
JBW.
The PHQ-9: validity of a brief depression severity measure
.
J Gen Intern Med
.
2001
;
16
:
606
613
.
36.
Spitzer
RL,
Kroenke
K,
Williams
JBW,
Löwe
B.
A brief measure for assessing generalized anxiety disorder
.
Arch Intern Med
.
2006
;
166
:
1092
.
37.
PHQ-9 Depression Scale.
University of Washington AIMS Center website.
Accessed May 1,
2021
.
38.
Lambert
MJ,
Harmon
C,
Slade
K,
et al
Providing feedback to psychotherapists on their patients' progress: clinical results and practice suggestions
.
J Clin Psychol
.
2005
;
61
:
165
174
.
39.
Shimokawa
K,
Lambert
MJ,
Smart
DW.
Enhancing treatment outcome of patients at risk of treatment failure: meta-analytic and mega-analytic review of a psychotherapy quality assurance system
.
J Consult Clin Psychol
.
2010
;
78
:
298
311
.
40.
Gondek
D,
Edbrooke-Childs
J,
Fink
E,
et al
Feedback from outcome measures and treatment effectiveness, treatment efficiency, and collaborative practice: a systematic review
.
Adm Policy Ment Health
.
2016
;
43
:
325
343
.
41.
Waldrop
J,
McGuinness
TM.
Measurement-based care in psychiatry
.
J Psychosoc Nurs Ment Health Serv
.
2017
;
55
:
30
35
.
42.
Lewis
CC,
Boyd
M,
Puspitasari
A,
et al
Implementing measurement-based care in behavioral health
.
JAMA Psychiatry
.
2019
;
76
:
324
.
43.
Fortney
JC,
Unützer
J,
Wrenn
G,
et al
A tipping point for measurement-based care
.
Psychiatr Serv
.
2017
;
68
:
179
188
.
44.
Vahabzadeh
A,
Sahin
N,
Kalali
A.
Digital suicide prevention: can technology become a game-changer?
Innov Clin Neurosci
.
2016
;
13
:
16
20
.
45.
Larsen
ME,
Nicholas
J,
Christensen
H. A
Systematic assessment of smartphone tools for suicide prevention
.
PLoS One
.
2016
;
11
:
e0152285
.
46.
Kearney
LK,
Wray
LO,
Dollar
KM,
King
PR.
Establishing measurement-based care in integrated primary care: monitoring clinical outcomes over time
.
J Clin Psychol Med Settings
.
2015
;
22
:
213
227
.
47.
Baumel
A,
Muench
F,
Edan
S,
Kane
JM.
Objective user engagement with mental health apps: systematic search and panel-based usage analysis
.
J Med Internet Res
.
2019
;
21
:
e14567
.
48.
Parks
AC,
Williams
AL,
Tugade
MM,
et al
Testing a scalable web and smartphone based intervention to improve depression, anxiety, and resilience: a randomized controlled trial
.
Int J Wellbeing
.
2018
;
8
:
22
67
.
49.
Owen
JE,
Jaworski
BK,
Kuhn
E,
et al
mHealth in the wild: using novel data to examine the reach, use, and impact of PTSD coach
.
JMIR Ment Health
.
2015
;
2
:
e7
.
50.
Torous
J,
Staples
P,
Shanahan
M,
et al
Utilizing a personal smartphone custom app to assess the patient health questionnaire-9 (PHQ-9) depressive symptoms in patients with major depressive disorder
.
JMIR Ment Health
.
2015
;
2
:
e8
.
51.
Davidson
L.
The recovery movement: implications for mental health care and enabling people to participate fully in life
.
Health Affairs
.
2016
;
35
:
1091
1097
.
52.
Jacob
K.
Recovery model of mental illness: a complementary approach to psychiatric care
.
Indian J Psychol Med
.
2015
;
37
:
117
.
53.
Bentley
KH,
Franklin
JC,
Ribeiro
JD,
et al
Anxiety and its disorders as risk factors for suicidal thoughts and behaviors: a meta-analytic review
.
Clin Psychol Rev
.
2016
;
43
:
30
46
.
54.
Angstman
KB,
Rasmussen
NH,
MacLaughlin
KL,
Staab
JP.
Inter-relationship of the functional status question of the PHQ-9 and depression remission after six months of collaborative care management
.
J Psychiatr Res
.
2013
;
47
:
418
422
.
55.
Culpepper
L,
Muskin
PR,
Stahl
SM.
Major depressive disorder: understanding the significance of residual symptoms and balancing efficacy with tolerability
.
Am J Med
.
2015
;
128
:
S1
S15
.
56.
Guo
T,
Xiang
Y-T,
Xiao
L,
et al
Measurement-based care versus standard care for major depression: a randomized controlled trial with blind raters
.
Am J Psychiatry
.
2015
;
172
:
1004
1013
.
57.
van Veen
T,
Binz
S,
Muminovic
M,
et al
Potential of mobile health technology to reduce health disparities in underserved communities
.
West J Emerg Med
.
2019
;
20
:
799
802
.
58.
Friedman
Y.
On recovery: re-directing the concept by differentiation of its meanings
.
Med Health Care Philos
.
2021
;
24
:
389
399
.
59.
Rodriguez
KA.
Maintaining treatment integrity in the face of crisis: a treatment selection model for transitioning direct ABA services to telehealth
.
Behav Anal Pract
.
2020
;
13
:
291
298
.
60.
Pincus
HA,
Spaeth-Rublee
B,
Sara
G,
et al
A review of mental health recovery programs in selected industrialized countries
.
Int J Ment Health Syst
.
2016
;
10
:
73
.

Competing Interests

Sources of Support: National Science Foundation Grant 1951062. Conflicts of Interest: Adam Pardes, William Lynch, Matthew Miclette, and Ellen McGeoch are employees of NeuroFlow, Inc. Brian P Daly is an investor in NeuroFlow, Inc.

This work is published under a CC-BY-NC-ND 4.0 International License.