## Abstract

Introduction:

Crowdsourcing is a method of data collection with possible benefits in assessing perceptions of mental illness in a large US population.

Methods:

The objective was to describe perceptions and trends of stigma surrounding mental illness in the United States using crowdsourcing. An online survey was conducted evaluating adults in the United States recruited via the online resource Amazon Mechanical Turk. Questions evaluated demographics and perceptions of mental illness. Survey data were adjusted for demographic variables and compared via logistic regression.

Results:

Respondents (n = 1422) were predominately 18 to 30 years of age (n = 743; 52.3%) and white (n = 1101; 77.4%). Over half reported an individual close to them had mental illness (n = 932; 65.5%), and more than one quarter (n = 397; 27.9%) reported having a current or previous mental illness. Non-whites were less likely to agree that: medications are effective (odds ratio [OR] 0.63); they would be comfortable around a coworker with mental illness (OR 0.66); and mental illness is inheritable (OR 0.74). They are also more likely to agree that mental illness is preventable (OR 1.49). Individuals reporting mental illness were more likely to agree that medications (OR 1.34; 95% confidence interval 1.03 to 1.74) and talk therapy (OR 1.46; 95% confidence interval 1.12 to 1.90) are effective. Those reporting some or no college were more likely to agree that the United States has good access to mental health treatment.

Discussion:

Crowdsourcing may be an effective way to obtain information regarding demographics, stigma, and mental illness. Personal experiences with mental illness, ethnicity, and educational level appear to continue to impact perceptions of mental illness.

Mental illness remains common across the United States. Data from the National Institute of Mental Health National Survey on Drug Use and Health1  indicated that 44.7 million US adults had a mental illness, and 43.1% of those received services in 2016. According to the National Alliance on Mental Illness,2  African, Hispanic, and Asian Americans used mental health services between half and one-third the rate of whites. In 2002, an estimated total direct and indirect societal cost of patients living with severe mental illness was greater than $300 billion yearly.3 Mental health care related expenditures ranked as high as the third costliest of medical conditions in 2006.4 Based on data collected by the Centers for Disease Control and Prevention5 in 2007, approximately 25% of adults with mental illness believed that others are sympathetic toward patients diagnosed with these conditions. The public's perceptions of mental health and mental illness have been studied in systematic reviews in regards to both individuals in public and in the workplace.6-9 The portrayal of mental illness in the media has also had an impact on the public's understanding of mental health and issues related to stigma.10,11 Reviews of university students described significant prevalence of illness, suggesting the need for both recognizing mental illness and improving education regarding stigma.12,13 Rusch and colleagues14 described 3 types of stigma: negative stereotypes, prejudice with negative emotional reactions, and discrimination. Considering prevalence of stigma that continues to the current day, it is imperative to continue to research these elements. A systematic review15 evaluated a number of health and medicine-based crowdsourcing studies. Their findings suggest that crowdsourcing can be a way to quickly and inexpensively produce a high-quality research project that can benefit many. Crowdsourcing is the process of obtaining information by soliciting contributions from a large number of people, typically from the internet. It has been defined as “an online, distributed, problem-solving, and production model that uses the collective intelligence of networked communities for specific purposes.”16 There are little data utilizing crowdsourcing as a primary instrument for assessing current stigma in mental health. Crowdsourcing may possibly be an untapped resource with a potential opportunity, which is one of the broad goals this study seeks to determine.16,17 The primary objective of this study was to explore characteristics of the US population regarding mental illness demographics using crowdsourcing. The secondary objective was to assess demographics that may impact perceptions, stigma, and knowledge of mental illness. ## Methods A pilot survey was developed by the researchers and imported into Amazon Mechanical Turk's (AMT; Seattle, WA) database (Table 1). Amazon Mechanical Turk is an online database that uses individuals to complete various online tasks, which can include responding to surveys, tagging items, or performing other tasks that cannot be completed using artificial intelligence. The electronic survey was posted on the internet through the AMT website and was available to individuals aged 18 to 65 in the United States. The project was approved by the Institutional Review Board at Sullivan University College of Pharmacy and was funded through an internal grant provided by Sullivan University. Due to the broader goals, the researchers developed the pilot survey to ask additional questions covering themes of cognitive and behavioral constructs in stigma.18 Areas covered included constructs regarding stereotyping, institutionalization, and mental health funding, and knowledge questions regarding treatment and disability. Survey items were independently evaluated by the institution's research committee members but not statistically validated prior to administration. Subjects were required to have an approval rating of at least 90% based on successful completion of previous AMT tasks, which is commonly used as a measure of reliability of workers. Prior to initiating the survey, respondents reviewed an informed consent document. Subjects were then given a 45-minute time limit to complete the survey and compensated US$0.50 for their participation, which is similar to other tasks available in AMT.

To ensure that subjects were engaged in the survey, an identical question regarding the percentage of Americans with depression was presented twice. The identity of AMT workers is anonymous with a randomized worker identification to exclude duplicate responses. Exclusion criteria were: incomplete survey submission, responders answering the duplicate question differently, and multiple submissions from the same responder. The first fully complete survey received was used in analysis.

Once collected, survey results were imported into an Excel® spreadsheet (Microsoft Office, Redmond, WA). Descriptive statistics were performed in Excel® for the primary objective. An independent biostatistician performed the statistical analyses of the secondary objective using SAS version 9.4 (SAS Institute, Cary, NC). The secondary objective was evaluated using ordered logistic regression models to quantify the strength of association while adjusting for variables. Models were adjusted for age, gender, reported close contact with mental illness, and presented as odds ratios (OR) and 95% confidence intervals (CIs). P values <.05 were considered statistically significant.

## Results

A total of 1872 responses were received between January 12 and 14, 2015. Of these, 450 were excluded from final analysis because of incomplete survey submission (n = 380; 20%), responders answering the duplicate question differently (n = 39; 2%), and multiple submissions from the same responder (n = 31; 1.6%). Of the responses received, 1422 (76%) were included in the final analysis. Demographic data are available in Table 2, frequency distribution of survey responses is found in Table 3, and statistical analyses of demographic influences are found in Table 4.

### Experience With Mental Illness

Over half of respondents reported an individual close to them had mental illness (n = 932; 65.5%), and more than one quarter (n = 397; 27.9%) reported having a current or previous mental illness.

### Disease Opinions

Participants were less likely to identify a mental illness as most dangerous, compared to the 81% who identified heart disease, colon cancer, or diabetes collectively as most dangerous. Anxiety was most often reported as least dangerous by participants (39%).

### Mental Health Status

Survey participants who reported having a mental illness were statistically more likely to agree with several statements, including: that medications (OR 1.34; 95% CI 1.03 to 1.74) and talk therapy (OR 1.46; 95% CI 1.12 to 1.90) are effective; they would feel comfortable around a coworker with mental illness (OR 2.05; 95% CI 1.56 to 2.69); and that mental illness is inheritable from your relatives (OR 1.50; 95% CI 1.14 to 1.96).

### Ethnicity

Non-white participants were more likely to report that mental illness is preventable (OR 1.49; 95% CI 1.18 to 1.88) and less likely to agree that medications are effective (OR 0.63; 95% CI 0.50 to 0.80). Non-whites were also less likely to report feeling comfortable around a coworker with mental illness (OR 0.66; 95% CI 0.52 to 0.84) and less likely to endorse that mental illness is inheritable (OR 0.74; 95% CI 0.58 to 0.95).

### Education

Compared to survey participants who completed 4-year college degrees or higher, survey participants who completed some college were more likely to report that people in the United States have good access to mental health treatment (OR 1.23; 95% CI 1.00 to 1.50) and less likely to agree that medication treatment is effective (OR 0.78; 95% CI 0.63 to 0.97). Participants who reported no college were also more likely to agree that people in the United States have good access to treatment (OR 1.52; 95% CI 1.11 to 2.06) and less likely to agree exercise can help treat mental illness (OR 0.54; 95% CI 0.39 to 0.74).

## Discussion

Stigma not only directly affects patients with mental illness, it may deter individuals from seeking help.15  Data from previous studies have also found significant differences in perceptions based on racial/ethnic identities.19  Other concerns to consider are that laypeople may be unable to recognize specific disorders or types of psychological distress.20  Studies examining and establishing themes with mental illness in the area of stigma is crucial.21  A survey by Yokoya and colleagues22  included 1085 respondents, and only 58.9% believed in the effectiveness of pharmacotherapy for depression. Given that disparities in perceptions and access to mental health among minority populations have been documented for many years, those disparities found within our survey are in line with previous experience in this field. In addition, a recently published study23  examining mental health literacy in college-aged males has reported higher degrees of education contributing to lower levels of stigma, which is similar to some of the associations found in this study.

Comparing this study's demographic data to 2010 US census data,24  gender and ethnicity appear similar, which gives credence to the hypothesis that crowdsourcing can provide a representative population. Participants in this survey were 46.1% female compared with 50.8% of the US population. Regarding ethnicity, 77.4% of subjects reported to be white, compared with 77.1% in census data. Asian (7.5%), black (6%), Hispanic/Latino (5.8%) compared to 4.8%, 12.6%, and 16.3% from census data respectively. It is more difficult to compare age groups as this study included respondents aged 18 to 65.

This study also supports the use of crowdsourcing to obtain data in an efficient way. Hundreds of responses were obtained from individuals over a short period of time with a relative low cost of approximately US$0.60 per response, including AMT fees. This is in line with suggestions from the systematic review by Ranard et al18 and incorporated areas they found to be lacking in crowdsourcing publications, such as demographics of the participating crowd. This method of collecting data may prove to be a cost-effective, high-quality, and under-utilized process that should continue to be investigated in the future. Based on findings, a follow-up study directed at a more targeted population, such as minority groups, may be of interest. It is well known that minorities are underrepresented in clinical research, and crowdsourcing may offer an alternative approach to access these populations that otherwise are underrepresented.25 The paucity of data that is available examining stigma in minority populations may be supported by this study. Cultural values of individuals may vary significantly, particularly from assumed cultural norms of whites in the United States.26 Others also support this strategy, as they found that AMT offers more demographically diverse samples than some traditional methods.27 Additional studies on minority groups may add valuable insights. This study has notable limitations. Respondents were inherently required to have access to the internet and have the ability to read and comprehend English. This method of data collection makes it difficult to assess those of low socioeconomic, educational, and/or English literacy level. This is evident by the increased percentage of respondents completing higher education compared to the US population. Additionally, the survey instrument itself contains wording that might be difficult to interpret. Specific areas involve the use of a non-validated or pretested measure. Responders were unable to ask to clarify a question that may not make sense to him or her, which could alter results. Also, data that required sophisticated adjustment for confounding variables and analysis of subgroups needed additional funding for biostatistician evaluation to confidently determine significance or lack thereof. Notable issues related to the specifics regarding remuneration can also be seen as limitations, as AMT are also cited as receiving US$0.10 hourly. Because of the study instrument and less-specific population examined, study findings across previously published data may be difficult to generalize. While other studies have examined cross-cultural beliefs regarding psychiatric illnesses, the differing constructs of the analysis make the data difficult to compare.28  Along those lines, there appears to be a gap in the literature regarding several points of discussion reported in this study. Furthermore, this study offered a specific compensation, therefore individuals who may not have been interested in the task or satisfied with the offered compensation may have declined to participate.

## Conclusion

Crowdsourcing may be a useful method of obtaining information regarding mental illness in the United States. These data indicate significant prevalence and awareness of mental illness, with varying perceptions of mental illness between white and minority populations. Factors that may influence perceptions of mental illness include personal experiences, ethnicity, and educational level. These data continue to describe a present force in US culture and a continuing concern. Future studies using crowdsourcing may be a practical and beneficial option in continued study of population perspectives.

## Acknowledgments

The research team would like to acknowledge Christina Pinkston, MS, from University of Louisville School of Public Health and Information Sciences, for assisting in statistical analysis of data.

## References

References
1
National Institute of Mental Health [Internet]
.
Bethesda (MD)
:
US Department of Health and Human Services
;
c2014 [updated 2017 Nov; cited 2018 Jul 26]
.
Any mental illness (AMI) among US adults
.
2
National Alliance on Mental Illness [Internet]
.
Arlington (VA)
:
National Alliance on Mental Illness
; c2015 [updated 2017 Jun; cited 2017 Jun 7]
.
Mental health by the numbers
.
3
National Institute of Mental Health [Internet]
.
Bethesda (MD)
:
US Department of Health and Human Services
; c2008 [updated 2008 Jun; cited 2017 Jun 7]
.
Annual total direct and indirect costs of serious mental illness (SMI) in 2002
.
4
National Institute of Mental Health [Internet]
.
Bethesda (MD)
:
US Department of Health and Human Services
; c2006 [updated 2009 Jul; cited 2017 Jun 7]
.
Total expenditures for the five most costly medical conditions (1996 vs 2006)
.
5
Centers for Disease Control and Prevention [Internet]
.
Atlanta
:
US Department of Health and Human Services
; c2013 [cited 2017 Jun 7]
.
Stigma of mental illness
.
6
Schomerus
G,
Schwahn
C,
Holzinger
A,
Corrigan
PW,
Grabe
HJ,
Carta
MG,
et al.
Evolution of public attitudes about mental illness: a systematic review and meta-analysis
.
Acta Psychiatr Scand
.
2012
;
125
(
6
):
440
-
52
. DOI: . PubMed PMID: 22242976.
7
Clement
S,
Schauman
O,
Graham
T,
Maggioni
F,
Evans-Lacko
S,
Bezborodovs
N,
et al.
What is the impact of mental health-related stigma on help-seeking? A systematic review of quantitative and qualitative studies
.
Psychol Med
.
2015
;
45
(
1
):
11
-
27
. DOI: . PubMed PMID: 24569086.
8
Mehta
N,
Clement
S,
Marcus
E,
Stona
A-C,
Bezborodovs
N,
Evans-Lacko
S,
et al.
Evidence for effective interventions to reduce mental health-related stigma and discrimination in the medium and long term: systematic review
.
Br J Psychiatry
.
2015
;
207
(
5
):
377
-
84
. DOI: . PubMed PMID: 26527664.
9
Hanisch
SE,
Twomey
CD,
Szeto
ACH,
Birner
UW,
Nowak
D,
Sabariego
C.
The effectiveness of interventions targeting the stigma of mental illness at the workplace: a systematic review
.
BMC Psychiatry
.
2016
;
16
:
1
. DOI: . PubMed PMID: 26739960; PubMed Central PMCID: PMC4704270.
10
Clement
S,
Lassman
F,
Barley
E,
Evans-Lacko
S,
Williams
P,
Yamaguchi
S,
et al.
Mass media interventions for reducing mental health-related stigma
.
Cochrane Database Syst Rev
.
2013
;7:CD009453.
DOI: . PubMed PMID: 23881731.
11
Horsefall
J,
Cleary
M,
Hunt
GE.
Stigma in mental health: clients and professionals
.
Issues Ment Health Nurs
.
2010
;
31
(
7
):
450
-
5
. DOI: . PubMed PMID: 20521914.
12
Ibrahim
AK,
Kelly
SJ,
CE,
Glazebrook
C.
A systematic review of studies of depression prevalence in university students
.
J Psychiatr Res
.
2013
;
47
(
3
):
391
-
400
. DOI: . PubMed PMID: 23260171.
13
Amarasuriya
SD,
Jorm
AF,
Reavley
NJ,
Mackinnon
AJ.
Stigmatising attitudes of undergraduates towards their peers with depression: a cross-sectional study in Sri Lanka
.
BMC Psychiatry
.
2015
;
15
:
1
. DOI: . PubMed PMID: 26087847; PubMed Central PMCID: PMC4472246.
14
Rüsch
N,
Thornicroft
G.
Does stigma impair prevention of mental disorders?
Br J Psychiatry
.
2014
;
204
:
249
-
51
. DOI: . PubMed PMID: 24692749.
15
Ranard
BL,
Ha
YP,
Meisel
ZF,
Asch
DA,
Hill
SS,
Becker
LB,
et al.
Crowdsourcing—harnessing the masses to advance health and medicine, a systematic review
.
J Gen Intern Med
.
2014
;
29
(
1
):
187
-
203
. DOI: . PubMed PMID: 23843021; PubMed Central PMCID: PMC3889976.
16
Brabham
DC,
Ribisl
KM,
Kirchner
TR,
Bernhardt
JM.
Crowdsourcing applications for public health
.
Am J Prev Med
.
2014
;
46
(
2
):
179
-
87
. DOI: . PubMed PMID: 24439353.
17
Paolacci
G,
Chandler
J.
Inside the Turk: understanding Mechanical Turk as a participant tool
.
Curr Dir Psychol Sci
.
2014
;
23
(
3
):
184
-
8
. DOI: .
18
Corrigan
PW,
Shapiro
JR.
Measuring the impact of programs that challenge the public stigma of mental illness
.
Clin Psychol Rev
.
2010
;
30
(
8
):
907
-
22
. DOI: . PubMed PMID: 20674114.
19
Rao
D,
Feinglass
J,
Corrigan
P.
Racial and ethnic disparities in mental illness stigma
.
J Nerv Ment Dis
.
2007
;
195
(
12
):
1020
-
3
. DOI: . PubMed PMID: 18091196.
20
Jorm
AF.
Mental health literacy. Public knowledge and beliefs about mental disorders
.
Br J Psychiatry
.
2000
;
177
(
05
):
396
-
401
. DOI: . PubMed PMID: 11059991.
21
Mestdagh
A,
Hansen
B.
Stigma in patients with schizophrenia receiving community mental health care: a review of qualitative studies
.
Soc Psychiatry Psychiatr Epidemiol
.
2014
;
49
(
1
):
79
-
87
. DOI: . PubMed PMID: 23835576.
22
Yokoya
S,
Maeno
T,
Sakamoto
N,
Goto
R,
Maeno
T.
A brief survey of public knowledge and stigma towards depression
.
J Clin Med Res
.
2018
;
10
(
3
):
202
-
9
. DOI: . PubMed PMID: 29416578; PubMed Central PMCID: PMC5798266.
23
Rafal
G,
Gatto
A,
DeBate
R.
Mental health literacy, stigma, and help-seeking behaviors among male college students
.
J Am Coll Health
.
2018
;
66
(
4
):
284
-
91
. DOI: . PubMed PMID: 29419361.
24
US Census Bureau [Internet]
.
Washington
:
US Census Bureau
; c2015 [cited 2017 Jun 6]
.
Quick facts (2010-2015 data)
.
25
Aponte-Rivera
V,
Dunlop
BW,
Ramirez
C,
Kelley
ME,
Schneider
R,
Blastos
B,
et al.
Enhancing Hispanic participation in mental health clinical research: development of a Spanish-speaking depression research site
.
Depress Anxiety
.
2014
;
31
(
3
):
258
-
67
. DOI: . PubMed PMID: 23959771.
26
Abdullah
T,
Brown
TL.
Mental illness stigma and ethnocultural beliefs, values, and norms: an integrative review
.
Clin Psychol Rev
.
2011
;
31
(
6
):
934
-
48
. DOI: . PubMed PMID: 21683671.
27
Buhrmester
M,
Kwang
T,
Gosling
SD.
Amazon's Mechanical Turk: a new source of inexpensive, yet high-quality, data?
Perspect Psychol Sci
.
2011
;
6
(
1
):
3
-
5
. DOI: . PubMed PMID: 26162106.
28
Altweck
L,
Marshall
TC,
Ferenczi
N,
Lefringhausen
K.
Mental health literacy: a cross-cultural approach to knowledge and beliefs about depression, schizophrenia and generalized anxiety disorder
.
Front Psychol
.
2015
;
6
:
1272
. DOI: . PubMed PMID: 26441699; PubMed Central PMCID: PMC4561812.

Disclosures: None of the authors report competing interests.