TBM ORIGINAL RESEARCH

A computational study of mental health awareness

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
campaigns on social media
Koustuv Saha,1,* John Torous,2,*, Sindhu Kiranmai Ernala,1 Conor Rizuto,2 Amanda Stafford,2
Munmun De Choudhury1

1
School of Interactive Computing, ABSTRACT
College of Computing, Georgia As public discourse continues to progress online, it is Implications
Institute of Technology, Atlanta, important for mental health advocates, public health officials, Practice: Online social media campaigns offer
GA 30332, USA and other curious parties and stakeholders, ranging from popular forums for the public to share their
2
Division of Digital Psychiatry, researchers, to those affected by the issue, to be aware of the thoughts, advice, and information on resources
Department of Psychiatry, Beth about mental health.
Israel Deaconess Medical Center,
advancing new mediums in which the public can share content
Harvard Medical School, Boston, ranging from useful resources and self-help tips to personal
MA 02115, USA struggles with respect to both illness and its stigmatization.
A better understanding of this new public discourse on Policy: Automatically classifying Twitter-based
*Co-first authors.
mental health, often framed as social media campaigns, mental health content remains a challenge that
can help perpetuate the allocation of sparse mental health limits the use of this data to inform policy.
resources, the need for educational awareness, and the
usefulness of community, with an opportunity to reach those
seeking help at the right moment. The objective of this study
was to understand the nature of and engagement around Research: New collaborations between patients,
mental health content shared on mental health campaigns, clinicians, and data scientists are necessary to
specifically #MyTipsForMentalHealth on Twitter around better understand the sentiment and spread of
World Mental Health Awareness Day in 2017. We collected social media–based mental health campaigns.
14,217 Twitter posts from 10,805 unique users between
September and October 2017 that contained the hashtag
Correspondence to: J Torous, #MyTipsForMentalHealth. With the involvement of domain INTRODUCTION
jtorous@bidmc.harvard.edu experts, we hand-labeled 700 posts and categorized them as With depression a leading cause of global disability
(a) Fact, (b) Stigmatizing, (c) Inspirational, (d) Medical/Clinical [1] and the burden of mental health conditions
Cite this as: TBM 2019;9:1197–1207 Tip, (e) Resource Related, (f) Lifestyle or Social Tip or Personal
View, and (g) Off Topic. After creating a “seed” machine
projected to continue to rise [2], there is an urgent
doi: 10.1093/tbm/ibz028
learning classifier, we used both unsupervised and semi need for new solutions and tools for mental health
© Society of Behavioral Medicine supervised methods to classify posts into the various expert [3]. Considering that 300 million people worldwide
2019. All rights reserved. For
permissions, please e-mail: journals.
identified topical categories. We also performed a content have depression, 60 million have bipolar disorder,
permissions@oup.com. analysis to understand how information on different topics and 23 million have schizophrenia [1]—it is clear
spread through social networks. Our support vector machine that any potential solution must leverage scalable
classification algorithm achieved a mean cross-validation
technology able to reach the millions in need and
accuracy of 0.81 and accuracy of 0.64 on unseen data. We
found that inspirational Twitter posts were the most spread billions at risk.
with a mean of 4.17 retweets, and stigmatizing content was Social media are described as Internet-based
second with a mean of 3.66 retweets. Classification of social applications, which allow people to share opinions
media–related mental health interactions offers valuable [4]. Social media are considered among the mass
insights on public sentiment as well as a window into the media communication channels—together with
evolving world of online self-help and the varied resources newspapers, magazines, billboards, radio, television,
within. Our results suggest an important role for social media–
and Internet—but they are distinct in that they enable
based peer support to not only guide information seekers
to useful content and local resources but also illuminate the people to be actively involved in the communication
socially-insular aspects of stigmatization. However, our results process and stay connected with other [4]. It has
also reflect the challenges of quantifying the heterogeneity of been well documented that social media constitutes
mental health content on social media and the need for novel an immensely powerful source of social influence
machine learning methods customized to the challenges of [5], with an ability to help individuals frame opinions
the field. on topics they care about, or to alter attitudes and
perceptions around events and issues [6]. With
Facebook harboring over a billion users, over 2.5
Keywords
billion active users of social media today, and with
Social media, Twitter, Mental health, Machine expanding penetration in high-, medium-, and low-
learning, Public health income countries [7], these technologies can be
TBM page 1197 of 1207
ORIGINAL RESEARCH

construed to be a key element of any technology- inspiration and hope, resources, facts, or outright
facilitated mental health solution [8]. inaccurate information. Understanding the types
One of the biggest strengths that social media of content being shared on Twitter, centered on
provides revolves around its ability to reach large mental health outreach is critical to a number of
populations quickly, inexpensively, and with low possible mental health applications and uses. These
effort [4]. Consequently, in recent years, a more may include, assessing its public health impact as
developed use of social media for improving well as public attitudes, identifying potential for

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
mental health has been through raising awareness, interventions such as fighting stigma, influencing
conducting outreach, and forecasting trends [9– public health policy decisions, tailoring mental
13]. Social media–based mental health campaigns health literacy efforts, and helping scale positive
such as the Bell Let’s Talk effort in Canada have uses such as peer or technology-driven or assisted
been temporally associated with an increased support or information on meaningful resources or
rate of mental health visits among youth [14]. In promoting positive behavior change [27].
fact, a review of such social media mental health Consequently, in recent years, a number of
campaigns has also found that these platforms hold mental health campaigns have surfaced that have
promise in changing user behaviors, and that they either primarily evolved via social media such
are highly effective in recruiting participants and as Twitter or used social media as a channel of
motivating them to take small, concrete actions [15]. communication and a mechanism to reach wide,
It has also been demonstrated that there is room in national and global populations. Many social
social media for targeted, inexpensive, small-scale media platforms and activists have spearheaded
projects, as well as large, well-funded, mass-reach these initiatives. For instance, in 2017, Instagram
marketing blitzes [16]. In other relevant work, the launched the #HereForYou campaign to help users
social media platform Twitter has been used to help find resources and support online and offline for
forecast the acuity of health emergencies in real how to get help with preventing and recovering from
time such as the 2013 Boston Marathon bombings mental illnesses.
[17] as well as offering population level of data To realize the potential of Twitter data to inform
on suicide risk factors [18]. Researchers have also public health campaigns, this article focuses on one
developed methods to predict individual-centric such social media–based mental health campaign
mental illness diagnoses based on Twitter posting that gained significant traction on Twitter around the
[19], identify spikes in use related to mental health World Mental Health Awareness Day in 2017. The
[20], study population-level mental health awareness research seeks to identify the nature of this content
on Facebook [9], and classify posts based on lexical shared on Twitter, and how different individuals
information and emotions [21]. However, less is engaged with it over a period of time. To do so, the
known about the actual content, types of messages, article leverages expert assessments of mental health
and information being shared on mental health content to provide automated, robust, and scalable
through Twitter, particularly surrounding raising machine learning and statistical methods for content
or improving awareness and understanding public categorization and for understanding engagement.
opinion and sentiment. This way, our novel approach goes beyond a
Relating to the potential of social media as a reliance on surveys and traditional media anchoring
platform to improve mental health literacy, recent effects to understand public attitudes surrounding
studies suggest that people use Twitter to discuss mental health—a gap noted in prior literature [28].
mental health to build community, raise awareness, In addition, this work innovates over state of the
have a safe place to express themselves and discuss art techniques to understand social media mental
personal struggles, serve as a coping mechanism, health content (e.g., see ref. [24]) that largely focuses
get peer advice and help, and appropriate it as on qualitative characterization and categorization of
a tool toward empowerment [22]. In fact, it has mental health awareness content on social media,
been argued that the diverse communities on requiring extensive effort- and time-consuming
social media help to make mental illnesses, which feedback from domain experts.
are often invisible to friends and family, visible
through postings, photos, and videos [16, 23], and METHODS
thereby can support altering perceptions of stigma Toward our goal of understanding the nature of and
[24]. But there is also evidence that mental health engagement around mental health content shared
stigma has migrated online and some Twitter posts on mental health campaigns, we focus on a recent
are inappropriate and condescending toward those Twitter campaign, which started close to the World
with mental health conditions [25]. Although it is Mental Health Awareness Day (October 10, 2017).
clear that those with mental illnesses use Twitter to The campaign was spearheaded by the hashtag,
talk about their lived experiences [26], it remains #MyTipsForMentalHealth. It was a grassroots
unclear if mental health tagged content is primarily campaign that trended globally with over 30,000
being shared to offer peer advice on treatment, Twitter posts in September 2017 [29, 30]. Our

page 1198 of 1207TBM

 ORIGINAL RESEARCH

rationale behind choosing this specific campaign two measures have been situated in prior literature
stems for two reasons: (a) Unlike other conditions, as reliable indicators of content engagement on
there have been very few mental health-specific Twitter [31]. Table 1 reports the descriptive statistics
campaigns on social media; and (b) this campaign of our dataset, Fig. 1a shows the daily volume of
had emerged as one of the largest ones in recent these posts in this period—we notice one sharp rise
history, and the context of its inception around in the number of Twitter posts (with these hashtags)
the World Mental Health Awareness Day allowed in the last week of September, and another one

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
greater reachability to and awareness among diverse exactly on the World Mental Health Day (October
audiences. With the support of the creator of this 10) [32]. Figure 1b shows the engagement distribu-
campaign as an author, this social media campaign tion of these Twitter posts. We also extracted the
represented the intersection of clinical interest, most frequently occurring hashtags in these posts
mental health advocacy, and data and computer (Fig. 1c) and find that these campaigns frequently
science. use other hashtags that are contextually related to
Surrounding this specific campaign, our technical the campaign, such as #mentalhealth (335 posts),
approach to address the proposed research #mondaymotivation (199 posts), #depression (72 posts),
goal involves the following steps: (a) collecting and #anxiety (49 posts).
relevant social media data; (b) generating thematic Further, we collected a variety of user attributes,
annotations on a sample of these collected Twitter such as the number of posts they had shared on
posts; (c) developing a machine learning framework Twitter, followers, and followees—this was essential
to leverage the expert annotations and automatically to understand who engages with these content. For
infer the topic of a mental awareness campaign posts; this, we could obtain the user attributes of 10,680
and finally (d) building an analytical framework that users who had public non-deleted accounts as
appropriates the outcomes of the machine learning of November 6, 2017. We found that these users
framework to explore the characteristics of content demonstrated varied social media usage patterns, as
shared around mental health awareness as well as indicated in terms of their number of posts ranging
the general Twitter audience’s engagement around between 1 and 1,140,627 (Median = 5605.50,
this content. stdev. = 3,6118.5), and their followers to followee
ratio, (which roughly estimates the popularity of
Twitter data collection a user is on Twitter) ranging between 0.01 and
We identified two trending hashtags on Twitter 54,802.47 (median = 0.97, stdev. = 723.3).
via which this mental health campaign spread: To further understand engagement, for each
#mentalhealthday and #mytipsformentalhealth [29, post in the dataset, we also obtained the number
30]. Using these hashtags as search queries, we of retweets and favorites received by that post
programmatically (and automatically without using the official Twitter API and via a technique
active human intervention) collected a large developed in-house, which used parsing the HTML
dataset of relevant Twitter posts. Specifically, content of the links corresponding to each Twitter
our data collection approach used a web crawler- post. Overall, the 14,217 posts collected using the
based Twitter Application Programming Interface awareness hashtags had a total number of 48,223
(API): GetOldTweets. Our data collection spanned retweets (mean = 3.39, stdev. = 29.3; Table 1),
between September 01, 2017 and November 05, whereas they were favorited a total of 145,682 times
2017, which collected all of the tweets associated (mean = 10.25, stdev. = 87.4; Table 1).
with this hashtag. Our motivation behind using this
specific API was that it provided all Twitter posts on Expert labeling
a given search query (here a hashtag). Hence, our Using a mini-modified Delphi process [33], the
dataset did not suffer from any biases resulting from following eight categories for classification of
specific sampling strategies. Twitter posts in our dataset were identified as: (a)
We obtained 14,217 Twitter posts that were Fact (F), (b) Stigmatizing (S), (c) Inspirational (I),
shared by 10,805 unique users at an average of 1.32 (d) Medical/ Clinical Tip (M), (e) Resource Related
posts per user. Corresponding to each post, we add- (R), (f) Lifestyle or Social Tip (LS), (g) Personal
itionally obtained their engagement metrics in terms View (PV), and (h) Off Topic (OT). Randomly
of the number of retweets (a signal of reshare) and sampled Twitter posts (700 in all) were individu-
favorites (a signal of endorsement). Note that, these ally hand-labeled by a board-certified psychiatrist

Table 1 | Descriptive statistics of our Mental Health Awareness Campaign dataset

Metric Value Metric Value Metric Value

No. of posts 14,217 Number of retweets 48,223 Number of favorites 145,682
No. of users 10,805 Median retweets 3.39 Median favorites 10.25
Avg. posts per user 1.32 Stdev. retweets 29.28 Stdev. favorites 87.40
TBM page 1199 of 1207
ORIGINAL RESEARCH

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
Fig. 1 | (a) Daily occurrence of MHAC posts; (b) distribution of retweets and favorites; and (c) top 20 most frequently used hashtags in the
mental health awareness campaign (MHAC) posts.

(coauthor Torous) and master’s level psychiatry re- structural, linguistic, and lexical aspects of Twitter
search assistant (coauthor Rizuto) who are familiar posts, drawing on prior work on social media and
with Twitter and social media. Any disagreements mental health [25]. A description of these features is
(n = 42) were discussed by co-authors Torous and given in section 2 of Supplementary Material. Using
Rizuto together in person until 100% consensus was a total of 634 features, we trained many classifiers
obtained. We provide example Twitter posts labeled with algorithms such as Random Forest, Support
corresponding to each of these categories in section Vector Machines (SVMs), and Logistic Regression,
1 of Supplementary Material document. as is standard practice for multi-class classification.
Our largest labeled topic was LS with which We used k-fold (k = 5) cross-validation [34] for par-
occurred in 31% (219) of the posts. However, as ameter tuning, and tested our seed classification
we found that only four posts belonged to the models on the held-out test dataset. A short tech-
Fact category, we did not consider this category nical description of these classification models is
as a distinct category for our downstream tasks. In given in section 2 of Supplementary Material.
addition, for better clarity and demarcation across
the categories, we merged PV and LS into a single Semi-supervised classifier (C)
category of PL (Personal, Lifestyle and Social Tip). The second phase of our machine learning
Our final set comprised six categories of Twitter approach seeks to improve on our ability to
posts: OT, S, I, M, R, and PL. categorize the Twitter posts, by leveraging the seed
classifier described earlier. This is essential due to
Machine learning approach to infer topics the fact that in many real-world scenarios involving
automatic categorization of content, like ours, and
Once we had expert-labeled a sample of 700
because labeled data are both expensive and time-
posts, our next objective was to use this sample to
consuming to gather as well as scarce (e.g., needs
automatically infer the topic of all (of the remaining
expert involvement and manual labor), although
13,517) Twitter posts in our data—this would give us
unlabeled data are comparatively huge and easier
a sense of the range of issues that surfaced in the
to gather. To build robust classification models that
specific Twitter mental health campaign of our
can generalize across datasets, settings, and mental
focus. To do so, this work built a two-phase multi-
health campaigns, we use semi-supervised learning
class machine learning classifier, as described
[35–37] in this second phase, that is able to leverage
next. In machine learning, classification is the
both labeled and unlabeled data in unison, thereby
problem of identifying to which of a set of categories
is to cover a better diversity of training examples
(subpopulations) a new observation belongs, on the
[38].
basis of a training set of data containing observations
Accordingly, from the 13,517 unlabeled posts,
(or instances) whose category membership is known.
we first obtained a random sample of 8,110 posts
(i.e., 60% unlabeled dataset), from which we
Seed classifier (C0) then found those posts that were similar to our
From the 696 manually labeled Twitter posts (after labeled examples [39], on the basis of their social
discarding posts labeled as F), we held out 140 posts media structural, linguistic structure and affect,
as our test dataset and used the remaining 556 posts psycholinguistics, and n-gram features (section 2 of
as our training data for building a first machine Supplementary Material elaborates on the features).
learning classifier. To build this classification model In particular, we obtained these similar examples
(C0), we used a variety of features that consider the using k-Nearest Neighbor technique to significantly
page 1200 of 1207TBM
 ORIGINAL RESEARCH

expand our limited sized hand-labeled training terms that are related to health and mental health
dataset of 696 posts with another 1,321 training (anxiety depression, stress, physical, etc.).
examples (see section 2 of Supplementary Material We note that the Linguistic Inquiry and Word
for more detail). Our net expanded training dataset Count (LIWC) category of second person pronoun
thus comprised 2,017 posts. We used the same set of and imperatives (e.g., look, ask, don’t listen, let tell)
features and repeated building several classifiers (C) plays a significant role in our classifier. Prior
according to the same methods as used for C0 (ref. research has found that the presence of second

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
section 2 of Supplementary Material), and tuned person pronouns in conjunction with interactive
their parameters with a k-fold (k = 5) cross-validation verbs (ask, tell, listen, etc.) is typically associated with
as described earlier. social processes and social interaction [41], aspects
key to raising awareness and dissemination of social
Analytic approach of machine-labeled content and their media information, and therefore likely to surface
engagement data in many categories in our dataset. The top features
To understand in what ways the categories of Twitter that belong to parts of speech (POS) sequences also
mental health campaign content, identified earlier, convey that our classifier is able to capture differences
differ in their content, we used an unsupervised in particular linguistic structures of expression. For
language modeling technique [40]. In section 3 instance, the sequence of “VB IN PRP” (verb–
of Supplementary Material, we provide details of preposition–personal pronoun) captures interactive
this approach. Finally, we studied the engagement and imperative opinions (e.g., “Know that you are not
received per each category of posts to understand alone,” “don’t worry about me,” “never forget that you
the reach and impact of different topical categories. are important”: the underlined segment denotes the
For each of the categories, we examined the particular parts-of-speech sequence). We refer the
probability distribution of retweets and favorites readers to section 2 of Supplementary Material for
received by posts belong to the category. an extended list of these top significant features.

RESULTS Analysis of machine-labeled content

Assessing the performance of the machine learning After using this well-performing semi-supervised
classifiers classifier C to automatically label all of the
We first report the k-fold (k = 5) cross-validation remaining 12,196 unannotated Twitter posts, we
and test accuracy metrics of our preliminary seed present an analysis of the different topic categories
classifiers (C0). We find that the best model (based characterizing these posts and the ways in which the
on mean and standard deviation of accuracy) Twitter community engages with this content. Figure
is an SVM classifier that achieves a mean cross- 2 presents the distribution of the posts per category
validation accuracy of 0.50 (stdev. = 0.01) and a test in our data, and Fig. 3 and Table 2 present the
accuracy of 0.52. This test accuracy is only slightly distribution of the dataset’s retweets and favorites
better than a baseline accuracy of 0.44, which is per category.
obtained by assigning all posts the label of the
largest sized category. Detailed information about Content analysis
the cross-validation accuracy of seed classifier (C0) We find that the Personal/Social Tip (PS) category
is given in section 2 of Supplementary Material. shows the greatest occurrence, occurring in 44%
Next, we examine the accuracy of our semi- posts (5,997 posts). This category includes per-
supervised classification approach. Our best model sonal-, social-, and lifestyle-oriented views and tips,
(C) used an SVM algorithm to achieve a mean cross- and over 96% posts in this category are associated
validation accuracy of 0.81 (stdev. = 0.08). This with the hashtag #mytipsformentalhealth. On the basis
model predicts with a test accuracy of 0.64 on the of the unsupervised language modeling approach
unseen held-out dataset, showing an improvement introduced in Methods section (also see section 3 of
by 25% from the seed classifier C0. We refer the Supplementary Material), we found this category to
reader to section 2 of Supplementary Material for include phrases of advice and guidance relating to
additional information about the performance of navigating one’s mental health, such as “dont let,”
these classifiers. “think positive,” “practice gratitude,” “avoid toxic
Considering the top significant features of the people,” and “avoid news”. Next, the Inspirational
classifier C, we find that certain hashtags such as (I) category occurs in over 28% posts (3,822 posts).
#mondaymotivation and #mentalhealthawareness and We find that the posts assigned to this category ex-
the presence of a URL are significant contributors. press the importance and positives of mental health
This observation pertains to the very definition of and aim to encourage the individuals to actively seek
our categories, where three categories are related to mental health care when in need, as evident from
resources or tips. The other important features are phrases such as “okay ask help” and “health matters”.
predominantly n-grams, among which we find many Here, we also find phrases that hint a motivational
tone such as “keep going” and “youre loved”. The
TBM page 1201 of 1207
ORIGINAL RESEARCH

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
Fig. 2 | Posts per category labeled in our entire dataset of 13,517 posts.

Fig. 3 | Distribution of retweets and favorites across the topical categories.

Resources-related (R) category occurs in almost 15% Tip category occurs the least and this kind of con-
posts (1,937), and over 82% of these posts contain a tent can be found in only about 2% posts. The top
URL. The top phrases in this category also demon- phrases in this category include keywords related to
strate that these posts point resources such as reports, clinical professionals, medicines, health conditions,
surveys, or information about mental health shows or treatment and therapy.
on Television and other media. Among the other
relevant categories, we find that the Stigmatizing Engagement analysis
category that occurs in 6% posts advice the audi- Across the entire corpus of 13,517 Twitter posts
ence to stay away from particular media sources and labeled by the classifier together with the 700
also express political content. The Medical/Clinical hand-labeled posts, there were 48,223 retweets
page 1202 of 1207TBM
 ORIGINAL RESEARCH

Table 2 | Engagement received in retweets and favorites per topic category

Category #Retweets Mean #RTs #Favorites Mean #Fav.

Stigmatizing (S) 3,350 3.66 9,199 10.05
Inspirational (I) 16,516 4.17 57,469 14.51
Medical/Clinical Tip (M) 560 2.20 1934 7.58
Resource Related (R) 6,378 3.17 15,544 7.72

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
Personal/Social Tip (PS) 20,127 3.17 57,691 9.09
Off Topic (OT) 1292 1.80 3,845 5.35

(mean = 2.36, stdev. = 23.3) and 145,682 favorites to coordinate a diverse, rapid response that can
(mean = 6.68, stdev. = 67.1). Applying the data lead to changes in attitudes, perspectives, and even
collection steps from earlier, we obtained a set regulations around important societal concerns,
of 14,642 unique users who retweeted the posts including health [15, 46]. Therefore, as we observe
and 45,744 unique users who liked or favorited in our work and as has been argued for other types
the posts, leading to a total of 60,386 users who of health concerns [23], social media campaigns on
engaged and responded to the 13,517 posts in all. mental health topics have an immense role to play to
Next, incorporating the machine labels from the raise awareness, reduce stigma, support individuals
classifier, the engagement received per category of living through these chronic conditions, or inexpen-
posts is presented in Table 2. Kruskal–Wallis tests sively but meaningfully engage otherwise passive
suggest statistical significance in both the number of bystanders.
retweets and favorites across the categories with p < Essentially, our work can pave the way for further
0.001, with corrected H-statistic values of 216.74 (for research on unraveling the role of social media and
retweets) and 534.31 (for favorites). other similar web-based channels in altering the
We observe that engagement received (per post) public discourse on mental illnesses. In this regard,
through favorites is higher than that received by as has also been noted by Korda and Itani [46] and
retweets, across all categories of posts. Examining Freeman et al. [15], although campaign impact
the engagement received per each category, evaluation measures are available, new measures are
stigmatizing and inspirational posts can be noted to needed to evaluate the effectiveness of social media
receive the highest engagement from an audience on campaigns, such as the one analyzed in this article.
Twitter. On the other hand, off-topic posts receive We are in agreement with other scholars [15] who
the lowest engagement both through retweets and noted that there is a need to incorporate outcomes,
favorites. research, and theory in designing social media–
based mental health promotion programs that can
build on empirical observations of campaigns like
DISCUSSION the one this article presents.
Our results underscore the potential of social media Methodologically speaking, a key novelty of
platforms such as Twitter to quantify the content, our work is that it reduces the need to incorporate
spread, and reach of mental health outreach efforts. extensive expert coded qualitative data for the
A large body of literature has examined the poten- purpose—an approach widely prevalent this far [22,
tial and power of social media to initiate, drive, and 24] —by making principled use of machine learning
engage publics around campaigns targeting various and access to vast amounts of unlabeled social media
social, political, and health issues [42, 43], including data. Nevertheless, using semi-supervised machine
instances when social media was the centerpiece of learning methods, the 0.64 mean accuracy of our
a campaign with print and television media used pri- methods reflects the challenges of quantifying the
marily to support the social media focus [43]. Our heterogeneity of mental health content. But these
work contributes to and extends that line of research, results also underscore the potential to automate
although it does not investigate how campaigns that classification in a manner suitable for real-time
unfold over social media compared with those that population-level insights.
are primarily concentrated on other mass media Our results raise questions regarding what level
channels. As the communications landscape gets of classification accuracy for social media content
denser, more complex, and more participatory, the is necessary to inform public health campaigns and
networked population is gaining greater access to in- measure their impact. There is currently no gold
formation, more opportunities to engage in public standard, and we propose that our results may set
speech, and an enhanced ability to undertake col- at least a starting point for future efforts to improve
lective action [44]. These increased freedoms can and upon. The difficulty in building a fully automated
have helped loosely coordinated public’s demand classifier may in part be because mental health
and drive change through increased reach [45]. In itself is a very heterogeneous term representing
fact, today millions of messages can be shared easily a series of further heterogeneous conditions.
TBM page 1203 of 1207
ORIGINAL RESEARCH

Although mental health conditions are brain-based influenced by exposure to such content. On the
illnesses [47], societal and cultural factors shape other hand, concerning individuals who might not
how individuals communicate and understand have any mental health challenges, engaging with
their experience of a mental illness [48]. Thus, the such content on their Twitter network, might lead to
biological heterogeneity combined with societal increased mental health literacy, awareness around
and cultural variation offers a plausible reason why experiences relating to mental health challenges and
semi-supervised, versus unsupervised, methods may reduced stigma. More broadly, how discussion of

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
be of more value and produce results with great stigma issues on social media affects mental health
accuracy as we found in this study. This suggests outcomes, both for those with and without lived
an important novel role for the peer community to experiences, constitutes an interesting direction for
share their lived experiences in helping in training future research.
and updating of semi-supervised models. Next, we observed that across all categories,
In terms of content, we found that personal engagement received (per post) by favorites
tips and inspiration were the most widely shared is higher than that received by retweets. This
categories in terms of volume and reach. This demonstrates that with regard to content related to
makes sense given the hashtag driving this content mental health awareness campaigns, endorsement
as #mytipsformentalhealth and reflects a wealth of and acknowledgment behaviors as characterized by
peer support and personal insights from the mental favorites are higher than information sharing and
health community. With nearly 45,000 Twitter broadcasting behaviors characterized by retweets.
posts in these two categories, there is a plethora of This poses a challenge to online mental health
valuable information. Beyond raising awareness, awareness campaigns whose primary goal is to
the significant volume of this content raises the initiate information sharing and diffusion behaviors
potential of matching people to the most relevant across a large network of individuals.
and useful messages. Although our methods focused We also found an overall lack of resources and
on classification of content, we propose a need for medical categorized posts compared to other
new methods to help deliver the most relevant peer categories. Although local resources will vary based
support messages to each individual as a next step on location, online resources such as the National
in increasing the utility and potential of social media Alliance of Mental Illness website are easily
for mental health. Matching the right content to accessible to most people, if not all [49]. Promoting
the right person could increase the impact of social and introducing more resource-based content into
media for health promotion [23], although will rely social media discussions represents an opportunity
on accurate classification of content as outlined in for reaching new audiences and raising awareness
this article. of existing services. The paucity of medical-based
Of concern, we found that stigmatizing content Twitter posts is also interesting as it may represent
was most engaging in terms of retweets and a lack of engagement by the medical community
favorites per Twitter post. Although the volume of in social media discussions. Although offering
stigmatizing content was relatively small compared medical advice over social media is dangerous and
to other categories, the higher social diffusion of unethical, information on the overall benefits of
stigmatizing content represents an opportunity for engaging in treatment, adhering to medications, and
the mental health community to combat it. Either maintaining healthy weight and diet are universal
in the form of public health educational campaigns, medical messages that have a role in mental health
direct outreach and education to those posting discussions like those happening on Twitter. The
stigmatizing content, or working with social media current lack of medical content may reflect clinicians
platforms to remove harmful or hateful content—it are concerned about engaging publicly on social
may be possible to curb the spread of this content media but may also represent a missed opportunity
early and thus preclude its broader diffusion. to partake in the modern dialogue around mental
Further, that stigmatizing and inspirational health. Together, our findings allow drawing a
posts using the awareness hashtag receive the variety of qualitative inferences as well as frame
highest amount of engagement from an audience hypotheses, which can be tested in future research.
on Twitter has implications to mental health Like all studies, ours study has several
awareness campaigns in two forms. First, concerning weaknesses that must be taken into consideration,
individuals on Twitter undergoing mental health however, many of which outline directions for
challenges, reading about others’ experiences (both future research. First, we only study a single
stigmatizing and inspirational ones) might lead to hashtag and campaign, and it is unclear how our
a space for shared experiences, reducing stigma results may generalize to a different mental health
and pushing them to open up about their own campaign that may gain significant traction on
experiences. To this end, it would be interesting to social media. Similar considerations of (a lack of)
observe whether individuals with certain mental generalization exist for other types of social media
illnesses versus others are less or more positively content and platforms as well—here we focused on

page 1204 of 1207TBM

 ORIGINAL RESEARCH

Twitter, English language posts, and analyzed the offline. Important next steps for the field will be
textual content of posts; however, other campaigns investigations comparing and contrasting different
may be more predominant on other platforms types of mental health campaigns released both on
(e.g., Facebook) and additional modalities may be social media and others via other mechanisms.
adopted for information dissemination and raising
awareness (e.g., via images or videos). Location CONCLUSION
analysis of these social media posts, whether Social media holds both promise and pitfalls for

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
Twitter or another platform, was also beyond the mental health. Although our results highlight
scope of this work, but in future work can provide the sheer volume of public discourse happening
additional rich insights into how engagement and online today, challenges in accurately classifying
content sharing across different topical categories this mental health content present barriers to
varied over geography. We also note that, although fully using this information to inform local public
millions of individuals use social media, we health campaigns or allocate resources. Although
cannot make the claim that insights gleaned from new methods will help overcome these barriers,
a framework like ours, while indicating public partnering with the peer community who is already
sentiment around mental health, may not be a creating and partaking in these online campaigns
true reflection of society [50, 51]. However, the offers the next step for the field to better study and
focus of this work was to specifically examine what support the new public discourse on mental health.
sentiments characterize social media campaigns
about mental health, hence our findings should be SUPPLEMENTARY MATERIAL
interpreted with that specificity in mind. Further,
Supplementary material is available at Translational
we worked with the creator of this campaign as an
Behavioral Medicine online.
author on the paper to ensure our interpretations
reflected both the challenges and successes of Acknowledgments: K.S., S.K.E., and M.D.C. were partly supported by
this campaign. Second, although we had expert a National Institutes of Health grant R01GM112697. J.T. is supported by a
review and classification of Twitter content and research fellowship from the American Psychiatric Association Foundation.

involved directly with the campaign’s creator,

Compliance with Ethical Standards
we acknowledge that there is no gold standard of
consensus for classification of mental health content Conflict of Interest: All authors declare that they have no conflicts of interest.
relating to campaigns. Future work can bolster our
approach by additionally surveying or interviewing Human Rights: For this type of study, formal consent is not required.
those who are active and prominent contributors in
an online campaign. Third, our machine learning Informed Consent: This study does not involve human participants and
informed consent was therefore not required.
classifiers have certainly provided a mechanism
to automatically scale analysis and understanding Welfare of Animals: This article does not contain any studies with animals
of social media content surrounding such performed by any of the authors.
campaigns; however, the performance metrics in
the current classifier do imply additional room for
improvement. We caution against conclusions that References
would directly use such a classifier to understand
people’s perceptions and attitudes around mental 1. World Health Organization. 2018. Mental disorders. [online] Available at
health, such as for infodemiology or infoveillance http://www.who.int/en/news-room/fact-sheets/detail/mental-disorders
Accessed 10 June 2018.
purposes [52], without any human or expert 2. Vigo D, Thornicroft G, Atun R. Estimating the true global burden of
involvement and intervention. Fourth and finally, mental illness. Lancet Psychiatry. 2016;3(2):171–178.
3. Kilbourne AM, Beck K, Spaeth-Rublee B, et al. Measuring and improving
our work is essentially a case study of a “natural” the quality of mental health care: A global perspective. World Psychiatry.
mental health campaign, however, mental health 2018;17(1):30–38.
promotion efforts may also motivate non-profits 4. Kietzmann JH, Hermkens K, McCarthy IP, Silvestre BS. Social media? Get
serious! Understanding the functional building blocks of social media.
and other stakeholder agencies and individuals Bus Horiz. 2011;54(3):241–251.
to conceive and launch targeted social media 5. Kaplan AM, Haenlein M. Users of the world, unite! The challenges and
opportunities of Social Media. Bus Horiz. 2010;53(1), 59–68.
campaigns. To this point, it is important to 6. Grajales FJ III, Sheps S, Ho K, Novak-Lauscher H, Eysenbach G. Social
appreciate that the social media environment is media: A review and tutorial of applications in medicine and health care.
J Med Internet Res. 2014;16(2):e13.
highly dynamic and rapidly evolving. Therefore, 7. Statista. 2018. Number of social media users worldwide 2010–
although our research does not provide a 2021 | Statista. [online] Available at https://www.statista.com/
mechanism to assess the return on investment of statistics/278414/number-of-worldwide-social-network-users/
Accessed 11 June 2018.
these targeted campaigns, future work is needed 8. Birnbaum ML, Ernala SK, Rizvi AF, De Choudhury M, Kane JM. A collab-
to develop quantifiable and objective measures orative approach to identifying social media markers of schizophrenia by
employing machine learning and clinical appraisals. J Med Internet Res.
of social media campaigns, that takes into account 2017;19(8):e289.
their unique attributes, which are often absent 9. Saha K, Weber I, Birnbaum ML, De Choudhury M. Characterizing
awareness of schizophrenia among facebook users by leveraging
in campaigns that are launched and driven

TBM page 1205 of 1207

ORIGINAL RESEARCH

facebook advertisement estimates. J Med Internet Res. 34. Kohavi R. A study of cross-validation and bootstrap for accuracy estima-
2017;19(5):e156. tion and model selection. IJCAI. 1995;4(2):1137–1145.
10. Haskins BL, Davis-Martin R, Abar B, Baumann BM, Harralson T, 35. Saha K, Chan L, De Barbaro K, Abowd GD, De Choudhury M. Inferring
Boudreaux ED. Health evaluation and referral assistant: A randomized mood instability on social media by leveraging ecological momentary
controlled trial of a web-based screening, brief intervention, and referral assessments. Proc ACM Interact, Mob, Wearable Ubiquitous Technol.
to treatment system to reduce risky alcohol use among emergency 2017;1(3):95.
department patients. J Med Internet Res. 2017;19(5):e119. 36. Chapelle O, Scholkopf B, Zien A. 2009. Semi-supervised learning
11. Luxton DD, June JD, Fairall JM. Social media and suicide: A public health (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Trans Neural Netw..
perspective. Am J Public Health. 2012;102(suppl 2):S195–S200. 2017;20(3):542–542.
12. Moorhead SA, Hazlett DE, Harrison L, Carroll JK, Irwin A, Hoving C. A new 37. Liu, J, Zhao S. Wang G. SSEL-ADE: A semi-supervised ensemble learning

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
dimension of health care: Systematic review of the uses, benefits, and framework for extracting adverse drug events from social media. Art
limitations of social media for health communication. J Med Internet Res. intel med. 2018;84:34–49.
2013;15(4):e85. 38. Zhou ZH. When semi-supervised learning meets ensemble learning. In:
13. Cassa CA, Chunara R, Mandl K, Brownstein JS. Twitter as a sentinel in International Workshop on Multiple Classifier Systems (pp. 529–538).
emergency situations: Lessons from the Boston marathon explosions. Springer, Berlin, Heidelberg. 2009.
PLOS Curr Dis. 2013. 1st edn. doi:10.1371/currents.dis.ad70cd1c8bc 39. Muja M, Lowe DG. Fast approximate nearest neighbors with automatic
585e9470046cde334ee4b algorithm configuration. VISAPP (1). 2009;2(331–340):2.
14. Booth RG, Allen BN, Bray Jenkyn KM, Li L, Shariff SZ. Youth mental health 40. Eisenstein J, Ahmed A, Xing EP. Sparse additive generative models of text.
services utilization rates after a large-scale social media campaign: In: Proceedings of the 28th International Conference on International
Population-based interrupted time-series analysis. JMIR Ment Health. Conference on Machine Learning. Omnipress. 2011:1041–1048.
2018;5(2):e27. 41. Golder SA, Macy MW. Diurnal and seasonal mood vary with
15. Freeman B, Potente S, Rock V, McIver J. Social media campaigns that work, sleep, and daylength across diverse cultures. Science.
make a difference: What can public health learn from the corporate 2011;333(6051):1878–1881.
sector and other social change marketers? Public Health Res Pract. 42. Enli G. Twitter as arena for the authentic outsider: Exploring the social
2015;25(2):e2521517. media campaigns of Trump and Clinton in the 2016 US presidential
16. Neiger BL, Thackeray R, Van Wagenen SA, et al. Use of social media in election. Eur J Commun. 2017;32(1):50–61.
health promotion: Purposes, key performance indicators, and evaluation 43. Hanna R, Rohm A, Crittenden VL. We’re all connected: The power of the
metrics. Health Promot Pract. 2012;13(2):159–164. social media ecosystem. Bus Horiz. 2011;54(3):265–273.
17. Kuehn BM. Twitter streams fuel big data approaches to health 44. Shirky C. The political power of social media: Technology, the public
forecasting. JAMA. 2015;314(19):2010–2012. sphere, and political change. Foreign Aff. 2011;90(1):28–41.
18. Jashinsky J, Burton SH, Hanson CL, et al. Tracking suicide risk factors 45. Guo C, Saxton GD. Tweeting social change: How social media are chan-
through twitter in the US. Crisis 2014;35(1):51–59. ging nonprofit advocacy. Nonprofit Volunt Sect Q. 2014;43(1):57–79.
19. Coppersmith G, Dredze M, Harman, C. Quantifying mental health signals 46. Korda H, Itani Z. Harnessing social media for health promotion and be-
in Twitter. In: Proceedings of the Workshop on Computational Linguistics havior change. Health Promot Pract. 2013;14(1):15–23.
and Clinical Psychology: From Linguistic Signal to Clinical Reality (pp. 47. Insel T, Cuthbert B, Garvey M, Heinssen R, Pine DS. Research domain
51–60). 2014. criteria (RDoC): Toward a new classification framework for research on
20. McClellan C, Ali MM, Mutter R, Kroutil L, Landwehr J. Using social media mental disorders. Am J Psyc. 2010;167:7.
to monitor mental health discussions—evidence from Twitter. J Am Med 48. Taylor SE, Brown JD. Illusion and well-being: a social psychological per-
Inform Assoc. 2017;24(3):496–502. spective on mental health. Psychol Bull. 1988;103(2):193.
21. Mowery D, Bryan C, Conway M. Feature studies to inform the classifi- 49. Hollis C, Morriss R, Martin J, et al. Technological innovations in
cation of depressive symptoms from Twitter data for population health. mental healthcare: Harnessing the digital revolution. Br J Psychiatry.
2017. arXiv preprint arXiv:1701.08229. 2015;206(4):263–265.
22. Berry N, Lobban F, Belousov M, Emsley R, Nenadic G, Bucci S. 50. Chou WY, Hunt YM, Beckjord EB, Moser RP, Hesse BW. Social media
#WhyWeTweetMH: Understanding why people use twitter to discuss use in the United States: Implications for health communication. J Med
mental health problems. J Med Internet Res. 2017;19(4):e107. Internet Res. 2009;11(4):e48.
23. Hawn C. Take two aspirin and tweet me in the morning: How Twitter, 51. Tufekci Z. Big questions for social media big data: Representativeness,
Facebook, and other social media are reshaping health care. Health Aff validity and other methodological pitfalls. ICWSM. 2014;14:505–514.
(Millwood). 2009;28(2):361–368. 52. Eysenbach G. Infodemiology and infoveillance: Framework for an
24. Robinson P, Turk D, Jilka S. et al. Measuring attitudes towards mental emerging set of public health informatics methods to analyze search,
health using social media: Investigating stigma and trivialisation. communication and publication behavior on the Internet. J Med Internet
Soc Psychiatry Psychiatr Epidemiol 2019;54:51. doi:10.1007/ Res. 2009;11(1):e11.
s00127-018-1571-5 53. Zhu X. Semi-supervised learning literature survey. Comput Sci
25. Joseph AJ, Tandon N, Yang LH, et al. #Schizophrenia: Use and misuse on (University of Wisconsin-Madison). 2006;2(3):4.
Twitter. Schizophr Res. 2015;165(2–3):111–115. 54. Friedman JH, Bentley JL, Finkel RA. An algorithm for finding best
26. Hswen Y, Naslund JA, Brownstein JS, et al. Online communication matches in logarithmic expected time. ACM Trans Math Softw.
about depression and anxiety among twitter users with schizophrenia: 1977;3(3):209–226.
Preliminary findings to inform a digital phenotype using social media. 55. Bentley JL. Multidimensional binary search trees used for associative
Psychiatr Q. 2018;89:569. doi:10.1007/s11126-017-9559-y searching. Commun ACM. 1975;18(9):509–517.
27. Gulliver A, Griffiths KM, Christensen H, Mackinnon A, Calear AL, 56. Paul MJ, Dredze M. Drug extraction from the web: Summarizing drug
Parsons A, Stanimi rovic, R. Internet-based interventions to promote experiences with multi-dimensional topic models. In: Proceedings of the
mental health help-seeking in elite athletes: An exploratory randomized 2013 Conference of the North American Chapter of the Association for
controlled trial. J Med Internet Res. 2012;14(3):e69. Computational Linguistics: Human Language Technologies (pp. 168–
28. Wright KB. Researching Internet-based populations: Advantages and 178), 2013.
disadvantages of online survey research, online questionnaire authoring 57. Sharma E, Saha K, Ernala SK, Ghoshal S, De Choudhury M. Analyzing
software packages, and web survey services. J Comput-Mediat Commun. ideological discourse on social media: A case study of the abortion
2005;10(3):JCMC1034. debate. In: Proceedings of the 2017 International Conference of The
29. Shanahan M. 2018. #MyTipsForMentalHealth is trending on Twitter Computational Social Science Society of the Americas(p. 3). ACM,
and people actually gave really valuable advice. [online] BuzzFeed. 2017, October.
Available at https://www.buzzfeed.com/morganshanahan/people- 58. Chandrasekharan E, Pavalanathan U, Srinivasan A, Glynn A, Eisenstein J,
are-sharing-their-tips-for-maintaining-mental-heath Accessed 20 Gilbert E. You can’t stay here: The efficacy of reddit’s 2015 ban
December 2018. examined through hate speech. Proc ACM Hum-Comput Interact.
30. Shanahan M. 2018. People are sharing their tips for maintaining mental 1(CSCW). 2017;31.
health on Twitter and it’s honestly so inspiring. [online] BuzzFeed. 59. Laniado D, Mika P. Making sense of twitter. In: International Semantic
Available at https://www.buzzfeed.com/morganshanahan/people- Web Conference (pp. 470–485). November, 2010; Berlin, Heidelberg:
are-sharing-their-tips-for-maintaining-mental-heath?utm_term=. Springer.
riNkAajZnN#.puad6QxErY Accessed 11 June 2018. 60. Zhou Y, Zhan J, Luo J. Predicting multiple risky behaviors via multimedia
31. Cha M, Haddadi H, Benevenuto F, Gummadi PK. Measuring user influence content. In: International Conference on Social Informatics (pp. 65–73).
in twitter: The million follower fallacy. ICWSM. 2010;10(10–17):30. September, 2017; Cham: Springer.
32. World Health Organization. 2018. World Mental Health Day—10 61. De Choudhury M, Kiciman E, Dredze M, Coppersmith G, Kumar M.
October [online] BuzzFeed. Available at https://www.who.int/mental_ Discovering shifts to suicidal ideation from mental health content
health/world-mental-health-day/en/ Accessed 20 December 2018. in social media. In Proceedings of the 2016 CHI Conference on
33. Brown BB. Delphi process: A methodology Used for the Elicitation of Human Factors in Computing Systems (pp. 2098–2110). ACM,
Opinions of Experts (No. RAND-P-3925). Santa Monica, CA: RAND Corp.; 2016, May.
1968.

page 1206 of 1207TBM

 ORIGINAL RESEARCH

62. Schwartz HA, Eichstaedt JC, Kern ML, et al. Personality, gender, and age In: Proceedings of the Twelfth International Conference on Information and
in the language of social media: The open-vocabulary approach. PLoS Knowledge Management (pp. 394–401). ACM, 2003, November.
One. 2013;8(9):e73791. 67. Manning C, Surdeanu M, Bauer J, Finkel J, Bethard S, McClosky D.
63. Saha K, De Choudhury M. Modeling stress with social media around The Stanford CoreNLP natural language processing toolkit. In:
incidents of gun violence on college campuses. Proc ACM Hum-Comput Proceedings of the 52nd Annual Meeting of the Association for
Interact 1(CSCW). 2017;92:1–92:27. Computational Linguistics: System Demonstrations (pp. 55–60),
64. Pennebaker JW, Francis ME, Booth RJ. Linguistic inquiry and word 2014.
count: LIWC 2001. Mahway: Lawrence Erlbaum Associates. 68. Chung C, Pennebaker JW. The psychological functions of function words.
2001;71(2001):2001. Soc Commun. 2007;1:343–359.
65. Nagatsuka T, Miyachi T, Shimada A, Takeya K, Kemmochi E, Nakajima A, 69. Marcus MP, Marcinkiewicz MA, Santorini B. Building a large

Downloaded from https://academic.oup.com/tbm/article/9/6/1197/5369573 by Royal Melbourne Institute of Technology user on 19 July 2024
Fujita, K. U.S. Patent No. 7,194,471. Washington, DC: U.S. Patent and annotated corpus of English: the Penn Treebank. Comput Linguist.
Trademark Office.2007. 1993;19(2):313–330.
66. Calado P, Cristo M, Moura E, Ziviani N, Ribeiro-Neto B, Gonçalves MA. Combining 70. Loader BD, Vromen A, Xenos MA. The networked young citizen: social
link-based and content-based methods for web document classification. media, political participation and civic engagement. Inf Commun Soc.
2014;17(2):143–150.

TBM page 1207 of 1207