Skip to main content

Advertisement

Springer Nature Link
Log in

Building personalized machine learning models using real-time monitoring data to predict idiographic suicidal thoughts

  • Article
  • Published:

From Nature Mental Health

View current issue Submit your manuscript

Abstract

Suicide risk is highest immediately after psychiatric hospitalization, but the field lacks methods for identifying which patients are at greatest risk, and when. We built personalized models predicting suicidal thoughts after psychiatric hospital visits (N = 89 patients), using ecological momentary assessment (EMA; average EMA responses per participant = 311). We built several idiographic models, including baseline autoregressive and elastic net models (using single train/test split) and Gaussian process (GP) models (using an iterative rolling-forward prediction method). Simple GP models provided the best prediction of suicidal urges (R2average = 0.17), outperforming baseline autoregressive (R2average = 0.10) and elastic net (R2average = 0.06) models. Similarly, simple GP models provided the best prediction of suicidal intent (R2average = 0.12) compared to autoregressive (R2average = 0.08) and elastic net (R2average = 0.04). Here we show that idiographic prediction of suicidal thoughts is possible, although the accuracy is currently modest. Building GP models that iteratively update and learn symptom dynamics over time could provide important information to inform the development of just-in-time adaptive interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price includes VAT (France)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1: Each panel represents descriptive visualizations (violin plots) of the distribution of idiographic model performance predicting suicidal urges.
Fig. 2: Illustrative example of idiographic prediction of suicidal urges for one participant.
Fig. 3: Descriptive visualizations (violin plots) of the distribution of idiographic model performance predicting suicidal intent.
Fig. 4: Illustrative example of idiographic prediction of suicidal intent for one participant.

Similar content being viewed by others

Data availability

This study includes data from a larger existing project. Access to anonymized data for the larger project (of which this study is a component) will be available through the National Institute of Mental Health Data Archive upon its completion.

Code availability

All code is publicly available at github.com/ShirleyBWang/idiographic_prediction (ref. 25).

References

  1. CDC. Facts About Suicide https://www.cdc.gov/suicide/facts/index.html (CDC, 2021).

  2. Nock, M. K. et al. Suicide and suicidal behavior. Epidemiol. Rev. 30, 133–154 (2008).

    Article  PubMed  Google Scholar 

  3. WHO. National Suicide Prevention Strategies: Progress, Examples and Indicators (World Health Organization, 2018).

  4. Franklin, J. C. et al. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol. Bull. 143, 187–232 (2017).

    Article  PubMed  Google Scholar 

  5. Kleiman, E. M. et al. Examination of real-time fluctuations in suicidal ideation and its risk factors: results from two ecological momentary assessment studies. J. Abnorm. Psychol. 126, 726–738 (2017).

    Article  PubMed  Google Scholar 

  6. Wang, S. B. et al. A pilot study using frequent inpatient assessments of suicidal thinking to predict short-term postdischarge suicidal behavior. JAMA Netw. Open 4, e210591 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  7. Bryan, C. J., Rozek, D. C., Butner, J. & Rudd, M. D. Patterns of change in suicide ideation signal the recurrence of suicide attempts among high-risk psychiatric outpatients. Behav. Res. Ther. 120, 103392 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  8. Wright, A. G. C. & Woods, W. C. Personalized models of psychopathology. Annu. Rev. Clin. Psychol. 16, 49–74 (2020).

    Article  PubMed  Google Scholar 

  9. Piccirillo, M. L. & Rodebaugh, T. L. Foundations of idiographic methods in psychology and applications for psychotherapy. Clin. Psychol. Rev. 71, 90–100 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  10. Molenaar, P. C. A manifesto on psychology as idiographic science: bringing the person back into scientific psychology, this time forever. Measurement 2, 201–218 (2004).

    Google Scholar 

  11. Molenaar, P. C. M. & Campbell, C. G. The new person-specific paradigm in psychology. Curr. Dir. Psychol. Sci. 18, 112–117 (2009).

    Article  Google Scholar 

  12. Fisher, A. J., Medaglia, J. D. & Jeronimus, B. F. Lack of group-to-individual generalizability is a threat to human subjects research. Proc. Natl Acad. Sci. USA 115, E6106–E6115 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  13. Berman, A. L., King, R. A. & Apter, A. in Suicide in Children and Adolescents (eds King, R. A. & Apter, A.) 198–210 (Cambridge Univ. Press, 2003).

  14. Leenars, A. A. In defense of the idiographic approach: studies of suicide notes and personal documents. Arch. Suicide Res. 6, 19–30 (2002).

    Article  Google Scholar 

  15. Barlow, D. H. & Nock, M. K. Why can’t we be more idiographic in our research? Perspect. Psychol. Sci. J. Assoc. Psychol. Sci. 4, 19–21 (2009).

    Article  Google Scholar 

  16. Ozomaro, U., Wahlestedt, C. & Nemeroff, C. B. Personalized medicine in psychiatry: problems and promises. BMC Med. 11, 132 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  17. Simon, G. E. & Perlis, R. H. Personalized medicine for depression: can we match patients with treatments? Am. J. Psychiatry 167, 1445–1455 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  18. Kaurin, A., Dombrovski, A. Y., Hallquist, M. N. & Wright, A. G. Integrating a functional view on suicide risk into idiographic statistical models. Behav. Res. Ther. 150, 104012 (2022).

    Article  PubMed  Google Scholar 

  19. Soyster, P. D., Ashlock, L. & Fisher, A. J. Pooled and person-specific machine learning models for predicting future alcohol consumption, craving, and wanting to drink: a demonstration of parallel utility. Psychol. Addict. Behav. 36, 296–306 (2021).

    Article  PubMed  Google Scholar 

  20. Fisher, A. J. & Soyster, P. D. Generating accurate personalized predictions of future behavior: a smoking exemplar. Preprint at https://doi.org/10.31234/osf.io/e24v6 (2019).

  21. Beck, E. D. & Jackson, J. J. Personalized prediction of behaviors and experiences: an idiographic person–situation test. Psychol. Sci. 33, 1767–1782 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  22. Nahum-Shani, I. et al. Just-in-Time Adaptive Interventions (JITAIs) in mobile health: key components and design principles for ongoing health behavior support. Ann. Behav. Med. 52, 446–462 (2018).

    Article  PubMed  Google Scholar 

  23. Howe, E. S. & Fisher, A. J. Identifying and predicting posttraumatic stress symptom states in adults with posttraumatic stress disorder. J. Trauma. Stress 35, 1508–1520 (2022).

    Article  PubMed  Google Scholar 

  24. Cohen, J. Statistical Power Analysis for the Behavioral Sciences (Academic Press, 2013).

  25. Wang, S. B. et al. Idiographic prediction of suicidal thoughts (GitHub); https://github.com/ShirleyBWang/idiographic_prediction

  26. Rasmussen, C. E. & Williams, C. K. I. Gaussian Processes for Machine Learning (MIT Press, 2006).

  27. Roberts, S. et al. Gaussian processes for time-series modelling. Philos. Trans. R. Soc. Math. Phys. Eng. Sci. 371, 20110550 (2013).

    Google Scholar 

  28. Schulz, E., Speekenbrink, M. & Krause, A. A tutorial on Gaussian process regression: modelling, exploring and exploiting functions. J. Math. Psychol. 85, 1–16 (2018).

    Article  Google Scholar 

  29. Torous, J. & Hsin, H. Empowering the digital therapeutic relationship: virtual clinics for digital health interventions. Npj Digit. Med. 1, 16 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  30. Rodriguez-Villa, E. et al. The digital clinic: implementing technology and augmenting care for mental health. Gen. Hosp. Psychiatry 66, 59–66 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  31. Coppersmith, D. D. L. et al. Mapping the timescale of suicidal thinking. Proc. Natl Acad. Sci. USA 120, e2215434120 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  32. Intille, S., Haynes, C., Maniar, D., Ponnada, A. & Manjourides, J. μEMA: microinteraction-based Ecological Momentary Assessment (EMA) using a smartwatch. Proc. ACM Int. Conf. Ubiquitous Comput. 2016, 1124–1128 (2016).

    PubMed  PubMed Central  Google Scholar 

  33. Ponnada, A., Haynes, C., Maniar, D., Manjourides, J. & Intille, S. Microinteraction ecological momentary assessment response rates: effect of microinteractions or the smartwatch? Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 92 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  34. Adams, L. et al. Assessing the real-time influence of racism-related stress and suicidality among black men: protocol for an ecological momentary assessment study. JMIR Res. Protoc. 10, e31241 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  35. Alvarez, K., Polanco-Roman, L., Samuel Breslow, A. & Molock, S. Structural racism and suicide prevention for ethnoracially minoritized youth: a conceptual framework and illustration across systems. Am. J. Psychiatry 179, 422–433 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  36. Kleiman, E. M. et al. Can passive measurement of physiological distress help better predict suicidal thinking?. Transl. Psychiatry 11, 611 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  37. Nock, M. K., Holmberg, E. B., Photos, V. I. & Michel, B. D. Self-injurious thoughts and behaviors interview: development, reliability and validity in an adolescent sample. Psychol. Assess. 19, 309–317 (2007).

    Article  PubMed  Google Scholar 

  38. Fortgang, R. G. et al. Increase in suicidal thinking during COVID-19. Clin. Psychol. Sci. 9, 482–488 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  39. Bentley, K. H. et al. Do patterns and types of negative affect during hospitalization predict short-term post-discharge suicidal thoughts and behaviors? Affect. Sci. 2, 484–494 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  40. Mou, D. et al. Negative affect is more strongly associated with suicidal thinking among suicidal patients with borderline personality disorder than those without. J. Psychiatr. Res. 104, 198–201 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  41. R Core Team. R: A language and environment for statistical computing (R Foundation for Statistical Computing, 2019); https://www.R-project.org/

  42. Wickham, H. et al. Welcome to the Tidyverse. J. Open Source Softw. 4, 1686 (2019).

    Article  Google Scholar 

  43. Harris, C. R. et al. Array programming with NumPy. Nature 585, 357–362 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  44. Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).

    Google Scholar 

  45. Hunter, J. D. Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9, 90–95 (2007).

    Article  Google Scholar 

  46. Kuhn, M. & Wickham, H. Tidymodels: a collection of packages for modeling and machine learning using tidyverse principles (Tidymodels, 2020); https://www.tidymodels.org

  47. Kuhn, M. & Johnson, K. Applied Predictive Modeling (Springer, 2013).

Download references

Acknowledgements

This research was supported by funding from the National Institute of Mental Health (F31MH125495 to S.B.W., U01MH116928 to M.K.N., K23MH120436 to K.H.B., K23MH132766 to R.G.F., K23MH120439 to K.L.Z. and R01MH117599 to J.W.S.). This content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author information

Authors and Affiliations

  1. Department of Psychology, Yale University, New Haven, CT, USA

    Shirley B. Wang & Jordyn R. Ricard

  2. Institute for Experiential Artificial Intelligence, Northeastern University, Boston, MA, USA

    Ruben D. I. Van Genugten

  3. Department of Computer Science, Wellesley College, Wellesley, MA, USA

    Yaniv Yacoby

  4. John A. Paulson School of Engineering and Applied Sciences, Harvard University, Boston, MA, USA

    Weiwei Pan

  5. Department of Psychology, Harvard University, Cambridge, MA, USA

    Kate H. Bentley, Alexis Christie, Merryn Daniel, Dylan DeMarco, Rebecca G. Fortgang, Alexander J. Millner, Narise Ramlal, Tida Tambedou, Kelly L. Zuromski & Matthew K. Nock

  6. Center for Precision Psychiatry, Massachusetts General Hospital, Boston, MA, USA

    Kate H. Bentley, Rebecca G. Fortgang & Jordan W. Smoller

  7. Department of Psychiatry, Massachusetts General Hospital/Harvard Medical School, Boston, MA, USA

    Kate H. Bentley, Suzanne A. Bird, Rebecca G. Fortgang & Jordan W. Smoller

  8. Mental Health Research, Franciscan Children’s, Boston, MA, USA

    Ralph J. Buonopane, Alexander J. Millner & Kelly L. Zuromski

  9. National Institute of Mental Health, Rockville, MD, USA

    Adam Haim

  10. Department of Psychology, University of Georgia, Athens, GA, USA

    Lia Follet

  11. Department of Psychology, University of Tennessee, Knoxville, TN, USA

    Flynn Kelly-Brunyak

  12. Department of Psychology, Rutgers University, New Brunswick, NJ, USA

    Evan M. Kleiman

  13. Department of Psychology, The Pennsylvania State University, University Park, PA, USA

    Onyinye Obi-Obasi

  14. Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA

    J. P. Onnela

Authors
  1. Shirley B. Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruben D. I. Van Genugten
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yaniv Yacoby
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weiwei Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kate H. Bentley
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Suzanne A. Bird
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ralph J. Buonopane
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Alexis Christie
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Merryn Daniel
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Dylan DeMarco
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Adam Haim
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Lia Follet
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Rebecca G. Fortgang
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Flynn Kelly-Brunyak
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Evan M. Kleiman
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Alexander J. Millner
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Onyinye Obi-Obasi
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. J. P. Onnela
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Narise Ramlal
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Jordyn R. Ricard
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Jordan W. Smoller
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Tida Tambedou
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Kelly L. Zuromski
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Matthew K. Nock
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All of the authors made a substantial contribution to this study. S.B.W. was responsible for data analysis and writing the paper. M.K.N. was responsible for study conception, design, funding acquisition and supervision of all activities. R.D.I.V.G., Y.Y. and W.P. contributed to data analysis. A.H. contributed to study design. K.H.B., S.A.B., R.J.B., R.G.F., E.M.K., A.J.M., J.W.S. and K.L.Z. contributed to study design and supervision. D.D. and J.P.O. contributed to software and data management. A.C., M.D., L.F., F.K.-B., O.O.-O., N.R., J.R.R. and T.T. contributed to data collection. All authors contributed to reviewing and revising the manuscript, and all authors approved the final version of the paper for submission.

Corresponding author

Correspondence to Shirley B. Wang.

Ethics declarations

Competing interests

M.K.N. receives publication royalties from Macmillan, Pearson and UpToDate. He has been a paid consultant in the past three years for Microsoft Corporation, the Veterans Health Administration and COMPASS Pathways, and for legal cases regarding a death by suicide. He has stock options in Cerebral Inc. He is an unpaid scientific advisor for Empatica, Koko and TalkLife. E.M.K. has been a paid consultant in the past three years for Boehringer Ingelheim Pharmaceuticals. J.W.S. is a member of the Scientific Advisory Board of Sensorium Therapeutics (with equity), and has received grant support from Biogen, Inc. He is PI of a collaborative study of the genetics of depression and bipolar disorder sponsored by 23andMe for which 23andMe provides analysis time as in-kind support but no payments. J.P.O. has been a paid consultant in the past three years for Boehringer Ingelheim and has received research funding from them. D.D. is the founder and CEO of Apoth.

Peer review

Peer review information

Nature Mental Health thanks Zhongzhi Xu and the other, anonymous reviewers for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, S.B., Van Genugten, R.D.I., Yacoby, Y. et al. Building personalized machine learning models using real-time monitoring data to predict idiographic suicidal thoughts. Nat. Mental Health 2, 1382–1391 (2024). https://doi.org/10.1038/s44220-024-00335-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s44220-024-00335-w

  • Springer Nature America, Inc.

Search

Navigation