Biometrics and Beyond: Harnessing computer vision and machine learning to measure real world social interactions in psychiatric populations

The NIMH Division of Neuroscience and Basic Behavioral Science, with support from the NIH Office of Behavioral and Social Sciences Research, convened a workshop to address the needs for quantitative measurement of human social interactions in psychiatric populations.

Deficits in social interactions are at the core of many mental disorders and can significantly impact how well patients function in day-to-day life.  Yet our ability to objectively measure these deficits and to track changes with time or treatment is quite limited.  There is a need for new analytic technologies to allow quantitative assessment of human social interactions in real-time and in real-world settings. Such “social biometric” tools could expand our capacity to detect, assess, and monitor both social deficits and the effects of clinical interventions on social behavior. Given the complexity of social interactions, the ideal set of tools would have the capacity to integrate and analyze input from multiple modalities. The ideal real-time, real-world social biometric tools would also need to be non-intrusive, unlikely to change naturalistic behavior, and acceptable to patients and their families.

The NIMH Division of Neuroscience and Basic Behavioral Science, with support from the NIH Office of Behavioral and Social Sciences Research, convened a workshop to address this need. The overall goal of the workshop was to create a community of individuals dedicated to the development of social biometric tools for the collection and analysis of objective, detailed, real-time data in ecological settings. The workshop focused specifically on Autism Spectrum Disorders as a clinical ‘use case’. The workshop was co-chaired by Dr. Cathy Lord and Dr. Gregory Abowd. Participants included individuals from the full range of relevant research sectors, including clinical experts in ASD and technical experts from the fields of engineering, computer science, and machine learning. A select group of trainees was also included. Investigators presented current efforts and perceived challenges within and across four areas: emotion/affect, eye gaze/attention, speech/auditory analyses, and body movement.