Opportunities and Challenges of Developing Information Technologies on Behavioral and Social Science Clinical Research

Table of Contents

• EXECUTIVE SUMMARY
  Introduction
• CHAPTER 1
  Developing Technologies for Understanding the Life Course and Etiology of Mental Disorders
• CHAPTER 2
  Using Technology to Predict and Prevent Mental Illness
• CHAPTER 3
  Using Technologies for Diagnosing and Treating Mental Illnesses
• CHAPTER 4
  Using Technology to Improve Quality in Mental Health Practice
• CHAPTER 5
  Using Technology for More Rapid and Nimble Research
• CHAPTER 6
  Facilitating Ethical Conduct of Technology-Enhanced Research and the Protection of Participants
• CHAPTER 7
  Additional Cross-Cutting Considerations and Recommendations
• CONCLUSIONS
• TABLE 1
  Digital Health Tools for Collecting Data
• REFERENCES
• NATIONAL ADVISORY MENTAL HEALTH COUNCIL WORKGROUP
• NATIONAL INSTITUTES OF HEALTH STAFF

EXECUTIVE SUMMARY
Introduction

Digital Health is no longer limited to the technologically savvy. Around the world, 2.19 billion people own a smartphone^1,2 including approximately 60% of minority and low income populations who use these devices^3-5 for Internet-based activities⁶. In addition, wearable sensors with wireless connections (e.g., autographs (smartwatch devices that verify the wearer’s handwritten signatures), pedometers, devices with GPS, continuous heart rate monitors) are growing in availability and adoption for health and fitness purposes⁷.

In the realm of mental health, Digital Health tools such as assessment and tracking application software (apps), online communities, and asynchronous coaching have increased dramatically in the last few years, with 29% of health-related apps focused specifically on mental health ⁸. Sensors, such as health bands, have also been used to measure behavioral correlates of mental health (e.g., sleep, physical activity). Large health care systems offer their members online tools for managing mental health problems and co-occurring conditions such as PSTD, sleep disturbance, depression, and even psychosis management.

Because of their ubiquitous nature, Digital Health tools have the potential to truly improve our understanding of mental illness, to track the course of illnesses, and to provide as-needed mental health care. For the purposes of science, these tools can help amass large quantities of physiological, social, emotional, and behavioral data in real time with limited burden on the consumer.

Over the past five years, there has been a notable increase in the use of Internet-based and mobile device-based data collection for studies ranging from simple survey studies to intervention studies^9-29. Industry has also contributed tools for mobile research, with innovative efforts like open source research kits, that help with the recruitment of very large samples into medical research. Indeed, the rapid recruitment of large samples has been demonstrated several times. For example, one recent remote clinical trial not only rapidly recruited 1,200 adults into a randomized controlled trial, but recruited a sample that was identical to the demographics of the United States, without any special efforts to recruit these populations⁹. The promise of these technologies is such that the NIH Office of the Director has invested significant resources into the All of Us Research Program  (formerly the Precision Medicine Initiative Cohort Program) that will rely heavily on mobile technology to recruit, retain, and collect behavioral information, and will also leverage information available in electronic health records, with an eye toward better understanding the causes and consequences of disease.

At several open sessions of the National Advisory Mental Health Council questions were raised about future directions for behavioral and clinical research in the context of new and developing digital tools. Council members suggested a workgroup be convened to review the state of behavioral research and offer guidance to researchers on how behavioral science should move forward. In response to this discussion, Dr. Insel, then Director of NIMH, elected to form a Workgroup of Council to address the topic.

THE NATIONAL ADVISORY MENTAL HEALTH COUNCIL

Workgroup on Behavioral and Social Science Research was tasked with exploring the opportunities and challenges of using new information technologies to study human behaviors relevant to the NIMH mission. The Workgroup began by reviewing the existing NIMH clinical portfolio in behavioral and social sciences to identify areas in which NIMH is already using 21st century tools and to identify potential new opportunities. Next, the workgroup looked to what may be possible in the context of the NIMH Strategic Plan for Research and Strategic Research Priorities. Finally, the workgroup approached the question of how to develop and use technology-driven information science at several levels of analysis, including the individual (cognition and behavior), social, and cultural contexts.

FRAMING THE WORKGROUP AGENDA

Workgroup members met by teleconference to set the agenda and identify participants who could discuss opportunities to leverage technology, including:

• Technologies to advance assessment (including the assessment related to the risk, etiology, diagnosis, and course of psychopathology);
• Technologies to advance research on and delivery of preventive and therapeutic interventions; and
• Technologies to study and improve service delivery (i.e., to improve the reach, efficiency, and quality of mental health services).

Workgroup members also identified many cross-cutting issues, including those related to research, infrastructure, privacy, safety, and ethics. These topics were organized into six key questions:

1. What technologies currently exist and/or could be developed to understand the life course and etiology of mental disorders in terms of their developmental trajectory, course, and epidemiology?
2. How can these new technologies be used to predict and prevent mental illness?
3. How can these new technologies be used to achieve more efficient and effective diagnoses and treatments of mental illnesses?
4. How can these new technologies be used to improve quality in mental health practice?
5. How can these new technologies address new questions as well as enable more rapid and nimble research?
6. How can technology-enhanced research be conducted ethically and in a manner that protects participant data?

OVERVIEW OF THE REPORT

The remaining sections of this report provide an overview of the discussion and recommendations from the workgroup’s deliberations. The next seven chapters summarize the discussion and corresponding recommendations related to each of the six key questions as well as a set of cross-cutting recommendations, respectively. The report includes a discussion on how Digital Health technology can facilitate mental health research (Chapters 1 and 2), potential avenues for research into the use of digital technology to improve treatment access and outcomes (Chapters 3 and 4) and the challenges of a Digital Health approach to research, including ethical and human subject protection concerns (Chapters 5 and 6). Additional cross-cutting considerations (Chapter 7) and recommendations are addressed, followed by an overall conclusion.

CHAPTER 1
Developing Technologies for Understanding the Life Course and Etiology of Mental Disorders

Important to this Strategic Research Priority is the ability to capture cognitive, social, emotional, and functional behavior accurately and continuously, with as little burden to the consumer as possible. Historically, these data have been captured using self-report and clinical and lab-based assessment tools (e.g., sleep laboratory tools). These traditional methods of assessment have proven to be invaluable in the study of developmental psychopathology but are limited in important ways. First, self-report measures are subject to inaccuracies due to retrospective memory, and to survey bias (the tendency to respond to questions in a prosocial way). Second, the clinical and lab- based assessments measure behavior outside the typical environment in which it occurs, thus introducing potential artifacts in the data. Finally, both methods collect information at specific time intervals, rather than continuously, potentially missing critical information about how mental illness develops and progresses. Even with the use of ecological momentary assessment (EMA), where research participants report on symptoms at random intervals during the day or after pre-determined times during the day, assessment may be burdensome and demands a level of engagement that may present a challenge to many people^30,31.

Digital Health technology deployed over smartphones and wearable sensors can mitigate many of the problems of traditional assessment. These technologies allow scientists to capture participants’ symptoms and functioning in real time, with far less burden than traditional methods. New technologies can improve engagement of patients, including their participation in therapeutic practice skills that may enhance the effectiveness of the therapy^32,33. These methods also allow scientists to measure how behavior varies over time, and between different social contexts (work versus home)^34,35. Digital Health measures can take the form of brief surveys delivered over text messaging or an assessment app. Such measures can also capture information on physical and spatial activity from the GPS and accelerometers in smartphones or wearable devices, as well as collect physiological data such as heart rate, blood pressure, and respiration, which are good measures of disability associated with many mental illnesses^36,37. Digital Health tools in the form of electronic health records, can also provide valuable information about illness, injuries, and toxin exposures early in the life span, or prior to the onset of mental illness. Table 1 provides a list of Digital Health tools and the information they collect.

As is noted in Table 1, there is a considerable amount of data available that has the potential to measure social, environmental, and behavioral markers of psychopathology. However, there is still work to be done to determine the construct validity and reliability of these measures, and to determine their predictive value in isolating deviations from normal function. Studies, such as the Sleep, Networks, Affect, Performance, Stress, and Health using Objective Techniques study, SNAPSHOT  study, are needed to test the validity and reliability of these measures. (SNAPSHOT is an NIH-funded collaborative research project between the Affective Computing group  and Macro Connections group at the MIT Media Lab and Brigham & Women’s Hospital.) As data become available, it will be important to warehouse and open-source the data, such as through the NIMH Data Archive, so that investigators can begin developing algorithms that indicate normal developmental processes, algorithms that signal aberrant development, and algorithms that identify toxic exposures, such as sedentary lifestyle and negative social interaction. Approaches might include recording multiple signals at single points in time as well as passively collecting data over months/years to evaluate long term outcomes and trajectories. To make sense of these data, the development of new analytic tools for compiling, analyzing, and combining data with other sources (e.g., genetic databanks, health registries, NIMH Data Archive) is necessary. The use of digital analytics for precision health is now possible given advances in machine learning models that can extract complex patterns from multiple sources of high-dimensional data over time. For example, recent work has shown that building mathematical models not only of groups, but also of individuals, longitudinally, can lead to stronger models that can be used to forecast changes tomorrow, based on behavior today³⁸.

CHAPTER 2
Using Technology to Predict and Prevent Mental Illness

As discussed in Chapter 1 and outlined in Table 1, Digital Health technologies can be very useful for measuring behaviors associated with mental health states and for identifying strategies for preventing impending mental health problems. Digital Health technology could be used to identify early behavioral warning signs, such as disturbed sleep, pressured speech on video chats, failure to respond to texts, and changes in activity, as they occur in real time. This could, in turn, offer opportunities to intervene before warning signs turn into mental health problems.

Although promising, there is still a considerable amount of research to be done to determine if Digital Health data of this nature can accurately predict when someone may be in the early phases of a new mental illness or relapsing from the recovery of a mental illness. Most research to date has focused on proof of principle, feasibility, and acceptability of these tools for prediction purposes. These studies find that many technology-based approaches (e.g., texting one’s clinical status, using web or mobile self-management software, behavioral sensing) are feasible and acceptable assessment approaches in clinical populations, particularly when designed with the characteristics of the target population in mind. Therefore, while the use of highly novel technology or innovative combinations of technology are exciting new directions that warrant continued development, scientific efforts in the Digital Health space can progress towards larger evaluations of efficacy and effectiveness in predicting and preventing mental illness, as well as developing and evaluating technology implementation models in the context of real-world care.

To support this research, many of the resources and methods discussed in Chapter 1 apply here. First, although digital data collection is acceptable by clinical populations participating in research, their construct validity and reliability still must be determined. Second, large repositories of Digital Health data are needed to create predictive algorithms, both at the group level (how do people at risk for depression behave differently than the general population?) and at the individual level (what is an individual’s digital signature like, and how well does deviation from that signature predict an event?). The data analytic methods discussed in Chapter 1 (e.g., machine learning approaches) could be applied to these needs as well. Therefore, multi-modal data collection is encouraged to fully capitalize on what contemporary personal devices offer (e.g., accelerometry, GPS, device use logs, in-home sensors, social media data).

Specific to prevention research is identifying strategies that could mitigate the occurrence or reoccurrence of a mental health event. As an example, just-in-time-interventions for people who are coping with active symptoms, need to be further studied in order to identify:

• The most efficacious interventions,
• The optimal timing between early indication of an event and the actual event, and
• The best way to alert patients, family or clinicians that an event is imminent.

Just In Time Adaptive Intervention (JITAI) methods³⁹ are ideal for developing decision rules that link a patient’s information to intervention options, specifying if, when, and what type/dose of an intervention should be offered. Developing these decision rules requires large-scale clinical trials. Likewise, the basket (treating to a phenotype) and umbrella (treating to a specific disease) trial designs can save valuable resources by matching the right intervention to the right subgroup of patients through Digital Health markers. While basket and umbrella trial designs are used almost exclusively in oncology trials, these trial designs can be used effectively in large scale mental health trials. Participant-based research methods will also be critical to determining whether the way warnings about impending illness states and information about preventative interventions are delivered will impact patients’ reactions to warnings, and their subsequent use of recommended strategies.

Researchers will need to plan for new types of data storage and data management challenges (e.g., raw data, feature extraction, data security measures, data transmission) and prepare to use and develop new data modeling and analytics as discussed above. New digital sources of information will provide more nuanced and granular information about illness and expression of illness within the individual. For example, in the NIMH-funded CrossCheck study, individuals with schizophrenia were given a multi-modal smartphone system that collected comprehensive behavioral sensing, device use, and self-report data to identify behavioral indicators of relapse and hospitalization⁴⁰. The results showed that digital warning signs of hospitalization were idiosyncratic, with data combinations and patterns that were meaningful indicators for one individual, but not for all. Researchers will need to look beyond group differences and develop new data modeling and inference techniques that will enable more tailored clinical signal detection—examining changes within each individual, over time.

CHAPTER 3
Using Technologies for Diagnosing and Treating Mental Illnesses

To this end, Digital Health technology, particularly online interventions, mobile health applications (apps), and electronic patient portals have tremendous potential to increase the reach of existing evidence-based treatments, to deliver interventions targeted to the unique symptom profile of the individual, and to do so in a manner that allows for as-needed and timely delivery. Technology can extend care to those who cannot access it by overcoming access barriers related to time constraints, transportation problems, and cost. Many people who are hesitant or unwilling to talk with mental health professionals due to problems such as social anxiety or mistrust of therapists may be willing to use technology based tools^41,42. Technologies may be able to improve the efficiency of existing treatments, by reducing the amount of therapist time required in psychotherapy⁴³. They may also be able to improve processes and quality of pharmacotherapy for depression in primary care⁴⁴. By embedding data collection in the context of people’s lives, technology may aid in the diagnosis, as well as the identification of actionable treatment strategies.

Web-based technologies have been investigated for more than 15 years^45-48. Evidence shows that coach- facilitated, Internet-based treatment is as effective as face-to-face treatment, but far better than stand- alone Internet interventions, where no coaching or interaction with a clinician is offered⁴⁹, and that this form of treatment is largely acceptable. The evidence for smartphone app interventions is emerging. While some of the early interventions used the phone primarily for in-the-moment mood assessment⁵⁰ or text messaging⁵¹ in the context of other forms of web-based or standard treatments, more recently apps have been developed that provide interventions in ways that are quite different from web-based treatments. These apps are designed to be used in short but frequent interactions targeting specific behavioral strategies. Rather than using psychoeducation, these apps can successfully reduce symptoms and improve well-being by prompting beneficial behaviors^52,53. An emerging area of investigation is the use of Digital Health with more traditional face-to-face therapy. Both web-based interventions and mobile-based intervention apps have been shown to decrease the amount of therapist time required^43,54, although some studies have also suggested that the introduction of technology into standard face-to-face therapies may increase the amount of time in treatment⁵⁵.

In spite of the overwhelming evidence that technology-based treatments can be effective, particularly when coupled with some human support, there are almost no instances of successful and sustainable implementation in real world clinical settings⁵⁶. There are many potential reasons for this.

One reason is that much of the design and development of technology-based interventions has been top down, with clinicians designing tools that try to get patients to do things that treatment models suggest would reduce symptoms. Other design methods, such as user-centered or participatory design, engage patients from the very earliest phases of development and keep them involved throughout the design process. Indeed, user-centered design could also help to develop targeted behavioral treatments by combining Digital Health assessment and digital phenotypic responses to treatments, in conjunction with patients’ reports on the helpfulness of the interventions. User-centered design should involve all key stakeholders in the design of those services, including patients, providers, administrators, and IT experts.

Another issue is that while there may be some people who can fully benefit from standalone digital interventions, the vast majority require human support or coaching to obtain more consistent engagement and outcomes, as was noted earlier in this chapter. Historically, web-based tools for computers did not work because people did not consistently use them⁵⁷, and therefore the addition of human support to get individuals to use the tools was developed⁵⁸. Thus, rather than thinking of these tools as digital interventions (or intervention technologies), they should be thought of as technology-enabled services⁵⁶. The use of this term has several implications. Most importantly, it focuses on the service to be provided and what technology or technologies are needed to support that service (rather than developing the service to match and be supported by the technology).

Moving forward, there are a number of opportunities and caveats in pursuing the conduct of mental health assessments and interventions through electronic technologies. As the real time, low burden detection of behaviors and states related to mental health and mental health treatment becomes possible, there is an opportunity to develop treatments that harness this information to support JITAI, as discussed in Chapter 2^41,56. Advances in assessment could also lead to stepped approaches to care, maximizing the workforce capacity by offering online tools to consumers who both like and can benefit from this form of treatment, reserving the scarce workforce for those who require more intensive, face-to-face care. Natural language processing is improving rapidly, to the point where chatbots can interact with people on specific tasks. It is unlikely in the near term that chatbots will be able to provide therapy, as artificial intelligence cannot yet manage the required complexities of human speech and experience^60,61. However, chatbots may be able to perform more limited functions such as assessment or training of the individuals with whom they interact.

There are opportunities in other technologies as well. Virtual reality technology has become increasingly accessible in recent years. A growing number of smaller studies have demonstrated its use in supporting the treatment of anxiety disorders, PTSD⁶², and persecutory delusions in schizophrenia⁶³. Serious gaming, that is, the use of games designed for purposes other than pure entertainment, such as the use of interactive games to teach treatment principles and promote behavior change, has shown, for example, good potential in the treatment of depression among youth⁶⁴.

There are a few caveats to consider when exploring technology for diagnosis and treatment. To succeed, investigators need to carefully consider their approaches to research and development. While it is likely that the use of technology will continue to have a transformative effect on mental health, it is essential that technology-enabled services focus on the use of evidence-based treatment models. While technology can accomplish some things clinicians cannot do (e.g., provide immediate information about function in real time readouts), the burden that a real-time intervention has on a consumer is relatively unknown. While some in-the-moment interventions may be highly effective, some may also be highly disruptive and unacceptable to some participants. The heterogeneity of needs, preferences, and types of responses make it difficult to fully anticipate what the future of Digital Health holds. To some degree this will depend on research interventionists, and their ability to embrace looking forward rather than being reactive to new technological developments. Closer collaborations with experts outside of the mental health field (e.g., engineers, computer scientists, anthropologists, ethicists, business entrepreneurs, marketers) will not only provide their expertise, but will likely stimulate mental health researchers to consider new strategies and manage assumptions. For example, clinicians/ researchers will need to adapt as new technologies develop and make current ones obsolete or ineffective. One possible consideration along these lines is whether apps will continue to be the dominant tools on the smartphone, or will there be another shift to a new form of interaction. Whatever the case, the researchers will also need to nimbly adapt to such changes.

CHAPTER 4
Using Technology to Improve Quality in Mental Health Practice

Since health organizations first began to adopt telehealth in the early 1990s, improvements in technology have contributed to a rapid growth and usefulness of these modes of treatment, with an even more rapid expansion anticipated in the coming years⁶⁵. Telehealth services delivered via telephone, videoconferencing, and mHealth appear to be well- accepted and many are as effective as face-to-face visits for delivering mental health care^66-69, as noted in prior chapters. Asynchronous modalities such as remote consultation between providers and secured emails between providers and patients are also being used to supplement clinical visits⁷⁰. Medicare, Medicaid, and private insurers increasingly support billing for these services⁷¹.

Once patients are in treatment, tools such as electronic health records (EHRs) and personal health records can be used to improve quality and outcomes of mental health care. Electronic health records coupled with a registry function can be used to track and monitor symptoms for patients who are not improving^72,73. Provider dashboards can provide clinicians with: real-time data about the course of patient’s illnesses and responses to treatment; passive data about patient activity; and reminders and prompts about indicated clinical interventions. Personal health records can integrate information across multiple systems of care, including medical and mental health records, supporting coordination of care and patient engagement⁷⁴. Insurance claims data can be used to identify high users of service and guide quality improvement efforts⁷⁵.

NIMH is supporting several large-scale efforts to pool data from health care systems to advance knowledge. The Mental Health Research Network, a consortium of 13 health system research centers, is currently being used as a platform to conduct research on mental health: epidemiology; health services; economics; disparities; outcomes and quality assessment; and pragmatic clinical trials⁷⁶.

In the public sector, NIMH has sponsored collaborative research projects to pool data from all-payer warehouses across states to advance policy research⁷⁷. NIMH is currently leading efforts to better understand the epidemiology and best clinical practices for individuals with clinical high-risk (CHR) and first episode psychosis (FEP) patients through the Early Psychosis Intervention Network (EPINET) program, which will collect and aggregate clinical, biological, and administrative data from academic and community-based early psychosis treatment programs around the country⁷⁸.

Several factors have limited the adoption and clinical utility of health information technologies in routine clinical practice. First, due to lack of funding and fragmentation, the use of health information technology in mental health care delivery has lagged behind the general medical sector, particularly in the public mental health sector⁷⁹. Second, most commercial mental health EHRs are primarily used as a means of storing medical information, and lack capabilities such as registry functions that can be used to drive better care⁷². Third, many of these systems lack the interoperability and privacy safeguards needed to exchange information with medical EHRs^80,81.

Finally, many health information technologies have not paid adequate attention to the preferences, needs, and time constraints of end users. Google discontinued its personal health record, Google Health, in 2012, due to low perceived usefulness and high burden involved in data entry⁸². Many practices continue to struggle to achieve meaningful use of their EHRs which are often viewed by providers as time consuming to complete, and a barrier to interacting with patients⁸⁰.

Population-based health information technology is likely to continue to migrate from provider-generated electronic health records to cloud-based, patient- owned records that combine health care records with passively collected health data from wearable devices, as well as data that are manually entered by patients on symptoms, functioning, and health goals. These platforms will create new opportunities for both coordinating care, and for recruiting and tracking patients enrolled in research studies⁸³. They will also create new challenges for patient consent and privacy, as lines blur between research and care delivery.

Using health technologies to improve population health requires that they be both effective and widely utilized. Research is needed to test the best strategies, including training and payment structures, for encouraging meaningful use of health technologies by the full range of end users including patients, providers, and managers.

CHAPTER 5
Using Technology for More Rapid and Nimble Research

Over the past several years, there have been a number of developments in research that are leveraging digital and passive sensing strategies through personal mobile technologies such as smartphones and wearables. Apple’s ResearchKit  studies have shown that it is possible to recruit tens of thousands of people into research in as little as a day. The NIMH-funded BRIGHTEN study, a feasibility randomized controlled trial comparing three mobile apps for depression, recruited and randomized 1,200 adults representative of the United States population and completed data collection in under one year⁹. Studies like these also find that smartphones and wearable sensors can capture a more accurate picture of participants’ symptoms in real time, with far less burden than is typical in a clinical or behavioral study. Clinical and behavioral research may be conducted more quickly, efficiently, and to greater scale than ever imagined.

Methods for collecting biometric, behavioral, and survey data discussed in previous sections are all potentially useful tools for collecting clinical and behavioral research data. However, the process of recruiting, consenting, and compiling these data necessitates accessible platforms. Online research platforms typically include a web-based or app- based landing page explaining the purpose of the study and an initial eligibility screening process. Study information to aid in the consent process is provided here, either using a text modality (e.g., PDF) or a brief video. Assessment and treatment tools are downloaded by the participant from this page. An example of this is the Center for Behavioral Intervention Technologies  (CBITS) Participant Registration Intervention Management (PRIM) system at Northwestern University. PRIM offers a standardized toolkit for research websites that include the capacity to screen and consent participants for eHealth and mHealth behavioral intervention research trials. Also, the University of California, San Francisco was recently awarded funds from the NIH to develop a robust recruitment platform for conducting research into common health problems using digital and mobile technologies. These platforms are customizable for specific projects, allowing research teams to develop their own online research portals without having to rebuild the necessary infrastructure and databases from scratch. Commercial and open-sourced research platforms are also available through companies such as Apple’s ResearchKit  and Arivale. Arivale currently collects a combination of biological, physical activity, and Ecological Momentary Assessment (EMA)-driven survey data to better understand the development and physical health of the population. ResearchKit has specific research projects (e.g., on cancer, heart disease, Parkinson’s Disease) that collect passive and EMA data, which are available via open-access protocols to any investigator interested in using the data. ResearchKit also provides guidelines to researchers to develop their own tools and research platform using CareKit . These commercial- and academic-led projects have the potential to facilitate the development of novel methods for conducting mental health research, either by providing a wealth of existing data to be analyzed, or by offering easily accessed methods for large scale, mobile data collection.

CHAPTER 6
Facilitating Ethical Conduct of Technology-Enhanced Research and the Protection of Participants

RECRUITMENT/RETENTION & EVOLUTION
Researchers embarking on research leveraging mobile and digital data must be aware of three key issues:

• Challenges with participant recruitment and retention, and measurement coherence
• Protection of participant data (privacy, security) and safety, and
• The rapid evolution of the field.

Addressing this range of challenges in recruitment, retention, measurement coherence, privacy, security and the ever rapid changes to the field, is critical to the success of research leveraging mobile technology.

RECRUITMENT: Although there has been success in using technology platforms to recruit participants from the general population to participate in research, the evaluation processes for including/ excluding participants with specific characteristics (e.g., age, diagnostic history, chronicity, etc.) and retaining participants throughout the study lifecycle remains problematic. While researchers have been successful in rapidly recruiting participants in large numbers, the success of remote recruitment strategies changes drastically and unpredictably over time. Early mobile research easily relied on search engines as a method for Internet-based recruitment⁸⁴. However, within a year, that method was no longer successful, and it is currently deemed by many in the field as a low yield recruitment source⁹. Furthermore, the ability to remotely assess and screen individuals to confirm eligibility criteria remains a significant challenge.

RETENTION: Some mobile studies suffer from poor long-term retention, with dropout rates as high as 90% over the course of only a few weeks⁸⁵.
A recent study of mental health app use found that even with push reminders, retention dropped from a low of 63% at one month to 46% by three months’ use⁸⁶. Conversely, some studies demonstrate relatively high levels of user engagement in mobile interventions over longer timeframes. For example, Ben-Zeev and colleagues (2016)⁸⁷ reported on a technology-assisted relapse prevention program for 342 people with schizophrenia that involved the use of an mHealth intervention called FOCUS. In the study, engagement was encouraged and supported by regular check-in calls and meetings with clinical research staff and approximately three quarters of study participants used the intervention regularly, for 3 to 6 months. However, even the most motivated participants exhibit variable use patterns, such as switching apps or devices as their goals or lives change⁸⁸.

It is clear there are many opportunities for behavioral scientists, working with experts in human centered design, to study how to best involve, motivate, and continuously engage participants in research that is conducted remotely. Exploring better research designs, including evaluating incentives and allowing participants to choose which data to share with researchers would be invaluable in improving the retention rates in these studies.

Another important challenge in conducting research of this nature is the pace at which technology changes. The technology specified in an initial research application may be out-of-date by the time the study is funded and ready to launch. This means that researchers and funders will need to be flexible about allowing for changes in technologies for data collection, during the conduct of the study. It also means that studies will need to move quickly for the outcomes to be meaningful. There currently exist several barriers to modifying an intervention during the course of the research study including limitations in study designs, data analytic challenges, and navigating lengthy Institutional Review Board (IRB) approval processes each time the intervention undergoes a new iteration.

PROTECTION OF PARTICIPANT DATA AND SAFETY: Electronic technology studies are subject to problems such as cyber-attacks, physical disasters, submission of corrupted or fraudulent data, loss and corruption of collected data, insider data theft, unauthorized data sharing, Private Health Information (PHI) leaks, and participant re-identification. There are additional challenges with the volume and granularity of data collected via passive sensing technologies (e.g., wearable or phone app) and the lack of standards to securely store and share these data.

Participant protection is a key issue. The data collected from research participants may not be part of their medical record and, as such, privacy protections offered through HIPAA do not apply. The personal health data collected by researchers using both research grade and commercial wearable sensors or mobile-based apps introduce new concerns for data management. For devices controlled by the researcher, data management is an internal process that can be dictated by institutional policy and IRB mandates, however, for many institutions, policies detailing how to manage the volume of data produced by these technologies do not yet exist. When using a commercial product as a research tool (e.g., Fitbit, MapMyFitness, etc.), research participants may be asked to set up an account with the vendor, which likely involves the review of Terms and Conditions and Privacy Agreements. Even if the participant opted to review these in detail, they are not written at the 4-6th grade reading level that IRBs strive for in documents related to informed consent and may lead to difficulty in fully understanding the risks. Additionally, the legal context of Terms and Condition agreements is likely to conflict with regulations to protect research participants’ rights to litigate, if harmed as a direct result of study participation. These are a few safety and security concerns that should be carefully considered to ensure appropriate protections of research participants.

While the focus has been on phone apps and mobile/wearable technologies, it would be remiss not to acknowledge data that can be leveraged for research that is collected on social network platforms like Instagram, Facebook and Twitter. For example, Tweets are publicly available and can be used for research purposes; however, most are geo-tagged and, with somewhat sophisticated analytics, researchers can identify public health threats and make inferences about individuals who may be at risk.

At this time, the ideal approach is to inform research participants about the precautions employed to protect data and to clearly state that the researchers are not able to guarantee the confidentiality of data and that techniques to anonymize data are limited in terms of protecting participants’ private information. Researchers may want to consider the use of software that can remove information that identifies a participant even if the person chooses to self-identify during their electronic participation.

PROTECTION OF PARTICIPANTS AND INSTITUTIONAL REVIEW BOARDS: Researchers also must present their research plan to an IRB responsible for applying regulations and ethical principles designed to protect human research participants. The IRB is charged with evaluating the probability and magnitude of risks and benefits associated with the research. New technologies and methods may introduce new challenges for IRBs. Given there are upwards of 6,000 IRBs in the United States, it may be unreasonable to expect IRB members to be sufficiently well versed with all technologies, which may contribute to either overlooking potential risks of harm to participants or making overly conservative decisions⁸⁹. There is considerable variability in how IRBs view the ethical dimensions because of the novelty of passive sensing and mobile research methods and the lack of guidelines. Furthermore, an underlying problem for IRBs is that the federal regulations used to guide research participant protections (45 CFR 46) have not been updated since 1991. An exploration of the gap between 21st century research methods and static regulations should include individuals with expertise in research, privacy, bioethics, research ethics, technology, computer science, and research regulations convened to imagine a responsive and dynamic ethics review process⁹⁰.

CHAPTER 7
Additional Cross-Cutting Considerations and Recommendations

INFRASTRUCTURE RECOMMENDATIONS:

• Facilitate access to existing mHealth data sources. Access to mHealth data sources may be promoted through data sharing, which would include incentivizing researchers to share their data e.g., through the use of the NIMH Data Archives and/ or repositories .
• Support conferences and workshops. Leverage existing conferences on mHealth, wireless health technology, allied health fields, and consumer technology. The researchers and stakeholders at these meetings could come together to discuss mental health applications in the context of those meetings.

TRAINING/COLLABORATION RECOMMENDATIONS:

• Develop new training models that integrate a wide range of experts. Encourage new training models for researchers, physicians, and psychologists (particularly those facing the challenges of mid-career ) in the areas of data use and intervention development, from a wide perspective. Encourage the collaborations of multi-disciplinary experts.
• Develop and test generalizable strategies for task shifting. Test approaches for training and supporting the implementation of technology-assisted approaches by para-professionals, including those who can translate and interpret data and know how to respond when technology is deployed on a large scale.

STUDY DESIGN/METHODOLOGY RECOMMENDATIONS:

• Utilize study designs that allow technology to evolve during the research lifecycle. The phases of traditional research contrast with studies on technology-enhanced assessment and intervention. Technologies adapt to how people use them and to users’ demands, therefore it is critical that ability for making modifications are built into the study designs, allowing for quick adaptation.⁹¹
• Consider the balance between pilot studies and later stage research. There should be a well-considered balance of pilot studies versus full-scale, late-stage research, as well as a consideration of the nature and amount of pilot data that are optimal before larger-scale testing is proposed.
• Test principles rather than technology. Support research that tests principles (e.g., behavioral science principles) rather than products or applications.

DISSEMINATION/IMPLEMENTATION RECOMMENDATIONS:

• Evaluate existing mental health apps. Numerous mental health applications are available online, but there is a need for rigorous evaluation, and development of an evaluation structure of these apps.
• Promote new models for implementation. The large-scale use of technology in this area will not happen on its own. There is a need for well- articulated models for implementation of these technologies in the real world.
• Promote Dissemination. Since dissemination is an important driver of uptake, various avenues for dissemination should be stimulated, including those of an open-source environment.
• Consider factors that impact the ultimate adoption of technology. Within the context of the research, larger provider organizations are well equipped to promote support of technology, for example, providing smartphone technology as part of insurance plans, especially for people with serious mental illness. Such support will likely help to scale up adoption of this technology.

TECHNOLOGY DESIGN RECOMMENDATIONS:

• Weave interventions into existing technology. Emphasize user-centered design. The field should ask patients about their needs and develop technological solutions that are built into the fabric of their lives (i.e., are invisible) to support behavioral change.
• Consider user factors. Technology users need to have options. Factors such as literacy levels and dexterity must be considered during hardware and software development.
• Simple is sometimes better. Capitalize on simple, user-friendly technologies—for both patient and clinician-facing tools.

CONCLUSIONS

There is considerable promise to the new approaches brought about by the technology, in assessment, treatment, service delivery, and research methods. Nonetheless, investigators must be careful to consider the limitations and challenges to this form of data collection, the role that technology plays in treatment and service delivery, and the potential burdens of Digital Health technology for end-users. Investigators should employ user-centered designs in developing tools, use novel data analytic techniques that are better suited for Big Data analysis, and pursue an interdisciplinary team science approach to research, that will involve collaborations with engineers, data scientists, health system experts, and technology designers. This research should specifically be conducted in collaboration with health systems and communities, so that the data collected, and the interventions developed have the greatest chance of being generalized and adopted.

With this report’s recommendations and cautions in mind, collecting behavioral data objectively in real-time on potentially thousands of consumers and participants, will result in far-reaching mental health treatment and service innovations in the future.

Table 1: Digital Health Tools for Collecting Data

DATA TOOL/TYPE	USEFUL FOR MEASURING	STATUS FOR USE IN RESEARCH
GPS/Wi-Fi Time spent at different locations Distance covered during a day Variability in geospatial activity	Social and occupational functioning (time spent at school, friends, work), time engaged in activities outside of home and work, change in routine.	Pros: technology exists Cons: requires participant to have phone and sensors with them and GPS activated
Physiological sensors Heart rate Heart rate variability Galvanic Skin Response Blood pressure	Autonomic activity, cardiac and muscular function, arousal	Pros: devices and apps widely available Cons: Devices and apps vary widely in accuracy
Accelerometers/ Actigraphy Movement	Activity (active vs. sedentary, walking, running, cycling) and sleep inference	Pros: data are available Cons: correlation with activity and sleep still in development. Does not currently measure quality of sleep or activity
Light sensor	Ambient brightness	Pros: data are available Cons: what activities (e.g., sleep) can be accurately inferred from these data are still TBD
Ingestible sensor Ingestion detection	Oral medication use	Pros: Objective indicator of medication use/adherence Cons: technology exists but is proprietary. Acceptability by different clinical populations TBD
Social media Sites Frequency of activity Group membership Photos posted	Degree of on-line social engagement, changes in activity	Pros: technology exists to have participants download data for up to 2 years Cons: security
Calendar # events event type	Work and activity demands	Pros: technology exists Cons: requires participant to enter data
Device use when and how often device is open time on device	Nighttime activity as proxy for sleep quality	Pros: technology exists Cons: may need to capture data from various devices
SMS/MMS/Phone/Email # of contacts messages sent messages received time spent in response when contacting and for how long	Social network, social behavior	Pros: data can be accessed Cons: but on only Android devices
Keyboard Typing errors Typing speed	Cognitive load	Pros: data can be captured Cons: unclear utility, requires a baseline
Microphone Human speech Heart rate Ambient sound	Conversation turn taking, speech rate and fluency, time in social environment (proximal to human speech) affect, cognition, sleep inferences	Pros: much could be learned about social, emotional and cognitive performance from vocal behavior Cons: still in proof of concept phase; ethical issues related to collecting data from people interacting with participant
Search Engine data Websites visited Website types On-line shopping	Interests, concerns	Pros: Participant can download search data from 18 months to two years Cons: Cannot distinguish if different person uses same computer, or if participant uses different computers for work or personal use
Video data Facial expression Eye movement Vocal data	Cognitive load, social behavior, emotional state	Pros: data is available Cons: not ready for routine use, association with cognition, emotion and social behavior still being researched
Health history Illness exposure Injuries Frequency of common illnesses (colds, flu, infections) Family illness history Prescription	Health status	Pros: Many big health plans have this data electronically Cons: for many, past history is still retrospective (existed before electronic health records), data hard to combine
Service Use Frequency of visits Types of services used Time between appointments Interventions prescribed Provider type Clinical notes	Quality of care; effects of treatment	Pros: Data currently being collected Cons: Missed information when participant moves or switches between hospitals and clinics, as well as insurance plans

References

1. Mogg T. US smart phone use over 100 million, Android increases market share. Digital Trends. 2012.
2. eMarketer. 2 Billion Consumers Worldwide to Get Smart(phones) by 2016. 2014;http://www.emarketer.com/Article/2-Billion-Consumers-Worldwide-Smartphones-by-2016/1011694, 2014 .
3. Quintanilla E. Cell Phones Helping Minorities close the gap on Internet access. Christian Science Monitor. 2012.
4. Marloew C. Research: ethnic minorities quickest to adopt smartphones [with charts]. Digital Media Wave. 2012.
5. Lewis T., Lagomarsino, G., Schweitzer, J. E-health in low- and middle-income countries: findings from the Center for Health Market Innovations. Bulletin of the World Health Organization. 2012;90:332-340. doi: 10.2471/BLT.11.099820. PMID: 22589566.
6. Smith, A. U.S Smartphone Use in 2015. April 1, 2015.
7. Statista. Share of U.S. consumers interested in buying a smart watch. 2013.
8. Anthes E. Mental Health: There’s an app for that. Nature. 2016;532:20-23. PMID: 27078548.
9. Anguera JA, Jordan JT, Castaneda D, Gazzaley A, Areán PA. Conducting a fully mobile and randomised clinical trial for depression: access, engagement and expense. BMJ Innov. 2016;2(1)14-21. PMID:27019745.
10. Glick G, Druss B, Pina J, Lally C, Conde M. Use of mobile technology in a community mental health setting. J Telemed Telecare. 2016;22(7):430-5. PMID: 26519378.
11. Aschbrenner KA, Naslund JA, Shevenell M, Mueser KT, Bartels SJ. Feasibility of Behavioral Weight Loss Treatment Enhanced with Peer Support and Mobile Health Technology for Individuals with Serious Mental Illness. Psychiatr Q. 2016; 87(3):401-15. PMID:26462674.
12. Chamberlain D, Elcock M, Puligari P. The use of mobile technology in health libraries: a summary of a UK-based survey. Health Info Libr J. 2015;32(4):265-275. Epub 2015 Aug 21. PMID: 26292980.
13. Sahu M, Grover A, Joshi A. Role of mobile phone technology in health education in Asian and African countries: a systematic review. Int J Electron Healthc. 2014;7(4):269-286. PMID: 25161104.
14. Shore JH, Aldag M, McVeigh FL, Hoover RL, Ciulla R, Fisher A. Review of mobile health technology for military mental health. Mil Med. 2014;179(8):865-878. PMID: 25102529.
15. Lee JA, Nguyen AL, Berg J, et al. Attitudes and preferences on the use of mobile health technology and health games for self- management: interviews with older adults on anticoagulation therapy. JMIR Mhealth Uhealth. 2014;2(3):e32. PMID: 25098413.
16. Teng JE, Thomson DR, Lascher JS, Raymond M, Ivers LC. Using Mobile Health (mHealth) and geospatial mapping technology in a mass campaign for reactive oral cholera vaccination in rural Haiti. PLoS Negl Trop Dis. 2014;8(7):e3050. PMID: 25078790.
17. Surka S, Edirippulige S, Steyn K, Gaziano T, Puoane T, Levitt N. Evaluating the use of mobile phone technology to enhance cardiovascular disease screening by community health workers. Int J Med Inform. 2014;83(9):648-654. PMID: 25002305.
18. Quanbeck AR, Gustafson DH, Marsch LA, et al. Integrating addiction treatment into primary care using mobile health technology: protocol for an implementation research study. Implement Sci. 2014;9:65. PMID: 24884976.
19. McCartney P. Mobile technology for parent tracking of child health. MCN Am J Matern Child Nurs. 2014;39(2):133. PMID: 24566165.
20. Praveen D, Patel A, McMahon S, et al. A multifaceted strategy using mobile technology to assist rural primary healthcare doctors and frontline health workers in cardiovascular disease risk management: protocol for the SMARTHealth India cluster randomised controlled trial. Implement Sci. 2013;8:137. PMID: 24274431.
21. Sankaranarayanan J, Sallach RE. Rural patients’ access to mobile phones and willingness to receive mobile phone-based pharmacy and other health technology services: a pilot study. Telemed J E Health. 2014;20(2):182-185. PMID: 24205837.
22. Purnell JQ, Griffith J, Eddens KS, Kreuter MW. Mobile technology, cancer prevention, and health status among diverse, low-income adults. Am J Health Promot. 2014;28(6):397-402. PMID: 24200336.
23. Ehteshami A, Hachesu PR, Esfahani MK, Rezazadeh E. Awareness and using of medical students about mobile health technology in clinical areas. Acta Inform Med. 2013;21(2):109- 112. PMID:24058250.
24. Brown W, 3rd, Yen PY, Rojas M, Schnall R. Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology. J Biomed Inform. 2013;46(6):1080-1087. PMID: 23973872.
25. Kumar S, Nilsen WJ, Abernethy A, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med. 2013;45(2):228-236. PMID: 23867031.
26. Richardson JE, Reid MC. The promises and pitfalls of leveraging mobile health technology for pain care. Pain Med. 2013;14(11):1621-1626. PMID: 23865541.
27. Braun R, Catalani C, Wimbush J, Israelski D. Community health workers and mobile technology: a systematic review of the literature. PLoS One. 2013;8(6):e65772. PMID: 23776544.
28. Logan AG. Transforming hypertension management using mobile health technology for telemonitoring and self-care support. Can J Cardiol. 2013;29(5):579-585. PMID: 23618506.
29. Free C, Phillips G, Galli L, et al. The effectiveness of mobile-health technology- based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS Med. 2013;10(1):e1001362. PMID:23349621.
30. Asselbergs J, Ruwaard J, Ejdys M, Schrader N, Sijbrandij M, Riper H. Mobile Phone-Based Unobtrusive Ecological Momentary Assessment of Day-to-Day Mood: An Explorative Study. J Med Internet Res. 2016;18(3):e72. Published online 2016 Mar 29. PMID: 27025287.
31. Intille S, Haynes C, Maniar D, Ponnada A, Manjourides J. MicroEMA: Microinteraction- based Ecological Momentary Assessment (EMA) Using a Smartwatch. UbiComp ‘16. September 12 - 16, 2016, Heidelberg, Germany. DOI: http://dl.acm.org/citation.cfm?doid=2971648.2971717 .
32. Morris RR, Schueller SM, Picard RW. Efficacy of a Web-Based, Crowdsourced Peer-To-Peer Cognitive Reappraisal Platform for Depression: Randomized Controllled Trial. J Med Internet Res. 2015;17(3):e72. PMID: 25835472.
33. Doré BP, Morris RR, Burr DA, Picard RW, Ochsner KN. Helping Others Regulate Emotion Predicts Increased Regulation of One’s Own Emotions and Decreased Symptoms of Depression. Pers Soc Psychol Bull. 2017;43(5):729-739. DOI: https://doi.org/10.1177/0146167217695558 .
34. Juengst SB, Graham KM, Pulantara IW, et al. Pilot feasibility of an mHealth system for conducting ecological momentary assessment of mood-related symptoms following traumatic brain injury. Brain Inj. 2015;29(11):1351-1361. PMID: 26287756.
35. van Os J, Delespaul P, Barge D, Bakker RP. Testing an mHealth momentary assessment Routine Outcome Monitoring application: a focus on restoration of daily life positive mood states. PLoS One. 2014;9(12):e115254. PMID: 25513813.
36. Tucker I, Smith LA. Topology and mental distress: self-care in the life spaces of home. J Health Psychol. 2014;19(1):176-183. PMID: 24155185.
37. Mason MJ, Korpela K. Activity spaces and urban adolescent substance use and emotional health. J Adolesc. 2009;32(4):925-939. PMID: 18851877.
38. Jaques N*, Taylor S*, Nosakhare E, Sano A, Picard R. Multi-task learning for predicting health, stress, and happiness. NIPS Workshop on Machine Learning for Healthcare. December 2016. Barcelona, Spain. (*equal contribution) PDF
39. Nahum-Shani I, Smith SN, Spring BJ, Collins LM, Witkiewitz K, Tewari A, Murphy SA. Just-in- Time Adaptive Interventions (JITAIs) in Mobile Health: Key components and Design Principles for Ongoing Health Behavior Support. Ann Behav Med. 2016 Sep 23. [Epub ahead of print] PMID: 27663578.
40. Ben-Zeev D, Brian RM, Wang R, Wang W, Campbell AT, Min SHA, Merrill M, Tseng WS, Choudhury T, Hauser M, Kane JM, Scherer EA. CrossCheck: Integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatric Rehab Journal [Epub ahead of print] PMID: 28368138.
41. Mohr DC, Ho J, Duffecy J, et al. Perceived barriers to psychological treatments and their relationship to depression. J Clin Psychol. 2010;66(4):394-409. PMID: 20127795.
42. Mohr DC, Hart SL, Howard I, et al. Barriers to psychotherapy among depressed and nondepressed primary care patients. Ann Behav Med. 2006;32(3):254-258. PMID: 17107299.
43. Wright JH, Wright AS, Albano AM, et al. Computer-assisted cognitive therapy for depression: maintaining efficacy while reducing therapist time. Am J Psychiatry. 2005;162(6):1158-1164. PMID: 15930065.
44. Corden ME, Koucky EM, Brenner C, et al. MedLink: A mobile intervention to improve medication adherence and processes of care for treatment of depression in general medicine. Digital Health. 2016;2:1-10. PMID: 26640740.
45. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. 2012;32(4):329-342. PMID: 22466510.
46. Richards D, Richardson T, Timulak L, McElvaney J. The efficacy of Internet-delivered treatment for generalized anxiety disorder: A systematic review and meta-analysis. Internet Interventions. 2015;2:272-282.
47. Kuester A, Niemeyer H, Knaevelsrud C. Internet- based interventions for posttraumatic stress: A meta-analysis of randomized controlled trials. Clin Psychol Rev. 2016;43:1-16. PMID: 26655959.
48. Baumeister H, Reichler M, Munzinger M, Lin J. The impact of guidance on Internet-based mental health interventions — A systematic review. Internet Interventions. 2014;1(4):205-215.
49. Mohr DC, Ho J, Duffecy J, et al. Effect of telephone-administered vs face-to-face cognitive behavioral therapy on adherence to therapy and depression outcomes among primary care patients: a randomized trial. JAMA. 2012;307(21):2278-2285. PMID: 22706833.
50. Proudfoot J, Clarke J, Birch MR, et al. Impact of a mobile phone and web program on symptom and functional outcomes for people with mild- to-moderate depression, anxiety and stress: a randomised controlled trial. BMC Psychiatry. 2013;13:312. PMID: 24237617.
51. Agyapong VI, Ahern S, McLoughlin DM, Farren CK. Supportive text messaging for depression and comorbid alcohol use disorder: single-blind randomised trial. J Affect Disord. 2012;141(2- 3):168-176. PMID: 22464008.
52. Ben-Zeev D, Brenner CJ, Begale M, Duffecy J, Mohr DC, Mueser KT. Feasibility, Acceptability, and Preliminary Efficacy of a Smartphone Intervention for Schizophrenia. Schizophr Bull. 2014. PMID: 24609454.
53. Mohr DC, Tomasino KN, Lattie EG, et al. IntelliCare: An Eclectic, Skills-based App Suite for the Treatment of Depression and Anxiety. J Med Internet Res. 2017;19(1):e10. PMID: 28057609.
54. Ly KH, Topooco N, Cederlund H, et al. Smartphone-Supported versus Full Behavioural Activation for Depression: A Randomised Controlled Trial. PLoS One. 2015;10(5):e0126559. PMID: 26010890.
55. Kenter RF, van de Ven PM, Cuijpers P, et al. Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: Results from a naturalistic study. Internet Interventions. 2015;2(1):77-83.
56. Mohr DC, Weingardt K, Reddy M, Schueller SM. Reevaluating our research approach for technology-enabled mental health services. Psychiatr Serv. Under Review.
57. Clarke G, Reid E, Eubanks D, et al. Overcoming depression on the Internet (ODIN): a randomized controlled trial of an Internet depression skills intervention program. J Med Internet Res. 2002;4(3):E14. PMID: 12554545.
58. Clarke G, Eubanks D, Reid E, et al. Overcoming Depression on the Internet (ODIN) (2): a randomized trial of a self-help depression skills program with reminders. J Med Internet Res. 2005;7(2):e16. PMID: 15998607.
59. Nahum-Shani I, Smith, S. N., Tewari, A., Witkiewitz, K., Collins, L. M., Spring, B., & Murphy, S. Just in time adaptive interventions (jitais): An organizing framework for ongoing health behavior support. Methodology Center technical report. 2014:14-126. https://cmapspublic2.ihmc.us/rid=1P0PJNP7V-12LPKBQ-273F/ .
60. Bohannon J. Artificial intelligence. The synthetic therapist. Science. 2015;349(6245):250-251. PMID: 26185240.
61. Swartout W, Artstein R, Forbell E, et al. Virtual Humans for Learning. Ai Mag. 2013;34(4):13-30. http://www.aaai.org/ojs/index.php/aimagazine/article/view/2487 .
62. Botella C, Serrano B, Banos RM, Garcia-Palacios A. Virtual reality exposure-based therapy for the treatment of post-traumatic stress disorder: a review of its efficacy, the adequacy of the treatment protocol, and its acceptability. Neuropsychiatr Dis Treat. 2015;11:2533-2545. PMID: 26491332.
63. Freeman D, Bradley J, Antley A, et al. Virtual reality in the treatment of persecutory delusions: randomised controlled experimental study testing how to reduce delusional conviction. Br J Psychiatry. 2016;209(1):62-67. PMID: 27151071.
64. Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Lucassen MF. The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: randomised controlled non-inferiority trial. BMJ. 2012;344:e2598. PMID: 22517917.
65. Wilson LS, Maeder AJ. Recent Directions in Telemedicine: Review of Trends in Research and Practice. Healthc Inform Res. 2015;21(4):213- 222. PMID: 26618026.
66. Gros DF, Morland LA, Greene CJ, et al. Delivery of Evidence-Based Psychotherapy via Video Telehealth. J Psychopathol Behav. 2013;35(4):506-521. http://link.springer.com/article/10.1007/s10862-013-9363-4 .
67. Fortney JC, Pyne JM, Mouden SB, et al. Practice- based versus telemedicine-based collaborative care for depression in rural federally qualified health centers: a pragmatic randomized comparative effectiveness trial. Am J Psychiatry. 2013;170(4):414-425. PMID: 23429924.
68. Hilty D, Yellowlees PM, Parrish MB, Chan S. Telepsychiatry Effective, Evidence-Based, and at a Tipping Point in Health Care Delivery? Psychiatr Clin North Am. 2015;38(3):559-592. PMID: 26300039.
69. Chan SR, Torous J, Hinton L, Yellowlees P. Mobile Tele-Mental Health: Increasing Applications and a Move to Hybrid Models of Care. Healthcare (Basel). 2014;2(2):220-233. PMID: 27429272.
70. Yellowlees P, Richard Chan S, Burke Parish M. The hybrid doctor-patient relationship in the age of technology - Telepsychiatry consultations and the use of virtual space. Int Rev Psychiatry. 2015;27(6):476-489. PMID: 26493089.
71. Horton K, Malcarney MB, Seiler N. Medicare Payment Rules and Telemedicine. Public Health Rep. 2014;129(2):196-199. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3904900/ .
72. Bauer AM, Thielke SM, Katon W, Unutzer J, Arean P. Aligning health information technologies with effective service delivery models to improve chronic disease care. Prev Med. 2014;66:167-172. PMID: 24963895.
73. Druss BG, Dimitropoulos L. Advancing the adoption, integration and testing of technological advancements within existing care systems. Gen Hosp Psychiatry. 2013;35(4):345-348. PMID: 23643032.
74. Druss BG, Ji X, Glick G, von Esenwein SA. Randomized trial of an electronic personal health record for patients with serious mental illnesses. Am J Psychiatry. 2014;171(3):360-368. PMID: 24435025.
75. Minkoff K, Parks J. Primary Health–Behavioral Health Integration for the Population of Individuals with Serious Mental Illness. In: O’Donohue W, Maragakis A, eds. Integrated Primary and Behavioral Care: Springer; 2015:171- 199.
76. Rossom RC, Simon GE, Beck A, et al. Facilitating Action for Suicide Prevention by Learning Health Care Systems. Psychiatr Serv. 2016;67(8):830- 832. PMID: 27032667
77. Hoagwood KE, Essock S, Morrissey J, et al. Use of Pooled State Administrative Data for Mental Health Services Research. Adm Policy Ment Health. 2016;43(1):67-78. PMID: 25578511
78. Heinssen R. NAMHC Concept Clearance. Early Psychosis Intervention Network (EPINET): A Learning Healthcare System for Early Serious Mental Illness. Bethesda, MD: NIMH; February 6, 2015 2015. https://www.nimh.nih.gov/funding/grant-writing-and-application-process/concept-clearances/2015/early-psychosis-intervention-network-epinet-a-learning-healthcare-system-for-early-serious-mental-illness.shtml.
79. Health NCfCB. HIT Adoption and Readiness for Meaningful Use in Community Behavioral Health. 2012. https://www.google.com/webhp?sa=N&hl=en&tab=lw&gws_d=ssl#hl=en&q=HIT+Adoption+and+readi-ness&spf=1 .
80. The Office of the National Coordinator for Health Information Technology (ONC) Office of the Secretary USDoHaHS. UPDATE ON THE ADOPTION OF HEALTH INFORMATION TECHNOLOGY AND RELATED EFFORTS TO FACILITATE THE ELECTRONIC USE AND EXCHANGE OF HEALTH INFORMATION. 2016. https://www.healthit.gov/sites/default/files/Attachment_1_-_2-26-16_RTC_Health_IT_Progress.pdf .
81. Cifuentes M, Davis M, Fernald D, Gunn R, Dickinson P, Cohen DJ. Electronic Health Record Challenges, Workarounds, and Solutions Observed in Practices Integrating Behavioral Health and Primary Care. J Am Board Fam Med. 2015;28 Suppl 1:S63-72. PMID: 26359473.
82. Spil T, Klein R. Personal Health Records Success; Why Google Health failed and what does that mean for Microsoft HealthVault? P Ann Hicss. 2014:2818-2827. http://ieeexplore.ieee.org/document/6758953/ .
83. New technology will allow patients to become partners in research. Yale News. 05-05-2016, 2016. http://news.yale.edu/2016/05/05/new-technology-will-allow-patients-become-partners-research .
84. Morgan AJ, Jorm AF, Mackinnon AJ. Internet- based recruitment to a depression prevention intervention: lessons from the Mood Memos study. J Med Internet Res. 2013;15(2):e31. PMID: 23403043.
85. Karyotaki E, Kleiboer A, Smit F, et al. Predictors of treatment dropout in self-guided web-based interventions for depression: an ‘individual patient data’ meta-analysis. Psychol Med. 2015;45(13):2717-2726.PMID: 25881626.
86. eMarketer. Push notification lifts retention rates. 2015. https://www.emarketer.com/Article/Push-Notifications-Lift-Retention-Rates/1012782 .
87. Ben-Zeev D, Scherer EA, Gottlieb JD, Rotondi AJ, Brunette MF, Achtyes ED, Mueser KT, Gingerich S, Brenner CJ, Begale M, Mohr DC, Schooler N, Marcy P, Robinson DG, Kane JM. mHealth for Schizophrenia: Patient Engagement With a Mobile Phone Intervention Following Hospital Discharge. JMIR Ment Health. 2016;3(3):e34. doi: 10.2196/mental.6348. PMID: 27465803.
88. Epstein DA, Ping A., Fogarty J., Munson, S.A. A Lived Informatics Model of Personal Informatics. Proceedings of the International Conference on Pervasive and Ubiquitous Computing (UbiComp). 2015:731-742. http://dl.acm.org/citation.cfm?id=2750858.2804250 .
89. Nebeker C, Linares-Orozco R, Crist K. A multi- case study of research using mobile imaging, sensing and tracking technologies to objectively measure behavior: Ethical issues and insights to guide responsible research practice. J Res Adm. 2015;46(1):118-137.
90. Bloss C, Nebeker C, Bietz M, Bae D, Bigby B, Devereaux M, Fowler J, Waldo A, Weibel N, Patrick K, Klemmer S, Melichar L. Re-Imagining Human Research Protections for 21st Century Science. J Med Internet Res. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5216254/ .
91. Hekler EB, Klasnja P, Riley WT, Buman MP, Huberty J, Rivera DE, Martin CA. Agile science: creating useful products for behavior change in the real world. Transl Behav Med. 2016;6(2):317- 28. doi: 10.1007/s13142-016-0395-7. PMID: 27357001.

National Advisory Mental Health Council Workgroup

Co-Chairs

PATRICIA A. AREÁN, PH.D.
Professor
Director of Targeted Treatment
Development Department of Psychiatry and Behavioral Sciences
University of Washington
Seattle, WA

ALAN LESHNER, PH.D.
American Association for the Advancement of Science
Washington, DC

Members

DEANNA M. BARCH, PH.D.
Gregory B. Couch Professor of Psychiatry
Department of Psychology, Psychiatry and Radiology
Washington University
St. Louis, MO

DROR BEN-ZEEV, PH.D.
Associate Professor of Psychiatry
Director, mHealth for Mental Health Program
Dartmouth Psychiatric Research Center Department of Psychiatry
Geisel School of Medicine at Dartmouth College
Lebanon, NH

DAVID A. BRENT, M.D.
Academic Chief
Child & Adolescent Psychiatry Endowed Chair in Suicide Studies
Professor of Psychiatry, Pediatrics and Epidemiology
Director, Services for Teens at Risk
University of Pittsburgh School of Medicine
Pittsburgh, PA

RANDALL L. CARPENTER, M.D.
Chief Scientific Officer
Rett Syndrome Research Trust
Trumbull, CT

BJ CASEY, PH.D.
Sackler Professor
Department of Psychiatry and Neuroscience
Sackler Institute for Developmental Psychobiology
Weill Medical College of Cornell University
New York, NY

JOHN W. DAVISON, M.B.A., PH.D.
Chief, Conditioned-Based Specialty Care Section Clinical Support Division
Defense Health Agency
Department of Defense
Office of the Chief Medical Officer (OCMO)
TRICARE Management Activity, OASD (HA)
Falls Church, VA

BENJAMIN G. DRUSS, M.D., M.P.H.
Rosalynn Carter Chair in Mental Health and Professor
Department of Health Policy and Management
Rollins School of Public Health
Emory University
Atlanta, GA

MICHAEL F. HOGAN, PH.D.
Consultant and Advisor
Hogan Health Solutions LLC
Delmar, NY

MARSHA M. LINEHAN, PH.D.
Professor and Director
Behavioral Research and Therapy Clinics
Department of Psychology
University of Washington
Seattle, WA

DAVID C. MOHR, PH.D.
Director
Northwestern University Center for Behavioral Intervention Technologies
Chicago, IL

CAMILLE NEBEKER, ED.D., M.S.
Assistant Professor,
Behavioral Medicine and Global Health
Department of Family Medicine and Public Health
University of California, San Diego School of Medicine
La Jolla, CA

ROSALIND PICARD, SC.D.
Director of Affective Computing Research
Co-director of Advancing Wellbeing Initiative
Professor of Media Arts and Sciences
Massachusetts Institute of Technology
Cambridge, MA

RUSSELL POLDRACK, PH.D.
Professor of Psychology
Department of Psychology
Stanford University
Stanford, CA

GENE E. ROBINSON, PH.D.
Director,
Carl R. Woese Institute for Genomic Biology
Swanlund Chair Center for Advanced Study
Professor in Entomology and Neuroscience
University of Illinois at Urbana-Champaign
Urbana, IL

RHONDA J. ROBINSON BEALE, M.D.
Senior Vice President and Chief Medical Officer
Senior Vice President and Chief Medical Officer
Blue Cross of Idaho
Meridian, ID

MARY JANE ROTHERAM, PH.D.
Bat-Yaacov Professor of Child Psychiatry and Behavioral Sciences
Director, Global Center for Children and Families
Director, Center for HIV Identification Prevention and Treatment Services (CHIPTS)
Semel Institute and the Department of Psychiatry
University of California, Los Angeles
Los Angeles, CA

KARAN SINGH, M.B.A.
Co-founder
Ginger.io
San Francisco, CA

National Institutes of Health Staff

YANCY BODENSTEIN, M.S., M.P.H.
Office of Science Policy, Planning and Communications
National Institute of Mental Health
NSC Building
Rockville, MD

MINDY CHAI, PH.D.
Office of Science Policy, Planning and Communications
National Institute of Mental Health
NSC Building
Rockville, MD

DAVID CHAMBERS, D.PHIL.
Deputy Director for Implementation Science
Division of Cancer Control & Population Sciences
National Cancer Institute
Rockville, MD

KAREN GAVIN-EVANS, PH.D.
Division of Extramural Activities
National Institute of Mental Health
NSC Building
Rockville, MD

ADAM HAIM, PH.D.
Office of Clinical Research
National Institute of Mental Health
NSC Building
Rockville, MD

JEAN NORONHA, PH.D.
Director Division of Extramural Activities
National Institute of Mental Health
NSC Building
Rockville, MD

JANE PEARSON, PH.D.
Chief, Suicide Preventive Interventions Programs
National Institute of Mental Health
NSC Building
Rockville, MD

BILL RILEY, PH.D.
Director
Office of Behavioral and Social Sciences Research
National Institutes of Health
Rockville, MD

JOEL SHERRILL, PH.D.
Acting Deputy Director Adult Treatment Intervention Research Branch
Division of Services and Intervention Research
National Institute of Mental Health
NSC Building
Rockville, MD

JANINE SIMMONS, M.D., PH.D.
Behavioral Science and Integrative Neuroscience Research Branch
Division of Neuroscience and Basic Behavioral Science
National Institute of Mental Health
NSC Building
Rockville, MD

TRACY WALDECK, PH.D.
Acting Deputy Director
Division of Extramural Activities
National Institute of Mental Health
NSC Building
Rockville, MD
(now, Deputy Director, Association for Psychological Science)