National Advisory Mental Health Council Workgroup on High-Dimensional Datasets

Rationale

High-dimensional phenotypes research

Basic and applied research on high-dimensional phenotypes has dramatically increased in recent years. This research includes studies targeting the genome, epigenome, transcriptome, proteome, metabolome, and microbiome, as well as neuroimaging studies and their combination.

Research on high-dimensional phenotypes typically seeks to identify markers of diagnostic, prognostic, and therapeutic value or examine the pathophysiological processes underlying mental illnesses.

For example, NIMH has funded several studies examining the links between epigenetic changes that result from stress and other environmental factors and downstream behavioral effects.

NIMH Council Workgroup on Genomics

Progress in the field of genomics provides an example of how NIMH has successfully navigated a similar issue by creating the NIMH Council Workgroup on Genomics. The workgroup collaborated with leading extramural investigators and sought help and advice from the National Advisory Mental Health Council (NAMHC).

This effort culminated in a series of recommendations meant to define priorities and rigorous approaches in acquiring, analyzing, and interpreting genetic findings and their application to subsequent biological and clinical inquiries.

Based on the recommendations from that report, NIMH provided guidance to the scientific community clarifying issues such as:

• The strength of required evidence for association
• Priorities for future studies of genetic association
• Acceptable designs and approaches for biological and clinical studies that seek to build on these findings

The challenges of generating reproducible findings for high-dimensional datasets

By contrast, there has been limited success in generating reproducible findings from studies using other “omics” or similarly complex, high-dimensional datasets. The complexity and size of these datasets and the relative novelty of statistical and machine learning approaches to analyze them have led to concerns regarding rigor and reproducibility.

One challenge is that newer high-dimensional phenotypes such as gene expression, microbiome composition, or whole-brain neural activity are dynamically regulated in a context dependent manner. Therefore, a single snapshot in time may not capture a reliable picture of a disease state or its underlying pathobiology.

A second issue is the field has not identified a standardized framework for experimental design and statistical inference, leading to the persistence of confounding effects and difficulties in interpreting the significance of findings.

Finally, many functional studies of high-dimensional phenotypes lack a coherent conceptual framework for understanding the relationship between molecular, neural, and behavioral variation.

There are considerable costs associated with acquiring and maintaining these data. NIMH needs to define priorities and approaches to ensure rigor and reproducibility while enabling fiscal responsibility.

High-Dimensional Datasets Workgroup Charge

NIMH convened an ad hoc workgroup to seek advisory input from the NAMHC. The group is charged with developing recommendations regarding appropriate conceptual frameworks and experimental and analytic designs for experiments using high-dimensional datasets.

Potential areas for consideration by the workgroup include:

• Ensuring statistical rigor and enhancing reproducibility
  • Are there proven approaches or experimental designs that can be used to maximize or verify the statistical rigor of findings from high-dimensional datasets?
  • Can we define a hierarchy of approaches for ensuring reproducibility? For example, “repeated leave one out” vs “reproduce in an independent dataset” for verifying machine learning approaches?
  • What are the safeguards necessary when using dimension reduction, clustering, and other data-driven approaches to reducing complexity?
• Understanding the role of hypotheses and conceptual frameworks
  • What is the role of hypothesis-based vs. data-driven studies?
  • What are the different design elements that would apply to these two different approaches?
  • What is an acceptable level of evidence for using hypotheses to reduce the dimensionality/complexity of datasets?
• Considering studies involving peripheral biomarkers
  • Does the availability of “omics” and other high-dimensional approaches, alone or in combination, increase the potential to discover and validate peripheral biomarkers? Or should we continue to discourage these?
  • What approaches or designs might be used to justify or enhance the potential relevance of experiments that utilize peripheral biomarker data?
  • How might peripheral and central high-dimensional measures be rigorously evaluated for predictive or mechanistic relevance?
• Considering studies of potential clinical utility
  • To what extent can high-dimensional measures be of clinical value in psychiatry (predictive, diagnostic, therapeutic, and prognostic value)?
  • Are there design differences that would be appropriate for studies aimed at clinical utility as opposed to those aimed at understanding disease mechanisms?

NAMHC Workgroup on High Dimensional Datasets - Roster

Co-Chairs

Members

NIMH Staff