Technology Research & Development Project 1 (TR&D1): Discovery

Enabling the Discovery of Temporally-Precise Intervention Targets and Timing Triggers from mHealth Biomarkers via Uncertainty-Aware Modeling of Personalized Risk Dynamics

TR&D1 Lead: Jim Rehg

Co-Investigators: Benjamin Marlin, Santosh Kumar

The past decade has seen tremendous advances in the ability to compute a diverse array of mobile sensor-based biomarkers in order to passively estimate health states, activities, and associated contexts (e.g. physical activity, sleep, smoking, mood, craving, stress, and geospatial context). Researchers are now engaged in the conduct of both observational and interventional field studies of increasing complexity and length that leverage mHealth sensor and biomarker technologies combined with the collection of measures of disease progression and other outcomes. As a result of the expansion of the set of available mHealth biomarkers and the push toward long-term, real-world deployment of mHealth technologies, a new set of critical gaps has emerged that were previously obscured by the focus of the field on smaller-scale proof-of-concept studies and the investigation of single biomarkers in isolation.

First, the issue of missing sensor and biomarker data in mHealth field studies has quickly become a critical problem that directly and significantly impacts many of our CPs. Issues including intermittent wireless dropouts, wearables and smartphones running out of battery power, participants forgetting to carry or wear devices, and participants exercising privacy controls can all contribute to complex patterns of missing data that significantly complicate data analysis and limit the effectiveness of sensor-informed mHealth interventions. Second, with increasing interest in the use of reinforcement learning methods to provide online adaptation of interventions for every individual, there is an urgent need for high-quality, compact and interpretable feature representations that can enable more effective learning under strict budgets on the number of interactions with patients. Finally, as in other areas that are leveraging machine learning methods to drive scientific discovery and support decision making, mHealth needs methods that can be used to derive high-level knowledge and support causal hypothesis generation based on complex, non-linear models fit to biomarker time series data.

Research Goals

TR&D1 is conducting the following innovative research to address the technological challenges described above:

  • Model and represent uncertainty in mHealth biomarkers to account for multifaceted uncertainty during momentary decision making in selecting, adapting, and delivering temporally-precise mHealth interventions.
  • Derive uncertainty-aware composite risk scores to identify timing triggers for delivering temporally-precise interventions.
  • Model the time-varying dynamic relationships between personalized drivers of momentary risk and disease progression to identify targets of temporally-precise interventions.

Technological Resources

TR&D1 is producing the following technological resources for the community:

  • Reference imputation models for widely used sensor data modalities including IMU, PPG, RIP, GPS, and key biomarkers including stress, steps, and cigarette smoking
  • A computational toolbox consisting of generalizable methods for estimating and personalizing risk scoring models from uncertain and incomplete risk factors
  • Reference implementations for several specific risk scores, including risk for smoking lapse, intervention disengagement, sedentary behavior, and alert fatigue
  • A computational toolbox and cloud-based analysis platform consisting of methods for analyzing risk score dynamics and the relationships between risk factors and risk scores reflecting the uncertainty across all prior levels of analysis

Impact on Science & Society

TR&D1's technologies have a significant potential to advance the fundamental understanding of health and behavior by supporting the analysis of complex, longitudinal, mHealth data. Improvements in modeling and propagating uncertainty due to missing data will lead to enhanced longitudinal coverage of computed biomarker values in addition to well-calibrated uncertainty in biomarker values. Research on data-efficient personalization of dynamic risk scores will lead to novel ideographic insights about the temporal relationships between context, risk factors, intervention receptivity and engagement, and health outcomes. Methods for analyzing temporal dynamics of risk will be used to identify the most opportune timing for delivering interventions, while the identification of the most salient risk factors that drive adverse outcomes at the current moment can be used to optimize the intervention content. By identifying the modifiable risk factors whose change is likely to have the greatest impact on the risk of adverse outcomes at specific times, interventions can be designed with maximum temporal precision and efficacy. By enabling the delivery of interventions in the face of data loss and uncertainty due to a wide range of adverse real-world conditions, TR&D1 will improve the overall robustness of mHealth interventions in real-world deployments, leading directly to improved health outcomes.

Recent Publications

Coming soon


TR&D1 Team

Jim Rehg, Ph.D.

Deputy Center Director, TR&D1 Lead

Jim Rehg

Georgia Tech

Dr. Jim Rehg (pronounced 'ray") is a Professor in the School of Interactive Computing at the Georgia Institute of Technology, where he is co-Director of the Computational Perception Lab (CPL) and Director of the Center for Behavioral Imaging. He received his Ph.D. from CMU in 1995 and worked at the Cambridge Research Lab of DEC (and then Compaq) from 1995-2001, where he managed the computer vision research group. He received an NSF CAREER award in 2001 and a Raytheon Faculty Fellowship from Georgia Tech in 2005. He and his students have received best student paper awards at ICML 2005, BMVC 2010, Mobihealth 2014, and Face and Gesture 2015, and a 2013 Method of the Year Award from the journal Nature Methods. Dr. Rehg serves on the Editorial Board of the Intl. J. of Computer Vision, and he served as the Program co-Chair for ACCV 2012 and General co-Chair for CVPR 2009, and will serve as Program co-Chair for CVPR 2017. He has authored more than 100 peer-reviewed scientific papers and holds 25 issued US patents. His research interests include computer vision, machine learning, pattern recognition, and robot perception. Dr. Rehg is the lead PI on an NSF Expedition to develop the science and technology of Behavioral Imaging, the measurement and analysis of social and communicative behavior using multi-modal sensing, with applications to developmental disorders such as autism. He also serves as the Deputy Director of the NIH Center of Excellence on Mobile Sensor Data-to-Knowledge (MD2K). Visit Google Scholar page


Benjamin Marlin, Ph.D.

Co-I, TR&D1, TR&D2

Benjamin Marlin

U Mass Amherst

Dr. Benjamin Marlin joined the College of Information and Computer Sciences at the University of Massachusetts Amherst in 2011. There, he co-directs the Machine Learning for Data Science lab. His current research centers on the development of customized probabilistic models and algorithms for time series with applications to the analysis of electronic health records and mobile health data. His recent work includes probabilistic models for analyzing wireless ECG data, detection of cocaine use from wireless ECG, hierarchical activity recognition from on-body sensor data with applications to smoking and eating detection, and methods for mitigating lab-to-field generalization loss in mobile health studies. Marlin is a 2014 NSF CAREER award recipient and a 2013 Yahoo! Faculty Research Engagement Program award recipient. His research has also been supported by the National Institutes of Health, the Patient-Centered Outcomes Research Institute, and the US Army Research Laboratory. Prior to joining UMass Amherst, Marlin was a fellow of the Pacific Institute for the Mathematical Sciences and the Killam Trusts at the University of British Columbia. He completed his PhD in machine learning in the Department of Computer Science at the University of Toronto. Visit Google Scholar page


Santosh Kumar, Ph.D.

Lead PI, Center Director, TR&D1, TR&D2, TR&D3

Santosh Kumar

University of Memphis

Dr. Santosh Kumar is the Lillian and Morrie Moss Chair of Excellence Professor in the Department of Computer Science at the University of Memphis and the Director of the NIH Center of Excellence for Mobile Sensor Data-to-Knowledge (MD2K), which is headquartered at the University of Memphis. He received his Ph.D. in Computer Science and Engineering from The Ohio State University in 2006, where his dissertation won a presidential fellowship. In 2010, Popular Science magazine named him one of America’s ten most brilliant scientists under the age of 38 (called “Brilliant Ten”). In 2011, he chaired the “mHealth Evidence” meeting jointly organized by NIH, NSF, RWJF, and McKesson Foundation to establish evidence requirements for mHealth. In 2013, he was invited to meet with the NIH Director to advise him on NIH efforts in the area of mHealth and was invited to the White House to give a talk on the future of Biosensors. In 2014, he co-organized and co-chaired the NSF-NIH Workshop on Computing Challenges in Future Mobile Health (mHealth) Systems and Applications. He holds the distinction of receiving the largest grants from both NIH ($10.8 million in 2014) and NSF ($4 million In 2016) in the history of the University of Memphis. Santosh’s research seeks to define new frontiers in the discipline of mobile health (mHealth). His decade-long work has involved collecting mobile sensor data from over 100 human volunteers for 25,000+ hours in their natural environments as part of various scientific user studies. His collaborative research involves more than twenty faculty members from fifteen institutions, spanning a variety of disciplines, making his projects highly transdisciplinary. Visit Google Scholar page