Rose Faghih

Wearable monitoring is likely to play a key role in the future of healthcare. In many cases, wearable devices may monitor our physiological signals that can indicate mental states, such as emotions. The lab of Rose Faghih has been developing a system called MINDWATCH, algorithms and methods for wearable sensors that collect information from electrical signals in the skin to make inferences about mental activity. While their lab has been successful in translating these physiological signals quickly and effectively, they didn't incorporate direct feedback from the individual’s subjective experiences.

Now, the researchers are incorporating feedback and labels from the users, enhanced with machine-learning, and combining it with their existing model for a fuller, more accurate picture of mental states.

At a high level, the human body can be viewed as a complex dynamical system. It is a complex conglomeration of control systems that each works in turn to manage different variables or states.  Unfortunately, a number of the states researchers are interested in — particularly the ones that are more abstract — cannot be measured directly. These include states of emotion, cognition and consciousness. Nevertheless, changes in these unobserved states do give rise to corresponding changes in different physiological signals that can be measured more easily. For instance, we may not be able to directly observe or measure a person’s emotional state, but we can measure subtle changes in a person’s heart rate, breathing or sweat secretions (which in turn affects the conductivity of the skin). These signals can then be used to estimate the states we wish to. 

However, the signals researchers use to estimate these unobserved states are “spikey” or “pulsatile” in nature. These spikey signals can be used to estimate the various states of the human body and brain without direct observation.  With already-existing methods, you could obtain state estimates, but still wouldn’t have any means to have those estimates agree with the more direct state-related information in possession. This is especially important in experiments involving human subjects where the subjects can indeed provide information related to the unobserved state. This type of more-direct state-related information can be called a “label.”

Incorporating direct feedback from users offers information that can’t be gleaned from biological data alone. For instance, a person with PTSD could have their skin conductance continuously monitored to provide an emotion estimate, but ideally, the final estimate should rely both on these signals and information perhaps obtained on rating scales or through regular questionnaires. It is likewise the case for patients with hormone disorders. Hormone measurements do provide valuable information, but should likely be combined with personal feedback (e.g. regarding feelings of energy/lethargy) to obtain a single complete picture. The authors met this need through a proof-of-principle work on a hybrid type of estimator. 

Performing estimation on some data such that what is predicted agrees with available labels falls within the domain of supervised machine learning. This work adapted an existing neural network method for state estimation by adding a penalization term for not agreeing with the labels to enable a hybrid estimator. The proposed hybrid estimator was utilized to determine an aspect of emotion tied to changes in skin conductance (through changes in sweat secretions) and to determine energy states within the body based on pulsatile hormone secretions. A wearable monitoring system that incorporates verbal feedback from the user with physiological signals for hybrid estimation can eventually provide a more complete picture of the user to eventually provide more comprehensive closed-loop care. 
 

Stress has a negative impact on physical health, reduces work productivity, and results in significant annual costs for industries and healthcare. While high stress is known to raise the risk of cardiovascular disease and have negative effects on mental health, It also has key effects on the ability of one to complete tasks by both excessively high or excessively low stress. There has been growing research interest on understanding how real-world stress impacts our body and performance, at work and across life activities

Unfortunately, attempts to simulate their impact in the laboratory or elsewhere are less useful than datasets gathered in real-world circumstances. As a result, researchers have access to fewer real-world stress datasets. Even rarer indeed are such datasets used in longitudinal investigations on the same subjects over time.

Real-world situations are also unrestricted environments. Research-grade equipment is frequently inaccessible, and motion artifact contamination is pervasive. These continue to be some of the biggest barriers to automated emotion decoders outside of the research labs in daily life.

To address the above-mentioned gap, Rose Faghih and her former PhD students Md. Rafiul Amin and Dilranjan Wickramasuriya performed an experiment, in which a set of students' physiological data was gathered over the course of three exams. They used a smartwatch-like wearable device and collected multimodal physiological data. The use of the smartwatch-like wearable device was to provide a seamless data collection experience for the students participating in the experiment. 

The investigation shows that it is possible to link the variations in the physiological signals to the exam performance. More details about this study can be found in the corresponding publication titled "A Wearable Exam Stress Dataset for Predicting Grades using Physiological Signals."

To enable other researchers, use this dataset for additional investigations, the research team has made the de-identified data publicly available on the PhysioNet platform. A Wearable Exam Stress Dataset for Predicting Cognitive Performance in Real-World Settings is available at: physionet.org/content/wearable-exam-stress

Ultimately, the researchers believe it would be extremely beneficial to consider how exam performance and the stress that goes along with it interact. It will allow for a wide range of potential applications with the aim of enhancing personal performance. This may, for instance, assist scientists  in developing effective interventions to improve each person's performance and increase productivity within a company. Additionally, the knowledge may be used in online and remote learning contexts to connect with students effectively and improve learning outcomes.

Emotions directly influence the way we think and interact with others in different situations. Additionally, when it interferes with rationality in our decision-making or perception, emotions play an undeniable role. 

From another perspective, studies from different fields indicate that human emotions could be impacted by any changes in the environment. Thus, having a solid understanding of the dynamics of emotions could benefit individuals in multiple ways: from the enhancement of therapeutic solutions and maintaining well-being to the prevention of unpleasant emotions and inducing favorable ones. Decoding emotions has been an ongoing research among researchers, dictating joint efforts from behavioral, physiological, and computational angles.

According to Russell's circumplex model, emotions are defined in a two-dimensional arousal-valence circular space. Compared to the arousal index, which designates the severity of the emotional state, internal valence levels determine the quality of emotions. In a high arousal mood, feeling excited or happy is opposed to nervousness and feeling stressed. On the flip side and in low arousal conditions, staying calm stands on the opposite side of feeling bored. Hence, feeling pleasant, which refers to the positive valence, is one of the key factors in one’s health condition and could result in a better quality of life. 

In a new piece of research, a team led by Rose Faghih, Associate Professor of Biomedical Engineering, follows the goal of establishing a mechanism to track an individual's emotional valence level and regulate it in a closed-loop framework. Since the emotional valence state is hidden and cannot be measured directly, the authors need to measure it indirectly. 

The use of wearable devices and designing decoders to gain insight into the internal brain states provides a good alternative to directly measuring brain activity. Inspired by recent advances in wearable technologies which could measure various physiological signals from humans in the loop, there exists possibilities to track emotional valence state indirectly and in a more practical manner. Among different physiological signals, it has been shown that facial electromyography (EMG) carries valuable information about the internal valence state. According to various research, facial EMG measurements of the zygomaticus muscle (zEMG) give the most distinct indicator of emotional valence state. In fact, changes in the zygomaticus muscle, which could be captured via wearables, could provide researchers with valuable information regarding one’s valence level.

In this research, the authors employ a mathematical framework to relate changes in zEMG signal for inferring the internal valence state. They extract corresponding features from the zEMG signal and estimate the internal valence state. Taking advantage of tracking of emotional valence state in a systematic way, they then propose a knowledge-based control architecture to close the loop and keep the estimated valence state within a favorable range. Within a closed-loop approach, biomarkers are collected to provide information regarding the individual's valence state to then automatically adjust the actuators in response to the current emotional state.

In silico results verify that, by making use of physiological data, the proposed controller could effectively regulate the estimated valence state. Ultimately, it is envisioned that the future outcomes of this research will include supporting alternative forms of self-therapy by using wearable machine interface architectures capable of mitigating periods of pervasive emotions and maintaining daily well-being and welfare. This study is the very first closed-loop control framework for regulating emotional valence state using facial muscle activity. This research suggests an important new step toward new clinical applications and the self-management of mental health. In the practical implementation of the proposed architecture, a wearable device would collect zEMG signal, and the appropriate practices (e.g., listening to music, breathing exercises) would be suggested to maintain the emotional valence levels within the favorable range.

Faghih, who joined NYU earlier in 2022, has been integrating research into her classroom teaching. The research covered in this Frontiers in Computational Neuroscience paper started as a course project in Faghih’s “State- Space Estimation with Physiological Applications” course at the University of Houston. In this class students had the opportunity to use advanced methods learned in the course to analyze physiological data in hands-on biomedical engineering projects designed by Faghih. Student authors Luciano Branco and Arian Ehteshami were graduate students in the course, and Hamid Fekri Azgomi, who recently completed his PhD under the supervision of Faghih, was a PhD student mentor for the project.

In the modern world, any challenge might be a source of stress. The fast-paced life has the potential to induce emotional and cognitive stress. Feeling overwhelmed, anxious, and agitated are among the symptoms associated with high levels of cognitive stress. Conversely, loss of cognitive engagement might also prevent individuals from following their goals. A low level of positive stress, which is also called eustress, might cause memory problems, lack of motivation, and poor concentration. It can also negatively affect individuals’ productivity in the workplace. According to multiple reports from “Mental Health America”, the percentage of people with depression and anxiety is growing in the United States. Thus, it is crucial to establish a mechanism for tracking one's cognitive stress level and keeping it within a favorable range. 

Influenced by the recent advances in wearable technologies and inspired by the fact that skin conductance signal carries important information regarding one’s internal arousal state, there could be practical approaches to handle cognitive stress levels in daily life. In response to any environmental stimuli, the human brain controls the autonomic nervous system in multiple ways. As changes in electrodermal activity (EDA) signals are impacted by the changes in sweat glands activation, variations in skin conductance signal can be tracked in real-time using wearable devices. 

In order to eventually enable wearable technologies to track and regulate one's internal stress state, the lab of Rose Faghih has pioneered methods for estimating the internal arousal state by monitoring one's EDA, i.e., fluctuations in their skin conductivity. They have developed a mathematical framework to model the internal arousal state and relate it to the changes in skin conductance signal (i.e., a signal that can be measured via wrist-worn wearable devices). Moreover, to estimate the hidden arousal state, they have developed methods that analyze the EDA signal and extract the underlying arousal events that activate sweat glands.  

Systematic modeling of the internal arousal state enables developing controllers to close the loop and regulate the estimated stress state. In this regard, well-established control techniques (i.e., linear quadratic regulator and model predictive controller) can be applied. In the process of finding the optimal control policy to regulate the stress state within these approaches, a corresponding cost function needs to be minimized. Hence, the performance of these control systems highly depends on the selection of corresponding objective functions. 

As the researchers are dealing with a human-in-the-loop problem, the process of finding the appropriate objective functions becomes more complicated. To overcome this challenge and enhance the performance of the closed-loop system, Faghih and Fekri Azgomi, who recently completed his PhD under the supervision of Faghih, developed a supervised controller that includes a knowledge-based supervised layer that adjusts the objective function and tunes parameters in real-time. In this supervised architecture, a fuzzy system is utilized to continuously update parameters in the objective function to adjust environmental variations. The results verify the effectiveness of proposed architecture in keeping the estimated stress state within a desired range by designing a supervised layer on top of the optimal control formulation.  

Furthermore, the proposed supervised control architectures provide an excellent setting to incorporate the relevant medical expertise to enhance the closed-loop system. These novel supervised control approaches could be further expanded to deliver adaptive and robust closed-loop characteristics to address inter- and intra-subject variations. 

There also exists great potential in implementing the supervised control architectures in other closed-loop applications such as adaptive deep brain stimulation for the treatment of neurodegenerative brain disorders and automating medication intake in a wide range of diseases. Once one models and tracks one's internal states, utilizing supervised control methodologies would result in enhancing the closed-loop treatment practices in a more personalized manner.