Wearable-based human flow experience recognition enhanced by transfer learning methods using emotion data

Background: Flow experience is a specific positive and affective state that occurs when humans are completely absorbed in an activity and forget everything else. This state can lead to high performance, well-being, and productivity at work. Few studies have been conducted to determine the human flow experience using physiological wearable sensor devices. Other studies rely on self-reported data. Methods: In this article, we use physiological data collected from 25 subjects with multimodal sensing devices, in particular the Empatica E4 wristband, the Emotiv Epoc X electroencephalography (EEG) headset, and the Biosignalplux RespiBAN – in arithmetic and reading tasks to automatically discriminate between flow and non-flow states using feature engineering and deep feature learning approaches. The most meaningful wearable device for flow detection is determined by comparing the performances of each device. We also investigate the connection between emotions and flow by testing transfer learning techniques involving an emotion recognition-related task on the source domain. Results: The EEG sensor modalities yielded the best performances with an accuracy of 64.97%, and a macro Averaged F1 (AF1) score of 64.95%. An accuracy of 73.63% and an AF1 score of 72.70% were obtained after fusing all sensor modalities from all devices. Additionally, our proposed transfer learning approach using emotional arousal classification on the DEAP dataset led to an increase in performances with an accuracy of 75.10% and an AF1 score of 74.92%. Conclusion: The results of this study suggest that effective discrimination between flow and non-flow states is possible with multimodal sensor data. The success of transfer learning using the DEAP emotion dataset as a source domain indicates that emotions and flow are connected, and emotion recognition can be used as a latent task to enhance the performance of flow recognition.