Current Research

Design and Mining of Gaze-Based Interaction Making Cognitive State Explicit

[The eyes are the windows of the mind? The eyes say more than the mouth?]
Achieving harmony between human beings and the information environment is one of the major issues in the field of computer science. Recognizing the cognitive state of people such as interest or intent by observation of their behavior is very important for achieving the harmonization. As you know, the eyes are the window of the mind. Eye movement is a clue for the recognition. So many researchers in various field such as psychophysics and engineering have analyzed the eye movement from the late 19th century.

Since the eyes are an interface to not only input vast amounts of information but also output the cognitive state, the eye movement should have a valuable role in designing a visual information-mediated human machine interaction. The machine can measure the eye movement as temporally continuous data, while the user unconsciously leaks information reflecting the cognitive state on the measurement. They will experience a natural interaction like casual communication between people. “The eyes say what the mouth doesn’t say at every moment” might be correct though people in the past said “the eyes say more than the mouth.” It is, however, difficult to recognize the cognitive state of human by only passive observation.

[Design of gaze-based interaction for probing responses reflecting cognitive state]
How do we recognize other’s cognitive state in everyday life? We adopt proactive behaviors so that we probe the cognitive state of conversational partners. My past study, for example, showed that observing their reaction timing to the probe made the recognition easy. Based on the findings, Mind Probing has been proposed as a gaze-based interaction strategy, which a machine gives the user a proactive behavior and then recognizes a cognitive state by observing his/her reaction.

[Mining of interaction among visual environment, gaze behavior, and cognitive state]
Mind Probing has the potential to be applied to not only human-to-human and human-to-machine interaction but also human-to-environment interaction. People shift gaze in a reflection of cognitive state and in response to visual stimuli present in our environment. However, eye movement and gaze behavior do not have a simple relationship to cognition because they represent low dimensional information, whereas the cognition seems to be a high dimensional internal state. A key to understanding the relationship is focusing on visual environment dynamics. Many researchers have analyzed visual attention and gaze behavior in laboratory environment with controllable visual stimuli for years. Contrary to the well-established approach, my study attempt is quantitatively describing the visual environment dynamics uncontrollable in everyday space. It opens the way for not only hypothesis testing that analyzes eye movement and gaze behavior for a specific scene but also data mining that extracts characteristic findings form big data. Through this analytic paradigm shift, my study aims to model relationships among the visual environment, the gaze behavior, and the cognitive state.

[Other research themes]
・Modeling timing structure of multimodal data
(Application: Facial image recognition to distinguish intentional smiles from spontaneous smiles, Detection of cognitively distracted driving)
・Extracting distinctive patterns from big data
(Application: Representation of individual facial features, Analysis of the knack and habit of gaze behavior, Retrieval of atypical recipes)