Current Research

Modeling of Human Speech and Behavior

1. Audio-visual speech recognition
A speech recognition interface is attracting attention as a hands-free computer interface for controlling such in-car systems as navigation and audio. However, speech recognition performance is degraded when affected by such background noises as engine, wind, road, and music. We are investigating audio-visual speech recognition to improve the robustness and accuracy of speech recognition by utilizing the visual information of the speaker's mouth region. For research of audio-visual speech recognition in real environments, we collected driver speech and facial images using the data collection vehicle shown in the figure while driving on city roads and expressways. Preliminary experiments have shown that speech recognition performance can be improved by using visual information with acoustic speech. Our future research topics include :
a) Accurate endpoint detection using audio-visual information
b) Robust visual feature extraction from facial images
c) Intelligent feature integration of audio and visual information
2. Person recognition
2.1. Speaker recognition
Biometric authentication systems have proved valuable over many years in security applications. Speaker recognition is a biometric recognition technique that identifies individuals based on their voice. We have proposed a new framework for designing a feature extractor in speaker recognition systems based on frequency warping using a second-order all-pass function and demonstrated that the proposed method enables us to find a frequency scale relevant to accurate speaker classification. We have also proposed a speaker modeling technique using speech spectra and fundamental frequencies (F0) in a unified framework and showed that speaker recognition accuracy can be improved by using F0 information with spectral information.
2.2. Driver recognition
All drivers have habits when behind the wheel. We focus on modeling driving behaviors of each driver such as car-following and pedal operation patterns. The relationship between following distance and velocity mapped into a two-dimensional space was modeled for each driver with a monotonically increasing nonlinear function or with a statistical method of a Gaussian mixture model (GMM). Gas and brake pedal operation patterns were also modeled with GMMs that represent the distributions of raw pedal operation signals or spectral features extracted through spectral analysis of the raw pedal operation signals. Driver models were evaluated in driver identification experiments using driving signals collected in a simulator and in a real vehicle on city roads. Experimental results showed that the driver model based on the spectral features of pedal operation signals efficiently models driver individual differences and achieved an identification rate of 76.8% in a field test with 276 drivers.
3. Sign language recognition
We are focusing on sign language recognition and text-to-sign animation synthesis. We collected 3D motion data of Japanese words consisting of finger alphabets signed by novices and experts. 98% and 92% recognition rates were obtained in signer-dependent and -independent large-vocabulary sign language recognition, respectively. We are integrating the sign language recognition system with a text-to-sign animation system. They are expected to be applied to sign language learning and translation systems to bridge the communication gap between deaf individuals and the hearing community who do not understand sign language.
Keywords : Lip reading, driving behavior, cepstal analysis, Gaussian mixture model, sign language, finger alphabet, hidden Markov model