My laboratory, the Bio-ASP Lab, focuses on three main research areas:
(1) Deep learning-based speech signal processing,
(2) Multimodal speech signal processing, and
(3) AI-based assistive oral communication technologies.

Over the past five years, our work has received several prestigious domestic and international awards, including the 2019 MERRY Acoustic Paper Award (Gold Prize), the 2019–2020 Rotary Education Foundation Outstanding Talent Award, the National Innovation Award in 2019, 2020, and 2021, and the 2022 NSTC Future Tech Award. Internationally, we won the IEEE Signal Processing Society Young Author Best Paper Award, the Gold Prize in the Machine Learning Challenges for Hearing Aids Challenge (competing among 15 systems from multiple international research institutions, including Japan’s NTT and JASIT, the University of Sheffield, University College London, Imperial College London, and Medizinische Hochschule Hannover, Germany), and First Place in the 2023 Cough Diagnostic Algorithm for Tuberculosis DREAM Challenge – SubChallenge 2 (with participants from the University of Cambridge, University of California Berkeley, Harvard University, University of Wisconsin, University of Michigan, Brown University, and the University of Munich, Germany). Below, I provide a detailed summary of our achievements in these three areas.

(1) Deep Learning–Based Speech Signal Processing

Building on deep learning theory, we have developed novel algorithms for speech denoising, dereverberation, and channel compensation, all aimed at improving human–human and human–machine communication. Our first—and the world’s first—deep learning–based speech enhancement method was published at Interspeech 2013. This paper has drawn significant attention since its release and has been cited 1,152 times on Google Scholar, demonstrating its high impact in the speech signal processing field. Since then, we have published over 80 related papers (70 in the past five years alone), many of which have had a major influence on this research area. To date, 20 of our papers have been cited more than 50 times, and 12 have been cited over 100 times, with a total of over 5,000 citations in this direction. Notable representative works include:

(Paper #1) “Speech enhancement based on deep denoising autoencoder”, Interspeech 2013, pp. 436–440, Sept. 2013. This is the first paper worldwide to apply deep learning to speech enhancement. It has received over 1,152 citations to date and is recognized as a paradigm-shifting work that profoundly influenced subsequent research in this field.

(Paper #2) “End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics Optimization by Fully Convolutional Neural Networks”, IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 26(9), pp. 1570–1584, April 2018. Published in a top-tier journal, this was the first end-to-end speech enhancement system to operate directly on the waveform, differing from previous spectrum-based methods. It has received 363 citations and earned the IEEE Signal Processing Society Young Author Best Paper Award.

(Paper #3) “MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement”, Proc. ICML 2019 (Long Oral Presentation, with Travel Grant). This was the first GAN-based approach to significantly improve both speech intelligibility and quality in speech enhancement. Since its publication, it has received 428 citations. Due to its outstanding performance, the open-source toolkit SpeechBrain (developed by Prof. Yoshua Bengio’s team) has incorporated MetricGAN+ as a standard enhancement model, which maintained state-of-the-art performance on international benchmarks for an extended period.

(Paper #4) “Multichannel Speech Enhancement by Raw Waveform-mapping using Fully Convolutional Networks”, IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 28, pp. 1888–1900, Feb. 2020. This is the first end-to-end multichannel speech enhancement system published in a top-tier journal. Experimental results confirmed that incorporating multichannel information further improves enhancement performance, opening a new research direction. It has received 68 citations since its publication.

(2) Multimodal Speech Signal Processing

Human communication—both human–human and human–machine—consists of verbal and nonverbal components. While speech conveys verbal content, visual cues can complement and enhance understanding. We have published a series of multimodal speech enhancement systems. A notable work is:

(Paper #5) “Audio-visual Speech Enhancement using Multimodal Deep Convolutional Neural Networks”, IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 2(2), pp. 117–128, April 2018. This is the world’s first journal paper to apply deep learning to combine audio and visual modalities for speech denoising. Results showed that visual information significantly improves the performance of deep learning–based denoising systems. It has received 314 citations since publication and is widely recognized as a pioneering work in multimodal speech enhancement. Building on this direction, we have published several follow-up studies and recently co-organized the COG-MHEAR Challenge at IEEE SLT 2022 in collaboration with Edinburgh Napier University, where I served as International Scientific Committee Co-Chair. I was also invited as a Keynote Speaker at the IEEE ICASSP 2023 Satellite Workshop.

(3) AI-Based Assistive Oral Communication Technologies

In recent years, we have focused on developing AI- and advanced signal processing–based assistive technologies for oral communication, including wireless FM systems, hearing aids, and cochlear implants. Our goal is to use cutting-edge AI algorithms to improve existing hearing-assistive devices, enhancing auditory performance and quality of life for individuals with hearing loss.

In addition to hearing-assistive technologies, we have developed novel speaking aids aimed at improving the intelligibility of speech produced by individuals with articulation disorders, thereby facilitating more effective communication. We are also working on AI-based intelligibility prediction models that estimate how well hearing-impaired listeners can understand different speech signals, enabling optimal assistive device configurations.

Our technologies have achieved international recognition, winning the Machine Learning Challenges for Hearing Aids Challenge Gold Prize . We have also created a deep learning–based abnormal voice detection platform, enabling at-home monitoring for potential voice disorders. Many of these projects have yielded highly positive results, including one representative paper that earned the Elsevier Journal of Voice’s “Most Cited Article” honor in 2022. Since its 2019 publication, this work has received 284 citations on Google Scholar .