September 9, 2024

Ruidong Zhang is a doctoral candidate in information science from Lianyungang, Jiangsu, China. He earned his  bachelor’s degree at Tsinghua University and now studies the combination of novel sensing and AI to build wearable devices under the guidance of Cheng Zhang at Cornell.

What is your area of research and why is it important?

My research focuses on combining novel sensing and AI to build wearable devices that can better understand human behavior, empower novel interactions, and promote health. Presently, my primary research focus lies in recognizing silent speech and synthesizing voices. When people speak with moving lips but no sound, such speech is referred to as silent speech. Silent speech could be particularly useful as a natural way of communication in quiet or noisy scenarios. What’s more, for people who have lost their voices, silent speech and voice synthesis based on silent speech represent an effective way to help them regain voices.

What are the larger implications of this research?

One direct impact of my research on silent speech and voice synthesis is that it provides a way for people who have lost their voices to regain access to voiced speech with minimal pain and cost. This could be important for their communication and social participation. In addition, my research provides a non-surgical approach compared with traditional voice restoration practices, which could be particularly helpful for people with limited access to medical resources. For the general public, as an emerging novel interaction interface, silent speech provides an alternative approach to interact with computers that is natural and private. When combined with recent advancements in AI, this could open up brand new opportunities for new forms of human-computer interaction.

You recently won a 2024 North America Qualcomm Innovation Fellowship for your work on “Communication for People with Impairments Using AI-powered Synthesis on Intelligent Wearables.” Can you describe the project?

The project aims to synthesize voices for people who have lost their voices after vocal cord removal (laryngectomy). Traditional solutions usually involve painful surgical procedures, while only being able to recover limited voices that are coarse or machine-like. In this project, we aim to develop a pair of normal-looking eyeglasses to generate high-quality voices without invasive procedures. The eyeglasses will be equipped with miniature sensors (speakers and microphones), which will work as sonar on the face, to capture facial movements when the user is speaking. We will then use generative models to synthesize voices from these captured moments. This project aims to combine the power of novel sensing and AI for social good to bring voice back to people in need.

What does it mean to you to have won the fellowship?

It is an honor for me to receive the renowned Qualcomm Innovation Fellowship award. Directly, the fellowship provides me with one year research funding together with various other resources and support from Qualcomm, allowing me to focus on my research more efficiently. As a growing researcher, it is important to know that my research is being recognized and supported by academia and industry. This award is a recognition of my past research and future visions in this area. It is also an incentive for me to keep pursuing my research and do greater good for our world.

What are your hobbies or interests outside of your research or scholarship?

My favorite sport is rock climbing (bouldering). I normally go twice a week at Lindseth Climbing Center. During winter, snowboarding is another of my hobbies. In addition, Ithaca has many gorgeous trails and a beautiful lake. I sometimes go hiking when the weather is good. Other than sports, my biggest hobby is stargazing. There are many dark sites around Ithaca to relax and contemplate under the brilliant night sky. I even have a large telescope for stargazing. I also do puzzles in my spare time.

Why did you choose Cornell to pursue your degree?

I started conducting research when I interned in Dr. Cheng Zhang’s lab in the summer of my junior year. I had a great time here and my research interests were highly-aligned with Cheng’s visions. So I did not hesitate to return here one year later to pursue my Ph.D. degree with Cheng as my advisor. Cornell also offers a wide range of resources and opportunities for me as a Ph.D. student. Plus, Ithaca is great for a nature lover like me.