It’s a Sign: AI Platform for Teaching American Sign Language Aims to Bridge Communication Gaps

American Sign Language is the third most prevalent language in the United States — but there are vastly fewer AI tools developed with ASL data than data representing the country’s most common languages, English and Spanish.

NVIDIA, the American Society for Deaf Children and creative agency Hello Monday are helping close this gap with Signs, an interactive web platform built to support ASL learning and the development of accessible AI applications.

Sign language learners can access the platform’s validated library of ASL signs to expand their vocabulary with the help of a 3D avatar that demonstrates signs — and use an AI tool that analyzes webcam footage to receive real-time feedback on their signing. Signers of any skill level can contribute by signing specific words to help build an open-source video dataset for ASL.

The dataset — which NVIDIA aims to grow to 400,000 video clips representing 1,000 signed words — is being validated by fluent ASL users and interpreters to ensure the accuracy of each sign, resulting in a high-quality visual dictionary and teaching tool.

“Most deaf children are born to hearing parents. Giving family members accessible tools like Signs to start learning ASL early enables them to open an effective communication channel with children as young as six to eight months old,” said Cheri Dowling, executive director of the American Society for Deaf Children. “And knowing that professional ASL teachers have validated all the vocabulary on the platform, users can be confident in what they’re learning.”

NVIDIA teams plan to use this dataset to further develop AI applications that break down communication barriers between the deaf and hearing communities. The data is slated to be available to the public as a resource for building accessible technologies including AI agents, digital human applications and video conferencing tools. It could also be used to enhance Signs and enable ASL platforms across the ecosystem with real-time, AI-powered support and feedback.

Supporting ASL Education, Exploring Language Nuance

During the data collection phase, Signs already provides a powerful platform for ASL language acquisition, offering opportunities for individuals to learn and practice an initial set of 100 signs so they can more effectively communicate with friends or family members who use ASL.

“The Signs learning platform could help families with deaf children quickly search for a specific word and see how to make the corresponding sign. It’s a tool that can help support their everyday use of ASL outside of a more formal class,” Dowling said. “I see both kids and parents exploring it — and I think they could play with it together.”

While Signs currently focuses on hand movements and finger positions for each sign, ASL also incorporates facial expressions and head movements to convey meaning. The team behind Signs is exploring how these non-manual signals can be tracked and integrated in future versions of the platform.

They’re also investigating how other nuances, like regional variations and slang terms, can be represented in Signs to enrich its ASL database — and working with researchers at the Rochester Institute of Technology’s Center for Accessibility and Inclusion Research to evaluate and further improve the user experience of the Signs platform for deaf and hard-of-hearing users.

“Improving ASL accessibility is an ongoing effort,” said Anders Jessen, founding partner of Hello Monday/DEPT, which built the Signs web platform and previously worked with the American Society for Deaf Children on Fingerspelling.xyz, an application that taught users the ASL alphabet. “Signs can serve the need for advanced AI tools that help transcend communication barriers between the deaf and hearing communities.”

The dataset behind Signs is planned for release later this year.

Start learning or contributing with Signs at signs-ai.com, and learn more about NVIDIA’s trustworthy AI initiatives. Attendees of NVIDIA GTC, a global AI conference taking place March 17-21 in San Jose, will be able to participate in Signs live at the event.