Inside Nvidia’s New ‘Signs’ Platform and the Meeting of ASL And Artificial Intelligence

I’ve been fortunate to cover a bevy of tech companies in my career as a journalist—Intel, OpenAI, Salesforce, amongst countless others—that, were you to play a quick game of word association, accessibility likely wouldn’t be one you’d blurt out. As it turns out, however, all three Bay Area-based companies indeed do care an awful lot about making technology usable by disabled people. In fact, to discover that a chipmaker like Intel and an enterprise software company in Salesforce—both areas which ostensibly have absolutely zero pertinence to accessibility—works so hard to make their wares inclusive to the disability community is simultaneously enlightening and heartening.

So it goes for Nvidia.

In a blog post published on Thursday, the Santa Clara-based company announced its new Signs platform. The software, which Nvidia describes as a “validated dataset for sign language learners and developers of ASL-based AI applications,” was conceived and developed in collaboration with the American Society for Deaf Children and creative agency Hello Monday, in an effort to increase representation of American Sign Language in AI-powered datasets. Nvidia notes ASL ranks third in the United States in terms of prevalence, behind only English and Spanish, yet there exist “vastly fewer AI tools developed with ASL data” compared to the aforementioned top two languages.

Nvidia has posted a video demonstrating Signs on its YouTube channel.

“Sign language learners can access the platform’s validated library of ASL signs to expand their vocabulary with the help of a 3D avatar that demonstrates signs—and use an AI tool that analyzes webcam footage to receive real-time feedback on their signing. Signers of any skill level can contribute by signing specific words to help build an open-source video dataset for ASL,” Nvidia wrote in part in its announcement. “The dataset—which NVIDIA aims to grow to 400,000 video clips representing 1,000 signed words—is being validated by fluent ASL users and interpreters to ensure the accuracy of each sign, resulting in a high-quality visual dictionary and teaching tool.”

In a brief interview conducted earlier this week ahead of today’s news, Nvidia’s manager of trustworthy AI product Michael Boone—coincidentally, he’s credited with the byline for the company’s blog post—explained to me Nvidia decided to work on the Signs project because they saw a need for it. A “large majority” of parents who have deaf children, Boone said, don’t know ASL and aren’t learning it; children are developing their signing skills outside of the home, he added, but Nvidia seized on an opportunity to help bridge the proverbial gap in terms of communicating with one’s nuclear family.

“We want [Signs] to help bridge the communication gap,” Boone said. “Looking at what had been done for [signing] individual letters, we figured it would be helpful to take it to the next step and create a database and user dictionary for words and short phrases.”

When asked about the technical aspects of Signs, Boone told me the state-of-the art for learning ASL is to watch a bunch of YouTube videos and maybe work with a live interpreter. What makes Nvidia’s work so novel, and so interesting, he said to me, is there heretofore hasn’t existed a way for ASL learners to garner real-time feedback on their language for free. According to Boone, Signs is an example of computer vision: using artificial intelligence, Signs has the ability not only to detect where the different parts of the body are, it’s also able to understand how a user is placing their hand as well as the “sweeping movements” of signs. All told, Boone said Nvidia’s overarching goal, technologically speaking, is increasing fluidity and ensuring the software is properly instructing the user to become proficient at speaking sign language.

At a macro level, Boone said the primary goal with Signs is twofold. The first, of course, is pedagogical. Nvidia (and its partner organizations) wants to teach people ASL. Secondly, the platform also exists as a conduit through which Boone a team can “curate a data set that can then be used to enable more accessible AI technology.” Crucially, Boone said Signs has been purposefully built by and for the ASL community, telling me the platform is expressly designed to involve and engage the community.

As I wrote at the outset, the reality is Nvidia is not an institution necessarily revered for assistive technologies. For his part, Boone acknowledged the company’s relatively nascent reputation in this realm and said Signs indeed does align with Nvidia’s “founding principles” that “enable everyone.” From ideation to delivery, he told me, Nvidia’s North Star always is to build technology which “enables as many groups as possible to benefit.” Specific to Signs, Boone said it’s something that’s “high quality [and] has robust data and [involves] all stakeholders within the community, from research to product and our future partners who will benefit from using this data.”

Besides Boone, I also had the chance to connect with Cheri Dowling. Dowling serves as executive director of the American Society for Deaf Children. In a short interview with me, she called Signs “a great tool” and said when she first learned of the project, she immediately knew her organization should back it. Dowling expounded further, saying Signs is a tool with which ASL learners can hone their fluency through what she characterized as a “fun website.” She emphasized how Signs gives users grace if they forget a sign, as they’re allowed to look it up and practice while taking solace they’re getting the mechanics right. As to future versions, Dowling said she’d like Signs to grow from single words to phrases and someday even full-on sentences. “With technology the way it is these days, the possibilities [for improvement] are endless,” she said.

When asked about feedback on the new software, Boone told me Signs has undergone extensive testing both internally and externally “for several months” now. He’s been nearly anticipating today’s public launch, telling me Signs marks the beginning of a journey that sees great potential to “create additional, accessible technologies.” Boone further noted Nvidia has partnered with the Rochester Institute of Technology for help with Signs’ user interface and user experience. The big idea here, he said, was to ensure the creation of a “transparent, safe, and explainable solution” for learning ASL. “I’m excited for what this dataset will be able to be used—not just for the user dictionary, but also an eye towards [making] future technologies as well,” Boone said.

For Dowling’s part, she said her team is “really excited” to see Signs set forth unto the world. Her organization offers online ASL classes and she noted Signs will be heartily recommended henceforth as a way to bolster and supplement the classwork. Dowling said the Nvidia team has been “great” to work alongside on making Signs a reality.

“It’s so important for families who have children using ASL to learn the language,” she said. “This is a fun way to do it together.”

Looking towards the future, Boone told me he portends it “full of possibilities.” He reiterated Nvidia’s raison d’être of building technologies for everyone, saying Nvidia was founded to “solve the world’s greatest challenges.” He also expressed appreciation of, and enthusiasm for, artificial intelligence’s capacity for doing genuine good. The advent of Signs, Boone told me, is an exciting development for Nvidia that “not only teaches but is also helping to enable other members of the ecosystem.”