The tool will be available on other devices in the future.
Smartphones have long offered voice-to-text transcription, but the same cannot be said for sign-to-text language. That changes today with the launch of the Pixel 11 family, which marks the debut of DeepMind’s new Sign Language to Text (SL2T) model. Google has integrated the model into Gboard and Live Transcribe, where it will allow deaf and hard of hearing users to connect to their phone anywhere they would have normally typed in the past. In practice, this will give these users a faster, more natural way to search the web, write messages, and converse with Google’s Gemini chatbot.
“There is great diversity among deaf people in terms of their level of mastery of signs, speaking, reading and writing. It is therefore important to support access in all modalities,” explains DeepMind. “Deaf people can benefit from sign language treatment in the same way that hearing people benefit from spoken language treatment, and technology is opening up new possibilities to bridge the communication gap between the deaf and hearing communities.”
DeepMind trained SL2T on over 100,000 hours of multilingual sign language data, with approximately a quarter of the dataset representing American Sign Language (ASL) information. For this reason, the template only supports ASL to English transcription at launch. Google plans to support more languages and devices in the future.
SL2T does not interpret raw video. Instead, a separate model on the device converts the images into a sort of wireframe of geometric coordinates sent to Google’s servers. The company claims that the model is designed this way to protect user privacy. SL2T also differs from older sign language to text conversion systems in that it translates the wireframes it receives directly into text, instead of producing intermediates called glosses.
“Glossies fail to capture the rich, non-linear aspects of sign languages, such as non-manual markers and spatial constructions,” explains DeepMind. “Direct translation from benchmarks removes artificial vocabulary boundaries and allows translation quality to evolve directly with the data.”
While support for ASL is a good start, DeepMind recognizes that there is still work to be done. Worldwide, more than 70 million deaf and hard of hearing people communicate in some 200 sign languages. The good news is that the company had a rocky start as SL2T was trained on multiple languages simultaneously so it could learn the underlying structures shared between them.
