According to Mashable, the feature integrates into Gboard and Live Transcribe, enabling users to sign searches, draft messages, or communicate in real time using their phone’s camera.
The technology aims to bridge communication gaps by turning upper-body movements into functional text without requiring a physical keyboard. Google DeepMind announced the tool ahead of the company’s hardware event.
How SL2T Translates Sign Language
Teaching a mobile device to process sign language requires tracking more than just hand positions. According to Google DeepMind, American Sign Language utilizes grammar and meaning conveyed through the hands, arms, torso, head, and face simultaneously.
To capture these nuances, an on-device computer vision system maps 130 distinct points across a user’s face, body, and hands. The system transforms the visual feed into a set of geometric coordinates while discarding the raw video footage immediately. Those coordinates travel to Google’s servers for translation.
Older sign-language translation systems typically converted signs into written labels known as “glosses” before generating text. Google DeepMind noted that glosses fail to capture non-linear elements like facial expressions and spatial constructions. By skipping the gloss step, SL2T processes raw landmarks directly to preserve the full context of the signing.
Training Data and Accessibility Limitations
Google trained the SL2T model on more than 100,000 hours of data spanning over 50 sign languages, with roughly 25 percent of that material dedicated to American Sign Language. The training data also includes one-handed signing—useful when a user holds the phone with one hand—along with left-handed signing styles.
Despite the extensive training dataset, Google and its AI Sign Language Advisory Committee emphasize that SL2T functions strictly as an assistive input tool for low-stakes communication. According to DeepMind’s reporting, the model currently faces several operational limitations:
- Struggles with regional signs, slang, rapid fingerspelling, and complex grammar.
- Vulnerable to accuracy drops in low-light environments, partial framing, or poor contrast between the signer and their background.
- Prone to generating “ghost text” during pauses or when another person enters the camera frame.
- Lacks formal evaluation for users under 18 and extensive testing across signers with motor disabilities.
Because of these constraints, the tool does not meet legal requirements for reasonable accommodations. Google states that the feature is not designed for medical appointments, legal proceedings, classrooms, job interviews, or government hearings where errors could carry serious consequences.
Device Availability and Future Support
The sign-to-text feature launches natively on the Pixel 11 family through Gboard and Live Transcribe. Initially, the system supports American Sign Language translations into English.
Google stated that the capability will expand to more devices at no additional cost. The company also plans to introduce support for additional sign languages in future updates.