Apple’s new accessibility feature lets Vision Pro users control a wheelchair with their eyes

Apple is previewing new accessibility features, including updates based on Apple Intelligence, such as natural language voice typing, as well as a new Vision Pro app that allows visual control of motorized wheelchairs. The updates, revealed just before Global Accessibility Awareness Day (GAAD) on May 21, will arrive later this year. “With Apple Intelligence, we are bringing powerful new capabilities to our accessibility features while maintaining our fundamental commitment to privacy by design,” said Tim Cook, Apple CEO.

VoiceOver’s Image Browser now uses Apple Intelligence to give more detailed descriptions of system-wide images, including photos (which often lack detailed alternative descriptions), scanned invoices, personnel records, and other content. “With updates to Live Recognition, VoiceOver users can press the Action button on iPhone to quickly ask a question about what’s in the camera’s viewfinder and get a detailed answer” and ask follow-up questions, Apple said.

Magnifier, meanwhile, will use Apple Intelligence to bring visual descriptions to its high-contrast interface for visually impaired users. It also works with the Action button, allowing users to quickly get answers to questions and control the application itself with voice requests such as “zoom” or “turn on flashlight”.

Another important AI addition is natural language for Apple’s voice control feature. “With new flexible input, users can describe on-screen buttons and controls in natural language instead of memorizing exact labels or numbers,” Apple wrote. This will work for navigating any app, including those with visual layouts like Apple Maps or Files. For example, users can say things like “tap the best restaurant guide” or “tap the purple folder.” The feature could also help users when elements in the app are not properly labeled for accessibility.

Accessibility Reader also gets a dose of Apple Intelligence. It can now work on more complex documents like scientific articles or texts with multiple columns, images and tables, providing article summaries on demand so readers can get an overview before diving into the details. Another new feature is built-in translation which allows users to read in their native language while maintaining custom formatting, fonts and colors.

Apple

Generated captions provide captions for “home videos, content shared by friends and family, and other media,” rather than just videos from platforms like YouTube and Facebook. Customizable captions are generated on-device to ensure privacy and appear automatically for uncaptioned videos on iPhone, iPad, Mac, Apple TV, and Apple Vision Pro.

Finally, Apple introduced a new wheelchair control feature for Apple Vision Pro. Utilizing the precision eye tracking system on the headsets, the new power wheelchair control feature offers a “responsive input method for compatible alternative driving systems.” It does not require frequent recalibration and works in a variety of lighting conditions, providing compatibility with TOLT and LUCI alternative drive systems with accessory support for Bluetooh and wired connections.

“The ability to control my electric wheelchair on my own is gold to me,” said Pat Dolan, who has lived with ALS for 10 years and is one of the founders of GeoALS. “With this new feature, Apple is developing technology that improves the lives of people who need it most.”