Apple's AI accessibility upgrades: Video captioning, eye-powered wheelchairs

Apple Intelligence is coming to the company's industry-leading accessibility tools, including upgraded vision, captioning, and mobility features now enhanced with machine learning.

In honor of Global Accessibility Awareness Day, which falls on May 21, the tech giant announced a suite of new AI-powered capabilities for the Apple ecosystem, covering its most popular accessibility offerings like VoiceOver, Magnifier, Voice Control, and Reader.

"Apple’s approach to accessibility is unlike any other. Now, with Apple Intelligence, we are bringing powerful new capabilities into our accessibility features while maintaining our foundational commitment to privacy by design," said CEO Tim Cook.

Upgraded access tools

Many of the new upgrades come to Apple's vision and voice control tools, designed for users who are blind or have low vision and those with various ranges of mobility.

With Image Explorer, part of Apple's gesture-based screen reader known as VoiceOver, users can get more detailed descriptions of device displays and content. VoiceOver's Live Recognition feature can be activated by the iPhone Action button and respond to detailed follow-up questions.

Mashable Light Speed

Two phone screens. The screen on the left shows Apple's magnifier app camera, framing a utility bill. The screen on the right shows a chat log with Apple Intelligence. The users asks "How much is this bill for?"

Credit: Apple

Instead of relying on memorizing specific commands or locations on a screen, iPhone and iPad users will be able to use natural language commands to navigate their devices using Voice Control and on-device tools like Magnifier. That means a user can simply ask their device to "tap the orange folder" or "zoom in on that word," and the device will respond — Apple calls it the power to just "say what you see."

New way to caption content

Apple's AI can also automatically generate on-device subtitles for uncaptioned video, including content taken on-device by Apple users, video sent to users without captions, and even streamed video.

The company's Accessibility Reader tool is getting an upgrade that will make it possible to navigate complicated text, like scientific studies with columns, images, and tables, provide more on-demand summaries, and translate languages without changing custom formatting.

Apple Vision Pro for power wheelchair users

Last, but certainly not least, Apple is adding a feature to Apple Vision Pro that will allow power wheelchair users to navigate their mobility devices using just the virtual reality headset.

The company explained that the feature leverages the device's eye-tracking — technology already used to power alternative drive controls, device navigation, and communication tools for people with various disabilities — but requires less frequent calibration than typical drive control devices. Apple still recommends users only use Apple Vision Pro headsets in controlled environments without potential obstacles or inclement weather, however.

The view of a room from a pair of Apple Vision Pro goggles. In the middle of the screen is a pop up menu with various commands including one that reads "wheelchair."

Credit: Apple

Additionally, Apple is making it easier to pair and hand off Made for iPhone hearing aids when switching devices, and add in human ASL interpreters to ongoing FaceTime calls. Apple tvOS is getting larger text support for people with low vision, and Name Recognition is expanding to 50 languages.

Apple said the upgraded tools and features will roll out later this year. The company also said it was expanding its collaborative Hikawa Grip & Stand line, a third-party accessibility accessory designed by artist Bailey Hikawa and released for a limited time last year.