When Matthew Sherwood goes shopping for clothes, he needs help to ensure that what he’s picking up is the color or style he’s looking for.
Sherwood has been blind for more than 15 years; he has a family, a successful investing career and a dog, Chris, who helps him navigate the world. But he says everyday tasks like shopping still present hurdles to his independence.
Artificial intelligence could soon help.
Currently, Sherwood says he sometimes uses an app called Be My Eyes, which pairs visually impaired users with sighted volunteers who provide help, through live video, with things like checking whether a shirt matches the rest of an outfit or if a carton of milk has expired. But advancements in AI technology are already beginning to remove the need for volunteer helpers on the other end.
Be My Eyes partnered with OpenAI last year to enable its AI model, rather than another human, see and describe what’s in front of a user. In OpenAI’s latest product demo, the company showed a clip of a person using the AI-powered version of Be My Eyes to hail a taxi — the app told the user exactly when to raise their arm for the car. Google in May announced a similar feature for its app “Lookout,” which is designed to help visually impaired users.
Applications for blind users are just one area where AI is helping to advance what’s known as “assistive technology,” tools designed to help people who are disabled or elderly.
Apple, Google and other tech companies have rolled out a growing slate of AI-powered tools to make life easier for people with a range of impediments, from eye-tracking tools that let physically disabled users control their iPhones with their eyes to detailed voice guidance for blind users of Google Maps.
Since the stunning launch of ChatGPT more than a year ago, it has been clear that AI will change our world by upending how we work, how we communicate and even what we perceive as reality. But for people with disabilities, AI also has the potential to be life-altering in an entirely different way.
“It used to be that if you were in business and you were blind, you had to have an administrative assistant reading to you,” Sherwood said. “But now, you have this new power … For some, this is great technology. For blind people, this is an opportunity to gain employment and an opportunity to compete in business, an opportunity to succeed.”
Tech companies have been using early forms of AI to make their products more accessible for years — think, automated closed captioning on videos or screen readers.
But experts say that the huge data sets and powerful computing systems behind more recent AI models are accelerating what’s possible in the assistive tech space. For instance, in order for an AI tool to reliably help blind people hail taxis, it needs to be very good at recognizing what a taxi does or does not look like, which requires training the model on a huge corpus of examples.
Another example: a Google tool that tells blind or low-vision users about what’s on their screen, has been upgraded with a “question and answer” feature that incorporates the company’s generative AI technology.
“The promise of AI has been evident for many, many years but it has to reach this quality level before it can be a viable thing that you include in products,” Eve Andersson, Google’s senior director of product inclusion, equity, and accessibility, told CNN.
New generative AI tools are especially promising for accessibility applications because they’re designed to understand and produce information in various formats, including text, audio, photos and videos. That means if a person needs to consume information in a certain medium, AI can act as a go-between; for instance, turning a piece of audio into written text for a hearing-impaired user.
“(People’s) accessibility needs take many different forms, but a large class of disabilities are really about input and output, it’s about how a person perceives information,” Andersson said. “There are hearing disabilities, vision, motor, speech, cognitive and all of these can involve a need for different modalities (of information) and one thing that AI is fantastic at is translating between modalities.”
Ensuring that AI systems continue to serve all kinds of users requires ongoing investment.
Because AI models are trained on human-created data, experts have warned that they may replicate the same biases present among humans. And early examples have already cropped up, including AI image generators that appeared to struggle with the concept of race, or an algorithm that allegedly showed job advertisements based on gendered stereotypes.
In one effort to address that risk, a group of Big Tech companies, including Apple, Google, Microsoft and others, have partnered with researchers at the University of Illinois Urbana-Champaign to create a training dataset for AI speech recognition tools that includes a diversity of speech patterns. Speech recognition tools, such as translators, voice assistants and voice-to-text apps can be especially important and useful for users with disabilities.
The effort, called the Speech Accessibility Project, involves collecting recordings from volunteers with conditions such as Parkinsons, Down Syndrome, ALS and other disabilities that can affect speech. With the help of the project’s now more than 200,000 recordings, a sample speech recognition tool created by the researchers misunderstands speech only 12% of the time, down from 20% prior to being trained on the new dataset.
“The more diverse types of speech we can get into those machine learning systems and the greater variety of severity, the better those systems are going to be at understanding individuals that don’t have ‘audiobook narrator’ speech,” said Clarion Mendes, a speech language pathologist and clinical assistant professor who helps lead the project.
“I have talked to so many people throughout this project who face huge barriers to life participation because of their communication, individuals with impressive degrees who can’t find employment because of their communication barriers,” Mendes said. “If something like assistive technology can make it possible for individuals to find enrichment in their hobbies, in their jobs … all of a sudden these activities that used to take excessive amounts of time or require the person to rely on other individuals, that has increased their independence exponentially.”
Andersson added that investing in AI for accessibility is not just the right thing to do, it also makes good business sense.
“We don’t want to leave people behind … technology in general has the ability to level the playing field,” Andersson said. “But there are also financial reasons like being able to sell your products to government entities, to educational institutions.”
