Assistive Technology

The Future of AAC and XR

A boy wearing an Augmented Reality headset.

If you have no idea what the term AAC means, you aren’t alone. AAC is not in the mainstream consciousness of most people even though it should be. AAC stands for Augmentative and Alternative Communication. This means people use ways other than speech to communicate.

AAC has been around for centuries in one form or another, like gestures and sign language, but modern AAC has developed over the past one hundred years or so. AAC as a physical device came about around one hundred years when the first communication board was created. Over the subsequent decades, different methods were developed to allow people to communicate either with assistance (aided) or on their own (unaided). The electronic AAC device evolved starting in the 1960s and in the past ten years has really come into its own.

The Current State of AAC

An iPad in a red case running the AAC program Proloquo2Go
The AAC program Proloquo2Go on an iPad.

There are dozens if not hundreds of options for AAC these days. You can use the traditional AAC style boards or go with a myriad of electronic or computer-based programs. In fact, you can run AAC right on your iPhone or iPad. With the advent of the internet, communicating in different ways has become easier but still not ideal for everyone.

Challenges with AAC have been multi-faceted: supporting sufficient vocabulary, effective access with or without support, physical adaptations, learning curves, and social and cultural acceptance.

AAC of the Now and Future

One of the challenges of AAC for many is the fact that you need to carry a board or a tablet or computing device and touch it, use electronic switches, or eye tracking from a distance. This works well but can be problematic for some people or in some situations. Say you are walking and carrying things but need to speak using your iPad. Or you have a motor disability and require a wheelchair and using a tablet with keyguard, but access can be slow and frustrating.

There is a new wave of computing that is using BCI—brain computer interface—to give more people access to computer control to speed up and enhance communication. AAC is taking advantage of this in creative ways.

A boy and his mother trying on the Cognixion One AAC headset. A man is adjusting the headset on the boy's head.
My son and wife trying the Cognixion One AAC headset.

One up-and-coming company is Cognixion. They have created an augmented reality (AR) headset that includes BCI via EEG (electroencephalography) sensors. The Cognixion One uses custom EEG sensors along with an AR visor to display the AAC application to the user hands-free.

AR projects 2D or 3D imagery in front of the eyes as a heads up display so the user can see both the information and their surroundings at the same time.

EEG uses electrodes on the outside of the head to sense electrical brain activity. These have traditionally been used for neurological evaluations in the medical field. Over the past few decades, researchers have studied the use of EEG to control computers—brain computer interface. This turns the brain into a switch of sorts, giving the ability to control a computing device without a keyboard or mouse. Once the brain is trained, it can work like a hands-free keyboard and mouse.

This focuses on two of the challenges of traditional AAC: carrying an device and physical access. They are also rethinking the user experience of AAC in the headset.

My son and wife got to try the Cognixion One. My son, who has cerebral palsy, communicates with minimal speech and relies on sign language, gestures, and an iPad with the program Proloquo2Go. Using something like the Cognixion One could open up more options for him to connect with others.

Imagine the Possibilities

This raises the question I’ve been asking myself for a while: how do people with disabilities engage in XR (extended reality—augmented and virtual reality) and the newfangled metaverse? While many of the common AAC methods could work in AR, not many or really any have been adapted to the medium. This needs to happen and needs to happen now.


What if everyone was able to communicate with each other seamlessly? That is no longer a pipe dream. We have the technology. We have the willingness. We even have some of the social and cultural support. We just need it all to come together. I feel this will happen in the coming years and decades, if not sooner.

Also See

Accessible Gaming: The Xbox Adaptive Controller

Xbox Adaptive Controller and Logitech kit

In September 2018, Microsoft released its Xbox Adaptive Controller to the world. This new controller provides alternative ways for gamers to interact with both the Xbox and Windows games. The controller has its own built in controls for some of the main buttons and inputs for up to 19 switches that use the 3.5mm standard and three USB ports.

Xbox Adaptive Controller and Logitech kit

Microsoft started development of the Xbox Adaptive Controller way back in 2015, and it took three years of iterations and collaboration to get to production. The company worked with gamers to develop and tweak the design, and it ended up being compatible with a wide array of switches, joysticks, and mounts and is massively customizable.

My son playing on the Xbox Adaptive Controller

Both of my boys love gaming. We got an Xbox One S last year and while my older son was quick to adapt to the new system, my younger son had challenges. He has cerebral palsy and had trouble using the standard Xbox controller for more complex games like NHL ’18. He loves hockey and really wants to play with his brother. So we got the Xbox Adaptive controller for Christmas along with the brand new Logitech Adaptive Gaming Kit.

The whole point of devices like this is to give people the choice of modifying input methods to fit their needs. You can use the base controller and a few 3.5mm switches, or a whole array of switches along with USB joysticks. There are switches for wheelchairs, hand mounts, foot controls, mouth controls, and more. You can remap the inputs any way you like. This is what true accessibility is about.

The Xbox Adaptive Controller gives my son the ability to interact with the games in his own way. The larger buttons of the Adaptive Controller are easier for him to push. The customizable switches allow us to space out additional button controls in a more accessible layout. The last step is to add some USB joysticks—this will allow complete control in NHL ’18, for example.

Microsoft is the only major game system maker that has done this in a comprehensive way, though adaptive controllers have been around in various forms for decades. My hope is that the success of Microsoft’s effort will trickle over to the other companies. I am looking at you Sony, Nintendo, and Apple.

One great effect of these new adaptive devices is their eventually adaptation (*ahem*) for other uses, including PC control and an area I am currently working in, XR Access: accessibility for virtual reality, augmented reality, and mixed reality systems, aka eXtended Reality (XR).

Notes from “Innovations in Accessible Technology” at the USBLN Conference

These are some of the topics and questions we covered in today’s  “Innovations in Accessible Technology” at the USBLN – US Business Leadership Network Conference. I may or may not have discussed all of these things, but I wanted to put it down here for posterity.

How do you engage users/customers with disabilities into the innovation and design process? How do the retail outlets (or other channels) act as a feedback loop?

We are not a product company like Microsoft or IBM, at least not in the same way. Traditionally our work has been performed directly for government and commercial clients. More recently we have ventured into product development on the investment and innovation sides of the house. This has been very exciting.

From our website: We are a global firm of 22,600 diverse, passionate, and exceptional people driven to excel, do right, and realize positive change in everything we do. We bring bold thinking and a desire to be the best in our work in consulting, analytics, digital solutions, engineering, and cyber, and with industries ranging from defense to health to energy to international development.

We work across many different government agencies that focus on health and accessibility, like the VA, Military Health, NIH, CDC, NSF, and Transportation.

No matter where we start with a particular project, we always engage the end users as part of the design process. We like to use the Agile method whenever possible as well. That means building something in iterations with a feedback loop from the stakeholders and end users.

In my role as the Solutions Evangelist and Lab Manager in the Booz Allen Innovation Center, I do get to run a tech “store” of sorts. We have hundreds of employees, clients, and other visitors in the Center every week. They come to work, to meetings, for events, and to experience our products and our partner’s products. I spend a lot of time giving demonstrations, but I make sure to allow the guest to get involved. I feel this is one of the best ways to get buy-in and input. Trying something for yourself is always a bigger impact than just reading about it or even seeing a video. For example, we show off virtual reality, which most people have not experience yet, and people go “wow” every time.

Research enables us to work on big ideas that sometimes turn into products. How do you know which project to invest in? What does the process from idea to product look like?

We do this in a few ways. One, we have an investment system across the firm within each major team (or market team) as well as our Strategic Innovation Group. Each team makes its own priorities, but we also work across teams in certain areas to pool money, people, and resources.

We also have programs like The Garage, a crowdsourcing platform that runs regular challenges for new ideas to solve big problems like the opioid epidemic, better access to healthcare for our veterans, or providing cybersecurity to protect our business.  We also have an annual Ideas Festival where our employees get together to share what they are passionate about. Apps like Microsoft’s Yammer and Teams people across the firm to communicate and to help build out ideas and teams to solve all kinds of problems.

One of my favorite things that we have introduced in recent years is the Summer Games internship program. Each intern team works with a Challenge lead to choose a problem to solve. Many of the topics are near and dear to their hearts. The teams are given autonomy and support over the ten-week internship span to build a business plan, prototype, and final presentation. Many of these projects end up as continued investments and become client deliverables, branded products, and internal or external open source projects.

I got to engage with many of the Summer Games teams this summer and help support them with advice, feedback, and devices they could borrow from our Solutions Lab in the Innovation Center. This is where I love to be: at the intersection of people, process, and technology.

A few of this year’s Summer Games projects had to do with accessible technology:

  • IRIS – An outdoor navigation system for the blind
  • SAMI – Solution for Accessible Mapping Indoors
  • Assistive Keyboard – Customizable input app

As I mentioned before, as part of my role as the Solutions Evangelist and Lab Manager at our Innovation Center I get to show off many of the cool products that we have built as well as partner products and other off-the-shelf technology that we are trying out. Getting exposure to these things is key: trying something on is much more impactful than just reading about it. This also stimulates new ideas. Many visitors have come through and seen our demos and had an ah-ha moment that applied to their own situation. I see this all the time while showing off tech like augmented and virtual reality, location tracking devices, and eye tracking.

I also do this at home with my family, partly because I like it but also to see what we can find to help my son with special needs engage with the world, like with the touchscreen iPad, Philips Hue lights, and music.

Putting accessibility at the heart of inclusive design produced products that are accessible to people with disabilities, but it also delivers great usability that customers love. How are you using inclusive design to meet the needs of people with and without disabilities? 

Universal or inclusive design is key in any design. From websites to desktop and mobile software, to hardware, furniture, and architecture, it makes the experience better for everyone.

We have a Section 508 group that supports software development across all teams at the firm. As we traditionally have been a tech firm focused on software and websites, this is probably our largest area where universal design comes into play. I’m sure I’m preaching to the choir here, but it’s common sense that something made to follow accessibility design rules will also help a wider audience.

To this effect, we have design and user experience groups that work across all of the markets and investments to ensure that usability and accessibility are addressed from the beginning to the end of the project. It is key to engage with design and UX experts the entire way. This must not be forgotten. You can’t just bring someone in at the end to “fix” a product. You can’t just have a designer apply UX to the original design. You have to engage with them the entire time, including testing, revisions, and final production. Iterations are key, and those cannot be done in a vacuum.

How do you take advantage of the intellect and talent in your companies to drive research and innovation?

Our Innovation Center in Washington, DC is a great example of how we can bring people together to create a sum that’s greater than its parts. The Center enables us to connect people that may not have been connected before or work in completely different markets with different skills and networks, it encourages cross collaboration and idea-sharing. This includes the project teams that apply to work there, the people who come to attend meetings, and the visitors that come to experience our demos.

As I mentioned before, we have a few other ways of connecting ideas with investment and support. That includes The Garage crowdsourcing platform, the annual Ideas Festival, and our Summer Games internship program. Investments are created across the firm as ideas funnel through the processes. Communication platforms like Microsoft’s Yammer and Teams help make those connections. (See above sections for more information.)

As the Solutions Evangelist and Lab Manager for the Innovation Center, I get to connect with many people across the firm. This brings in new demos and content to the Center, as well as helps incubate new ideas and expand on current solutions. Getting people out of their silos is important. I enjoy being part of opening people’s minds and hopefully inspiring some new ideas.

Aging is the fastest growing demographic group. No other groups suffer from disability than the elders. With such global challenge, how can technology help people aging in place?  

This is an exciting area for developments with technology. We have so many new technologies on the consumer and enterprise side that can easily be applied to solutions for those with the disabilities and our seniors.

Tech like mobile phones, home automation, health and wellness wearables, sensors, telepresence robots (like the Suitable Technologies Beams we use in the IC), cameras, RFID, beacons, autonomous cars, and implants can help with these challenges.

Machine learning, AI are hot topics.  How can we not talk about this?

This is also a very exciting area. Machine learning and AI will fuel advances in such things as voice control and concierges, robotics, monitoring and tracking, and even tech like our own IRIS blind navigation platform. The more computers can learn about our environment and how we use it, the better than can aid us with traveling, experiences, and keeping us safe.

Speaking at the USBLN Conference in Orlando

How many of you have struggled with finding assistive technology for your employees, your family, or yourselves?

It is more than just about technology. We have done a lot with accessible technology but we can do more. There are new technologies out there that can help, but we need a cohesive way to bring them together to affect change for people with disabilities.

At Booz Allen, I advocate for Assistive Technology at the Innovation Center and with our Diversity & Inclusion team. At home of a father of a nine year old son with special needs. Every day I think what his life will be like when he graduates from high school. I look around the people in this room and I realize the impact that we could have together and am excited at the possibilities.

I’d really like to stress three key points on how we can make an impact together.

  1. People: People are critical. People are passionate. However people don’t know what they don’t know. I can advocate for my son, but I can’t be at every school for every child. We need to educate everyone on the technologies, tools, and policies that are out there.
  2. Process: Connecting online. Connecting with vendors. Connecting parents with each other. Connecting with teachers. The connections are key at bringing awareness and also incubating new ideas. This is tied together with process.
  3. Tech: Technology is already here, it just needs to be applied. As new technology comes out, we need to evaluate it and be prepared to move on from older systems into the new ones. There is no one answer for anything. This applies to assistive technology as well.

If we think about this as a collective, we can organize to be bigger than the sum of our parts. We need that for our families, friends, and colleagues but also the next generation of our children. I want to continue this discussion with you.

I will be part of the Technology Roundtable “Innovations in Accessible Technology” on Thursday, August 24th at 9am at the USBLN – US Business Leadership Network Conference. I will speak alongside MicrosoftBloombergDeque Systems, and IBM Research. I am the Solutions Evangelist and Lab Manager for the Booz Allen Hamilton Innovation Center in Washington, DC.

Look for us at 9am on Thursday, August 24th on the agenda.