D1: Developing multimodal technologies to improve access

eye w dotsProject Leaders:  T. Jakobs (InvoTek), D. Hershberger (Saltillo), S. Fager, J. Light, D. Beukelman, D. Erdogmus
Consumer Team: J. Baker, A. Wilson, M. Cuttlers, B. Arnold

Grant is 10 years old and has severe cerebral palsy. He is fully included with his typical peers in a grade 5 classroom. He is currently struggling to keep pace with the curriculum, not because of the learning required, but rather because of his slow access to AAC and other technologies. He lacks the range and precision of movement in any single modality to use direct selection and has to rely on scanning, a very inefficient method of access. He is exhausted by the end of the school day. He requires a better, more efficient access technique.

Challenge: Significant advances in AAC access technologies  have enabled more individuals with complex communication needs (CCN) who have severe motor impairments to operate computer technology using eye tracking, head tracking, and touch sensitive interfaces. However, all of these methods rely on a single access modality. There are significant challenges associated with using a single access modality (e.g., using eye gaze alone) to control technology, including fatigue, over-use injuries, and inefficiency.

Goals: In this development project, we will create new integrated access methods that combine multiple access modalities  to control smart technologies (smartphones, tablet devices, and computers).

Objectives:  RERC on AAC engineers will build on our prior work and will integrate multiple access modalities such as speech, gestures, eye tracking, and head tracking to best meet the needs of individuals with CCN and severe motor impairments. Evaluation will focus on comparing multimodal access strategy to single access modalities to determine the impact on rate, efficiency of access, and personal preference of these individuals. Specifically, we will test two hypotheses:

  1. These individuals will demonstrate increased accuracy and efficiency using multimodal access strategies compared to a single modality; and
  2. They will report higher ratings of consumer satisfaction for multimodal access compared to single mode access. The evaluation will be conducted in a series of separate but parallel studies, each designed to test the effects of multimodality access on three populations with severe motor impairments who require alternate access:
    • children with CP,
    • adults with CP, and
    • adults with cervical spinal cord injury.


Fager, S., Jakobs, T., & Beukelman, D. (2015, November). Integrating Speech Recognition Into AAC Technology. Presentation at the annual conference of the American Speech and Hearing Association, Denver CO. Handout as a pdf

Fager, S. & Jakobs, T. (January, 2015). Integrating speech recognition into AAC technology. Presentation at Annual Conference of the  Assistive Technology Industry Association (ATIA), Orlando, FL. Handout as a pdf

Powered by WordPress. Designed by WooThemes