D3: Developing a Smart Predictor app for AAC conversation

Figure D3 cartoonTeam Leaders: T. Jakobs (InvoTek),  M. Fried-Oken,
Consumer Team: J. Staehely, P. Kolden

Andy is a 37 year old man with athetoid cerebral palsy who relies on scanning software and a single switch placed at his right temple to select letters displayed on a speech generating tablet. Andy holds a job at the local bookstore. He uses his tablet and switch to update inventory: he scans in new book titles that arrive daily. But when it comes to speaking with others, his coworkers often don’t take the time to wait for him to express himself. This limits his ability to contribute to his workplace and he is socially isolated. Andy needs a way to speed up his communication. The proposed smart prediction AAC software offers him a solution. It enables people who know Andy well, like his personal assistant (PA), to suggest contextually appropriate vocabulary while he is forming words on his device, improving prediction accuracy and communication rate.

Challenge: Persons with severe speech and physical impairment (SSPI) not only rely on AAC strategies to express themselves. They also rely on communication partners such as family members, friends, or personal assistants to co-construct messages. We seek to take advantage of the physical skills, language skills and shared knowledge of the communication partners to enhance the user’s message production and communication performance while allowing the person with SSPI to maintain control over the conversation.

Goals: The goal of this project is to design a new AAC technology to increase the speed and informativeness of face-to-face communication. We will  develop a new app, called SmartPredictor, that is used by persons with severe speech and physical impairment (SSPI) and includes a third party (e.g. caregiver, friend, family member) as one source for language input during message generation.

We propose to improve a new, technology-based application for AAC conversation, for persons with SSPI, that is grounded in co-construction (one person using shared knowledge to help form messages with another) and word prediction. We have already developed two Android prototype apps that communicate wirelessly, Smart Predictor-AAC for the AAC user and Smart Predictor for the non-disabled communication partner through other funding mechanisms. The AAC user spells out a message with Smart Predictor-AAC. The second app, Smart Predictor, enables the co-constructor to suggest words or phrases in real time by either typing out words, or by choosing words from a prediction list. The word and phrase predictions, along with other words stored in the device, are used by a language model within the AAC user’s app, and presented in standard word prediction lists to the AAC user. The AAC user maintains independence to choose or ignore the co-construction suggestions during message generation. This written interaction between the tablet devices mimics the co-construction behavior observed in typical speaking conversations. We will improve the language models for this prototype and evaluate the technology in real-life communication environments.  This new AAC system will be tested using a within subjects repeated measures experimental design. The study will evaluate the effects of the prediction condition (i.e., Smart Predictor or standard prediction) and session (session 1 or session 2) on communication accuracy and rate.


Powered by WordPress. Designed by WooThemes