Team Leaders: T. Jakobs (InvoTek), M. Fried-Oken,
Consumer Team: J. Staehely, P. Kolden
Andy is a 37 year old man with athetoid cerebral palsy who relies on scanning software and a single switch placed at his right temple to select letters displayed on a speech generating tablet. Andy holds a job at the local bookstore. He uses his tablet and switch to update inventory: he scans in new book titles that arrive daily. But when it comes to speaking with others, his coworkers often don’t take the time to wait for him to express himself. This limits his ability to contribute to his workplace and he is socially isolated. Andy needs a way to speed up his communication. The proposed smart prediction AAC software offers him a solution. It enables people who know Andy well, like his personal assistant (PA), to suggest contextually appropriate vocabulary while he is forming words on his device, improving prediction accuracy and communication rate.
Challenge
During spontaneous conversation, persons with severe speech and physical impairment (SSPI) who rely on AAC and their partners are faced with unique timing challenges which are dictated by the rate at which the person using AAC formulates messages. For a person who relies on single switch scanning with an alphabet display, there are often long periods for message generation that often violate pragmatic rules of conversation and increase the likelihood that conversational partners will become disengaged and distracted during the conversation. To speed up message preparation, the conversational partner may participate in joint formulation of the message. However, partners provide spoken word choices to the person who is composing a written message. There is yet another violation here that creates a communication asymmetry through the use of two different production modalities.
Goals
Smart Predict was developed to increase efficiency and effectiveness of AAC message production while reducing the violations of conversational rules during multi-modality co-construction, and increasing partner engagement. We propose a novel dual-tablet application for AAC conversation that provides personalized vocabulary supplementation with a dual tablet AAC system. First, we seek to take advantage of the shared knowledge of the communication partners to enhance the user’s message efficiency while the person who relies on AAC maintains control over word choices. Second, by offering vocabulary support in the same written modality in real time, we increase the likelihood of continued partner engagement during the conversation. Our objectives were:
- Develop the proof-of-concept for Smart Predict using two Android tablets.
- With people who rely on AAC, iteratively refine Smart Predict into a prototype proof-of-product.
- With people who rely on AAC, evaluate the Smart Predict proof-of-product through single case alternating treatment research designs.
Key Development Points and Implications
- The Smart Predict system consists of two applications, each on a separate Android tablet: (1) Smart Predict is a tablet-based keyboard with word prediction that facilitates message generation through spelling using single switch access. (2) Smart Predict Partner app enables the partner to supplement personalized vocabulary by either typing out words or phrases, or by choosing words from a prediction list, and sending them in real time through Bluetooth to the AAC application.
- The supplemental words and phrases are presented in standard word prediction areas to the person using AAC who maintains independence to choose or ignore the partner’s suggestions during message generation. The typist does not know which words are proposed by the system’s language database or the conversational partner.
- Smart Predict has been evaluated with two alternating treatments single case design experiments.
- Experiment 1 evaluated partner engagement during message generation with and without the Smart Predict + Partner app. An alternating-treatments single-case design, counterbalanced across conditions, was used to compare the effects of the Smart Predict app alone vs. Smart Predict app + Smart Predict Partner app on the level of communication partner engagement during conversations for the three dyads. There was strong evidence that the Smart Predict app + Smart Predict Partner app resulted in fewer distractions that Smart Predict app only. These results were replicated within each participant pair and between all three participant pairs, demonstrating greater partner engagement with the dual tablet vocabulary supplementation system.
- Experiment 2 evaluated message efficiency during message generation with and without the Smart Predict + Partner app. A single case alternating treatments design was implemented, with characters per minute and selections per character as the dependent variables in two conversational conditions: Smart Predict alone, and Smart Predict + Partner app. For three participant pairs, there was strong to moderate evidence that the Smart Predict + Partner app resulted in more characters per minute and fewer selections per character that Smart Predict app only. These results were replicated within each participant pair and between all three participant pairs, demonstrating greater message efficiency with the dual tablet vocabulary supplementation system.
The video below depicts the use of SmartPredict during a research session. The Smart Predict app system utilizes two Android apps that communicate wirelessly: an AAC user app (for persons with CCN and motor impairments), and a co-constructor app (for non-disabled conversation partners). The AAC user spells out a message with Smart Predict user app. The Smart Predict co-constructor app, displays a copy of the AAC user’s message as it is being created. This enables the co-constructor to suggest words or phrases in real time by either typing out words, or by choosing words from a prediction list. The word and phrase predictions, along with other words stored in the device, are used by a language model within the Smart Predict AAC user’s app, and presented in standard word prediction lists to the AAC user. The AAC user maintains independence to choose or ignore the co-construction suggestions during message generation. This written interaction between the tablet devices mimics the co-construction behavior observed in typical speaking conversations.
Resources
Fried-Oken, M., Jakobs, T., & Jakobs, E. (2018, February). SmartPredict: AAC app that integrates partner knowledge into word prediction. Presentation at the Annual Conference of the Assistive Technology Industry Association (ATIA). Orlando, FL.
Fried–Oken, M., Jakobs, E., Jakobs, T., Kinsella, M., & Pryor, R. (2018, July). SmartPredict: AAC app that integrates partner knowledge into word prediction. Poster presented at the State of the Science Conference of the Rehabilitation Engineering Research Center on Augmentative and Alternative Communication (RERC on AAC), Arlington, Va.
Pryor, R. Kinsella, M., Jakobs, E., Jakobs, T., & Fried-Oken (2019, November). Smart Predict: AAC App that Integrates Partner Knowledge into Word Prediction . Poster presented at the Annual Conference of the American Speech-Language and Hearing Association (ASHA). Orlando, FL.
