D2: Developing AAC technology to support interactive video visual scene displays

Figure D2-1 Pict and textProject LeadersT. Jakobs (InvoTek team), J. Light,  D. McNaughton

Challenge: Individuals with complex communication needs (CCN) who have cognitive and language limitations benefit from AAC technologies that utilize visual scene displays (VSDs), that is, photos of meaningful events within the individual’s life with relevant language concepts embedded as hotspots within the scene. VSDs reduce cognitive and linguistic demands and provide visual supports to maximize communication and participation of beginning communicators.However, communication events are dynamic, not static, and current apps only support static photo VSDs that cannot capture the dynamic routines that require communication.

Goal: In this project, we will develop a prototype app to capture video of daily events and create communication VSDs within these videos, thus providing a technical solution to the problem of supporting the communication of individuals with severe disabilities effectively within daily activities.

Invotek will develop the prototype app and the Penn State team will evaluate it in a series of studies with children and adolescents/adults who have severe cognitive and language limitations (e.g., ASD, CP, IDD). Specifically we will investigate the impact of the new app that supports video VSDs on the participants’ frequency of communication turns and the success of their communication attempts in school, work and home environments.


Light, J. & McNaughton, D. (2015). Designing AAC research and intervention to improve outcomes for individuals with complex communication needs. Augmentative and Alternative Communication, 31, 85-96.

Powered by WordPress. Designed by WooThemes