D2: Developing AAC technology to support interactive video visual scene displays

Figure D2-1 Pict and textProject LeadersT. Jakobs (InvoTek team), J. Light,  D. McNaughton

Challenge: Individuals with complex communication needs (CCN) who have cognitive and language limitations benefit from AAC technologies that utilize visual scene displays (VSDs), that is, photos of meaningful events within the individual’s life with relevant language concepts embedded as hotspots within the scene. VSDs reduce cognitive and linguistic demands and provide visual supports to maximize communication and participation of beginning communicators.However, communication events are dynamic, not static, and current apps only support static photo VSDs that cannot capture the dynamic routines that require communication.

Goal: In this project, we will develop a prototype app to capture video of daily events and create communication VSDs within these videos, thus providing a technical solution to the problem of supporting the communication of individuals with severe disabilities effectively within daily activities.

Invotek will develop the prototype app and the Penn State team will evaluate it in a series of studies with children and adolescents/adults who have severe cognitive and language limitations (e.g., ASD, CP, IDD). Specifically we will investigate the impact of the new app that supports video VSDs on the participants’ frequency of communication turns and the success of their communication attempts in school, work and home environments.

Summary Statement

Redesigning AAC apps to capture video of daily events and integrate communication VSDs within these videos may better support the participation and communication of individuals with complex communication needs. In theory, VSDs embedded within a video will be even more effective at facilitating participation and communication than static photo VSDs because video VSDs capture both the spatial and temporal contexts of activities and communication opportunities, thereby preserving the dynamic relationships and engagement cues found in real world interactions. Furthermore, automatic pausing of the video at key segues in the event marks the appropriate opportunity for participation and communication, and provides the necessary vocabulary within the VSD for the individual who uses AAC to fulfill the communication demands at that point. Apps that support videos with integrated VSDs capture the dynamic routines within the learner’s life (e.g., school, work, community activities) and step the learner through the activities, one step at a time, fostering greater participation and communication.

Cite as: Light, J., McNaughton, D., & Jakobs, T. (2014). Developing AAC technology to support interactive video visual scene displays. RERC on AAC: Rehabilitation Engineering Research Center on Augmentative and Alternative Communication. Retrieved from https://rerc-aac.psu.edu/development/d2-developing-aac-technology-to-support-interactive-video-visual-scene-displays/


Light, J. & McNaughton, D. (2015). Designing AAC research and intervention to improve outcomes for individuals with complex communication needs. Augmentative and Alternative Communication, 31, 85-96.

Powered by WordPress. Designed by WooThemes