Project Leaders: T. Jakobs (InvoTek team), J. Light, D. McNaughtonChallenge: Individuals with complex communication needs (CCN) who have cognitive and language limitations benefit from AAC technologies that utilize visual scene displays (VSDs), that is, photos of meaningful events within the individual’s life with relevant language concepts embedded as hotspots within the scene. VSDs reduce cognitive and linguistic demands and provide visual supports to maximize communication and participation of beginning communicators.However, communication events are dynamic, not static, and current apps only support static photo VSDs that cannot capture the dynamic routines that require communication.
Goal: In this project, we will develop a prototype app to capture video of daily events and create communication VSDs within these videos, thus providing a technical solution to the problem of supporting the communication of individuals with severe disabilities effectively within daily activities.
Invotek will develop the prototype app and the Penn State team will evaluate it in a series of studies with children and adolescents/adults who have severe cognitive and language limitations (e.g., ASD, CP, IDD). Specifically we will investigate the impact of the new app that supports video VSDs on the participants’ frequency of communication turns and the success of their communication attempts in school, work and home environments.
Summary Statement
Redesigning AAC apps to capture video of daily events and integrate communication VSDs within these videos may better support the participation and communication of individuals with complex communication needs. In theory, VSDs embedded within a video will be even more effective at facilitating participation and communication than static photo VSDs because video VSDs capture both the spatial and temporal contexts of activities and communication opportunities, thereby preserving the dynamic relationships and engagement cues found in real world interactions. Furthermore, automatic pausing of the video at key segues in the event marks the appropriate opportunity for participation and communication, and provides the necessary vocabulary within the VSD for the individual who uses AAC to fulfill the communication demands at that point. Apps that support videos with integrated VSDs capture the dynamic routines within the learner’s life (e.g., school, work, community activities) and step the learner through the activities, one step at a time, fostering greater participation and communication.
Cite as: Light, J., McNaughton, D., & Jakobs, T. (2014). Developing AAC technology to support interactive video visual scene displays. RERC on AAC: Rehabilitation Engineering Research Center on Augmentative and Alternative Communication. Retrieved from https://rerc-aac.psu.edu/development/d2-developing-aac-technology-to-support-interactive-video-visual-scene-displays/
Resources
Light, J. & McNaughton, D. (2015). Designing AAC research and intervention to improve outcomes for individuals with complex communication needs. Augmentative and Alternative Communication, 31, 85-96.
Demonstration of Use
The clip below provides a demonstration of the use of a video VSD approach, as used to support communication and participation for a young adult with ASD who was learning to work in a library. At the beginning of the clip, the model has just demonstrated completion of the step ‘Sort the books into piles’ and hits the play button to continue to the next step. The next step, ‘Ask a staff member to check your work’ has a hot spot and provides a communicative opportunity. The model touches the hot spot and activates the message (spoken by the tablet), “Can you check my work?”. The model then continues on to the next step in the task by pressing the play button.
Additional information on this research project is available at Use of Interactive Video VSD for an Adolescent with ASD (Babb et al., 2017)
