Team Leaders: Melanie Fried-Oken, Deniz Erdogmus, Steven Bedrick,
Consumer Team: Greg Bieker, John Simpson
In 1995, Jean-Dominique Bauby, the editor of French Elle magazine, experienced a stroke. He lapsed into a coma and awoke days later with locked-in syndrome (LIS), his entire body paralyzed aside from some limited movement in his head and eyes. Unable to speak, Bauby learned to communicate by blinking: he could blink to answer yes/no questions, or to spell words using partner-assisted scanning (someone recited the letters of the alphabet and Bauby blinked to indicate his desired letter). Using this painstaking method, Bauby dictated his memoir, The Diving Bell and the Butterfly (adapted into a film of the same name), relating his experiences living with LIS. He died in 1997, three days after the book’s publication. Bauby’s story illustrates the need for a safe, reliable, and innovative access method for people who have minimal movement. As the population of adults with locked-in syndrome continues to increase, so will the demand for research based training tools and technologies that enhance communication performance and participation in medical decisions and management.
Challenge
In recent years, brain-computer interface (BCI) has provided a potential means for individuals with minimal movement to control a computer (for spelling, Internet access, or other functions) using only their brain waves, with no motor output required. There are limited data about BCI usability with clinical populations and we do not yet have research-based training protocols to help individuals learn to use this challenging access method. Most BCI research has been accomplished in engineering labs, with non-disabled participants and investigators. Most reports of BCI use do not include training protocols or evidence-based treatment goals. Unfortunately, people with minimal movement have tested very few BCI devices, and their input has been largely absent from the efforts guiding the development and use of this technology. BCI is just entering the AAC research and development community.
Goals
Two separate studies are proposed to investigate different brain signals that can be used for intent selection with a spelling BCI. A third study is proposed to examine factors that impact BCI use by people with severe disabilities.
Study 1. To investigate the effects of visual acuity and ocular motility impairments on use of the Shuffle Speller brain-computer interface by participants with and without severe disabilities. Shuffle Speller BCI was developed by the joint OHSU + Northeastern University team, and uses the SSVEP (steady state visual evoked potential) for intent selection. The first experiment for Study 1 will explore the effects of simulated vision and ocular motility impairments on users’ typing accuracy and rate. The second will include users with CCN and visual impairments. Study 1 has several hypotheses: 1) Simulated visual impairment of 20/200 in participants without disabilities will not significantly affect typing accuracy (correct selections out of total selections) with the Shuffle Speller. 2) For participants without disabilities, imposed ocular motility impairments will not affect typing accuracy (correct selections out of total selections) with the Shuffle Speller. 3) People with SSPI who present with visual impairments will attain better typing accuracy (correct selections out of total selections) using the Shuffle Speller than using standard eye gaze AAC devices.
Study 2. To examine use of the motor imagery paradigm of the mindBEAGLE BCI by individuals with total Locked-In Syndrome, and explore the use of customized motor imagery prompts as a means of improving communication performance. mindBEAGLE is a brain-computer interface that is manufactured by g.tec medical engineering, and utilizes the sensorimotor rhythm (SMR) for intent selection. The hypothesis for Study 2 is that customized, personally relevant motor imagery prompts will lead to better calibration performance (% accuracy for SMR) and higher accuracy in responses to yes/no questions (% accuracy to 10 known yes/no questions), as compared to a standard motor imagery prompt.
Study 3. To investigate the possibility of building a drowsiness detector based on neurophysiologic signals, an important factor in BCI performance. The first experiment for Study 3 will involve participants without disabilities, and will explore how drowsiness impacts RSVP Keyboard™ performance and P300 amplitude over time, how drowsiness detector results compare to participant self-reports, and what approaches to calculation of drowsiness are most robust across and within individuals. The RSVP Keyboard™ is a BCI spelling system designed by the OHSU + Northeastern University team. The second experiment will involve similar investigations with participants with CCN. The study 3 hypothesis is that increased drowsiness as assessed physiologically and by self-report (Karolinska Sleepiness Scale) is correlated with worse performance as assessed by Area Under the Curve (AUC) measurements during repeated RSVP KeyboardTM calibration sessions.
Key Findings and Implications
AAC methods can help people with severe speech and physical impairments participate in BCI research, supporting cognitive, language, and sensory screening, informed consent procedures, and user experience feedback.
Study 1
- Simulated visual acuity impairment of 20/200 did not reduce typing accuracy or speed for healthy control participants using a steady state visual evoked potential-based brain-computer interface system. Some participants were able to type accurately even under a simulated ocular motility impairment condition. These findings suggest that visual impairments do not necessarily preclude use of a visual BCI.
- Participants with ALS who are unable to use existing eyegaze-based speech-generating devices were able to use the Shuffle Speller typing interface with both eyegaze and steady state visual evoked potential-based brain-computer interface. Shuffle Speller’s unique design may support communication for individuals who have difficulty with other interfaces, and a hybrid of eyegaze and BCI input may be beneficial for some users.
Study 2
- Participants with severe speech and physical impairments were unable to use a motor imagery-based (SMR) brain-computer interface to answer yes/no questions, and their performance did not improve with customized, personally relevant motor imagery prompts.
Study 3
- Calibration accuracy with the RSVP Keyboard™ brain-computer interface decreased, and self-ratings of sleepiness and boredom increased, over five successive calibration sessions with 20 participants without disabilities. Physiological measures may help increase BCI performance by guiding feedback provided to users or the adaptation of system classifiers to user state.
Resources
Peters, B., Mooney, A., Oken, B., & Fried-Oken, M. (2016). Soliciting BCI user experience feedback from people with severe speech and physical impairments. Brain-Computer Interfaces, 3, 47-58.
Peters, B., Higger, M., Quivira, F., Bedrick, S., Dudy, S., Eddy, B. … & Erdogmus, D. (2018). Effects of simulated visual acuity and ocular motility impairments on SSVEP brain-computer interface performance: An experiment with Shuffle Speller. Brain-Computer Interfaces, 5(2-3), 58-72.
Oken, B., Memmott, T., Eddy, B., Wiedrick, J., & Fried-Oken, M. (2018). Vigilance state fluctuations and performance using brain–computer interface for communication. Brain-Computer Interfaces, 5, 146-156.
Peters, B., Kinsella, M., Eddy, B., Mooney, A., & Fried-Oken, M. (2018). A revised sensory/cognitive/communication screen for use with communication BCI study participants. 7th International BCI Meeting Abstract Book. Pacific Grove, CA.
Koch Fager, S., Fried-Oken, M., Jakobs, T., & Beukelman, D. R. (2019). New and emerging access technologies for adults with complex communication needs and severe motor impairments: State of the science. Augmentative and Alternative Communication, 35, 13-25.
Greg Bieker YouTube video from RERC State of the Science Conference https://www.youtube.com/watch?v=otwF3UkznSQ
