The ability of a robotic arm to be controlled only by human thought is made possible by the use of a brain-machine interface (BMI). This project sought to improve BMI user practicality by revising the Andersen Lab’s BMI robotic arm control system [1, 2]. It aimed to answer two questions: Does augmented reality benefit BMIs? Can BMIs be controlled by decoded verbs from the posterior parietal cortex brain region? This project found that augmented reality significantly improved the functionality of BMIs and that using motor imagery was the most effective way to control BMI motion.
Author: Sydney Hunt
Duke University
Mentors: Richard Andersen, Jorge Gamez de Leon
California Institute of Technology
Editor: Alex Bardon
Introduction
BMIs function by connecting brain activity to a machine through the use of sensors that are surgically implanted in an individual’s brain. When these sensors are connected to a computer, the electrical activity of the brain is measured. This neuronal information is then used to control an external device through a ‘neural decoding task,’ or a machine learning algorithm that programs the external device to perform a specific function when the computer reads a specific electrical activity from the brain.
Neural Sensors
Early versions of intracortical BMIs in humans focused on using signals recorded from arrays implanted in the primary motor cortex (M1) or posterior parietal cortex (PPC) of tetraplegic humans [3]. Data from these studies showed that both the M1 and PPC can encode the body part a person plans to move and a person’s nonmotor intentions [4]. Yet sometimes, it is difficult to predict whether the PPC or M1 will provide stronger signals, as these strengths vary depending on an individual’s brain and the location of the neurally implanted sensors.
In this project, JJ–a Photoshop artist, steak lover, and father of four who became tetraplegic in a flying go-kart accident–had two 4×4 mm array sensors [5] surgically implanted in his brain. The first sensor was implanted in his M1 to detect the neural information that controls his voluntary movement control and execution [3]. The second sensor was implanted in his PPC to detect his planned movement, spatial reasoning, and attention [4, 6].
JJ’s electrical brain activity was measured by utilizing these two sensors in combination with the NeuroPort Biopotential Signal Processing System [7]. This collected neuronal information was later used to control a Kinova JACO robotic arm [8].
Neural Decoding Task for JACO Robotic Arm Control
A limited amount of research has found that thinking about action verbs (e.g., “grasp”) could be decoded from M1 or PPC, in addition to the desired bodily movement [4]. This decoded information could potentially reduce the energy needed to control a BMI system; JJ could simply think of the word “grasp” rather than imagining the grasping action–which includes reaching, grabbing, and picking up–when controlling the JACO Robotic Arm [9].
A neural decoding task was therefore developed to translate JJ’s measured electrical brain activity to an action the Kinova JACO robotic arm would perform (e.g. “grasp”). This neural decoding task trained a machine learning algorithm to correctly associate JJ’s electrical brain activity to robotic arm movement (see Figures 1-2). As a result, JJ was able to control the Kinova JACO robotic arm’s trajectory with only his thoughts.
The color of each square in the figure corresponds to the percentage value of the accuracy of the machine learning algorithm between the Predicted Class and True Class. As seen by the blue to pink gradient scale on the right side of each graph, these values can range from 0% accurate (light blue) to 100% accurate (hot pink).
In each graph, the diagonal composed of the six squares from the top left corner to the bottom right corner corresponds to the machine learning algorithm being correct (what the algorithm predicted JJ commanded/imagined matched what JJ commanded/imagined). The left graph represents the results of the neuronal signals collected from JJ’s sensor in his M1 and the right graph represents the results of the neuronal signals collected from JJ’s sensor in his PPC. The algorithm’s overall accuracy is displayed above each graph (36.67% in M1, 55.00% in PPC).
Object Selection via Augmented Reality
The Andersen Lab’s BMI interface was further modernized by introducing augmented reality technology into its system. This incorporation allowed JJ to independently select any object in a 360-degree space using the object detection features of the Microsoft HoloLens2 [10] augmented reality device camera (see Figure 3). BMI practicality was consequently increased; the spatial limitations of object selection when using a BMI were reduced since predefined objects or predefined locations were no longer required in this BMI system.
Results
This BMI system allowed JJ to independently select, pick up, move, and set down a water bottle without verbalization (see Figures 4-5). Consequently, this cognitive-based neural prosthesis reduced the social anxiety paralyzed individuals may experience when using voice-controlled prostheses.
The functioning BMI system also demonstrated that augmented reality benefitted the Andersen Lab’s BMI system by reducing its spatial limitations. It allowed JJ to select objects of his choice to be manipulated by an external device, rather than be constrained to using predefined objects and predefined start/end locations.
Data analysis of the neural decoding task showed that motor imagery of action verbs (e.g. JJ imagining himself grasping something) was better represented than both commanding action verbs (e.g. JJ said the word “grasp” aloud) and imagining action verbs (e.g. JJ imagined saying the word “grasp”) in the area of the M1 and PPC where JJ’s particular arrays were implanted (see Figure 2). When comparing the decoding accuracy between the two areas of the brain, the PPC had a higher decoding accuracy of motor imagery than the M1.
Future Applications
The Andersen Lab had previously developed a neural decoding task that accurately decoded JJ’s neuron signals when he imagined moving his thumb in the up, down, forward, or backward direction. Due to time constraints, this thumb control was implemented into this project in order to let JJ control the Kinova JACO robotic arm (see Figures 4-5).
Future work can include implementing the neural decoding task developed in this project into the Andersen Lab’s BMI. Using this strong PPC motor imagery representation of action verbs may make it easier for JJ to navigate the Kinova JACO robotic arm. Rather than meticulously imagining thumb movement, JJ can just think about performing the grasping action and the Kinova JACO robotic arm will execute a hardcoded movement associated with the “grasp” action verb [9].
Additional improvements can also include the implementation of multiple unique game objects appearing on the HoloLens2 screen, providing JJ with visual feedback that multiple real-world objects were being detected. Therefore, JJ could theoretically create a queue of objects to move, which would better represent real-life scenarios.
Acknowledgments
Thank you to the following individuals and groups for their support. I greatly appreciate your mentorship, guidance, and confidence in me both during and after this project.
Andersen Lab; California Institute of Technology; Caltech WAVE Fellows Program; Friends; Family; JJ; Jorge Gamez de Leon; Richard Andersen; Tianqiao and Chrissy Chen Institute for Neuroscience; Tyson Aflalo.
References
- Andersen, R. (2019). The Intention Machine. Scientific American, 320(4), 24–31. https://doi. org/10.1038/scientifcamerican0419-24
- Katyal, K. D., Johannes, M. S., Kellis, S., Aflalo, T., Klaes, C., McGee, T. G., Para, M. P., Shi, Y., Lee, B., Pejsa, K., Liu, C., Wester, B. A., Tenore, F., Beaty, J. D., Ravitz, A. D., Andersen, R. A., & McLoughlin, M. P. (2014). A collaborative BCI approach to autonomous control of a prosthetic limb system. 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 1479–1482. https://doi.org/10.1109/SMC.2014.6974124
- Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., & Donoghue, J. P. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature, 442(7099), 164–171. https://doi.org/10.1038/ nature04970
- Andersen, R. A., Aflalo, T., & Kellis, S. (2019). From thought to action: The brain-machine interface in posterior parietal cortex. Proceedings of the National Academy of Sciences, 116(52), 26274–26279. https://doi.org/10.1073/pnas.1902276116
- “NeuroPort Array IFU.” NeuroPort Array PN 4382, 4383, 6248, and 6249 Instructions for Use, 29 June 2018, https://blackrockneurotech.com/research/wp-content/ifu/LB-0612_NeuroPort_Array_IFU.pdf.
- Aflalo, T., Kellis, S., Klaes, C., Lee, B., Shi, Y., Pejsa, K., Shanfield, K., Hayes-Jackson, S., Aisen, M., Heck, C., Liu, C., & Andersen, R. (2015). Decoding motor imagery from the posterior parietal cortex of a tetraplegic human. Science, 348(6237), 906–910. https://doi.org/10.1126/ science.aaa5417
- Blackrock Microsystems, LLC. NeuroPort Biopotential Signal Processing System User’s Manual, 2018. Accessed on: May 31, 2021. [Online]. Available: https://blackrockneurotech.com/research/wp-content/ifu/LB-0175_NeuroPort_Biopotential_Signal_Processing_System_Users_Manual.pdf
- KINOVA JACO™ Prosthetic robotic arm User Guide, 2018. Accessed on: May 31, 2021. [Online]. Available: https://github.com/Kinovarobotics, https://www.kinovarobotics.com/sites/default/files/PS-PRA-JAC-UG-INT-EN%20201804-1.0%20%28KINOVA%20JACO%E2%84%A2%20Prosthetic%20robotic%20arm%20user%20guide%29_0.pdf
- T. Aflalo, C. Y. Zhang, E. R. Rosario, N. Pouratian, G. A. Orban, & R. A. Andersen (2020). A Shared Neural Substrate for Action Verbs and Observed Actions in Human Posterior Parietal Cortex. Science Advances. https://www.vis.caltech.edu/documents/17984/Science_Advances_ 2020.pdf
- Microsoft HoloLens: https://docs.microsoft.com/es-es/windows/mixed-reality/
Further Reading
Please click the links below or visit the Andersen Lab website for more information about BMIs.
The Intention Machine: A new generation of brain-machine interfaces can deduce what a person wants
Annual Review of Psychology: Exploring Cognition with Brain–Machine Interfaces
Single-trial decoding of movement intentions using functional ultrasound neuroimaging
CURJ 