Robotic Arm Control Through Algorithmic Neural Decoding and Augmented Reality Object Detection

The ability of a robotic arm to be controlled only by human thought is made possible by the use of a brain-machine interface (BMI). This project sought to improve BMI user practicality by revising the Andersen Lab’s BMI robotic arm control system [1, 2]. It aimed to answer two questions: Does augmented reality benefit BMIs? Can BMIs be controlled by decoded verbs from the posterior parietal cortex brain region? This project found that augmented reality significantly improved the functionality of BMIs and that using motor imagery was the most effective way to control BMI motion.

Author: Sydney Hunt
Duke University
Mentors: Richard Andersen, Jorge Gamez de Leon
California Institute of Technology
Editor: Alex Bardon


BMIs function by connecting brain activity to a machine through the use of sensors that are surgically implanted in an individual’s brain. When these sensors are connected to a computer, the electrical activity of the brain is measured. This neuronal information is then used to control an external device through a ‘neural decoding task,’ or a machine learning algorithm that programs the external device to perform a specific function when the computer reads a specific electrical activity from the brain.

Neural Sensors

Early versions of intracortical BMIs in humans focused on using signals recorded from arrays implanted in the primary motor cortex (M1) or posterior parietal cortex (PPC) of tetraplegic humans [3]. Data from these studies showed that both the M1 and PPC can encode the body part a person plans to move and a person’s nonmotor intentions [4]. Yet sometimes, it is difficult to predict whether the PPC or M1 will provide stronger signals, as these strengths vary depending on an individual’s brain and the location of the neurally implanted sensors.

In this project, JJ–a Photoshop artist, steak lover, and father of four who became tetraplegic in a flying go-kart accident–had two 4×4 mm array sensors [5] surgically implanted in his brain. The first sensor was implanted in his M1 to detect the neural information that controls his voluntary movement control and execution [3]. The second sensor was implanted in his PPC to detect his planned movement, spatial reasoning, and attention [4, 6].

JJ’s electrical brain activity was measured by utilizing these two sensors in combination with the NeuroPort Biopotential Signal Processing System [7]. This collected neuronal information was later used to control a Kinova JACO robotic arm [8].

Neural Decoding Task for JACO Robotic Arm Control

A limited amount of research has found that thinking about action verbs (e.g., “grasp”) could be decoded from M1 or PPC, in addition to the desired bodily movement [4]. This decoded information could potentially reduce the energy needed to control a BMI system; JJ could simply think of the word “grasp” rather than imagining the grasping action–which includes reaching, grabbing, and picking up–when controlling the JACO Robotic Arm [9]. 

A neural decoding task was therefore developed to translate JJ’s measured electrical brain activity to an action the Kinova JACO robotic arm would perform (e.g. “grasp”). This neural decoding task trained a machine learning algorithm to correctly associate JJ’s electrical brain activity to robotic arm movement (see Figures 1-2). As a result, JJ was able to control the Kinova JACO robotic arm’s trajectory with only his thoughts.

Figure 1: Training the Neural Decoder. The developed neural decoding task consisted of one of five action verbs (“grasp”, “drink”, “drop”, “move”, “rotate”) randomly appearing on a monitor screen, followed by the randomly selected appearance of a red dot or blue square. When a red dot appeared on the monitor, JJ verbally commanded the word that previously appeared (e.g. “grasp”). When a blue square appeared, JJ nonverbally commanded or imagined doing the action of the word that previously appeared. JJ’s electrical brain activity was recorded during this exercise using the NeuroPort Biopotential Signal Processing System, which stored which of JJ’s neuronal signals were active throughout this task. This collected data was later spike sorted and analyzed to measure the accuracy of the machine learning algorithm (see Figure 2).
Figure 2: Data Analysis of Neural Decoder Accuracy (Motor Imagery). A Linear Discriminant Analysis Classification was performed on the neural decoding task for the verbally commanded and imagined verbs (“grasp”, “drink”, “drop”).  “V.Grasp” corresponds to “grasp” being verbally commanded (JJ said the word “grasp” aloud), while “I.Grasp” corresponds to the “grasp” action being imagined (JJ imagined himself grasping an object). This labeling system applies to all action verbs in the figure.

The color of each square in the figure corresponds to the percentage value of the accuracy of the machine learning algorithm between the Predicted Class and True Class. As seen by the blue to pink gradient scale on the right side of each graph, these values can range from 0% accurate (light blue) to 100% accurate (hot pink).

In each graph, the diagonal composed of the six squares from the top left corner to the bottom right corner corresponds to the machine learning algorithm being correct (what the algorithm predicted JJ commanded/imagined matched what JJ commanded/imagined). The left graph represents the results of the neuronal signals collected from JJ’s sensor in his M1 and the right graph represents the results of the neuronal signals collected from JJ’s sensor in his PPC. The algorithm’s overall accuracy is displayed above each graph (36.67% in M1, 55.00% in PPC).
Object Selection via Augmented Reality

The Andersen Lab’s BMI interface was further modernized by introducing augmented reality technology into its system. This incorporation allowed JJ to independently select any object in a 360-degree space using the object detection features of the Microsoft HoloLens2 [10] augmented reality device camera (see Figure 3). BMI practicality was consequently increased; the spatial limitations of object selection when using a BMI were reduced since predefined objects or predefined locations were no longer required in this BMI system.

Figure 3: HoloLens2 View of Unity Application. A screenshot of what JJ saw through the Microsoft HoloLens2 screen when using the revised Andersen Lab BMI. A Unity Cross-Platform Game Engine application was developed and run on the HoloLens2 augmented reality device. It caused a cube game object, similar to what is shown in this photo, to appear on the HoloLens2 screen when the HoloLens2’s camera detected a water bottle. When the eye-tracking feature of the HoloLens2 recognized that JJ was looking at the cube, the cube would rotate, signaling JJ’s object selection. The Kinova JACO robotic arm can also be seen on the left side of the figure. It, along with other objects in the real-world room, is covered in white mesh due to HoloLens2’s spatial mapping feature.


This BMI system allowed JJ to independently select, pick up, move, and set down a water bottle without verbalization (see Figures 4-5). Consequently, this cognitive-based neural prosthesis reduced the social anxiety paralyzed individuals may experience when using voice-controlled prostheses.

The functioning BMI system also demonstrated that augmented reality benefitted the Andersen Lab’s BMI system by reducing its spatial limitations. It allowed JJ to select objects of his choice to be manipulated by an external device, rather than be constrained to using predefined objects and predefined start/end locations.

Data analysis of the neural decoding task showed that motor imagery of action verbs (e.g. JJ imagining himself grasping something) was better represented than both commanding action verbs (e.g. JJ said the word “grasp” aloud) and imagining action verbs (e.g. JJ imagined saying the word “grasp”) in the area of the M1 and PPC where JJ’s particular arrays were implanted (see Figure 2). When comparing the decoding accuracy between the two areas of the brain, the PPC had a higher decoding accuracy of motor imagery than the M1.

Figure 4: BMI Trajectory. A high-level overview of how the BMI system functioned. “Camera” in this figure refers to the camera on the HoloLens2 augmented reality headset. See Figure 5 for more details on how the Kinova JACO Robotic Arm trajectory was determined.
Figure 5: Kinova JACO Arm Trajectory. A high-level overview of the three states the JACO robotic arm went through when executing the BMI system. The states are numbered on the rightmost side of the figure (1,2, and 3). The action the Kinova JACO Robotic Arm executed was determined based on the thumb motion JJ performed: “1.” in each stage corresponds to the action the Kinova JACO Robotic Arm performed if JJ moved his thumb in the upward direction; “2.”  in each stage corresponds to the action the Kinova JACO Robotic Arm performed if JJ moved his thumb in the downward direction.

Future Applications

The Andersen Lab had previously developed a neural decoding task that accurately decoded JJ’s neuron signals when he imagined moving his thumb in the up, down, forward, or backward direction. Due to time constraints, this thumb control was implemented into this project in order to let JJ control the Kinova JACO robotic arm (see Figures 4-5). 

Future work can include implementing the neural decoding task developed in this project into the Andersen Lab’s BMI. Using this strong PPC motor imagery representation of action verbs may make it easier for JJ to navigate the Kinova JACO robotic arm. Rather than meticulously imagining thumb movement, JJ can just think about performing the grasping action and the Kinova JACO robotic arm will execute a hardcoded movement associated with the “grasp” action verb [9].

Additional improvements can also include the implementation of multiple unique game objects appearing on the HoloLens2 screen, providing JJ with visual feedback that multiple real-world objects were being detected. Therefore, JJ could theoretically create a queue of objects to move, which would better represent real-life scenarios. 


Thank you to the following individuals and groups for their support. I greatly appreciate your mentorship, guidance, and confidence in me both during and after this project.

Andersen Lab; California Institute of Technology; Caltech WAVE Fellows Program; Friends; Family; JJ; Jorge Gamez de Leon; Richard Andersen; Tianqiao and Chrissy Chen Institute for Neuroscience; Tyson Aflalo.


  1. Andersen, R. (2019). The Intention Machine. Scientific American, 320(4), 24–31. https://doi. org/10.1038/scientifcamerican0419-24
  2. Katyal, K. D., Johannes, M. S., Kellis, S., Aflalo, T., Klaes, C., McGee, T. G., Para, M. P., Shi, Y., Lee, B., Pejsa, K., Liu, C., Wester, B. A., Tenore, F., Beaty, J. D., Ravitz, A. D., Andersen, R. A., & McLoughlin, M. P. (2014). A collaborative BCI approach to autonomous control of a prosthetic limb system. 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 1479–1482.
  3. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., & Donoghue, J. P. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature, 442(7099), 164–171. nature04970
  4. Andersen, R. A., Aflalo, T., & Kellis, S. (2019). From thought to action: The brain-machine interface in posterior parietal cortex. Proceedings of the National Academy of Sciences, 116(52), 26274–26279.
  5. “NeuroPort Array IFU.” NeuroPort Array PN 4382, 4383, 6248, and 6249 Instructions for Use, 29 June 2018, 
  6. Aflalo, T., Kellis, S., Klaes, C., Lee, B., Shi, Y., Pejsa, K., Shanfield, K., Hayes-Jackson, S., Aisen, M., Heck, C., Liu, C., & Andersen, R. (2015). Decoding motor imagery from the posterior parietal cortex of a tetraplegic human. Science, 348(6237), 906–910. science.aaa5417
  7. Blackrock Microsystems, LLC. NeuroPort Biopotential Signal Processing System User’s Manual, 2018. Accessed on: May 31, 2021. [Online]. Available: 
  8. KINOVA JACO™ Prosthetic robotic arm User Guide, 2018. Accessed on: May 31, 2021. [Online]. Available:, 
  9. T. Aflalo, C. Y. Zhang, E. R. Rosario, N. Pouratian, G. A. Orban, & R. A. Andersen (2020). A Shared Neural Substrate for Action Verbs and Observed Actions in Human Posterior Parietal Cortex. Science Advances. 2020.pdf 
  10. Microsoft HoloLens:

Further Reading

Please click the links below or visit the Andersen Lab website for more information about BMIs.

The Intention Machine: A new generation of brain-machine  interfaces can deduce what a person wants

Annual Review of Psychology: Exploring Cognition with Brain–Machine Interfaces

Single-trial decoding of movement intentions using functional ultrasound neuroimaging

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: