Experiment

Adaptive automation using Brain-Machine-Interface to recover from unpredicted perturbation in flight attitude.

Objective

The objective of this reseach is to be able to develop adaptive automation that can respond to a pilots command directly from the brain faster than the hand. Machine learning techniques over brain activity collected by 400 channel MEG are used to detect the intention to rapidly pull back on the control stick (pitch up) in response to an unpredicted perturbation. The BMI decoder is trained on 3 sessions in which the pilot flies over the ocean and tested on one additional session over the ocean and another session in which the pilot flies through the grand canyon. The grand canyon session is to determine if the BMI can generalize to new conditions in which the pilot is continuously moving the control stick. Independent component anaylsis was used over the 400 MEG channels and task related components were extracted. Least squares probabilistic classification was conducted over features in the task related component continuouly in a 80 ms moving window.

Task/Conditions/Example

The conditions consisted of trials in which there was an unexpected perturbation and trials in which there was not a perturbation. The pilot flew the plane (F22 Raptor) from the first person perspective. Four sessions were over the ocean and one session was in the grand canyon where the task was to maneuver over the river as close to the ground as possible while being able to recover from a perturbation.

Example Movie of Perturbation over Ocean

Please install the Flash Plugin

Example Movie of Perturbation over Grand Canyon

Please install the Flash Plugin


Results

The results for the test session over the ocean: 85% Correct Detection of Perturbation. Hits with faster response time than by hand = 28 out of 40. The number of False Alarms when BMI detected perturbation when there wasn't one = 0. The average response time improved from 369ms to 291ms (78ms).


Please install the Flash Plugin

The results for the test session over the grand canyon: 82.5% Correct Detection of Perturbation. Hits with faster response time than by hand = 40 out of 60. The number of False Alarms when BMI detected perturbation when there wasn't one = 1 out of 60. The average response time improved from 346ms to 286ms (60ms).


Please install the Flash Plugin

Decoding was over a single independent component with distributed source localized in precuneus, SMA, somatosensory cortex, and visual cortex.

Important Findings

The BMI is faster than the hand. Adaptive automation using a BMI can enhance Flight Performance.

The BMI works continuously over time without any a-priori knowledge of when a perturbation may occur.

The BMI can generalize to more complex tasks and differentiate between motor intention to an unexpected perturbation from that used during normal maneuvering.

Practical application in controlling drones in which timing may be critical due to inherent feedback and control delays in the system.