Journal of Neuroscience Methods 229 (2014) 33–43

Contents lists available at ScienceDirect

Journal of Neuroscience Methods journal homepage:

Computational Neuroscience

A hybrid brain computer interface system based on the neurophysiological protocol and brain-actuated switch for wheelchair control Lei Cao a,b , Jie Li a , Hongfei Ji a , Changjun Jiang a,∗ a b

Department of Computer Science and Technology, Tongji University, 201804 Shanghai, China Institute of Medical Psychology and Behavioral Neurobiology, University of Tuebingen, D-72074 Tuebingen, Germany

h i g h l i g h t s • • • •

A MI and SSVEP-based hybrid brain computer interface is first proposed to control a real wheelchair. A novel neurophysiological protocol is used for direction and speed control simultaneously and improve the effectiveness. A hybrid modalities-based control switch is first designed to start and stop the control system for the control safety. The threshold strategy provides the reliability of classification result and safety of wheelchair control.

a r t i c l e

i n f o

Article history: Received 27 January 2014 Received in revised form 21 March 2014 Accepted 22 March 2014 Keywords: Hybrid BCI Wheelchair control Motor imagery SSVEP Switch control

a b s t r a c t Background: Brain Computer Interfaces (BCIs) are developed to translate brain waves into machine instructions for external devices control. Recently, hybrid BCI systems are proposed for the multi-degree control of a real wheelchair to improve the systematical efficiency of traditional BCIs. However, it is difficult for existing hybrid BCIs to implement the multi-dimensional control in one command cycle. New method: This paper proposes a novel hybrid BCI system that combines motor imagery (MI)-based bio-signals and steady-state visual evoked potentials (SSVEPs) to control the speed and direction of a real wheelchair synchronously. Furthermore, a hybrid modalities-based switch is firstly designed to turn on/off the control system of the wheelchair. Results: Two experiments were performed to assess the proposed BCI system. One was implemented for training and the other one conducted a wheelchair control task in the real environment. All subjects completed these tasks successfully and no collisions occurred in the real wheelchair control experiment. Comparison with existing method(s): The protocol of our BCI gave much more control commands than those of previous MI and SSVEP-based BCIs. Comparing with other BCI wheelchair systems, the superiority reflected by the index of path length optimality ratio validated the high efficiency of our control strategy. Conclusions: The results validated the efficiency of our hybrid BCI system to control the direction and speed of a real wheelchair as well as the reliability of hybrid signals-based switch control. © 2014 Elsevier B.V. All rights reserved.

1. Introduction In the field of rehabilitation engineering, Electroencephalogram (EEG)-based BCIs for wheelchair control have attracted great attention because of their convenience, non-invasion and low expense (Tanaka et al., 2005; Nijholt and Tan, 2008; Rebsamen et al., 2010; Sellers, 2011; Lin et al., 2011). It is helpful for the paralyzed to

∗ Corresponding author. Tel.: +86 02169589867; fax: +86 02169589867. E-mail addresses: [email protected] (L. Cao), [email protected] (J. Li), [email protected] (H. Ji), [email protected] (C. Jiang). 0165-0270/© 2014 Elsevier B.V. All rights reserved.

improve the independence in the daily life (Rebsamen et al., 2010; Millán et al., 2010; Sellers, 2011). The EEG signals applied to control a wheelchair include the P300 potential (Rebsamen et al., 2007; Iturrate et al., 2009), SSVEPs (Middendorf et al., 2000; Muller-Putz et al., 2006; Müller et al., 2010; Stamps and Hamam, 2010) and event-related desynchronization/synchronization (ERD/ERS) produced by MI tasks (Leeb et al., 2007; Ferreira et al., 2010; Tsui et al., 2011; Huang et al., 2012). Up to now, synchronous and asynchronous protocols have been presented for BCI-based wheelchair control. Typically, the synchronous protocol was proposed for BCI-based wheelchairs by Rebsamen (Rebsamen et al., 2006, 2010). Then, the P300 potential


L. Cao et al. / Journal of Neuroscience Methods 229 (2014) 33–43

(Iturrate et al., 2009; Rebsamen et al., 2010) or SSVEP signal (Müller et al., 2010; Diez et al., 2013) evoked by visual stimuli, was used for selecting the pre-defined location of a destination. Moreover, an intelligent navigation system was employed to avoid obstacles by laser sensors and drive the wheelchair along the specific path (Iturrate et al., 2009). The user couldn’t modify the designated trajectory. The synchronous protocol showed high accuracy and safety (Rebsamen et al., 2006). However, low response efficiency and inflexible path option were unacceptable for wheelchair control in the real environment. Meanwhile, the asynchronous protocol was used for controlling the brain-actuated wheelchair by the independent manipulation (Leeb et al., 2007; Tsui et al., 2011; Hema et al., 2011). A representative MI-based control system was developed by Galán et al., to control a real wheelchair (Galán et al., 2008). In the system, the asynchronous protocol was applied to realize the real-time directional control. Recently, several BCI systems using the asynchronous protocol had been developed for commercial applications (Blatt et al., 2008; Carrera et al., 2011). Nevertheless, the available control commands from a single modality were not enough to meet the criteria of multi-dimensional control. In recent years, hybrid BCIs were proposed for multidimensional control. It combined different EEG signals to produce multiple control commands simultaneously or sequentially for applications. It was demonstrated that hybrid EEG signals, such as SSVEP and MI, could improve the classification accuracy of BCIs (Allison et al., 2010; Brunner et al., 2010). Lately, Li et al. incorporated the P300 potential and MI or SSVEP to control the 2-D cursor and brain-actuated wheelchair (Li et al., 2010, 2013; Long et al., 2012, 2013). This system provided multiply commands to realize multi-dimensional control (e.g., direction and speed). Simultaneously, the hybrid BCI combined MI and SSVEP signals had been developed for control engineering (Pfurtscheller et al., 2010; Horki et al., 2011; Allison et al., 2012; Choi and Jo, 2013). In these BCIs, the control commands were limited by a small number of classification categories. Moreover, Pfurtscheller et al. firstly designed a MI-based brain switch for activating and deactivating their hybrid BCI system (Pfurtscheller et al., 2010). These MI and SSVEP-based hybrid BCIs achieved the good control effect for external devices control. However, it was a challenge that hybrid BCI system was designed for high-efficiency wheelchair control. In this paper, we proposed a hybrid modalities-based and self-paced BCI system for wheelchair control. A novel neurophysiological protocol provided eight commands for sophisticated multi-dimension control. This protocol gave much more control commands than those of previous MI and SSVEP-based BCIs (Pfurtscheller et al., 2010; Horki et al., 2011; Allison et al., 2012; Choi and Jo, 2013). And a reliable hybrid modalities-based brain switch was developed for a real wheelchair firstly. In our system, left- and right-hand imageries were used for adjusting the direction of the wheelchair. The idle state without mental activities was decoded to keep the wheelchair moving at the straight direction. Synchronously, SSVEP signals induced by gazing specific flashing buttons were used to accelerate or decelerate the wheelchair according to actual conditions. Moreover, the control switch was triggered by hybrid signals (MI and SSVEP), as well a threshold strategy was provided for obtaining available classification results. Our study contained two experiments. First, a virtual wheelchair training task was conducted for evaluating the performance of our proposed hybrid BCI. Second, a real wheelchair was controlled for performing the assigned task by our hybrid BCI system. The efficiency of our methodology was demonstrated by our experimental results and data analysis. The rest of the paper is organized as follows: The methodology, including the system paradigm, neurophysiological protocol, classification algorithm and threshold strategy, is presented in Section

2. Experimental and result analysis are declared in Section 3. Further discussion is given in Section 4. Section 5 concludes the paper. 2. Methodology 2.1. System paradigm We propose a hybrid BCI system based on a novel interactive paradigm for wheelchair control. As shown in Fig. 1, the system is composed of a signal acquisition device, a stimulus panel, a BCI module, a wireless communication module and a wheelchair. A suitable EEG cap is used for data acquisition from 15 Cu electrodes. A high-performance bio-signal amplifier translates the raw signal into the computer-sensible data. On the stimulus panel, 4 flashing buttons are used to produce SSVEP signals. And MI is spontaneously carried out by the user simultaneously. Then the BCI module decodes EEG signals into control commands. These control commands are finally transmitted to control the wheelchair by the wireless communication module. 2.2. Neurophysiological protocol In our system, the user is able to manipulate the direction, speed and switch synchronously. Three options, straight driving, left turn and right turn, are available for directional control. As long as the speed is controlled within the limited range, the user could accelerate, decelerate or drive the wheelchair at the uniform velocity. In addition, the control switch is utilized for starting or stopping the wheelchair. In order to accomplish these functions, the hybrid BCI system provides eight commands in total: turn left, turn right, drive forward, accelerate, decelerate, drive at the uniform velocity, turn on and off the switch. As shown in Table 1, these commands are used for constructing the neurophysiological protocol to control the wheelchair effectively. The neurophysiological protocol for wheelchair control is illustrated in Fig. 2. When the control switch is turned on by the specific hybrid EEG signals, the BCI system starts to acquire EEG data for controlling the speed and direction of the wheelchair. The single control command is transmitted by sliding 2 s window with an 1.5-s overlap between consecutive computations (regulating the state of the wheelchair every 500 ms). During the driving process, the speed and direction are simultaneously modulated by our control strategy. The control switch will not be turned off to stop the wheelchair until the corresponding hybrid EEG signals are detected by our system. First, left-hand imagery and the specific SSVEP signal are concurrently detected to turn on the control system. Then the user can steer the wheelchair by using the direction and speed control commands simultaneously as follows: (1) Left- or right-hand imagery is detected to make a left or right turn respectively. The directional command triggers Table 1 The control command and their corresponding mental tasks (MI: motor imagery). Control commands

Mental tasks

Left turn Right turn Driving forward Acceleration Deceleration Driving at the uniform velocity Turning on the switch

Left-hand MI Right-hand MI Idle state without MI Focusing on the button flashed in the 11Hz Focusing on the button flashed in the 9Hz Ignoring flashing buttons

Turning off the switch

Focusing on the button flashed in the 8Hz and left-hand MI Focusing on the button flashed in the 7Hz and right-hand MI

L. Cao et al. / Journal of Neuroscience Methods 229 (2014) 33–43


Fig. 1. The system paradigm of our hybrid BCI for wheelchair control.

a pre-defined degree of rotation. And the wheelchair will keep going straight in absence of motor imagery patterns. (2) When a speed control command produced by SSVEP is sent to the control system, the driving speed (the rotation or straight velocity) is increased or reduced within the limited range. It is determined by focusing on a specific flashing button corresponding to the acceleration or deceleration. And the wheelchair will drive at the uniform speed if the system does not detect any specific SSVEP signal.

Furthermore, when the user needs to have a rest or stop the wheelchair at the specified position for a while, he/she must imagine right-hand movement and focus on the corresponding flashing button synchronously for switching off the system. Then, the wheelchair won’t be moved until the command for switching on is sent to the control system. The single modality, such as SSVEP or MI, is easily triggered by threshold strategy. The misoperations will affect the efficiency of wheelchair control seriously. Hence, we proposed hybrid modalities-based switch control for turning on or off the wheelchair.

Fig. 2. A flow chart for the neurophysiological protocol used in the wheelchair control. The left-hand imagery and the specific SSVEP signal for turning on the switch are detected to start the wheelchair. Accordingly, right-hand imagery and the specific SSVEP signal for turning off the control switch are extracted to stop the wheelchair. In the wheelchair control, the user imagines left- or right-hand movement to produce a command for directional control, respectively. The idle state without motor imagery is decoded to control the wheelchair to go straight ahead. Meanwhile, the user must focus on corresponding flashing buttons to regulate the driving speed. The rest state without focusing on any flashing button is used for keeping a constant driving.


L. Cao et al. / Journal of Neuroscience Methods 229 (2014) 33–43

2.3. Classification algorithm and threshold strategy The algorithm used for detecting MI and SSVEP signals are described in the following passages. For both modalities, we employ a threshold strategy based on prior experience to distinguish the idle state and control state. (1) MI signals: As described above, left- and right-hand imagery are used to control the direction and switch. To detect MI features, EEG signals are band pass filtered at 5–30 Hz first. Then the data are processed by using the common spatial patterns (CSP) algorithm proposed in (Müller-Gerking et al., 1999) and (Ramoser et al., 2000). The CSP method is useful for discriminating two populations of EEG. Based on training dataset collected in the training phase, a CSP transformation matrix is calculated for a spatial filter. The acquisition of training data is introduced in the next section. The first and last rows of the CSP transformation matrix could maximize the difference of two groups of data. Therefore, several rows from both ends of each class are used as a spatial filter commonly. If the number of feature vectors is too small, the classifier can’t fully extract the distinguishable features between two classes. And the classifier would overfit the training data if the number is too large (Blankertz et al., 2008). In our algorithm, the first and last two rows are selected to construct the transformation matrix for further data analysis. After projecting the EEG signals by CSP transformation matrix, the logarithmic variances of the projections are calculated for feature extraction. And a radial basis function kernel support vector machine (RBF-SVM) classifier is used for training these feature vectors. Furthermore, a 10-fold cross validation method is used for picking out an optimal classifier. In the experiment, the RBF-SVM classifier produces the class label and classification probability. The classification probability P ranges from 0.5 to 1, where larger value represents the higher classification accuracy. Therefore, we use a probability threshold Tclass (class = left − handimageryorright − handimagery) to estimate the present state S as



P < Tclass

controlstate, P  Tclass


where the idle state means that the classification result is invalid to reject the directional change while the control state suggests that the classification result is valid for directional control. In this paper, Tclass was set to 0.75 in according with the result analysis of their training data. Though the optimal threshold was slightly different for each subject, it was feasible that the identical threshold was selected strictly to avoid the error of false positive. (2) SSVEP signals: in our system, SSVEP signals are used for speed and switch control. From (Lin et al., 2006), we utilize the canonical correlation analysis (CCA) method for SSVEP detection. In this method, the CCA coefficient v(cof ) , which ranges from 0 to 1, is calculated to indicate the relevance between the subject’s EEG signals and reference signals of one certain frequency. The value 0 implies a weak correlation while 1 implies a strong one. Consequently, we select the maximum CCA coefficient max(v(cof ) ) to identify which flashing button the user is focusing on. Meanwhile, a threshold T(cof) is used for evaluating the mental condition M. It’s defined as



if max(v(cof ) ) < Tcof


if max(v(cof ) )  Tcof


In (2), if max(v(cof ) ) < Tcof , then the system identifies that the user does not paying attention to a certain flashing button. Conversely, the classification label corresponding to the maximum reference coefficient is valid for wheelchair control. In this paper, Tcof was set to 0.38 according to the data analysis in the training task.

Fig. 3. The names and the distribution of EEG electrodes. Fifteen channels are marked for data acquisition.

3. Experiment and result analysis 3.1. Experimental setup A high-performance bio-signal amplifier (g.Tec) was used to acquire scalp EEG signals from fifteen electrodes. The following 15 channels, including “FC3”, “FC4”, “C5”, “C3”, “C1”, “Cz”, “C2”, “C4”, “C6”, “CP3”, “CP4”, “POz”, “O1”, “Oz”, “O2”, were placed at the frontal, central, parietal and occipital regions. The distribution of these electrodes were showed in Fig. 3. The EEG signals were referenced to the unilateral ear. Impedances of all electrodes were kept below 5 k. The EEG data was amplified, digitalized with a sampling frequency of 256 Hz, notch-filtered with 50 Hz and bandpass-filtered between 0.1 and 30 Hz. To validate the proposed hybrid BCI system for wheelchair control, three healthy male subjects, aged from 21 to 30, participated in our experiments for data collection. None of them had prior experience with hybrid BCI. They all gave informed consent approved by the Ethics Committee. 3.2. Training classification model First, the subjects performed the MI task for training classification models with the RBF-SVM classifier. The workflow was illustrated in Fig. 4. Each subject was asked to seat in an armchair, keep their hands relaxed and focus on the center of the screen. After a left or right arrow appeared, the subject was instructed to imagine a movement of the left or right hand for about 2 s. This session consisted of three runs, each run had 10 trials for each class. The data were used to train RBF-SVM models off-line. 3.3. Training for hybrid BCI and wheelchair control tasks With the model obtained in the above section, visual and auditory stimuli were combined for a training task of hybrid BCI. An arrow cue with instructing left- or right-hand MI was presented on the center screen. At the same time, the auditory cue, one of four spoken digits in Chinese (i.e., 7, 8, 9, 11), which was defined as a corresponding SSVEP stimulus serial number oscillated at 7, 8,

L. Cao et al. / Journal of Neuroscience Methods 229 (2014) 33–43


Fig. 4. The paradigm of training data collection for training classification model in a trial. At the beginning of the trial (0–2 s), the screen is blank. From 2 to 6 s, an arrow cue is shown on the screen. The subject is instructed to perform the left- or right-hand imagery according to the cue. In order to eliminate the influence of the response delay, the data collection is started from 4 to 6 s.

9, 11 Hz, was presented for subjects to focus on the corresponding flashing button for about 2s. In some trials, there was only a single visual or auditory cue presented, then the subjects performed the MI or SSVEP task independently. The subjects were required to execute the corresponding tasks according to the visual cue on the monitor and the auditory cue presented by the loudspeaker. This session consisted of six runs, each run had 28 trials corresponding to different mental tasks performed in a random order. Before the real wheelchair control experiment, the subjects were trained for steering a simulated wheelchair in the virtual environment. We tested the straight and curve paths by three control methodologies(i.e., MI, SSVEP and hybrid signals). The paths were layout in a coordinate system and the scale was 1:550 pixels. These standard paths were plotted in Fig. 5. The straight path was defined as

x = 0.5 0.1  y  0.9


And the curve path was defined as

⎧ ⎨ x = 0.5 + 0.4 sin(t) ⎩

y = 0.5 − 0.4 cos(t)


0  t  2

The control strategies of MI and SSVEP were illustrated in Fig. 6. MI-based control strategy only could modulate the direction for a lack of control commands. And SSVEP-based control strategy was based on the sequential control protocol. That means, alternatively, the user had to accelerate or decelerate, and then made a left or right turn by the SSVEP-based BCI system. The control strategy of

Fig. 6. (A) The control strategy of MI. The simulated wheelchair is controlled for left turn, right turn and straight driving at a constant speed. (B) The control strategy of SSVEP. The simulated wheelchair is controlled for left turn, right turn, accelerating, decelerating and straight driving at a constant speed.

hybrid signals had been presented in Section 2. In this experiment, the high and low speeds were set to 11 and 2.75 pixels/s respectively. The maximum and minimum of rotation speeds were 30o and 10o per directional command. The changing values of the speed and direction were 2.75 pixels and 5o per control command. At the beginning, every subject was permitted to manipulate the simulated wheelchair by three control methodologies for half an hour. Then they were required to control the simulated car through these two paths for three trials. When all subjects finished the training of hybrid BCI system, they were permitted to address the challenge of a real wheelchair control task. The objective of this control experiment was to assess the performance of the participants as well as their ability to avoid obstacles and accomplish complex tasks in the real environment. In this experiment, each of them performed four trials to accomplish the real wheelchair control task. The control mechanism had been already described in Section 2. The user was required to leave

Fig. 5. The standard path for the simulated wheelchair test. The straight path is depicted on the left coordinate system and the curve path is depicted on the right coordinate system. Starting point and Stopping point are both marked in coordinates.


L. Cao et al. / Journal of Neuroscience Methods 229 (2014) 33–43 Table 2 The classification accuracy rates for 3 classifying methods.

Fig. 7. A top-down view of the path in the second experiment. All measures are in meters and the wheelchair is to scale. The shaded objects represent the static obstacles. The dash curve is the standard example of a real path.

the starting point and reach the stopping point by passing the breakpoint. The standard road map was marked in Fig. 7. At the breakpoint, the subject was required to switch off the wheelchair for 30 s. The parking task was designed for evaluating the reliability of the hybrid modalities-based switch. The subject needed to turn off the switch and avoid launching the wheelchair by the error of false positive in 30 s. The linear velocity was limited between 0.2 and 0.5 m/s and the angular velocity was set from 10o to 30o . In the real environment, the linear and angular velocities were lower than the theoretical values because of several factors including the terrain topography.

3.4. Experimental result In our study, a simulated wheelchair training task and a real wheelchair test were conducted to evaluate our hybrid BCI system. In the experiment, it was impossible to detect the classification accuracy because the subject could not remember all control intentions in a long time. Therefore, other indicators were used for evaluating the experimental performance. According to (Long et al., 2013), we selected the comparable indicators and proposed other significant indicators to assess our BCI system and control strategies. (1) Accuracy rate: the percentage of successful classification result. (2) Mean Error: the mean of distance errors. The distance error is the shortest length between the recording coordinate point and the standard path after one control command is transmitted by our BCI system. Especially, path length employed in (Iturrate et al., 2009; Long et al., 2013), is a good indicator to reflect the experimental performance. The longer the path length is, the lower the control efficiency will be. And it is same with the mean error. However, the mean error could give a visual interpretation of the control efficiency by the comparison between the actual trajectory and the standard path. (3) Standard deviation of errors: the standard deviation of distance errors. The distance error is defined at the indicator of the mean error. (4) Path length optimality ratio: the ratio of the traveled path length to the standard path length.


The classification accuracy rate for MI (%)

The classification accuracy rate for SSVEP (%)

The classification accuracy rate for hybrid modalities (%)

1 2 3

100 96.3 100

87.5 93.7 100

87.5 87.5 96.9

(5) Time Consumption: the time consumption to accomplish the task. It is calculated by the number of commands recorded by the BCI module. (6) Information transmission rate (ITR): the bit rate of information communicated per unit time. ITR is a standard measure of communication systems. More details can be found in (Cheng et al., 2002). (7) Collisions: the number of collisions incurred to the edges of the working coordinate by the simulated wheelchair or to the obstacle by the real wheelchair. (8) The useful command of switch control: the control command of switching off the BCI system used for stopping the wheelchair at the breakpoint and avoiding obstacles. (9) The useless command of switch control: the misoperation of switching off the BCI system in the task. Table 2 listed the accuracy rates for three classification methods in the training phase. The result suggested, the classification accuracy of hybrid modalities were lower than that of other classification methods slightly. However, the accuracy rate of hybrid modalities was more than 85% for all subjects. This conclusion indicated that, it was feasible for our hybrid BCI to control a wheelchair. The experimental result of the simulated wheelchair task was summarized in Tables 3 and 4. The tasks were successfully completed by all subjects. The average performances of all subjects were illustrated in Fig. 8. Because of the lack of control commands, the speed and direction were not simultaneously controlled by single modality-based method. The basic directional control was firstly designated for the functional completeness. Hybrid modalitiesbased commands were enough to control the speed and direction of the wheelchair at the same time. For the straight path, the average mean error and average standard deviation of errors by the SSVEP-based method were the least among three control methodologies. And the average mean error and average standard deviation of errors by the hybrid modalities-based method were minimum among three control methodologies for the curve path. It was demonstrated that the control precision of SSVEP tasks was optimal for steering the simulated wheelchair on the straight path. And the control strategy of hybrid tasks was fit for the curve path. However, the average time consumptions of hybrid tasks were the least for all control tasks in both situations, indicating that the control method of hybrid tasks was time-saving because of the benefit of synchronous multi-degree control (our hybrid BCI system allowed the user to regulate the speed and direction simultaneously while other BCI systems couldn’t accomplish the parallel control). And no collisions occurred in this experiment. Furthermore, the mean values of path length optimality ratios by hybrid BCI were lower than 1.2 for the straight and curve paths. This result excelled the performance of previous BCI systems proposed in (Iturrate et al., 2009; Long et al., 2013) for the simulated wheelchair task. In the real wheelchair task, the process of wheelchair control task was recorded by digital video camera for further analysis. The performance of the real wheelchair task was presented in Table 5. Apparently, it was difficult to obtain the precise trajectories without sensors. Nevertheless, the statistics of time consumption was available to assess the practicability for our BCI control system. In the experiment, Subject 1 spent more time finishing the task as a

L. Cao et al. / Journal of Neuroscience Methods 229 (2014) 33–43


Table 3 The performance of the simulated wheelchair task on the straight path. Methodology


The mean error (pixels)


1 2 3 1 2 3 1 2 3

14.3 18.7 4.4 7.7 14.3 8.3 17.1 6.6 14.9



± ± ± ± ± ± ± ± ±

2.2 1.7 1.7 2.8 1.7 2.8 3.3 2.2 2.8

Standard deviation of errors (pixels) 8.92 11.88 11.99 6.48 8.29 5.30 16.04 4.83 12.80

± ± ± ± ± ± ± ± ±

0.09 0.02 0.08 0.12 0.06 0.10 0.07 0.10 0.06

Path opti. ratio 1.15 1.34 1.05 1.02 1.03 1.02 1.14 1.01 1.03

± ± ± ± ± ± ± ± ±

0.03 0.03 0.04 0.01 0.02 0.02 0.05 0.01 0.01

Time consumption (s) 265 338 348 367 281 261 87 77 84

± ± ± ± ± ± ± ± ±

34 27 38 31 27 33 10 14 15

Collisions 0 0 0 0 0 0 0 0 0

Table 4 The performance of the simulated wheelchair task on the curve path. Methodology


The mean error (pixels)


1 2 3 1 2 3 1 2 3

285.5 274.5 281.6 260.7 287.7 282.7 224.4 221.7 256.9



± ± ± ± ± ± ± ± ±

6.6 8.3 7.2 7.7 6.6 6.1 6.1 4.4 7.2

Standard deviation of errors (pixels) 107.88 122.83 121.44 121.65 123.48 113.56 109.69 109.45 117.79

± ± ± ± ± ± ± ± ±

12.62 6.16 8.72 4.98 13.48 10.66 8.28 11.06 6.42

Path opti. ratio 1.20 1.04 1.08 1.07 1.04 1.26 1.26 1.18 1.08

± ± ± ± ± ± ± ± ±

0.06 0.02 0.03 0.04 0.03 0.07 0.09 0.05 0.04

Time consumption (s) 542 516 527 621 446 523 250 220 201

± ± ± ± ± ± ± ± ±

43 32 37 37 44 36 25 28 34

Collisions 0 0 0 0 0 0 0 0 0

Fig. 8. The average performances of all subjects by 3 control methods. The mean error and standard deviation of errors indicate the control precision of different control methods. Path length optimality ratio represents the cost of physical distance. Time consumption reveals the efficiency of different control methods.

Table 5 The performance of the real wheelchair test. Subject

Time consumption(s)

Useful commands of switch control

Useless commands of switch control

Time for stopping task(s)


1 2 3

568 ± 52 269 ± 40 275 ± 32

6±2 5±3 5±1

3±1 2±2 2±1

35 ± 4 35 ± 5 36 ± 4

0 0 0


L. Cao et al. / Journal of Neuroscience Methods 229 (2014) 33–43

Fig. 9. On the left side, the ITRs of three subjects for three different control methods are listed in the histogram. On the right side, the average ITRs of three control methods imply the difference between traditional BCIs and hybrid BCIs.

result of his weak control ability. By comparison, Subject 2 and Subject 3 controlled the wheelchair more effectively. In addition, few useless commands of switch control indicated that the wheelchair control wasn’t interfered by switch control. None of them left the breakpoint until the wheelchair stoped over 30 s (our experimental requirements compelled the user to park in the breakpoint for 30 s at least). And no collisions were observed in this experiment. These results proved that, all users controlled the real wheelchair by our hybrid BCI system successfully. 4. Discussion In this paper, a neurophysiological protocol was presented for hybrid BCI system to control a real wheelchair. The advantages of our proposed hybrid BCI system included the high-efficiency control strategy derived from the novel protocol as well as the reliability of hybrid modalities-based switch control. Previously, ITR was limited by the single modality of EEG signals for BCI-based wheelchairs controlled by the MI (Iturrate et al., 2009) or SSVEP signal (Muller-Putz et al., 2006). Fig. 9 showed the difference of ITR between single modality-based BCI systems and our hybrid BCI system in the experiment of the simulated wheelchair. The average ITR of hybrid BCI system was considerably higher than that of MIbased or SSVEP-based BCI system. Recently, other hybrid BCI had been developed for obtaining the considerable ITR (Choi and Jo, 2013; Li et al., 2013). Nevertheless, the comparison of ITR between our proposed BCI with them verified that our control system had the apparent advantage of information transmission. These results suggested that our hybrid BCI system had the excellent capacity of control efficiency. In the experiment of the simulated wheelchair (see Fig. 8), the index of path length optimality ratio was a main comparable factor due to the difference of the experimental conditions (e.g., parameters settings of the speed and the path). Comparing with the result in (Iturrate et al., 2009; Long et al., 2012, 2013), the superiority reflected by the index of path length optimality ratio validated the high efficiency of our control strategy, mainly because the control command sent by SSVEP signal had higher efficiency than that by P300 in time consumption. It is a great advantage for designing the hybrid BCI to use SSVEP and MI signals. To clearly show the control procedure, the subjects’ trajectories obtained from one trial were roughly drawn in Fig. 10. Generally, these trajectories were consistent with the predefined route. It implied that the direction and speed of the wheelchair could be effectively modulated by our synchronous control strategy. For previous similar hybrid modalities-based BCI systems, the control protocol only provided two-class classification for SSVEP recognition (Pfurtscheller et al., 2010; Horki et al., 2011; Allison et al., 2012; Choi and Jo, 2013). The control complexity of these

BCIs was lower than that of our BCI. However, the experimental result implied that our system was proper for multi-dimension wheelchair control. Moreover, the quantitative dominance of control commands was helpful for solving the problem of control complexity. Furthermore, we analyzed the usage of hybrid modalities, single modality and idle state as showed in Fig. 11. The average usage shares of hybrid modalities accounted for more than 30% of total commands for all subjects. It was proved that hybrid modalities were frequently applied to control the direction and speed of the real wheelchair simultaneously. For the sequential control strategy from (Long et al., 2013), The user had to take twice time to achieve the same control effect by our hybrid BCI system. It was demonstrated that our hybrid BCI system reduced the time expenditure greatly. From Table 2, the classification accuracy of MI was very high. However, the idle state was not used for training phase. The distance error was mainly caused by few control mistakes of the idle state, which represented for the straight driving. And the direction needed to be corrected continuously after one mistake was made. It was suggested that the experimental performance was dependent on the control strategy. In particular, Subject 2 used less multi-modality commands than other subjects, but his time consumption was the least among all participants in the real wheelchair control task. This difference was mainly due to the strategy selection in different road conditions. In

Fig. 10. The trajectories roughly obtained from one trial for all subjects. The colors of these curves are used to identify different subjects. The real line represents the route in the trial.

L. Cao et al. / Journal of Neuroscience Methods 229 (2014) 33–43


Fig. 11. The average usage shares of hybrid modalities, single modality and idle state for four trials.

the simulated wheelchair control task, the finding indicated that the control strategy of SSVEP had the advantage of control precision on the straight path. Conversely, the control efficiency of hybrid modalities was higher than that of the single modality on the curve path. From Tables 3 and 4, Paired t-test demonstrated that the control precision of hybrid modalities was significantly higher than that of MI (t = 4.04, p < 0.05) or SSVEP (t = 3.55, p < 0.05) for the curve path, but no significant difference were found for the straight path. This conclusion implied that the user should only use the commands for regulating the speed and avoid deflecting the route on the straight path. And the synchronous control strategy needed to be applied to improve the efficiency of wheelchair control on the curve path. Therefore, we argued that the balance between speed control and direction control must be kept for driving the wheelchair in complicated road conditions. In addition, the user didn’t need always to keep eyes on flashing buttons or imagine hand movement monotonously. In our system, the subject only implemented the mental task or gazed at a flashing button when the user had to change the direction or speed of the wheelchair. From Fig. 11, the amount of idle states accounted for about 10% of total commands for each subject. This phenomenon indicated that the user would have enough time to rest or relax at the right time. In our system, the threshold strategy provided the reliability of classification results for users. In previous studies, the likelihood probabilities of classification results were not always utilized in these BCI systems. Only when the likelihood probability reached a certain upper threshold, could the classification result be available to reflect the instantaneous mental activity exactly. Thus, we should employ the threshold strategy to produce credible control

commands precisely. On the basis of above principle, we used the probability thresholds provided by the RBF-SVM classifier and CCA method for identifying mental states. The thresholds were defined by the result of the training test. Fig. 12 showed the classification results for Subject 2 in the first trial of the real wheelchair control task. In the first half period of this task, the defined threshold of MI filtered most low-probability classification results to maintain the orientation of the wheelchair. This phenomenon was consistent with the pre-defined straight route for the first half path. And the valid directional commands (exceeding the threshold of MI) significantly increased for improving the driving quality of the curve path in the second half period. Meanwhile, the control commands of speed control were processed by the threshold of SSVEP for actual road conditions. These findings validated that the threshold strategy was flexible for adapting complicated road conditions. Especially, the typical classification results (from Fig. 12) were illustrated in Fig. 13. By contrast, the MI and SSVEP signals were obviously detected by temporal-spatial and spatial features when the likelihood probabilities of classification results were higher than pre-defined thresholds. These results indicated that the threshold strategy improved the systematical adaptability for different road conditions evidently. In this study, it was important to have a control switch to turn on or off the BCI control system for autonomous control. However, it was difficult to design an efficient (fast and precise) switch for wheelchair control. Previously, Long et al. proposed the possibility of combining SSVEP signals with electromyography (EMG) to produce a fast and accurate stop command in their control system (Long et al., 2013). Furthermore, a simple switch based on the state control was developed for virtual wheelchair control in

Fig. 12. In the first wheelchair control task for Subject 2, the thresholds for MI and SSVEP filter some low-probability classification results, respectively. The axis N indicates the sequential number of classification results and the axis P stands for the likelihood probability from 0 to 1.


L. Cao et al. / Journal of Neuroscience Methods 229 (2014) 33–43

Fig. 13. (A) The typical classification result is illustrated to discriminate MI signal and idle state. When the classification probability is higher than the pre-defined threshold of MI, the feature of ERD/ERS is obviously detected for MI. (B) The typical classification result is illustrated to discriminate SSVEP and non SSVEP. If the reference coefficient is higher than the pre-defined threshold of SSVEP, the spectral power is intensified at the corresponding frequency.

(Huang et al., 2012). And the MI-based brain switch had been desingated for hybrid BCI control (Pfurtscheller et al., 2010). In our BCI system, we first proposed the brain-actuated switch controlled by hybrid modalities and the trigger period was shortened to 500 ms for meeting the control requirements. Obviously, the control task was more difficult than those of above BCI systems. However, it was demonstrated that combining multiple brain signals, such as MI and SSVEP, could improve the classification accuracy (Allison et al., 2010). In order to ensure the switch not to be arbitrarily triggered, we made use of hybrid modalities to produce control commands for turning on or off the switch. First, left-hand imagery and the specific SSVEP signal for turning on the control switch must be simultaneously detected to start the control system. And the user should perform right-hand motor imagery and focus on the specific flashing button related to turn off the switch when he/she wants to stop the wheelchair. Subject 3’s performance of switch control was typically presented in Fig. 14. The number of misoperations was negligible in the whole control process. Generally, the inconvenience had been reduced to reach an acceptable level for

wheelchair control. In this experiment, the subject was required to accomplish the breaking task for 30 s. The tasks were completed within enough residence time by all participants. Moreover, the control switch was effectively utilized to brake for avoiding obstacles. These results implied that the hybrid modalities-based switch was easily controlled to improve the functionality of wheelchair control for meeting actual control demands. Although the results demonstrated the feasibility of applying a hybrid BCI system to wheelchair control, some issues were recognized while conducting experiments. Firstly, the recording electrodes are more than that of the previous MI and SSVEP-based hybrid BCI (Pfurtscheller et al., 2010). It is not feasible for developing portable BCI systems. In the future, the number of electrodes need to be reduced to design appropriate portable devices. Simultaneously, the classification accuracy should be acceptable for control precision and safety. Furthermore, all subjects had rich experience in BCI tasks. The workload of training was acceptable for these veteran users. However, thorough analyses of training workload will require more studies with more subjects. Evaluating vigilance, usability, other environmental factors is important in real-world settings for complex applications. 5. Conclusions This paper introduced a novel hybrid BCI system to control the direction and speed of a real wheelchair. The control system combined MI and SSVEP signals for the synchronous control strategy. Eight commands were obtained from the multi-modality mental tasks. A simulated wheelchair training task by 3 control methods was implemented firstly and a real wheelchair control experiment was performed for assessing our hybrid BCI system. Both experimental results demonstrated the effectiveness of our hybrid BCI system. Further data analysis revealed that our synchronous control strategy was useful for reducing time consumption. Moreover, a threshold strategy was proposed to provide the reliability of classification results for wheelchair control. Finally, a control switch was designed to improve the functionality of our BCI system and avoid obstacles. Our future work will be focus on the audiovisual hybrid BCI system for wheelchair control. Acknowledgements The work was supported by the Fundamental Research Funds for National Basic Research Program of China (Grant No. 2010CB328101), International cooperation program (2012DFG11580), China Postdoctoral Science Foundation (2013M541541), Central Universities (Grant No. 0800219202) and National Natural Science Foundation of China (Grant No. 61105122). References

Fig. 14. The sequential performance of switch control in a trial of wheelchair control experiment for Subject 3. Yellow parts indicate the system is active for normal control. Red parts indicate the system is closed by the misoperation. Blue parts indicate the system is closed for avoiding the obstacle. Green parts indicate the subject is instructed to stop the wheelchair at the breaking point. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Allison B, Brunner C, Altstätter C, Wagner I, Grissmann S, Neuper C. A hybrid ERD/SSVEP BCI for continuous simultaneous two dimensional cursor control. J Neurosci Meth 2012];209(2):299–307. Allison BZ, Brunner C, Kaiser V, Müller-Putz GR, Neuper C, Pfurtscheller G. Toward a hybrid brain-computer interface based on imagined movement and visual attention. J Neural Eng 2010];7(2):026007. Blankertz B, Tomioka R, Lemm S, Kawanabe M, Muller KR. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process Magazine 2008];25(1):41–56. Blatt R, Ceriani S, Seno BD, Fontana G, Matteucci M, Migliore D. Brain control of a smart wheelchair. In: 10th international conference on intelligent autonomous systems; 2008]. p. 221–8. Brunner C, Allison BZ, Krusienski DJ, Kaiser V, Müller-Putz GR, Pfurtscheller G. Improved signal processing approaches in an offline simulation of a hybrid brain-computer interface. J Neurosci Meth 2010];188(1):165–73. Carrera A, Alonso A, de la Rosa Steinz R, Gómez M, Val LD. Wheelchair-based system adapted to disabled people with very low mobility. Ambient Intell Softw Appl 2011];92:17–24.

L. Cao et al. / Journal of Neuroscience Methods 229 (2014) 33–43 Cheng M, Gao X, Gao S, Xu D. Design and implementation of a brain–computer interface with high transfer rates. IEEE Trans Biomed Eng 2002];49(10):1181–6. Choi B, Jo S. A low-cost EEG system-based hybrid brain–computer interface for humanoid robot navigation and recognition. PLoS ONE 2013];8(9):e74583. Diez P, Müller S, Mut V, Laciar E, Avila E, Bastos-Filho T, Sarcinelli-Filho M. Commanding a robotic wheelchair with a high-frequency steady-state visual evoked potential based brain–computer interface. Med Eng Phys 2013];35(8): 1155–64. Ferreira A, Bastos-Filho TF, Sarcinelli-Filho M, Sánchez JLM, García JCG, Quintas MM. Improvements of a brain–computer interface applied to a robotic wheelchair. Biomed Eng Syst Technol 2010];52:64–73. Galán F, Nuttin M, Lew PWFE, Vanacker G, Philips J. A brain-actuated wheelchair: asynchronous and non-invasive brain-computer interfaces for continuous control of robots. Clin Neurophysiol 2008];119(9):2159–69. Hema CR, Paulraj MP, Yaacob S, Adom AH, Nagarajan R. Asynchronous brain machine interface-based control of a wheelchair. Softw Tools Algorithms Biol Syst 2011];696:565–72. Horki P, Solis-Escalante T, Neuper C, Müller-Putz G. Combined motor imagery and SSVEP based BCI control of a 2 DoF artificial upper limb. Med Biol Eng Comput 2011];49(5):567–77. Huang D, Qian K, Fei DY, Jia W, Chen X, Bai O. Electroencephalography (EEG)-based brain-computer interface (bci): A 2-D virtual wheelchair control based on eventrelated desynchronization/synchronization and state control. IEEE Trans Neural Syst Rehabilit Eng 2012];20(3):379. Iturrate I, Antelis JM, Kubler A, Minguez J. A noninvasive brain-actuated wheelchair based on a p300 neurophysiological protocol and automated navigation. IEEE Trans Robot 2009];25(3):614–27. Leeb R, Friedman D, Müller-Putz GR, Scherer R, Slater M, Pfurtscheller G. Self-paced (asynchronous) BCI control of a wheelchair in virtual environments: a case study with a tetraplegic. Comput Intell Neurosci 2007];2007:7. Li Y, Long J, Yu T, Yu Z, Wang C, Zhang H. An EEG-based BCI system for 2-D cursor control by combining mu/beta rhythm and p300 potential. IEEE Trans Biomed Eng 2010];57(10):2495–505. Li Y, Pan J, Wang F, Yu Z. A hybrid BCI system combining p300 and SSVEP and its application to wheelchair control. IEEE Trans Neural Syst Rehabilit Eng 2013];60(11):3156–66. Lin CT, Euler C, Mekhtarian A, Gil A, Hern L, Prince D. A brain-computer interface for intelligent wheelchair mobility. In: Health care exchanges (PAHCE), 2011 Pan American; 2011 March. p. 316. Lin Z, Zhang C, Wu W, Gao X. Frequency recognition based on canonical correlation analysis for SSVEP-based BCIS. IEEE Trans Biomed Eng 2006];53(12): 2610–4. Long J, Li Y, Wang H, Yu T, Pan J. Control of a simulated wheelchair based on a hybrid brain computer interface. In: Engineering in medicine and biology society (EMBC), 2012; 2012]. p. 6727–30.


Long J, Li Y, Wang H, Yu T, Pan J, Li F. A hybrid brain computer interface to control the direction and speed of a simulated or real wheelchair. IEEE Trans Neural Syst Rehabilit Eng 2013];20(5):720–9. Middendorf M, McMillan G, Calhoun G, Jones KS. Brain-computer interfaces based on the steady-state visual-evoked response. IEEE Trans Rehabilit Eng 2000];8(2):211–4. Millán JR, Rupp R, Mller-Putz GR, Murray-Smith R, Giugliemma C, Tangermann M. Combining brain-computer interfaces and assistive technologies: state-of-theart and challenges. Front Neurosci 2010];4:1–15. Müller SMT, Celeste WC, Bastos-Filho TF, Sarcinelli-Filho M. Brain-computer interface based on visual evoked potentials to command autonomous robotic wheelchair. J Med Biol Eng 2010];30(6):407–15. Müller-Gerking J, Pfurtscheller G, Flyvbjerg H. Designing optimal spatial filters for single-trial EEG classification in a movement task. Clin Neurophys 1999];110(5):787–98. Muller-Putz GR, Scherer R, Neuper C, Pfurtscheller G. Steady-state somatosensory evoked potentials: suitable brain signals for brain-computer interfaces? IEEE Trans Neural Syst Rehabilit Eng 2006];14(1):30–7. Nijholt A, Tan D. Brain-computer interfacing for intelligent systems. IEEE Trans Intell Syst 2008];23(3):72–9. Pfurtscheller G, Solis-Escalante T, Ortner R, Linortner P, Muller-Putz G. Self-paced operation of an SSVEP-based orthosis with and without an imagery-based brain switch: a feasibility study towards a hybrid BCI. IEEE Trans Neural Syst Rehabilit Eng 2010];18(4):409–14. Ramoser H, Muller-Gerking J, Pfurtscheller G. Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabilit Eng 2000];8(4):441–6. Rebsamen B, Burdet E, Guan C, Zhang H, Teo CL, Zeng Q. A brain-controlled wheelchair based on p300 and path guidance. In: The first IEEE/RAS-EMBS international conference on biomedical robotics and biomechatronics, BioRob 2006; 2006]. p. 1101–6. Rebsamen B, Guan C, Zhang H, Wang C, Teo C, Ang MH. A brain controlled wheelchair to navigate in familiar environments. IEEE Trans Neural Syst Rehabilit Eng 2010];18(6):590–8. Rebsamen B, Teo CL, Zeng Q, Burdet E, Guan C. Controlling a wheelchair indoors using thought. IEEE Trans Intell Syst 2007];22(2):18–24. Sellers EW. Clinical applications of brain–computer interface technology. Clin EEG Neurosci 2011];42(4):iv–v. Stamps K, Hamam Y. Towards inexpensive BCI control for wheelchair navigation in the enabled environment – a hardware survey. Brain Informat 2010];6334:336–45. Tanaka K, Matsunaga K, Wang HO. Electroencephalogram-based control of an electric wheelchair. IEEE Trans Robot 2005];21(4):762–6. Tsui CSL, Gan JQ, Hu H. A self-paced motor imagery based brain–computer interface for robotic wheelchair control. Clin EEG Neurosci 2011];42(4):225–9.

A hybrid brain computer interface system based on the neurophysiological protocol and brain-actuated switch for wheelchair control.

Brain Computer Interfaces (BCIs) are developed to translate brain waves into machine instructions for external devices control. Recently, hybrid BCI s...
2MB Sizes 0 Downloads 3 Views