484
IEEE TRANSACTIONS ON HAPTICS,
VOL. 6,
NO. 4,
OCTOBER-DECEMBER 2013
Task Performance Evaluation of Asymmetric Semiautonomous Teleoperation of Mobile Twin-Arm Robotic Manipulators Pawel Malysz, Member, IEEE, and Shahin Sirouspour, Member, IEEE Abstract—A series of human factors experiments involving maneuvering and grasping tasks are carried out to evaluate the effectiveness of a novel asymmetric semiautonomous teleoperation (AST) control design framework for teleoperation of mobile twin-arm robotic manipulators. Simplified configurations are examined first to explore control strategies for different aspects of such teleoperation tasks. These include teleoperation of a nonholonomic mobile base, telemanipulation of a dual-arm robot, and dual-arm/dual-operator teleoperation task scenarios. In two sets of experiments with a planar nonholonomic mobile base, teleoperation via a 3DOF planar haptic interface with position mapping and force reflection of the nonholonomic constraint decreases task-completion-time (TCT) and reduces unwanted collisions. In dual-arm and dual-operator teleoperation maneuverability experiments, the assignment of decoupled and nonconflicting control frames reduces TCT and unwanted contacts. The use of so-called “soft” constraints via passive semiautonomous control reduces TCT and unwanted block drops in telegrasping experiments with a twin-arm manipulator. A final comprehensive experiment encompassing elements of the simplified configurations demonstrates the effectiveness of AST control framework in dual-operator teleoperation of a twin-arm mobile manipulator. Index Terms—Telerobotics, cooperative teleoperation, haptics, nonholonomic robot, multilateral control, human factors
Ç 1
INTRODUCTION
C
ONVENTIONAL teleoperation systems usually involve a single operator controlling a single robot where the master and slave end-effectors have the same degrees of freedom (DOF). Such system configurations have been extensively researched from the control and human factors perspectives [1], [2]. In reality, however, many practical teleoperation tasks require more complex system configurations than the standard symmetric single-master/singleslave system. The authors research group has recently introduced a new unified framework for design and control of a much broader class of haptic enabled asymmetric semiautonomous teleoperation (AST) systems [3]. In AST, the master and slave subsystems may have different DOF, multiple robots/users may coordinate in the task, and autonomous control may be mixed with human control. AST examples in Fig. 1 show a great degree of flexibility in the system design that could be exploited to enhance human-machine interface ergonomics and help operator(s) execute difficult tasks. Multiarm, kinematically redundant and mobile robots can improve operability in complex environments. Multimaster architectures not only enable multiuser cooperation, but they can also increase the
. The authors are with the Department of Electrical and Computer Engineering, McMaster University, ITB-A111, 1280 Main St. West, Hamilton, ONT L8S 4K1, Canada. E-mail: [email protected], [email protected]. Manuscript received 10 Jan. 2012; revised 28 Feb. 2013; accepted 29 Apr. 2013; published online 8 May 2013. Recommended for acceptance by S. Hirche. For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference IEEECS Log Number TH-2012-01-0003. Digital Object Identifier no. 10.1109/ToH.2013.23. 1939-1412/13/$31.00 ß 2013 IEEE
dexterity of the operator interface by employing multiarm haptic devices. Moreover, autonomous control can assist the operator(s) in difficult task scenarios. The focus of our earlier work in [3], [4], and [5] was on the development of a theoretical control framework for AST. This paper experimentally evaluates AST control strategies for a specific application, i.e., teleoperation of mobile twin-arm manipulators. This is achieved by examining the human operator(s) performance in a number of relevant elemental AST system configurations and an allencompassing task. Compared to conventional (hapticenaled) symmetric teleoperation systems, human factors aspects of asymmetric semiautonomous teleoperation systems have been relatively unstudied. One of the design goals of any teleoperation controller is to achieve high performance. This is usually addressed by providing high transparency or augmenting human capabilities. A multitude of control approaches and architectures have been developed for teleoperation to address issues ranging from time delay, robustness, asymmetries, to human perception and performance [1], [2]. The presence of an operator in the control loop in teleoperation inevitably requires human factors experiments on practical tasks to demonstrate effectiveness of controllers designed to improve user performance. A wide variety of experiments have been used for teleopereration performance evaluation. For example, a classical Fitts task involving point-to-point motion was used for early teleoperation studies experimenting with variable velocity mappings [6]. A cooperative peg-in-hole task has been employed to evaluate the effectiveness of haptic guides in bilateral teleoperation grasping [7]. Another Published by the IEEE CS, RAS, & CES
MALYSZ AND SIROUSPOUR: TASK PERFORMANCE EVALUATION OF ASYMMETRIC SEMIAUTONOMOUS TELEOPERATION OF MOBILE...
485
Fig. 1. Examples of asymmetric semiautonomous teleoperation configurations relevant to mobile twin-arm slave robots. (a) Single-master/singleslave teleoperation of a mobile kinematically deficient slave; the nonholonomic constraint on the slave TCF is reflected to the master device. (b) Multiple-master/single-slave teleoperation control of a kinematically redundant slave holding an object; each master can control a different teleoperation control frame on the slave. (c) Multiple-master/multiple-slave teleoperation with user-perceived autonomous control components; such components can for example provide grasping assistance. (d) Single-master/multiple-slave teleoperation of a twin-armed slave system where autonomous control enforces a desired formation; teleoperation is used to control the average position of the slave end-effectors.
peg-in-hole type task was performed by McLean et al. [8] to evaluate new performance metrics and to study effects of 2D, 3D, and direct viewing conditions. Goel et al. [9] and Hwang and Hannaford [10] have employed point-to-point motion and positioning tasks to evaluate redundancy resolution schemes considering joint failure and pseudoinverse inertia weighting, respectively. Cooperative motion tasks have also been performed to assess haptic skills training in teleoperation [11]. A block placement task was considered by Schauß et al. [12] in the evaluation of a coordinating controller to improve multiuser cooperation in teleoperation. Collision avoidance is another practical task that has been used for systems involving time delay [13] and mobile robot control [14]. More application-specific tasks such as those related to satellite module docking have also been used to evaluate perspective taking and camera placement [15]. Semere et al. [16] carried out experimental trials considering a blunt dissection task with partial force feedback. From the above examples, it is common to evaluate performance using metrics based on time/speed and completion of tasks such as inserting/stacking of blocks and objects. In this paper, the authors newly developed methods for trilateral teleoperation [5], [4], teleoperation of nonholonomic mobile robots [4], and semiautonomous grasping assistance in dual-slave teleoperation [17] are revisited and evaluated from an operator performance perspective. These different control approaches are special cases of a more general and systematic control strategy for hapticenabled AST systems presented in [3] that is ultimately applied for teleoperation of a mobile twin-arm robot manipulator. Given the focus of the paper, only a brief overview of the key elements and properties of the actual control framework are discussed here. The AST configurations depicted in Fig. 1 all arise in multioperator teleoperation of mobile-twin arm robots. Since mobile robot bases are typically nonholonomic, AST control strategies for such robots are developed and evaluated in this paper. These include dual-hand and dual-operator control in trilateral teleoperation system configurations. A dual-master/singleslave maneuverability task is presented that can help investigate the effect of coupling/decoupling between the two master devices in dual-operator and dual-hand settings. The aforementioned configurations can exhibit
kinematic constraints that can be haptically reflected to the operator(s); the multilateral and nonholonomic experiments in this paper help to evaluate the impact of such constraint forces on teleoperation performance. A dualmaster/dual-slave teleoperation scenario allows to independently assess the impact of semiautonomous grasping assistance for the operation of the slave manipulator arms, both in single-operator and dual-operator settings. A final combined task employing a mobile twin-arm slave is performed to validate the AST control framework under a realistic task scenario. The design of the task and AST control strategy is informed by the results of the earlier simplified AST configuration experiments. A maneuverability task is designed that involves elements of position control and collision avoidance is chosen to evaluate performance in the teleoperation control of a mobile robot and multiple-master/single-slave teleoperation. A new composite score metric is introduced to simultaneously evaluate both task speed and contact forces for these experiments. In slave dual-arm teleoperation experiments, performance is assessed through block grasping, transfer, and stacking tasks. Multimaster experiments are performed under both dual-hand and cooperative dualoperator scenarios. A subset of this work has been presented at the IEEE 2012 Haptics Symposium [18]. The scope and experiments of the conference paper were limited to teleoperation of nonholonomic slave robots and dual-master/dual-slave teleoperation cases, predominately in a single-operator setting. In this paper, new material related to multilateral teleoperation and multioperator teleoperation experiments are provided. The remainder of this paper is organized as follows: A brief overview of the teleoperation control and experimental philosophy is presented in Section 2. Sections 3-6 overview the control method and describe experimental details/results for teleoperation tasks for the different cases depicted in Figs. 1a, 1b, and 1c. Section 7 concludes the paper.
2
CONTROL AND EXPERIMENTAL PHILOSOPHY
2.1 Control Philosophy The theory of AST control developed by Malysz [3] is applied on a case by case basis in this paper to deal with
486
IEEE TRANSACTIONS ON HAPTICS,
master/slave asymmetry and/or semiautonomous teleoperation. This theory is based on three fundamental control concepts: Position tracking for each teleoperation control frames (TCF); each master TCF is mapped to TCF on the slave system; in general, both are 6D, however potentially only a subset of the frames DOFs are linked through teleoperation, for example, translational DOF. . Force tracking with intervening virtual tool dynamics; reflection of environment contact forces, any slave TCF constraint forces, and any semiautonomous control virtual forces to the operator(s). . Integration of velocity-based autonomous control to accommodate semiautonomous teleoperation. This velocity-based control is implemented on the slave system, but its effect is felt by the operator(s) through the enforcement of the position tracking objective. The first two items can be regarded as performance/ transparency objectives for AST systems. The position tracking objective has priority over the force tracking objective; as a result any kinematic constraints enforced by position tracking affect force tracking by reflection of constraint forces. Integration of velocity-based autonomous control also does not affect position tracking, and as a result force tracking is impacted by reflection of semiautonomous virtual forces. When the velocity-based semiautonomous control overlaps with a human controlled TCF, the control framework can adjust the authority between the two. This affects the force tracking objective by altering the magnitude of the semiautonomous virtual forces. In this paper, unless otherwise stated, a one-to-one position scaling is employed. The (second) force tracking objective will be briefly reviewed for different system configurations studied in this paper. In Section 3, it is shown how kinematic constraints and deficiency impact the force tracking objective. Section 4 will describe how kinematic constraints in a multimaster system affect force tracking for both master devices. Section 5 considers semiautonomous control aspects without kinematic constraints; it will be shown there how virtual forces modify force tracking. .
2.2 Overview of Experiments and Methods In mobile twin-arm robot applications, a realistic task may involve multiple operators controlling the slave robot to maneuver the mobile base and grasp/move objects. Such object transfer task will be studied in this paper. To gain insight for the design of a control strategy for such a complex system, the subtasks of driving the mobile base and manipulating the object by the robotic arms will be examined in isolation first. In driving a mobile base, control strategies will be developed and compared for both nonholonomic (see Section 3) and holonomic (see Section 4) platforms using various AST system configurations. In the case of a holonomic base, dual-hand (see Section 4.3) and dual-operator (see Section 4.4) driving will be also examined. Object telegrasping and telemanipulation is an important aspect of our intended application. A number of control strategies involving a simplified dual-master/dual-slave
VOL. 6,
NO. 4,
OCTOBER-DECEMBER 2013
system configuration with different levels of grasping assistance within AST control framework will be developed and assessed in Section 5. The experiments in this section will focus on a block stacking task using dual-hand and cooperative dual-operator system configurations. The lessons learned from the experiments in the simplified system configurations inform the design of control strategy in a final comprehensive experiment for dual-operator teleoperation of a mobile nonholonomic twin-arm robotic manipulator in Section 6. The experiment, which will involve both driving and object grasping/ manipulation, will not revisit every aspect of the simplified system configurations but rather will combine the most effective ones. For example, control strategies that decouple tasks among the operators turn out to be more effective than those that involve some level of coupling. Therefore, in the final experiment, one operator will be assigned to drive the base and another will be in charge of controlling the arms, and dual-operator driving and dual-operator grasping configurations will not be reexamined. Also another notable difference in the final experiment is that the operators will have less ideal 2D viewing conditions via computer screens compared with a direct view of the task environment in the simplified configurations. The operator and environment interaction forces were measured by ATI Mini40 and/or Nano25 force/torque sensors. The real-time control code was implemented in Matlab RTW Simulink and Quanser Quarc 2.0, running at sampling rates of at least 1,000 Hz. The data obtained from the experiments was statistically analyzed with Fisher’s least significant difference test using a repeated measures analysis of variance (ANOVA) [19] with a significance level of ¼ 0:05 for all cases. To mitigate order bias, different experimental control modes were presented to the volunteers in random order. For each experiment set, groups of volunteers were recruited from within the Electrical and Computer Engineering Department at McMaster University with ages ranging from 21 to 39. The subjects had no visual or physical impairments and were mostly male. None of the subjects had prior experience with teleoperation. Brief practice sessions, typically 2-5 minutes, for each task were given to the volunteers to get familiar with the interfaces prior to each experiment. The following metrics have been employed for performance evaluation: task completion time (TCT), mean absolute force (MAF) (the contact forces measured at the slave end-effector(s)), . composite score index logðTCT MAF), and . number of collisions and/or block drops. Subsets of the above metrics are used in different experiments and AST configurations. The TCT metric is commonly used in human performance evaluation. The MAF measures collision avoidance performance during haptic-enabled maneuverability tasks. As a result of force reflection MAF also represents a measure of some of the forces perceived by the operator(s). A similar metric has been used in [12]. A new composite score index introduced in this paper to study and compensate for any potential user performance tradeoff in TCT and MAF. . .
MALYSZ AND SIROUSPOUR: TASK PERFORMANCE EVALUATION OF ASYMMETRIC SEMIAUTONOMOUS TELEOPERATION OF MOBILE...
487
Fig. 3. Metallic maze used for experiments with an approximate example path/lap for the origin of TCF #1 shown as a long green curved arrow. Note in the first half of the path/lap TCF #1 leads TCF #2, in the second half return portion TCF #2 leads TCF #1.
Fig. 2. Experimental setup for maneuverability tasks. Two different TCFs are considered for the object that is maneuvered through the maze. In the experiments with simulated nonholonomic slave, TCF #1 is constrained to not allow motion along the downward pointing (red) arrow.
The logarithm function transforms this metric into a form that is more statistically favorable to a normal distribution. For some stacking tasks, the number of object drops is recorded as well.
3
MANEUVERABILITY TASK OF SIMULATED NONHOLONOMIC SLAVE
Here, the subtask of teleoperation of a nonholonomic mobile base of a mobile twin-arm slave robot is examined.
3.1 Control Overview The control approach originally developed in [4] is used for single-master/single-slave teleoperation of a nonholonomic mobile robot, for example, Fig. 1a) is applicable here. In this approach, either a 2DOF or 3DOF master device controls a corresponding frame/point on the mobile base. In all modes, a master/slave position correspondence results as one of the transparency properties, i.e., kp xm ¼ xs , where xm and xs are master/slave positions and kp a diagonal scaling matrix. The haptic forces conveyed to the operator satisfy P ½fm þ kf fs ¼ P ½Mt x€m þ Bt x_ m ;
ð1Þ
Aðxs Þx_ m ¼ 0;
ð2Þ
where fm and fs are user/environment forces, kf is a diagonal scaling matrix, Mt and Bt are virtual tool mass/ damper diagonal matrices, P is either a projector or identity matrix, and Aðxs Þ is a configuration dependent matrix that represents any constraint on the slave task-space. When the slave task-space xs is chosen to be 2D and unconstrained, its corresponding Jacobian is square and full-rank resulting in P ¼ I in (1); the constraint (2) would be nonexistent in such a case. However, when the slave task-space is chosen to be 3D, then (2) represents the
nonholonomic constraint reflected to the master device. Moreover, P is a rank-deficient projector matrix and (1) states that an intervening virtual mass/damper tool response is perceived along with scaled environment forces in the unconstrained directions.
3.2 Experimental Details The two leftmost Quanser Pantograph devices in Fig. 2 are employed for this task. Operators had direct view of the task environment. The slave was made to mimic a nonholonomic mobile robot by constraining its end-effector velocities to disallow motion along the (red) downward pointing arrow in Fig. 2. This was enforced using local velocity regulation control of the nonholonomic constraint. As a result the slave only has two joint/actuator motions, i.e., rotation and forward/reverse motion. When the slave task-space was chosen at TCF #1, the following slave task-space kinematics were assumed: 2 3 2 3 x_ 1 0 cosðÞ _ x_ s ¼ 4 y_ 1 5 ¼ 4 0 sinðÞ 5 _ ¼ J s ðqs Þq_s ; ð3Þ d _ 1 0 _ T are the slave task-space velocities and where x_ s ¼ ½x_ 1 y_ 1 T _ are the actuator velocities. When TCF #2 was q_ s ¼ ½_ d used, the slave task-vector x_ s was redefined as x_ 2 b sinðÞ cosðÞ _ ð4Þ x_ s ¼ ¼ ¼ Js ðqs Þq_s ; y_ 2 b cosðÞ sinðÞ d_ where b is distance of TCF #2 from TCF #1. The task involved maneuvering an object attached to the end-effector through a metallic maze made of steel strapping, as shown in Fig. 3. By attaching the object at one end rather than in the middle, the task was made more difficult when performing both forward and reverse motion of the object as shown in Fig. 3. Two different TCFs were designated on the object as shown in Fig. 2. The subjects were instructed to complete three successive laps of the path shown in Fig. 3 in the shortest time while avoiding contact with the maze wall. The performance metrics were the TCT of the three successive laps, the MAF of the average translational slave contact force magnitude, and the composite score log ðTCT MAFÞ.
488
IEEE TRANSACTIONS ON HAPTICS,
VOL. 6,
NO. 4,
OCTOBER-DECEMBER 2013
Fig. 4. Nonholonomic slave control modes. (a) 3DOF constrained. (b) 2DOF translational. (c) 2DOF nonholonomic.
Three different control strategies were employed in this experiment to compare the proposed approach in [4] to more conventional symmetric teleoperation-based strategies. The first control mode, analogous to Fig. 4a, is denoted as 3DOF constrained (3DC); it utilized the three DOF of the master device end-effector (x; y; ) to control TCF #1, i.e., (3) was assumed. In this control mode, the operator felt the nonholonomic constraint on object motion. Force feedback was also given along the unconstrained directions. The second control strategy, shown in Fig. 4b, is denoted as 2DOF translational (2DT). In this mode, the operator used the translational DOF of the master device (x; y) to control the position of TCF #2, with the slave task vector defined in (4). The rotation of the master device endeffector was immobilized. Translational haptic feedback was also provided to the operator. The nonholonomic nature of the slave constraint can result in multiple pathdependent object configurations for a given position of the origin of TCF #2. The other control mode in Fig. 4c is denoted as 2DOF nonholonomic (2DN) with J s ¼ I. It directly mapped the rotational velocity of the master end-effector to that of the slave rotational velocity and y-axis velocity of the master _ The x-axis motion of the device to the driving velocity d. master device was not needed and immobilized. In this case, the master position along its y-axis corresponds to the distance traveled, i.e., the green path in Fig. 3. Similar to 2DT control. Multiple path-dependent object configurations for a given master (y; ) position are also possible. To provide haptic force feedback along the y-axis of the master device, the measured translational slave end-effector force was first rotated and then projected. A video describing these control modes is available online [20]. To effectively manage the master workspace, a translational position scaling of kx;y p ¼ 2 was used in all three control strategies; the rotational mapping remained one-toone. Twelve volunteers were recruited for this maneuverability experiment.
3.3 Results and Discussions The results of the experimental trials are shown in Fig. 5. The 2DN control mode had the worst performance in every category; the differences were statistically significant. This is to be expected due to the mismatch between the movements of the object and hand, complicating eye-hand coordination. The best TCT values were obtained using the 3DC control; the differences in TCT were also statistically significant. Qualitatively, the subjects stated this control type was the most intuitive. Interestingly, the lowest MAF scores came from the 2DT control mode, however the differences compared to 3DC mode were not statistically significant. It
Fig. 5. Experimental results: control of nonholonomic slave (a) taskcompletion time, (b) mean absolute force, and (c) composite Score ¼ logðTCT MAFÞ. Control types: 3DOF constrained, 2DOF translation (2DT), 2DOF nonholonomic. The error bars are the standard error in the mean (SEM) and the sample size is 12.
may be possible that constraint forces perceived in the 3DC control mode may mask contact forces with the maze wall and cause some form of haptic confusion. Using the composite score index, the 3DC and 2DT modes were significantly better than 2DN mode. This is reasonable since both the 3DC and 2DT modes were consistent in terms of eye-hand coordination. However, the difference between the two were not statically significant. A separate comparison has been made between the 3DC and 2DT modes by differentiating between the forward and reverse half-laps with results shown in Table 1. In general, the forward direction half-lap was more difficult to complete using 2DT control since the object tended to “flip” more easily. Therefore, it can be argued that the 3DC control mode yields the best worst case performance. These results indicate it can be advantageous to reflect a constrained slave task-space to the user. This can be particularly beneficial for the control of mobile manipulators and robots in cluttered environments. These experiments also give justification to explore such constrained mapping approaches and human factors experiments in other teleoperation systems involving nonholonomic constraints such as steerable needles [21].
4
MULTILATERAL TELEOPERATION MANEUVERABILITY TASK
In this section, multilateral and multioperator aspects relevant to teleoperation of mobile twin-arm slave robots are examined in the context of maneuvering a simulated holonomic mobile base.
4.1 Control Overview The methods originally developed in [4] and [5] for multimaster/single-slave teleoperation are applicable here. The TABLE 1 Nonholonomic Slave—Forward versus Reverse—log ðTCT MAFÞ
Differences in these quantities are statistically significant, the represent the standard error in the mean, and sample size is 12.
MALYSZ AND SIROUSPOUR: TASK PERFORMANCE EVALUATION OF ASYMMETRIC SEMIAUTONOMOUS TELEOPERATION OF MOBILE...
T Pn ðk1p x_ 1m ÞT ðk2p x_ 2m ÞT ¼ 0;
Fig. 6. Control modes in trilateral teleoperation experiments (a) translational-rotational mode, (b) two different TCF points are used for the vonstrained-prioritized (CP) and constrained-equal modes, and (c) single-hand mode.
objective is to control the end-effector of the slave robot by manipulating two control frames attached to it, for example, at points A and C in Fig. 1b. One of the control objectives will be to enforce position tracking between corresponding master and slave TCFs, i.e., k1p x1m ¼ x1s and k2p x2m ¼ x2s , where x1m and x2m are master positions, x1s and x2s are slave positions, and k1p and k2p are scaling matrices. The first control method uses a so called prioritized trilateral teleoperation approach [5]. In this method, a primary master with priority can freely control its corresponding slave TCF and perceives no constraint force. A second master controls the second slave TCF such that its motions would be constrained by the configuration of the primary TCF and only the secondary master perceives the kinematic constraint between x1s and x2s . Intervening virtual tool dynamics are also reflected to each master. The force tracking objectives using this control approach can be stated as f1m þ k1f f1s ¼ M1t x€1m þ B1t x_ 1m ;
ð5Þ
Pu ½f2m þ k2f f2s ¼ Pu ½M2t x€2m þ B2t x_ 2m ;
ð6Þ
þ Pn k2p x_ 2m ¼ Pn J2s J1s x_ 1s ;
ð7Þ
where f1s and f2s are the projections of the measured environment force into the corresponding slave TCFs. The expression in (5) represents unconstrained force tracking with intervening virtual tool dynamics for the primary master. At the secondary master, haptic forces obey properties (6) and (7) where Pu and Pn are projector matrices orthogonal to each other, J1s and J2s Jacobian þ a pseudoinverse; note matrices for each slave TCF, and J1s these matrices are generally configuration dependent. Expression (6) describes the motion in the unconstrained directions of the secondary master whereas the unilateral constraint (7) applies to the noncontrollable directions. In this control approach, the primary master can influence the secondary master but not vice versa. The second control approach is based on so called concatenated multilateral teleoperation concept [4]. The idea is to give each master equal priority so that kinematic constraints between x1s and x2s are perceived bilaterally. In this case, (5)-(7) are replaced with force transparency properties f þ k1f f1s M1t x€1m þ B1t x_ 1m ¼ Pu ; ð8Þ Pu 1m f2m þ k2f f2s M2t x€2m þ B2t x_ 2m
489
ð9Þ
where Pu and Pn are configuration dependent projector matrices orthogonal to each other and (9) is a bilateral constraint perceived by both masters. Considering the example in Fig. 1b, the force transparency properties in (8) and (9) create an impression of each master “holding on” to Points A and C and experiencing “push/pull” from each other.
4.2 Experimental Details The entire setup shown in Fig. 2 was used to perform a maneuverability task similar to the one described in Section 3.2. In addition to the two Quanser planar pantographs, a Quanser integrated haptic actuator (IHA) was employed as the second master device. In these experiments, all 3DOF of the slave robot and at most 2DOF of each master device were employed. Three different dual master control modes were employed for multilateral teleoperation. The first is translational-rotation (TR) mode and is depicted in Fig. 6a. Master #2 controlled the translation of slave TCF #1 and Master #1 controlled the orientation. Here, Master #2 functioned as a 2DOF haptic interface with its endeffector locked at a constant height by control. Master #1 acted as a 1DOF haptic interface with its translation being locked by control. In this scenario, the slave taskspaces were completely decoupled and nonconflicting, therefore the two control approaches described in Section 4.1 behave identically. In the constrained-prioritized (CP) control mode, Master #2 controlled the translation of TCF #1 and Master #1 was designated to control the translation of TCF #2. This mode corresponds to Fig. 6b. Both master devices functioned as 2DOF translational haptic interfaces. Here, the two slave TCFs were coupled and conflicting since there was a distance constraint between them. TCF #1 was given top priority and the prioritized teleoperation control approach from [5] described in Section 4.1 was employed. This resulted in Master #2 experiencing a circular constraint about the origin of TCF #1; Master #1 reflects no constraint forces. The last two-arm mode was similar to CP in that Fig. 6b was still applicable, and Masters #1 and #2 controlled the translation of TCF #2 and #1, respectively. Unlike the previous case, both masters were given equal priority by employing the concatenated control approach described in Section 4.1, i.e., [4]. The distance constraint was felt equally by the two masters. This control type was denoted as constrained-equal (CE). 4.3 Dual-Hand Control Here, a single operator employed the dual-arm haptic interface to maneuver the object; each master device control DOF was less than the slave DOF. Four different control types were used in this experiment, three of which used the dual-arm methods. Another control type, denoted as single hand (SH) control, i.e., Fig. 6c, used only Master #1 in a traditional SMSS teleoperation approach and served as a reference for comparison. 4.3.1 Results and Discussions Sixteen volunteers were recruited for this experiment. The results are given in Fig. 7. The SH control type control had
490
IEEE TRANSACTIONS ON HAPTICS,
VOL. 6,
NO. 4,
OCTOBER-DECEMBER 2013
Fig. 7. Experimental results: dual-hand maneuverability (a) task completion time, (b) mean absolute force, and (c) composite Score ¼ logðTCT MAFÞ. Control types: single-hand, translationrotation, constrained-prioritized, and constrained-equal. The error bars are the SEM and the sample size is 16.
Fig. 8. Experimental results: dual-operator cooperative maneuverability (a) task completion time, (b) mean absolute force, (c) composite Score ¼ logðTCT MAFÞ. Control types: translation-rotation, constrained-prioritized, constrained-equal. The error bars are the SEM and the sample size is 14.
the best performance of all the configurations. All comparisons with the SH configuration were statistically significant except for the TCT comparison with the TR mode. The dualmaster control types TCT values were 16-46 percent greater and MAF 49-95 percent greater than those of the SH configurations. These results showed that for this task there was a tradeoff between operator performance and using two potentially simpler haptic interfaces instead of a single complex interface. Depending on application, this tradeoff may be acceptable to offset costs of the master system. This may be even be more significant for operation in 6DOF space with full haptic feedback where a 6DOF haptic interface is required in a traditional SMSS system. Further human factors experiments with 6D tasks are needed to investigate this tradeoff. Although the TR control type had the best numerical scores among dual-hand interfaces, the differences with the CE mode were not statistically significant. The CP mode was the worst performer using the composite score index; these difference comparisons were significantly different. Qualitatively, the volunteers stated that all control modes were intuitive, but did express that the CP mode was the worst among them. The exact reasons for CP possibly having the highest MAF scores are unclear, but two possible explanations can be offered: 1) The constraint on Master #1 potentially complicated maneuvers for avoiding collisions. 2) The presence of both constraint and wall contact forces may have also caused haptic confusion for the operator using Master #1.
multiuser teleoperation experiments by Schauß et al. [12]. Fourteen volunteers were recruited for this experiment and were placed into seven pairs. Each operator had an opportunity to use each master device; therefore, each pair completed the three test cases twice, resulting in six runs per pair and a sample data size of 14.
4.4 Dual-Operator Cooperative Control To further investigate the multimaster control strategies, cooperative experiments with two operators were conducted using the same task. Our target teleoperation application could typically involve multiple operators and, therefore, exploring multioperator performance in this task can generate insights into the effect of tasking coupling among the operators. Only three control types were applicable in this case, those being the TR, CP, and CE configurations. The level of coupling between operators is different in each case. The volunteers were instructed not to communicate with each other verbally or otherwise during the experiment. A similar protocol was employed in the
4.4.1 Results and Discussions Experimental results are given in Fig. 8. The results indicate that the CP mode had the worst performance in all categories; the differences were statistically significant. This showed that giving priority to one operator can actually reduce the overall task performance. The TR configuration had the lowest MAF values, although the difference with CE mode MAF scores was not statistically significant. The composite index scores yielded statistically significant differences among the different configurations with the TR mode having performed the best. There are two possible explanations for this behavior. The addition of constraint forces may have masked maze wall contact forces. Moreover, choosing decoupled task-spaces potentially prevented counterproductive motions between the operators. The results of the dual-hand and dual-operator experiments show that prioritized multilateral control can impede maneuverability of a planar object. Moreover, task and TCF decoupling may also be desirable. This task decoupling concept was suggested before by Katsura et al. [22]; however, their claims had not been supported by human factors studies.
5
DUAL-MASTER/DUAL-SLAVE GRASPING TASK WITH PASSIVE ASSISTANCE
In this section, the subtask of telemanipulating the twin robotic arms is examined in the context of tasks involving grasping.
5.1 Control Overview Teleoperation of a dual-master/dual-slave system is considered here, for example, Fig. 1c. The control is based on the semiautonomous approach from [17]. In this method, position tracking remains as one of the transparency
MALYSZ AND SIROUSPOUR: TASK PERFORMANCE EVALUATION OF ASYMMETRIC SEMIAUTONOMOUS TELEOPERATION OF MOBILE...
491
Fig. 10. Different stages of the grasping task. (a) Starting position. (b) One block placed. (c) Two blocks placed. (d) Task completed.
Fig. 9. Experimental setup for grasping task with slave end-effector configurations shown. For the dual-operator experiments, the slave endeffectors were outfitted with green pencil pillows.
properties, i.e., kip xim ¼ xis for i ¼ 1; 2. Passive assistance is provided via the “soft” velocity-constraints [17]. The “soft” constraints on the slave robotic arms are formulated based on their “hard” counterparts, i.e., T T T 0 ¼ Jc ðq1s ; q2s Þ q_1s q_2s ;
ð10Þ
where q1s , q_1s , q2s , and q_2s are the slave arms joint positions/ velocities and Jc is a Jacobian matrix used in the autonomous control part. The passive semiautonomous aspect of our controller aims to minimize velocities T T T Jc ½q_1s q_2s . This has the effect of increasing the perceived impedance along the directions that oppose the constraint specified by (10). The transparency objective is expressed as
f1m þ k1f f1s f2m þ k2f f2s
x€1m ¼ ðMt þ wc Mat Þ x€2m x_ 1m ; þ ðBt þ wc Bat Þ x_ 2m
ð11Þ
where for i ¼ 1; 2, fis , fim are user/environment forces and kif is a diagonal scaling matrix. Moreover, Mt , and Bt are virtual tool mass/damper diagonal matrices and Mat and Bat are virtual mass/damper matrices arising from passive semiautonomous control. The weighting parameter wc effectively controls the relative “hardness” of the “soft” constraint.
5.2 Experimental Details The experimental platform and the arrangement shown in Fig. 9 has been used. Two Quanser pantographs were employed as the master devices. The slave robots consisted of one Sensable Phantom and one Quanser IHA. The operator(s) had direct view of the task environment. The task involved moving and stacking three cubic wood blocks from one location to another as shown in Fig. 10. A barrier
made of steel strapping was placed and forced the operator(s) to lift up the blocks high enough to be able to clear the barrier prior to block placement. All the DOF of the master/slave robots were used in the experiment. The rotational DOF of each master device end-effector was mapped to the height of their corresponding slave robot with a kp ¼ 0:21 m/rad scaling, i.e., a 180-degree counterclockwise turn of the master endeffector would increase the slave height by a 20 cm. The force/torque scaling for these axes was kf ¼ 0:03 m, i.e., a vertical 1 N slave end-effector force would generate a 0.03 Nm torque on the master device end-effector. The horizontal axes maintained a one-to-one mapping of force and motion. The operators were instructed to perform the task in minimal time without dropping a block. In the event a block was dropped, the facilitator of the experiment reset the placement of the blocks to one of the configurations in Figs. 10a, 10b, or 10c) depending on whether the first, second, or third block was being grasped. Both the TCT and the number of block drops were used as metrics. Four different control modes were used. The first mode denoted as no assistance (NA) did not utilize any semiautonomous control. This was accomplished by setting wc ¼ 0 in the teleoperation task-space weighting matrix [17]. The remaining modes employed different soft constraints by having wc attain a maximum value of one when the soft constraint was at its maximum strength. The assistance was designed to be fully active when both slave end-effectors exerted sufficient contact force on the block. The magnitude of the slave end-effector contact forces determined activation of the soft-constraint via 8 kf1s k < flo OR < 0; kf2s k < flo ðkf1s k2 flo2 Þððkf2s k2 flo2 Þ wc ¼ ; otherwise; : min 1; ðf f Þ4 hi
ð12Þ
lo
where flo ¼ 0:15 N and fhi ¼ 0:6 N are thresholds that allow a smooth and subtle transition between free motion with no assistance and grasping with assistance. Both these thresholds were below what was needed to maintain a firm grasp without the block slipping; hence, the assistance was fully active when the operator desired a firm grasp. The passive grasping assistance functions such that there is partial coupling and increased virtual tool impedance along user-defined motions/directions of the two slave robots. The design of Jc in (10) determines these userdefined directions.
492
IEEE TRANSACTIONS ON HAPTICS,
VOL. 6,
NO. 4,
OCTOBER-DECEMBER 2013
Fig. 11. Graphical equivalents of soft-constraints for grasping assistance: (a) spherical-like 1D constraint (1C), (b) cylindrical-like 2D constraint (2C), and (c) box-like 3D constraint (3C).
The second control mode, denoted as 1D soft constraint (1C) and graphically interpreted as Fig. 11a, considered the following velocity constraint: ¼Jc
zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl ffl{ h i 0 q_1s J 1s 33 y z y x z 0 ¼ x ; d d d d d d 033 J2s q_2s ð13Þ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi d ¼ x2 þ y2 þ z2 ; x ¼ x1sx x2sx ; y ¼ x1sy x2sy ; z ¼ x1sz x2sz ;
ð14Þ
where x, y, and z are differences along the three translational axes. Note that J1s 2
IEEE TRANSACTIONS ON HAPTICS,
VOL. 6,
NO. 4,
OCTOBER-DECEMBER 2013
Task Performance Evaluation of Asymmetric Semiautonomous Teleoperation of Mobile Twin-Arm Robotic Manipulators Pawel Malysz, Member, IEEE, and Shahin Sirouspour, Member, IEEE Abstract—A series of human factors experiments involving maneuvering and grasping tasks are carried out to evaluate the effectiveness of a novel asymmetric semiautonomous teleoperation (AST) control design framework for teleoperation of mobile twin-arm robotic manipulators. Simplified configurations are examined first to explore control strategies for different aspects of such teleoperation tasks. These include teleoperation of a nonholonomic mobile base, telemanipulation of a dual-arm robot, and dual-arm/dual-operator teleoperation task scenarios. In two sets of experiments with a planar nonholonomic mobile base, teleoperation via a 3DOF planar haptic interface with position mapping and force reflection of the nonholonomic constraint decreases task-completion-time (TCT) and reduces unwanted collisions. In dual-arm and dual-operator teleoperation maneuverability experiments, the assignment of decoupled and nonconflicting control frames reduces TCT and unwanted contacts. The use of so-called “soft” constraints via passive semiautonomous control reduces TCT and unwanted block drops in telegrasping experiments with a twin-arm manipulator. A final comprehensive experiment encompassing elements of the simplified configurations demonstrates the effectiveness of AST control framework in dual-operator teleoperation of a twin-arm mobile manipulator. Index Terms—Telerobotics, cooperative teleoperation, haptics, nonholonomic robot, multilateral control, human factors
Ç 1
INTRODUCTION
C
ONVENTIONAL teleoperation systems usually involve a single operator controlling a single robot where the master and slave end-effectors have the same degrees of freedom (DOF). Such system configurations have been extensively researched from the control and human factors perspectives [1], [2]. In reality, however, many practical teleoperation tasks require more complex system configurations than the standard symmetric single-master/singleslave system. The authors research group has recently introduced a new unified framework for design and control of a much broader class of haptic enabled asymmetric semiautonomous teleoperation (AST) systems [3]. In AST, the master and slave subsystems may have different DOF, multiple robots/users may coordinate in the task, and autonomous control may be mixed with human control. AST examples in Fig. 1 show a great degree of flexibility in the system design that could be exploited to enhance human-machine interface ergonomics and help operator(s) execute difficult tasks. Multiarm, kinematically redundant and mobile robots can improve operability in complex environments. Multimaster architectures not only enable multiuser cooperation, but they can also increase the
. The authors are with the Department of Electrical and Computer Engineering, McMaster University, ITB-A111, 1280 Main St. West, Hamilton, ONT L8S 4K1, Canada. E-mail: [email protected], [email protected]. Manuscript received 10 Jan. 2012; revised 28 Feb. 2013; accepted 29 Apr. 2013; published online 8 May 2013. Recommended for acceptance by S. Hirche. For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference IEEECS Log Number TH-2012-01-0003. Digital Object Identifier no. 10.1109/ToH.2013.23. 1939-1412/13/$31.00 ß 2013 IEEE
dexterity of the operator interface by employing multiarm haptic devices. Moreover, autonomous control can assist the operator(s) in difficult task scenarios. The focus of our earlier work in [3], [4], and [5] was on the development of a theoretical control framework for AST. This paper experimentally evaluates AST control strategies for a specific application, i.e., teleoperation of mobile twin-arm manipulators. This is achieved by examining the human operator(s) performance in a number of relevant elemental AST system configurations and an allencompassing task. Compared to conventional (hapticenaled) symmetric teleoperation systems, human factors aspects of asymmetric semiautonomous teleoperation systems have been relatively unstudied. One of the design goals of any teleoperation controller is to achieve high performance. This is usually addressed by providing high transparency or augmenting human capabilities. A multitude of control approaches and architectures have been developed for teleoperation to address issues ranging from time delay, robustness, asymmetries, to human perception and performance [1], [2]. The presence of an operator in the control loop in teleoperation inevitably requires human factors experiments on practical tasks to demonstrate effectiveness of controllers designed to improve user performance. A wide variety of experiments have been used for teleopereration performance evaluation. For example, a classical Fitts task involving point-to-point motion was used for early teleoperation studies experimenting with variable velocity mappings [6]. A cooperative peg-in-hole task has been employed to evaluate the effectiveness of haptic guides in bilateral teleoperation grasping [7]. Another Published by the IEEE CS, RAS, & CES
MALYSZ AND SIROUSPOUR: TASK PERFORMANCE EVALUATION OF ASYMMETRIC SEMIAUTONOMOUS TELEOPERATION OF MOBILE...
485
Fig. 1. Examples of asymmetric semiautonomous teleoperation configurations relevant to mobile twin-arm slave robots. (a) Single-master/singleslave teleoperation of a mobile kinematically deficient slave; the nonholonomic constraint on the slave TCF is reflected to the master device. (b) Multiple-master/single-slave teleoperation control of a kinematically redundant slave holding an object; each master can control a different teleoperation control frame on the slave. (c) Multiple-master/multiple-slave teleoperation with user-perceived autonomous control components; such components can for example provide grasping assistance. (d) Single-master/multiple-slave teleoperation of a twin-armed slave system where autonomous control enforces a desired formation; teleoperation is used to control the average position of the slave end-effectors.
peg-in-hole type task was performed by McLean et al. [8] to evaluate new performance metrics and to study effects of 2D, 3D, and direct viewing conditions. Goel et al. [9] and Hwang and Hannaford [10] have employed point-to-point motion and positioning tasks to evaluate redundancy resolution schemes considering joint failure and pseudoinverse inertia weighting, respectively. Cooperative motion tasks have also been performed to assess haptic skills training in teleoperation [11]. A block placement task was considered by Schauß et al. [12] in the evaluation of a coordinating controller to improve multiuser cooperation in teleoperation. Collision avoidance is another practical task that has been used for systems involving time delay [13] and mobile robot control [14]. More application-specific tasks such as those related to satellite module docking have also been used to evaluate perspective taking and camera placement [15]. Semere et al. [16] carried out experimental trials considering a blunt dissection task with partial force feedback. From the above examples, it is common to evaluate performance using metrics based on time/speed and completion of tasks such as inserting/stacking of blocks and objects. In this paper, the authors newly developed methods for trilateral teleoperation [5], [4], teleoperation of nonholonomic mobile robots [4], and semiautonomous grasping assistance in dual-slave teleoperation [17] are revisited and evaluated from an operator performance perspective. These different control approaches are special cases of a more general and systematic control strategy for hapticenabled AST systems presented in [3] that is ultimately applied for teleoperation of a mobile twin-arm robot manipulator. Given the focus of the paper, only a brief overview of the key elements and properties of the actual control framework are discussed here. The AST configurations depicted in Fig. 1 all arise in multioperator teleoperation of mobile-twin arm robots. Since mobile robot bases are typically nonholonomic, AST control strategies for such robots are developed and evaluated in this paper. These include dual-hand and dual-operator control in trilateral teleoperation system configurations. A dual-master/singleslave maneuverability task is presented that can help investigate the effect of coupling/decoupling between the two master devices in dual-operator and dual-hand settings. The aforementioned configurations can exhibit
kinematic constraints that can be haptically reflected to the operator(s); the multilateral and nonholonomic experiments in this paper help to evaluate the impact of such constraint forces on teleoperation performance. A dualmaster/dual-slave teleoperation scenario allows to independently assess the impact of semiautonomous grasping assistance for the operation of the slave manipulator arms, both in single-operator and dual-operator settings. A final combined task employing a mobile twin-arm slave is performed to validate the AST control framework under a realistic task scenario. The design of the task and AST control strategy is informed by the results of the earlier simplified AST configuration experiments. A maneuverability task is designed that involves elements of position control and collision avoidance is chosen to evaluate performance in the teleoperation control of a mobile robot and multiple-master/single-slave teleoperation. A new composite score metric is introduced to simultaneously evaluate both task speed and contact forces for these experiments. In slave dual-arm teleoperation experiments, performance is assessed through block grasping, transfer, and stacking tasks. Multimaster experiments are performed under both dual-hand and cooperative dualoperator scenarios. A subset of this work has been presented at the IEEE 2012 Haptics Symposium [18]. The scope and experiments of the conference paper were limited to teleoperation of nonholonomic slave robots and dual-master/dual-slave teleoperation cases, predominately in a single-operator setting. In this paper, new material related to multilateral teleoperation and multioperator teleoperation experiments are provided. The remainder of this paper is organized as follows: A brief overview of the teleoperation control and experimental philosophy is presented in Section 2. Sections 3-6 overview the control method and describe experimental details/results for teleoperation tasks for the different cases depicted in Figs. 1a, 1b, and 1c. Section 7 concludes the paper.
2
CONTROL AND EXPERIMENTAL PHILOSOPHY
2.1 Control Philosophy The theory of AST control developed by Malysz [3] is applied on a case by case basis in this paper to deal with
486
IEEE TRANSACTIONS ON HAPTICS,
master/slave asymmetry and/or semiautonomous teleoperation. This theory is based on three fundamental control concepts: Position tracking for each teleoperation control frames (TCF); each master TCF is mapped to TCF on the slave system; in general, both are 6D, however potentially only a subset of the frames DOFs are linked through teleoperation, for example, translational DOF. . Force tracking with intervening virtual tool dynamics; reflection of environment contact forces, any slave TCF constraint forces, and any semiautonomous control virtual forces to the operator(s). . Integration of velocity-based autonomous control to accommodate semiautonomous teleoperation. This velocity-based control is implemented on the slave system, but its effect is felt by the operator(s) through the enforcement of the position tracking objective. The first two items can be regarded as performance/ transparency objectives for AST systems. The position tracking objective has priority over the force tracking objective; as a result any kinematic constraints enforced by position tracking affect force tracking by reflection of constraint forces. Integration of velocity-based autonomous control also does not affect position tracking, and as a result force tracking is impacted by reflection of semiautonomous virtual forces. When the velocity-based semiautonomous control overlaps with a human controlled TCF, the control framework can adjust the authority between the two. This affects the force tracking objective by altering the magnitude of the semiautonomous virtual forces. In this paper, unless otherwise stated, a one-to-one position scaling is employed. The (second) force tracking objective will be briefly reviewed for different system configurations studied in this paper. In Section 3, it is shown how kinematic constraints and deficiency impact the force tracking objective. Section 4 will describe how kinematic constraints in a multimaster system affect force tracking for both master devices. Section 5 considers semiautonomous control aspects without kinematic constraints; it will be shown there how virtual forces modify force tracking. .
2.2 Overview of Experiments and Methods In mobile twin-arm robot applications, a realistic task may involve multiple operators controlling the slave robot to maneuver the mobile base and grasp/move objects. Such object transfer task will be studied in this paper. To gain insight for the design of a control strategy for such a complex system, the subtasks of driving the mobile base and manipulating the object by the robotic arms will be examined in isolation first. In driving a mobile base, control strategies will be developed and compared for both nonholonomic (see Section 3) and holonomic (see Section 4) platforms using various AST system configurations. In the case of a holonomic base, dual-hand (see Section 4.3) and dual-operator (see Section 4.4) driving will be also examined. Object telegrasping and telemanipulation is an important aspect of our intended application. A number of control strategies involving a simplified dual-master/dual-slave
VOL. 6,
NO. 4,
OCTOBER-DECEMBER 2013
system configuration with different levels of grasping assistance within AST control framework will be developed and assessed in Section 5. The experiments in this section will focus on a block stacking task using dual-hand and cooperative dual-operator system configurations. The lessons learned from the experiments in the simplified system configurations inform the design of control strategy in a final comprehensive experiment for dual-operator teleoperation of a mobile nonholonomic twin-arm robotic manipulator in Section 6. The experiment, which will involve both driving and object grasping/ manipulation, will not revisit every aspect of the simplified system configurations but rather will combine the most effective ones. For example, control strategies that decouple tasks among the operators turn out to be more effective than those that involve some level of coupling. Therefore, in the final experiment, one operator will be assigned to drive the base and another will be in charge of controlling the arms, and dual-operator driving and dual-operator grasping configurations will not be reexamined. Also another notable difference in the final experiment is that the operators will have less ideal 2D viewing conditions via computer screens compared with a direct view of the task environment in the simplified configurations. The operator and environment interaction forces were measured by ATI Mini40 and/or Nano25 force/torque sensors. The real-time control code was implemented in Matlab RTW Simulink and Quanser Quarc 2.0, running at sampling rates of at least 1,000 Hz. The data obtained from the experiments was statistically analyzed with Fisher’s least significant difference test using a repeated measures analysis of variance (ANOVA) [19] with a significance level of ¼ 0:05 for all cases. To mitigate order bias, different experimental control modes were presented to the volunteers in random order. For each experiment set, groups of volunteers were recruited from within the Electrical and Computer Engineering Department at McMaster University with ages ranging from 21 to 39. The subjects had no visual or physical impairments and were mostly male. None of the subjects had prior experience with teleoperation. Brief practice sessions, typically 2-5 minutes, for each task were given to the volunteers to get familiar with the interfaces prior to each experiment. The following metrics have been employed for performance evaluation: task completion time (TCT), mean absolute force (MAF) (the contact forces measured at the slave end-effector(s)), . composite score index logðTCT MAF), and . number of collisions and/or block drops. Subsets of the above metrics are used in different experiments and AST configurations. The TCT metric is commonly used in human performance evaluation. The MAF measures collision avoidance performance during haptic-enabled maneuverability tasks. As a result of force reflection MAF also represents a measure of some of the forces perceived by the operator(s). A similar metric has been used in [12]. A new composite score index introduced in this paper to study and compensate for any potential user performance tradeoff in TCT and MAF. . .
MALYSZ AND SIROUSPOUR: TASK PERFORMANCE EVALUATION OF ASYMMETRIC SEMIAUTONOMOUS TELEOPERATION OF MOBILE...
487
Fig. 3. Metallic maze used for experiments with an approximate example path/lap for the origin of TCF #1 shown as a long green curved arrow. Note in the first half of the path/lap TCF #1 leads TCF #2, in the second half return portion TCF #2 leads TCF #1.
Fig. 2. Experimental setup for maneuverability tasks. Two different TCFs are considered for the object that is maneuvered through the maze. In the experiments with simulated nonholonomic slave, TCF #1 is constrained to not allow motion along the downward pointing (red) arrow.
The logarithm function transforms this metric into a form that is more statistically favorable to a normal distribution. For some stacking tasks, the number of object drops is recorded as well.
3
MANEUVERABILITY TASK OF SIMULATED NONHOLONOMIC SLAVE
Here, the subtask of teleoperation of a nonholonomic mobile base of a mobile twin-arm slave robot is examined.
3.1 Control Overview The control approach originally developed in [4] is used for single-master/single-slave teleoperation of a nonholonomic mobile robot, for example, Fig. 1a) is applicable here. In this approach, either a 2DOF or 3DOF master device controls a corresponding frame/point on the mobile base. In all modes, a master/slave position correspondence results as one of the transparency properties, i.e., kp xm ¼ xs , where xm and xs are master/slave positions and kp a diagonal scaling matrix. The haptic forces conveyed to the operator satisfy P ½fm þ kf fs ¼ P ½Mt x€m þ Bt x_ m ;
ð1Þ
Aðxs Þx_ m ¼ 0;
ð2Þ
where fm and fs are user/environment forces, kf is a diagonal scaling matrix, Mt and Bt are virtual tool mass/ damper diagonal matrices, P is either a projector or identity matrix, and Aðxs Þ is a configuration dependent matrix that represents any constraint on the slave task-space. When the slave task-space xs is chosen to be 2D and unconstrained, its corresponding Jacobian is square and full-rank resulting in P ¼ I in (1); the constraint (2) would be nonexistent in such a case. However, when the slave task-space is chosen to be 3D, then (2) represents the
nonholonomic constraint reflected to the master device. Moreover, P is a rank-deficient projector matrix and (1) states that an intervening virtual mass/damper tool response is perceived along with scaled environment forces in the unconstrained directions.
3.2 Experimental Details The two leftmost Quanser Pantograph devices in Fig. 2 are employed for this task. Operators had direct view of the task environment. The slave was made to mimic a nonholonomic mobile robot by constraining its end-effector velocities to disallow motion along the (red) downward pointing arrow in Fig. 2. This was enforced using local velocity regulation control of the nonholonomic constraint. As a result the slave only has two joint/actuator motions, i.e., rotation and forward/reverse motion. When the slave task-space was chosen at TCF #1, the following slave task-space kinematics were assumed: 2 3 2 3 x_ 1 0 cosðÞ _ x_ s ¼ 4 y_ 1 5 ¼ 4 0 sinðÞ 5 _ ¼ J s ðqs Þq_s ; ð3Þ d _ 1 0 _ T are the slave task-space velocities and where x_ s ¼ ½x_ 1 y_ 1 T _ are the actuator velocities. When TCF #2 was q_ s ¼ ½_ d used, the slave task-vector x_ s was redefined as x_ 2 b sinðÞ cosðÞ _ ð4Þ x_ s ¼ ¼ ¼ Js ðqs Þq_s ; y_ 2 b cosðÞ sinðÞ d_ where b is distance of TCF #2 from TCF #1. The task involved maneuvering an object attached to the end-effector through a metallic maze made of steel strapping, as shown in Fig. 3. By attaching the object at one end rather than in the middle, the task was made more difficult when performing both forward and reverse motion of the object as shown in Fig. 3. Two different TCFs were designated on the object as shown in Fig. 2. The subjects were instructed to complete three successive laps of the path shown in Fig. 3 in the shortest time while avoiding contact with the maze wall. The performance metrics were the TCT of the three successive laps, the MAF of the average translational slave contact force magnitude, and the composite score log ðTCT MAFÞ.
488
IEEE TRANSACTIONS ON HAPTICS,
VOL. 6,
NO. 4,
OCTOBER-DECEMBER 2013
Fig. 4. Nonholonomic slave control modes. (a) 3DOF constrained. (b) 2DOF translational. (c) 2DOF nonholonomic.
Three different control strategies were employed in this experiment to compare the proposed approach in [4] to more conventional symmetric teleoperation-based strategies. The first control mode, analogous to Fig. 4a, is denoted as 3DOF constrained (3DC); it utilized the three DOF of the master device end-effector (x; y; ) to control TCF #1, i.e., (3) was assumed. In this control mode, the operator felt the nonholonomic constraint on object motion. Force feedback was also given along the unconstrained directions. The second control strategy, shown in Fig. 4b, is denoted as 2DOF translational (2DT). In this mode, the operator used the translational DOF of the master device (x; y) to control the position of TCF #2, with the slave task vector defined in (4). The rotation of the master device endeffector was immobilized. Translational haptic feedback was also provided to the operator. The nonholonomic nature of the slave constraint can result in multiple pathdependent object configurations for a given position of the origin of TCF #2. The other control mode in Fig. 4c is denoted as 2DOF nonholonomic (2DN) with J s ¼ I. It directly mapped the rotational velocity of the master end-effector to that of the slave rotational velocity and y-axis velocity of the master _ The x-axis motion of the device to the driving velocity d. master device was not needed and immobilized. In this case, the master position along its y-axis corresponds to the distance traveled, i.e., the green path in Fig. 3. Similar to 2DT control. Multiple path-dependent object configurations for a given master (y; ) position are also possible. To provide haptic force feedback along the y-axis of the master device, the measured translational slave end-effector force was first rotated and then projected. A video describing these control modes is available online [20]. To effectively manage the master workspace, a translational position scaling of kx;y p ¼ 2 was used in all three control strategies; the rotational mapping remained one-toone. Twelve volunteers were recruited for this maneuverability experiment.
3.3 Results and Discussions The results of the experimental trials are shown in Fig. 5. The 2DN control mode had the worst performance in every category; the differences were statistically significant. This is to be expected due to the mismatch between the movements of the object and hand, complicating eye-hand coordination. The best TCT values were obtained using the 3DC control; the differences in TCT were also statistically significant. Qualitatively, the subjects stated this control type was the most intuitive. Interestingly, the lowest MAF scores came from the 2DT control mode, however the differences compared to 3DC mode were not statistically significant. It
Fig. 5. Experimental results: control of nonholonomic slave (a) taskcompletion time, (b) mean absolute force, and (c) composite Score ¼ logðTCT MAFÞ. Control types: 3DOF constrained, 2DOF translation (2DT), 2DOF nonholonomic. The error bars are the standard error in the mean (SEM) and the sample size is 12.
may be possible that constraint forces perceived in the 3DC control mode may mask contact forces with the maze wall and cause some form of haptic confusion. Using the composite score index, the 3DC and 2DT modes were significantly better than 2DN mode. This is reasonable since both the 3DC and 2DT modes were consistent in terms of eye-hand coordination. However, the difference between the two were not statically significant. A separate comparison has been made between the 3DC and 2DT modes by differentiating between the forward and reverse half-laps with results shown in Table 1. In general, the forward direction half-lap was more difficult to complete using 2DT control since the object tended to “flip” more easily. Therefore, it can be argued that the 3DC control mode yields the best worst case performance. These results indicate it can be advantageous to reflect a constrained slave task-space to the user. This can be particularly beneficial for the control of mobile manipulators and robots in cluttered environments. These experiments also give justification to explore such constrained mapping approaches and human factors experiments in other teleoperation systems involving nonholonomic constraints such as steerable needles [21].
4
MULTILATERAL TELEOPERATION MANEUVERABILITY TASK
In this section, multilateral and multioperator aspects relevant to teleoperation of mobile twin-arm slave robots are examined in the context of maneuvering a simulated holonomic mobile base.
4.1 Control Overview The methods originally developed in [4] and [5] for multimaster/single-slave teleoperation are applicable here. The TABLE 1 Nonholonomic Slave—Forward versus Reverse—log ðTCT MAFÞ
Differences in these quantities are statistically significant, the represent the standard error in the mean, and sample size is 12.
MALYSZ AND SIROUSPOUR: TASK PERFORMANCE EVALUATION OF ASYMMETRIC SEMIAUTONOMOUS TELEOPERATION OF MOBILE...
T Pn ðk1p x_ 1m ÞT ðk2p x_ 2m ÞT ¼ 0;
Fig. 6. Control modes in trilateral teleoperation experiments (a) translational-rotational mode, (b) two different TCF points are used for the vonstrained-prioritized (CP) and constrained-equal modes, and (c) single-hand mode.
objective is to control the end-effector of the slave robot by manipulating two control frames attached to it, for example, at points A and C in Fig. 1b. One of the control objectives will be to enforce position tracking between corresponding master and slave TCFs, i.e., k1p x1m ¼ x1s and k2p x2m ¼ x2s , where x1m and x2m are master positions, x1s and x2s are slave positions, and k1p and k2p are scaling matrices. The first control method uses a so called prioritized trilateral teleoperation approach [5]. In this method, a primary master with priority can freely control its corresponding slave TCF and perceives no constraint force. A second master controls the second slave TCF such that its motions would be constrained by the configuration of the primary TCF and only the secondary master perceives the kinematic constraint between x1s and x2s . Intervening virtual tool dynamics are also reflected to each master. The force tracking objectives using this control approach can be stated as f1m þ k1f f1s ¼ M1t x€1m þ B1t x_ 1m ;
ð5Þ
Pu ½f2m þ k2f f2s ¼ Pu ½M2t x€2m þ B2t x_ 2m ;
ð6Þ
þ Pn k2p x_ 2m ¼ Pn J2s J1s x_ 1s ;
ð7Þ
where f1s and f2s are the projections of the measured environment force into the corresponding slave TCFs. The expression in (5) represents unconstrained force tracking with intervening virtual tool dynamics for the primary master. At the secondary master, haptic forces obey properties (6) and (7) where Pu and Pn are projector matrices orthogonal to each other, J1s and J2s Jacobian þ a pseudoinverse; note matrices for each slave TCF, and J1s these matrices are generally configuration dependent. Expression (6) describes the motion in the unconstrained directions of the secondary master whereas the unilateral constraint (7) applies to the noncontrollable directions. In this control approach, the primary master can influence the secondary master but not vice versa. The second control approach is based on so called concatenated multilateral teleoperation concept [4]. The idea is to give each master equal priority so that kinematic constraints between x1s and x2s are perceived bilaterally. In this case, (5)-(7) are replaced with force transparency properties f þ k1f f1s M1t x€1m þ B1t x_ 1m ¼ Pu ; ð8Þ Pu 1m f2m þ k2f f2s M2t x€2m þ B2t x_ 2m
489
ð9Þ
where Pu and Pn are configuration dependent projector matrices orthogonal to each other and (9) is a bilateral constraint perceived by both masters. Considering the example in Fig. 1b, the force transparency properties in (8) and (9) create an impression of each master “holding on” to Points A and C and experiencing “push/pull” from each other.
4.2 Experimental Details The entire setup shown in Fig. 2 was used to perform a maneuverability task similar to the one described in Section 3.2. In addition to the two Quanser planar pantographs, a Quanser integrated haptic actuator (IHA) was employed as the second master device. In these experiments, all 3DOF of the slave robot and at most 2DOF of each master device were employed. Three different dual master control modes were employed for multilateral teleoperation. The first is translational-rotation (TR) mode and is depicted in Fig. 6a. Master #2 controlled the translation of slave TCF #1 and Master #1 controlled the orientation. Here, Master #2 functioned as a 2DOF haptic interface with its endeffector locked at a constant height by control. Master #1 acted as a 1DOF haptic interface with its translation being locked by control. In this scenario, the slave taskspaces were completely decoupled and nonconflicting, therefore the two control approaches described in Section 4.1 behave identically. In the constrained-prioritized (CP) control mode, Master #2 controlled the translation of TCF #1 and Master #1 was designated to control the translation of TCF #2. This mode corresponds to Fig. 6b. Both master devices functioned as 2DOF translational haptic interfaces. Here, the two slave TCFs were coupled and conflicting since there was a distance constraint between them. TCF #1 was given top priority and the prioritized teleoperation control approach from [5] described in Section 4.1 was employed. This resulted in Master #2 experiencing a circular constraint about the origin of TCF #1; Master #1 reflects no constraint forces. The last two-arm mode was similar to CP in that Fig. 6b was still applicable, and Masters #1 and #2 controlled the translation of TCF #2 and #1, respectively. Unlike the previous case, both masters were given equal priority by employing the concatenated control approach described in Section 4.1, i.e., [4]. The distance constraint was felt equally by the two masters. This control type was denoted as constrained-equal (CE). 4.3 Dual-Hand Control Here, a single operator employed the dual-arm haptic interface to maneuver the object; each master device control DOF was less than the slave DOF. Four different control types were used in this experiment, three of which used the dual-arm methods. Another control type, denoted as single hand (SH) control, i.e., Fig. 6c, used only Master #1 in a traditional SMSS teleoperation approach and served as a reference for comparison. 4.3.1 Results and Discussions Sixteen volunteers were recruited for this experiment. The results are given in Fig. 7. The SH control type control had
490
IEEE TRANSACTIONS ON HAPTICS,
VOL. 6,
NO. 4,
OCTOBER-DECEMBER 2013
Fig. 7. Experimental results: dual-hand maneuverability (a) task completion time, (b) mean absolute force, and (c) composite Score ¼ logðTCT MAFÞ. Control types: single-hand, translationrotation, constrained-prioritized, and constrained-equal. The error bars are the SEM and the sample size is 16.
Fig. 8. Experimental results: dual-operator cooperative maneuverability (a) task completion time, (b) mean absolute force, (c) composite Score ¼ logðTCT MAFÞ. Control types: translation-rotation, constrained-prioritized, constrained-equal. The error bars are the SEM and the sample size is 14.
the best performance of all the configurations. All comparisons with the SH configuration were statistically significant except for the TCT comparison with the TR mode. The dualmaster control types TCT values were 16-46 percent greater and MAF 49-95 percent greater than those of the SH configurations. These results showed that for this task there was a tradeoff between operator performance and using two potentially simpler haptic interfaces instead of a single complex interface. Depending on application, this tradeoff may be acceptable to offset costs of the master system. This may be even be more significant for operation in 6DOF space with full haptic feedback where a 6DOF haptic interface is required in a traditional SMSS system. Further human factors experiments with 6D tasks are needed to investigate this tradeoff. Although the TR control type had the best numerical scores among dual-hand interfaces, the differences with the CE mode were not statistically significant. The CP mode was the worst performer using the composite score index; these difference comparisons were significantly different. Qualitatively, the volunteers stated that all control modes were intuitive, but did express that the CP mode was the worst among them. The exact reasons for CP possibly having the highest MAF scores are unclear, but two possible explanations can be offered: 1) The constraint on Master #1 potentially complicated maneuvers for avoiding collisions. 2) The presence of both constraint and wall contact forces may have also caused haptic confusion for the operator using Master #1.
multiuser teleoperation experiments by Schauß et al. [12]. Fourteen volunteers were recruited for this experiment and were placed into seven pairs. Each operator had an opportunity to use each master device; therefore, each pair completed the three test cases twice, resulting in six runs per pair and a sample data size of 14.
4.4 Dual-Operator Cooperative Control To further investigate the multimaster control strategies, cooperative experiments with two operators were conducted using the same task. Our target teleoperation application could typically involve multiple operators and, therefore, exploring multioperator performance in this task can generate insights into the effect of tasking coupling among the operators. Only three control types were applicable in this case, those being the TR, CP, and CE configurations. The level of coupling between operators is different in each case. The volunteers were instructed not to communicate with each other verbally or otherwise during the experiment. A similar protocol was employed in the
4.4.1 Results and Discussions Experimental results are given in Fig. 8. The results indicate that the CP mode had the worst performance in all categories; the differences were statistically significant. This showed that giving priority to one operator can actually reduce the overall task performance. The TR configuration had the lowest MAF values, although the difference with CE mode MAF scores was not statistically significant. The composite index scores yielded statistically significant differences among the different configurations with the TR mode having performed the best. There are two possible explanations for this behavior. The addition of constraint forces may have masked maze wall contact forces. Moreover, choosing decoupled task-spaces potentially prevented counterproductive motions between the operators. The results of the dual-hand and dual-operator experiments show that prioritized multilateral control can impede maneuverability of a planar object. Moreover, task and TCF decoupling may also be desirable. This task decoupling concept was suggested before by Katsura et al. [22]; however, their claims had not been supported by human factors studies.
5
DUAL-MASTER/DUAL-SLAVE GRASPING TASK WITH PASSIVE ASSISTANCE
In this section, the subtask of telemanipulating the twin robotic arms is examined in the context of tasks involving grasping.
5.1 Control Overview Teleoperation of a dual-master/dual-slave system is considered here, for example, Fig. 1c. The control is based on the semiautonomous approach from [17]. In this method, position tracking remains as one of the transparency
MALYSZ AND SIROUSPOUR: TASK PERFORMANCE EVALUATION OF ASYMMETRIC SEMIAUTONOMOUS TELEOPERATION OF MOBILE...
491
Fig. 10. Different stages of the grasping task. (a) Starting position. (b) One block placed. (c) Two blocks placed. (d) Task completed.
Fig. 9. Experimental setup for grasping task with slave end-effector configurations shown. For the dual-operator experiments, the slave endeffectors were outfitted with green pencil pillows.
properties, i.e., kip xim ¼ xis for i ¼ 1; 2. Passive assistance is provided via the “soft” velocity-constraints [17]. The “soft” constraints on the slave robotic arms are formulated based on their “hard” counterparts, i.e., T T T 0 ¼ Jc ðq1s ; q2s Þ q_1s q_2s ;
ð10Þ
where q1s , q_1s , q2s , and q_2s are the slave arms joint positions/ velocities and Jc is a Jacobian matrix used in the autonomous control part. The passive semiautonomous aspect of our controller aims to minimize velocities T T T Jc ½q_1s q_2s . This has the effect of increasing the perceived impedance along the directions that oppose the constraint specified by (10). The transparency objective is expressed as
f1m þ k1f f1s f2m þ k2f f2s
x€1m ¼ ðMt þ wc Mat Þ x€2m x_ 1m ; þ ðBt þ wc Bat Þ x_ 2m
ð11Þ
where for i ¼ 1; 2, fis , fim are user/environment forces and kif is a diagonal scaling matrix. Moreover, Mt , and Bt are virtual tool mass/damper diagonal matrices and Mat and Bat are virtual mass/damper matrices arising from passive semiautonomous control. The weighting parameter wc effectively controls the relative “hardness” of the “soft” constraint.
5.2 Experimental Details The experimental platform and the arrangement shown in Fig. 9 has been used. Two Quanser pantographs were employed as the master devices. The slave robots consisted of one Sensable Phantom and one Quanser IHA. The operator(s) had direct view of the task environment. The task involved moving and stacking three cubic wood blocks from one location to another as shown in Fig. 10. A barrier
made of steel strapping was placed and forced the operator(s) to lift up the blocks high enough to be able to clear the barrier prior to block placement. All the DOF of the master/slave robots were used in the experiment. The rotational DOF of each master device end-effector was mapped to the height of their corresponding slave robot with a kp ¼ 0:21 m/rad scaling, i.e., a 180-degree counterclockwise turn of the master endeffector would increase the slave height by a 20 cm. The force/torque scaling for these axes was kf ¼ 0:03 m, i.e., a vertical 1 N slave end-effector force would generate a 0.03 Nm torque on the master device end-effector. The horizontal axes maintained a one-to-one mapping of force and motion. The operators were instructed to perform the task in minimal time without dropping a block. In the event a block was dropped, the facilitator of the experiment reset the placement of the blocks to one of the configurations in Figs. 10a, 10b, or 10c) depending on whether the first, second, or third block was being grasped. Both the TCT and the number of block drops were used as metrics. Four different control modes were used. The first mode denoted as no assistance (NA) did not utilize any semiautonomous control. This was accomplished by setting wc ¼ 0 in the teleoperation task-space weighting matrix [17]. The remaining modes employed different soft constraints by having wc attain a maximum value of one when the soft constraint was at its maximum strength. The assistance was designed to be fully active when both slave end-effectors exerted sufficient contact force on the block. The magnitude of the slave end-effector contact forces determined activation of the soft-constraint via 8 kf1s k < flo OR < 0; kf2s k < flo ðkf1s k2 flo2 Þððkf2s k2 flo2 Þ wc ¼ ; otherwise; : min 1; ðf f Þ4 hi
ð12Þ
lo
where flo ¼ 0:15 N and fhi ¼ 0:6 N are thresholds that allow a smooth and subtle transition between free motion with no assistance and grasping with assistance. Both these thresholds were below what was needed to maintain a firm grasp without the block slipping; hence, the assistance was fully active when the operator desired a firm grasp. The passive grasping assistance functions such that there is partial coupling and increased virtual tool impedance along user-defined motions/directions of the two slave robots. The design of Jc in (10) determines these userdefined directions.
492
IEEE TRANSACTIONS ON HAPTICS,
VOL. 6,
NO. 4,
OCTOBER-DECEMBER 2013
Fig. 11. Graphical equivalents of soft-constraints for grasping assistance: (a) spherical-like 1D constraint (1C), (b) cylindrical-like 2D constraint (2C), and (c) box-like 3D constraint (3C).
The second control mode, denoted as 1D soft constraint (1C) and graphically interpreted as Fig. 11a, considered the following velocity constraint: ¼Jc
zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl ffl{ h i 0 q_1s J 1s 33 y z y x z 0 ¼ x ; d d d d d d 033 J2s q_2s ð13Þ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi d ¼ x2 þ y2 þ z2 ; x ¼ x1sx x2sx ; y ¼ x1sy x2sy ; z ¼ x1sz x2sz ;
ð14Þ
where x, y, and z are differences along the three translational axes. Note that J1s 2
