Evaluation of Subjective and Objective Performance Metrics for Haptically Controlled Robotic Systems

This paper studies in detail how diﬀerent evaluation methods perform when it comes to describing the performance of haptically controlled mobile manipulators. Particularly, we investigate how well subjective metrics perform compared to objective metrics. To ﬁnd the best metrics to describe the performance of a control scheme is challenging when human operators are involved; how the user perceives the performance of the controller does not necessarily correspond to the directly measurable metrics normally used in controller evaluation. It is therefore important to study whether there is any correspondence between how the user perceives the performance of a controller, and how it performs in terms of directly measurable metrics such as the time used to perform a task, number of errors, accuracy, and so on. To perform these tests we choose a system that consists of a mobile manipulator that is controlled by an operator through a haptic device. This is a good system for studying diﬀerent performance metrics as the performance can be determined by subjective metrics based on feedback from the users, and also as objective and directly measurable metrics. The system consists of a robotic arm which provides for interaction and manipulation, which is mounted on a mobile base which extends the workspace of the arm. The operator thus needs to perform both interaction and locomotion using a single haptic device. While the position of the on-board camera is determined by the base motion, the principal control objective is the motion of the manipulator arm. This calls for intelligent control allocation between the base and the manipulator arm in order to obtain intuitive control of both the camera and the arm. We implement three diﬀerent approaches to the control allocation problem, i.e., whether the vehicle or manipulator arm actuation is applied to generate the desired motion. The performance of the diﬀerent control schemes is evaluated, and our ﬁndings strongly suggest that objective metrics better describe the performance of the controller, even though there is a clear correlation between subjective and objective performance metrics.


Introduction
Teleoperation and haptic control allow operators to control remotely located objects from a safe and comfortable location.The main motivation for remotely operated robots is to relieve humans from entering hos-tile and dangerous environments and to utilize robots in areas where humans do not have access.This kind of systems poses several challenges when it comes to the evaluation of the control scheme, as it is not only the directly measurable metrics that define the performance of the controller, but also how the operator perceives the controller.
All teleoperated systems have one thing in common; they are controlled by a human operator, so how the human operator perceives the controller performance should thus be an important criterion when designing the controller.It is not straightforward, however, to find suitable metrics for this kind of subjective performance evaluations.In this paper we will use the NASA-TLX evaluation scheme to get feedback from the operators.In addition we will compare the results of the NASA-TLX with that of the objective metrics, which in this paper are i) the time used to perform a given task, ii) the number of errors/failures, and iii) the manipulability of the manipulator arm during the task.The main objective of comparing the subjective and objective metrics is to gain insight into to what extent these correlate.It is interesting to investigate whether or not the actual performance of the system is reflected in the feedback that we get from the operator.
Human factor is an important part in humanrobot interaction (HRI) (Goodrich and Schultz (2007)).NASA-TLX has been widely used to study operators performance and workload in HRI (Steinfeld et al. (2006)).Measuring human mental workload when operators perform tasks with telerobotics, has been treated in for example Kiselev and Loutfi (2012), Adams and Kaymaz-Keskinpala (2004), Kaber et al. (2000a), andStefanidis et al. (2010).Grane and Bengtsson (2005) compare how different types of interfaces perform in terms of mental workload and Rook and Hogema (2005) used NASA-TLX to evaluate the effect of human-machine interface design.A relevant work on evaluating the HRI in vehicle navigation systems has been studied in Ross and Burnett (2001), and Kaber et al. (2000b) evaluate the effects of workload in a teleoperation task.Goodrich et al. (2004) proposed behavioral entropy as a technique to estimate human workload in HRI.
In this paper we use a haptically controlled mobile manipulator to perform the evaluation tests.The mobile manipulator allows for both locomotion and interaction tasks, both of which are challenging tasks for the operator and require the controller to efficiently transmit information about the remote environment to the operator in order to perform the task.In addition to the manipulator arm the mobile robot is equipped with a camera that is fixed in one direction in the robot frame, so the operator will have a restricted amount of visual information about the environment.
Teleoperated robotic manipulators have long been an active field of research and a wide variety of controllers have been proposed.Passivity-based controllers are commonly used to control bilateral teleoperation systems with two-port network representations (Hokayem and Spong (2006), Ryu et al. (2004b), Ryu et al. (2004a)).Energy-based approaches have also been proposed to obtain stable behavior of the two systems, for example in Hannaford (1989) and Franken et al. (2011).Over the last years, however, we have seen an increased interest also in teleoperation of mobile manipulators, i.e., a robotic manipulator mounted on a mobile base.This setup has great potential because it combines two important properties: the mobility of the mobile base and the dexterity and manipulability of the manipulator arm (Park and Khatib (2006), Seraji (1998), Farkhatdinov and Ryu (2008)).
We implement three different control schemes that cannot straightforwardly be separated in terms of performance and user evaluation, and find the correlation between the different performance metrics.As the control is implemented on the slave side, the actual difference between the approaches is rather subtle from the operator's point of view, so good performance metrics are essential to be able to distinguish between them.The main objective is to investigate whether objective and/or subjective metrics can distinguish between these control schemes, to find any correlation between the metrics, and to suggest a set of metrics that in a decisive way can quantify the performance of haptically controlled robotic systems.

System Setup and Problem Formulation
The system to be studied consists of a standard bilateral teleoperation setup with a haptic device controlled by a human operator which is used to control a remotely located robot.The robot consists of a wheeled vehicle with a manipulator arm attached to it.We will attach a frame F b to the vehicle and denote the location of F b with respect to the inertial frame F 0 by the homogeneous matrix g 0b and its velocity by the body velocity twist V B 0b = g −1 0b ġ0b .The configuration of the robotic arm is given by the joint variables q ∈ R n in the normal way, and the joint velocities as q = dq dt ∈ R n .The position of the end-effector frame F e in the world frame is found as g 0e = g 0b g be (q) (From et al. (2010)).We refer to From et al. (2014) for a detailed formulation of the kinematics of vehicle-manipulator systems.

The Control Allocation Problem for Mobile Manipulators
The setup described above calls for the integration of two rather distinct operation modes: i) accurate manipulation of objects using the robotic arm in the relatively limited workspace of the manipulator; and The main challenge is therefore to obtain a control allocation between the vehicle and the manipulator in such a way that the motion of both the vehicle and the manipulator arm can be controlled intuitively using the manipulator-like haptic device.We denote this the control allocation problem for mobile manipulators.
The distribution of control forces between the manipulator and the base to obtain both manipulation and locomotion is obtained through the control allocation algorithm, i.e., how to interpret the master reference (6 DoF) as both position and velocity references and how to distribute the control forces between the vehicle and the base (3+6 DoF), i.e., the control allocation problem for vehicle-manipulator systems.
In this paper we take the master reference and generate position or velocity references for the vehicle and manipulator, and we denote this the control allocation problem because the motion is distributed between the two systems.It is important to note, however, that we assume that the low-level controllers of the vehicle and manipulators are such that these references are followed, i.e., we are only concerned with kinematic control.Once the control allocation is in place, any method for stable teleoperation can be used, such as passivity-and energy-based approaches.

Problem Formulation
The main motivation of this paper is to gain more insight into how well different metrics describe the per-formance of user controlled mechanical systems.The main objective is to understand whether or not the way the user perceives the controller is reflected in the actual behavior of the robot in terms of objective and directly measurable metrics.We study the correlation between We will compare three control schemes with similar characteristics and endeavor to determine whether the objective or subjective metrics best describe the performance of the system, and in particular if they give the same result.

Motion Control
In this section, we will briefly introduce what we refer to as the control allocation problem for vehiclemanipulator systems, i.e., how a reference trajectory is allocated between the vehicle and the arm.This problem has been studied in detail in Pham and From (2013).

Control Modes
All of the approaches presented in this paper use the notion of control modes to determine distribution of control forces.The control modes are described in brief below.

Manipulation Mode
This mode is used for fine manipulation and interaction tasks.This is normally implemented as a positionto-position or velocity-to-velocity control scheme.Because the manipulator arm is generally much more accurate than the vehicle, manipulation mode is realized through the manipulator arm only while the vehicle is fixed.Thus, as the vehicle is fixed and we only control the slave robot which is kinematically similar to the master robot, we can apply any standard control scheme for haptic teleoperation in this mode.If larger motions are desired, vehicle actuation is required and we switch to locomotion mode.
3 The main challenge is therefore to obtain a control allocation between the vehicle and the manipulator in such a way that the motion of both the vehicle and the manipulator arm can be controlled intuitively using the manipulator-like haptic device.We denote this the control allocation problem for mobile manipulators.
The distribution of control forces between the manipulator and the base to obtain both manipulation and locomotion is obtained through the control allocation algorithm, i.e., how to interpret the master reference (6 DoF) as both position and velocity references and how to distribute the control forces between the vehicle and the base (3+6 DoF), i.e., the control allocation problem for vehicle-manipulator systems.
In this paper we take the master reference and generate position or velocity references for the vehicle and manipulator, and we denote this the control allocation problem because the motion is distributed between the two systems.It is important to note, however, that we assume that the low-level controllers of the vehicle and manipulators are such that these references are followed, i.e., we are only concerned with kinematic control.Once the control allocation is in place, any method for stable teleoperation can be used, such as passivity-and energy-based approaches.

Problem Formulation
The main motivation of this paper is to gain more insight into how well different metrics describe the per-formance of user controlled mechanical systems.The main objective is to understand whether or not the way the user perceives the controller is reflected in the actual behavior of the robot in terms of objective and directly measurable metrics.We study the correlation between We will compare three control schemes with similar characteristics and endeavor to determine whether the objective or subjective metrics best describe the performance of the system, and in particular if they give the same result.

Motion Control
In this section, we will briefly introduce what we refer to as the control allocation problem for vehiclemanipulator systems, i.e., how a reference trajectory is allocated between the vehicle and the arm.This problem has been studied in detail in Pham and From (2013).

Control Modes
All of the approaches presented in this paper use the notion of control modes to determine distribution of control forces.The control modes are described in brief below.

Manipulation Mode
This mode is used for fine manipulation and interaction tasks.This is normally implemented as a positionto-position or velocity-to-velocity control scheme.Because the manipulator arm is generally much more accurate than the vehicle, manipulation mode is realized through the manipulator arm only while the vehicle is fixed.Thus, as the vehicle is fixed and we only control the slave robot which is kinematically similar to the master robot, we can apply any standard control scheme for haptic teleoperation in this mode.If larger motions are desired, vehicle actuation is required and we switch to locomotion mode.

Locomotion Mode
Whenever a large displacement of the robot is needed the vehicle needs to take care of this motion.Normally a position-to-velocity control scheme is chosen to allow for an infinitely large slave workspace.In locomotion mode the vehicle and the arm are used to obtain large displacements of the end-effector.As the master robot is to control both the vehicle and the slave arm, we have two kinematically dissimilar systems.We solve this by virtually connecting the master end effector to the slave end effector, which is our primary control objective.

Control Strategies
In this section we present in brief the three control strategies used for the experiments in this paper.We refer to Pham and From (2013) for a more detailed description of the control laws.

Strategy I -Master workspace strategy
With this strategy, the robot will automatically change between the two control modes depending on the master position.If the robot is far from the goal, the operator will move the haptic device far and fast.It is thus natural to define a limit area in the master manipulator's workspace so that whenever the master is inside this area, the robot will be controlled in manipulation mode while we switch to locomotion mode when it moves out of the area.Using the limits as defined in Figure 2 the mode is chosen corresponding to the following law: where z m and x m are the master positions in the zxplane of the haptic device and v z is the master speed in the z-axis of the master frame.z 0 , x 0 and v 0 are constant parameters that define when switching will occur.
When in locomotion mode we allow only for motion of the vehicle which is given by where k v and k φ are proportionality constants; v s and φ s are the velocity and the heading angle of the vehicle in the body frame; and d 1 and d 2 are defined by the position of the haptic device, as shown in Figure 2, i.e., the distances from the master's tip position to the limit area that is used to define the manipulation mode.
Modeling, Identification and Control

Locomotion Mode
Whenever a large displacement of the robot is needed the vehicle needs to take care of this motion.Normally a position-to-velocity control scheme is chosen to allow for an infinitely large slave workspace.In locomotion mode the vehicle and the arm are used to obtain large displacements of the end-effector.As the master robot is to control both the vehicle and the slave arm, we have two kinematically dissimilar systems.We solve this by virtually connecting the master end effector to the slave end effector, which is our primary control objective.

Control Strategies
In this section we present in brief the three control strategies used for the experiments in this paper.We refer to Pham and From (2013) for a more detailed description of the control laws.

Strategy I -Master workspace strategy
With this strategy, the robot will automatically change between the two control modes depending on the master position.If the robot is far from the goal, the operator will move the haptic device far and fast.It is thus natural to define a limit area in the master manipulator's workspace so that whenever the master is inside this area, the robot will be controlled in manipulation mode while we switch to locomotion mode when it moves out of the area.Using the limits as defined in Figure 2 the mode is chosen corresponding to the following law: where z m and x m are the master positions in the zxplane of the haptic device and v z is the master speed in the z-axis of the master frame.z 0 , x 0 and v 0 are constant parameters that define when switching will occur.
When in locomotion mode we allow only for motion of the vehicle which is given by where k v and k φ are proportionality constants; v s and φ s are the velocity and the heading angle of the vehicle in the body frame; and d 1 and d 2 are defined by the position of the haptic device, as shown in Figure 2, i.e., the distances from the master's tip position to the limit area that is used to define the manipulation mode.

Strategy II -Slave workspace strategy
Alternatively we can use the slave workspace to determine the control mode.Like in Wrock and Nokleby (2011), the system changes automatically from the manipulation mode to the locomotion mode when the slave manipulator reaches the limit of the workspace.We thus have where x s and y s are the actual slave positions in the xand y-axes of the robot frame; x sd and y sd , that are computed from actual master positions, are the desired slave manipulator position; and x l and y l are the slave limit positions in the x-and y-axes of the robot frame, respectively.The locomotion mode using this approach is similar to the master workspace strategy presented in 3.2.1.

Strategy III -Control Allocation
In this section we describe the third control scheme, first presented in Pham and From (2013), which introduces artificial forces between the end-effector and the base.
First we find the manipulator workspace W M with respect to the vehicle frame F b .We define the workspace for position control as a workspace W P , somewhat smaller than the manipulator workspace W M , as illustrated in Figure 3. Whenever the manipulator is inside this workspace, position control is applied.This is equivalent to the manipulation mode in the previous sections.
If the master manipulator is outside the workspace W P , velocity control is applied.In this case the slave manipulator remains fixed at the limit of the 4

Strategy II -Slave workspace strategy
Alternatively we can use the slave workspace to determine the control mode.Like in Wrock and Nokleby (2011), the system changes automatically from the manipulation mode to the locomotion mode when the slave manipulator reaches the limit of the workspace.We thus have where x s and y s are the actual slave positions in the xand y-axes of the robot frame; x sd and y sd , that are computed from actual master positions, are the desired slave manipulator position; and x l and y l are the slave limit positions in the x-and y-axes of the robot frame, respectively.The locomotion mode using this approach is similar to the master workspace strategy presented in 3.2.1.

Strategy III -Control Allocation
In this section we describe the third control scheme, first presented in Pham and From (2013), which introduces artificial forces between the end-effector and the base.
First we find the manipulator workspace W M with respect to the vehicle frame F b .We define the workspace for position control as a workspace W P , somewhat smaller than the manipulator workspace W M , as illustrated in Figure 3. Whenever the manipulator is inside this workspace, position control is applied.This is equivalent to the manipulation mode in the previous sections.
If the master manipulator is outside the workspace W P , velocity control is applied.In this case the slave manipulator remains fixed at the limit of the workspace, while the vehicle velocity is so that the vehicle follows the master end-effector with a mass-springdamper characteristics.
Denote by xs the position of the end effector projected into the position workspace W P , as illustrated in Figure 3. Then the slave position with respect to this projected position is given by ∆ = x s − xs and we will let the vehicle be governed by (3) The following references will give the above characteristics: • Manipulator arm reference: • Vehicle reference: This control law is to be interpreted in the following way: The desired end-effector velocity in the inertial space is given by V B 0e,d .The manipulator reference is obtained by the Adjoint map Ad g (From et al. (2014)) and subtracting the vehicle motion V B 0b,r , i.e., so it only remains to find the reference for the vehicle motion from the desired end-effector motion.The position, velocity, and acceleration of the end effector with respect to the vehicle generates a force F given by eq. ( 3) that acts on the vehicle.This force is transferred into a vehicle motion, or rather the vehicle velocity by eq. ( 5) where d b can be interpreted as the damping on the vehicle.Note that this is different from d which is the desired damping characteristics as observed from the camera when watching the end effector.Finally the motion of the vehicle is removed from the desired motion passed on to the manipulator controller.Note also that the constants in the mass-spring-damper system (3) need to be tuned to avoid saturation in the manipulator workspace.
For a wheeled robot no instantaneous motion in the direction of the y-axis is allowed, in which case the torques that act on the vehicle will take the form Control Allocation for Mobile Manipulators workspace, while the vehicle velocity is so that the vehicle follows the master end-effector with a mass-springdamper characteristics.
Denote by xs the position of the end effector projected into the position workspace W P , as illustrated in Figure 3. Then the slave position with respect to this projected position is given by Δ = x s − xs and we will let the vehicle be governed by (3) The following references will give the above characteristics: • Manipulator arm reference: • Vehicle reference: This control law is to be interpreted in the following way: The desired end-effector velocity in the inertial space is given by V B 0e,d .The manipulator reference is obtained by the Adjoint map Ad g (From et al. (2014)) and subtracting the vehicle motion V B 0b,r , i.e., so it only remains to find the reference for the vehicle motion from the desired end-effector motion.The position, velocity, and acceleration of the end effector with respect to the vehicle generates a force F given by eq. ( 3) that acts on the vehicle.This force is transferred into a vehicle motion, or rather the vehicle velocity by eq. ( 5) where d b can be interpreted as the damping on the vehicle.Note that this is different from d which is the desired damping characteristics as observed from the camera when watching the end effector.Finally the motion of the vehicle is removed from the desired motion passed on to the manipulator controller.Note also that the constants in the mass-spring-damper system (3) need to be tuned to avoid saturation in the manipulator workspace.
For a wheeled robot no instantaneous motion in the direction of the y-axis is allowed, in which case the torques that act on the vehicle will take the form

Fe,m
Velocity generated by virtual spring Workspace with position control WP

Manipulator workspace WM
Figure 3: Definition of the workspaces in which the robot is controlled in the locomotion and manipulation modes.Note that the workspace is defined for the manipulator arm with respect to the vehicle frame F b , and not the world frame F 0 .The velocity is generated by the virtual spring between the master manipulator (gray) and the slave manipulator (black).The intuitive interpretation of the virtual spring is illustrated by the spring between the master manipulator and the vehicle.

Experiments-Rationale and Methods
Several inexperienced operators were asked to control the robot to perform a simple task which required both fine manipulation and locomotion.Even though the task itself is simple, it is hard to perform because the operator only sees the remote workspace through a narrow camera window.It is further complicated by the kinematic dissimilarity of the master and the slave.Due to these difficulties, particularly for inexperienced operators, we experience a high number of failures and long execution times for most operators.It is therefore difficult to compare the performance of the different approaches.The experiments are motivated by the observation that it is hard to distinguish the performance of a control law based on the feedback from the operators, and we would like to investigate further whether this low discrepancy is due to similar performance of the approaches or because it is not captured by simply interviewing the operators.To this end, we use subjective and objective measures to see what best captures the performance of the control laws, and if the two approaches of measuring performance give the 5 Figure 3: Definition of the workspaces in which the robot is controlled in the locomotion and manipulation modes.Note that the workspace is defined for the manipulator arm with respect to the vehicle frame F b , and not the world frame F 0 .The velocity is generated by the virtual spring between the master manipulator (gray) and the slave manipulator (black).The intuitive interpretation of the virtual spring is illustrated by the spring between the master manipulator and the vehicle.

Experiments-Rationale and Methods
Several inexperienced operators were asked to control the robot to perform a simple task which required both fine manipulation and locomotion.Even though the task itself is simple, it is hard to perform because the operator only sees the remote workspace through a narrow camera window.It is further complicated by the kinematic dissimilarity of the master and the slave.Due to these difficulties, particularly for inexperienced operators, we experience a high number of failures and long execution times for most operators.It is therefore difficult to compare the performance of the different approaches.The experiments are motivated by the observation that it is hard to distinguish the performance of a control law based on the feedback from the operators, and we would like to investigate further whether this low discrepancy is due to similar performance of the approaches or because it is not captured by simply interviewing the operators.To this end, we use subjective and objective measures to see what best captures the performance of the control laws, and if the two approaches of measuring performance give the same result.
We perform a series of experiments and measure the performance using both a subjective workload assessment and measurable metric values to characterize the performance of the control laws.For the subjective evaluation we use the NASA-TLX test which gives us an overall workload score calculated from the weighted average of six subcategories.This will give us an idea of how mentally challenging the operators find the task.The objective evaluation of the task is performed based on execution time, number of failures, and the mobility of the robot arm during task execution.Our main objective is to discover discrepancies between the approaches and, if such a discrepancy exists, evaluate what is the best way to evaluate the performance of an interaction task using a mobile manipulator.

Robotic Setup
A standard 6-DoF Phantom haptic device from Sensable was used to control a mobile manipulator consisting of a Pioneer 3-AT mobile robot with a 7-DoF Cyton arm attached to it.The local computer communicates with the remotely located on-board computer via a wireless network.The time delay is minimal and not treated in this paper.The control is, however, implemented so that it is robust with respect to time delays.
The operator's view of the remote workspace is through a video image displayed on a screen only, i.e., there is no direct visual of the robot.The video is captured by a camera and transmitted to a screen.

Methods
The participants were asked to conduct a specific task which consisted in traversing a room to pick up an object and put it into a bin.We also placed several obstacles between the starting point and the destination to enforce a change of direction during the locomotion.The operators have to control the robot to cross the room and avoid all obstacles to complete the task.When they arrive at the final destination they have to pick up an object and place it into the bin, which completes the task.The task is constructed to force switching between the two control modes.
To verify the control scheme presented we let several inexperienced operators control the robot.We let the operators perform several different tasks using three different strategies: S1.Automatic changing between locomotion and manipulation mode using master workspace, Section 3.2.1;S2.Automatic changing between locomotion and manipulation mode using slave workspace, Section 3.2.2;S3.Control allocation approach, Section 3.2.3.
To avoid learning effects the sequence of the control schemes is randomized: • 1/3 of the operators perform the experiments with the sequence of the control schemes S1-S2-S3 • 1/3 of the operators perform the experiments with the sequence of the control schemes S2-S3-S1 • 1/3 of the operators perform the experiments with the sequence of the control schemes S3-S1-S2 To evaluate the performance of the operators the following metrics were used:

Subjective metrics
The following subjective metrics were used: • Interviewthe operators were asked to describe how each control law performed.
• NASA-TLXthe operators filled in the NASA Task Load Index (NASA-TLX).The NASA-TLX uses six dimensions to assess mental workload: mental demand, physical demand, temporal demand, performance, effort, and frustration (Rubio et al., 2004).After performing each task, the operators provide ratings on each of the six subscales.
The operator is also asked to rate which factors he/she considers the most important.

Objective metrics
The following objective metrics were used: • Number of failuresthe number of failures for each approach was registered.
• Execution timethe time needed to complete the task (when successful) was recorded.
• Manipulabilitythe manipulability of the robot arm during the manipulation task was recorded, i.e., for the time interval starting when the gripper closes (when the object is grasped) and until the gripper opens (when the object is dropped into the bin), and not for the first part of the experiment when only locomotion mode is used.This gives us an idea of the mobility of the arm and the distance from singularities during motion.

Experimental Results and Discussion
In this section we first present the experimental results in Section 5.1, followed by a discussion in Section 5.2.

General Feedback
All the operators were interviewed during and after the experiments which gave valuable feedback regarding their "feel" during the experiments.This is important information when we later are to evaluate the teleoperation schemes and compare them.
For the master workspace strategy, almost all operators are confused whether it is the vehicle or the arm that is controlled.The reason for this is probably that the arm (which is visible for the operator on the screen) does not follow the master, i.e., it can stop moving as the master enters the locomotion mode.The operators report that this makes it difficult to control the system.
With the slave workspace, on the other hand, the operators know exactly when the vehicle will move because the arm has to move to the limit before the vehicle can move.They therefore report that they can perform the task more easily.However, since this is a rather simple task-just to grasp an objectthey almost only use the locomotion mode.The slave workspace strategy allows for this as the manipulator arm is stretched forward during locomotion mode.The master workspace strategy, on the other hand, does not necessarily allow for this as the arm may be retracted during locomotion mode.In principle the operators have to control the robot so that the end effector passes the object and then move the arm back to grasp the object.Because the arm is at the limit of its workspace when the system moves towards the object, some operators find it difficult to position the system close enough to the object.This motivates leaving the manipulator more in the middle of its workspace during locomotion mode.
The operators report that the control allocation approach is the most intuitive and find it fairly simple once they manage to think of the task as controlling the end-effector motion.They also report that they are able to disregard the vehicle motion when performing manipulation tasks and also when the vehicle is moving slowly.This makes the operation more efficient because the switching is hidden from the operator.With this approach, the operator can easily drive the system close enough to the object to execute the task.At this position, the arm is close to the center of its workspace so that it can be controlled in the manipulation mode.This strategy thus takes advantage of the slave workspace strategy and also eliminates some of the drawbacks of the same strategy.

Quantitative Metrics
To get a more quantitative evaluation of the different approaches we measured the median execution times, number of failures, and median manipulability for each operator performing the task.We also asked the operators to fill in the NASA-TLX form.A summary of the results is shown in Table 1.The executing times of the 24 operators are shown in Figure 4 and Figure 5.We see that the control allocation is the approach that performs the best quite consistently.There are three operators that perform the operation fastest with the master workspace strategy and one user who takes the shortest time with the slave workspace strategy.Figure 5 shows the overall performance in terms of execution times and we see that the control allocation has better performance.This confirms the feedback from the operators that the third method is the most intuitive.
The number of failures for the three strategies is shown in Figure 6 and Figure 7.The highest number of failures occurs for the master strategy.This corresponds well with the operators' "feel"; they reported that they felt confused when they control the robot using this strategy because the robot can change quite suddenly between the two control modes when the master moves in or out of the limit area, which can cause failures.Also the slave strategy has a high number of fail tries.Recall that the slave manipulator is at the limit of its workspace (stretched out) when the robot moves towards the object so that it is difficult for operators to put the robot in a good position to interact with the object.The control allocation strategy has the lowest number of failures.Also this is natural as the manipulator arm is drawn towards the center of its workspace and also corresponds well with the feedback from the operators.
For the manipulability metric the control allocation maintains good manipulability during the grasping operation, as can be seen from Figure 8 and Figure 9.The control allocation maintains its high manipulability due to the virtual spring.The master workspace has a little bit better manipulability than the slave workspace.We will not put too much into this, however, as the positioning of the arm for the master workspace approach is random.The slave workspace strategy has the lowest manipulability because the slave manipulator is normally fixed at the limit of the workspace when in locomotion mode, which is the main drawback of this strategy.Also for the NASA-TLX the control allocation performs slightly better than the other approaches, as can be seen from Table 1, Figure 10 and Figure 11.Once again the control allocation strategy has the best performance with a slight advantage over the other approaches.There are some minor variations in performance for the different subcategories, for example the operators clearly feel a higher level of frustration when using the slave and master workspace strategy compared to the control allocation, while they feel more stress on temporal demand with the control allocation.

Discussion
Several different metrics for evaluating the performance of the proposed control schemes were presented.
We divide the metrics into theoretical and directly measurable performance metrics on one hand, and subjective metrics such as stress and frustration on the other.The main purpose of this paper is to evaluate whether objective or subjective performance metrics best describe the performance of a control law for teleoperation of mobile manipulators with limited visual feedback from the remote environment, and whether there is any discrepancy between the approaches.The results presented in the previous section all suggest that the control allocation performs better than the other approaches.In this sense the results are fairly consistent, even though the number of experiments performed was quite low.It is fair to conclude from this that the number of experiments performed The control allocation maintains its high manipulability due to the virtual spring.The master workspace has a little bit better manipulability than the slave workspace.We will not put too much into this, however, as the positioning of the arm for the master workspace approach is random.The slave workspace strategy has the lowest manipulability because the slave manipulator is normally fixed at the limit of the workspace when in locomotion mode, which is the main drawback of this strategy.Also for the NASA-TLX the control allocation performs slightly better than the other approaches, as can be seen from Table 1, Figure 10 and Figure 11.Once again the control allocation strategy has the best performance with a slight advantage over the other approaches.There are some minor variations in performance for the different subcategories, for example the operators clearly feel a higher level of frustration when using the slave and master workspace strategy compared to the control allocation, while they feel more stress on temporal demand with the control allocation.

Discussion
Several different metrics for evaluating the performance of the proposed control schemes were presented.
We divide the metrics into theoretical and directly measurable performance metrics on one hand, and subjective metrics such as stress and frustration on the other.The main purpose of this paper is to evaluate whether objective or subjective performance metrics best describe the performance of a control law for teleoperation of mobile manipulators with limited visual feedback from the remote environment, and whether there is any discrepancy between the approaches.The results presented in the previous section all suggest that the control allocation performs better than the other approaches.In this sense the results are fairly consistent, even though the number of experiments performed was quite low.It is fair to conclude from this that the number of experiments performed 8 The control allocation maintains its high manipulability due to the virtual spring.The master workspace has a little bit better manipulability than the slave workspace.We will not put too much into this, however, as the positioning of the arm for the master workspace approach is random.The slave workspace strategy has the lowest manipulability because the slave manipulator is normally fixed at the limit of the workspace when in locomotion mode, which is the main drawback of this strategy.
Also for the NASA-TLX the control allocation performs slightly better than the other approaches, as can be seen from Table 1, Figure 10 and Figure 11.Once again the control allocation strategy has the best performance with a slight advantage over the other approaches.There are some minor variations in performance for the different subcategories, for example the operators clearly feel a higher level of frustration when using the slave and master workspace strategy com- The control allocation maintains its high manipulability due to the virtual spring.The master workspace has a little bit better manipulability than the slave workspace.We will not put too much into this, however, as the positioning of the arm for the master workspace approach is random.The slave workspace strategy has the lowest manipulability because the slave manipulator is normally fixed at the limit of the workspace when in locomotion mode, which is the main drawback of this strategy.Also for the NASA-TLX the control allocation performs slightly better than the other approaches, as can be seen from Table 1, Figure 10 and Figure 11.Once again the control allocation strategy has the best performance with a slight advantage over the other approaches.There are some minor variations in performance for the different subcategories, for example the operators clearly feel a higher level of frustration when using the slave and master workspace strategy compared to the control allocation, while they feel more stress on temporal demand with the control allocation.

Discussion
Several different metrics for evaluating the performance of the proposed control schemes were presented.We divide the metrics into theoretical and directly measurable performance metrics on one hand, and subjective metrics such as stress and frustration on the other.The main purpose of this paper is to evaluate whether objective or subjective performance metrics best describe the performance of a control law for teleoperation of mobile manipulators with limited visual feedback from the remote environment, and whether there is any discrepancy between the approaches.The results presented in the previous section all suggest that the control allocation performs better than the other approaches.In this sense the results are fairly consistent, even though the number of experiments performed was quite low.It is fair to conclude from this that the number of experiments performed The control allocation maintains its high manipulability due to the virtual spring.The master workspace has a little bit better manipulability than the slave workspace.We will not put too much into this, however, as the positioning of the arm for the master workspace approach is random.The slave workspace strategy has the lowest manipulability because the slave manipulator is normally fixed at the limit of the workspace when in locomotion mode, which is the main drawback of this strategy.Also for the NASA-TLX the control allocation performs slightly better than the other approaches, as can be seen from Table 1, Figure 10 and Figure 11.Once again the control allocation strategy has the best performance with a slight advantage over the other approaches.There are some minor variations in performance for the different subcategories, for example the operators clearly feel a higher level of frustration when using the slave and master workspace strategy compared to the control allocation, while they feel more stress on temporal demand with the control allocation.

Discussion
Several different metrics for evaluating the performance of the proposed control schemes were presented.We divide the metrics into theoretical and directly measurable performance metrics on one hand, and subjective metrics such as stress and frustration on the other.The main purpose of this paper is to evaluate whether objective or subjective performance metrics best describe the performance of a control law for teleoperation of mobile manipulators with limited visual feedback from the remote environment, and whether there is any discrepancy between the approaches.The results presented in the previous section all suggest that the control allocation performs better than the other approaches.In this sense the results are fairly consistent, even though the number of experiments performed was quite low.It is fair to conclude from this that the number of experiments performed 8 pared to the control allocation, while they feel more stress on temporal demand with the control allocation.

Discussion
Several different metrics for evaluating the performance of the proposed control schemes were presented.
We divide the metrics into theoretical and directly measurable performance metrics on one hand, and subjective metrics such as stress and frustration on the other.The main purpose of this paper is to evaluate whether objective or subjective performance metrics best describe the performance of a control law for teleoperation of mobile manipulators with limited visual feedback from the remote environment, and whether there is any discrepancy between the approaches.The results presented in the previous section all suggest that the control allocation performs better than the other approaches.In this sense the results are fairly consistent, even though the number of experi- is sufficient to distinguish between the different control approaches.
The experiments suggest that the operator actually has a fairly good intuition when it comes to what control scheme that performs the best, which is not obvious as the operator only has limited knowledge of what happens on the slave side.In fact, we get more useful information from the interview process than the NASA-TLX.While the operator certainly has a good notion of how well he does in terms of failures, the execution times are in many cases very similar and it is probably fair to say that the operator will not give much thought to the manipulability during the operation.It is therefore not obvious that these results are mirrored in the interview process and the NASA-TLX.On the other hand, we believe that the theoretical metrics such as number of failures, execution time, and manipulability give a better measure of the actual performance; the number of failures, for example, tells us that the control allocation approach clearly outperforms the other methods, but this is not clear from the NASA-TLX test.We see that there is a clear correspondence between the general feedback and the objective metrics, but this is only partially seen from the results of the NASA-TLX test.
The preliminary results give some early predictions regarding the usefulness of the evaluation metrics presented and the discrepancy between these.More importantly it serves as a motivation to investigate this further and shows the importance of being aware of two rather different ways of measuring the performance of different teleoperation control schemes.The number of operators that performed the test in this paper is limited, but sufficient to obtain a preliminary conclusion.Although we need more experiments to get a strong statistical foundation, the results presented give us a clear indication of the performance of the metrics.

Conclusion
In this paper we have compared subjective and objective performance metrics for evaluating the perfor- is sufficient to distinguish between the different control approaches.
The experiments suggest that the operator actually has a fairly good intuition when it comes to what control scheme that performs the best, which is not obvious as the operator only has limited knowledge of what happens on the slave side.In fact, we get more useful information from the interview process than the NASA-TLX.While the operator certainly has a good notion of how well he does in terms of failures, the execution times are in many cases very similar and it is probably fair to say that the operator will not give much thought to the manipulability during the operation.It is therefore not obvious that these results are mirrored in the interview process and the NASA-TLX.On the other hand, we believe that the theoretical metrics such as number of failures, execution time, and manipulability give a better measure of the actual performance; the number of failures, for example, tells us that the control allocation approach clearly outperforms the other methods, but this is not clear from the NASA-TLX test.We see that there is a clear correspondence between the general feedback and the objective metrics, but this is only partially seen from the results of the NASA-TLX test.
The preliminary results give some early predictions regarding the usefulness of the evaluation metrics presented and the discrepancy between these.More importantly it serves as a motivation to investigate this further and shows the importance of being aware of two rather different ways of measuring the performance of different teleoperation control schemes.The number of operators that performed the test in this paper is limited, but sufficient to obtain a preliminary conclusion.Although we need more experiments to get a strong statistical foundation, the results presented give us a clear indication of the performance of the metrics.

Conclusion
In this paper we have compared subjective and objective performance metrics for evaluating the perfor-9 ments performed was quite low.It is fair to conclude from this that the number of experiments performed is sufficient to distinguish between the different control approaches.
The experiments suggest that the operator actually has a fairly good intuition when it comes to what control scheme that performs the best, which is not obvious as the operator only has limited knowledge of what happens on the slave side.In fact, we get more useful information from the interview process than the NASA-TLX.While the operator certainly has a good notion of how well he does in terms of failures, the execution times are in many cases very similar and it is probably fair to say that the operator will not give much thought to the manipulability during the operation.It is therefore not obvious that these results are mirrored in the interview process and the NASA-TLX.On the other hand, we believe that the theoretical metrics such as number of failures, execution time, Control Allocation for Mobile Manipulators is sufficient to distinguish between the different control approaches.
The experiments suggest that the operator actually has a fairly good intuition when it comes to what control scheme that performs the best, which is not obvious as the operator only has limited knowledge of what happens on the slave side.In fact, we get more useful information from the interview process than the NASA-TLX.While the operator certainly has a good notion of how well he does in terms of failures, the execution times are in many cases very similar and it is probably fair to say that the operator will not give much thought to the manipulability during the operation.It is therefore not obvious that these results are mirrored in the interview process and the NASA-TLX.On the other hand, we believe that the theoretical metrics such as number of failures, execution time, and manipulability give a better measure of the actual performance; the number of failures, for example, tells us that the control allocation approach clearly outperforms the other methods, but this is not clear from the NASA-TLX test.We see that there is a clear correspondence between the general feedback and the objective metrics, but this is only partially seen from the results of the NASA-TLX test.
The preliminary results give some early predictions regarding the usefulness of the evaluation metrics presented and the discrepancy between these.More importantly it serves as a motivation to investigate this further and shows the importance of being aware of two rather different ways of measuring the performance of different teleoperation control schemes.The number of operators that performed the test in this paper is limited, but sufficient to obtain a preliminary conclusion.Although we need more experiments to get a strong statistical foundation, the results presented give us a clear indication of the performance of the metrics.

Conclusion
In this paper we have compared subjective and objective performance metrics for evaluating the perfor-9 is sufficient to distinguish between the different control approaches.
The experiments suggest that the operator actually has a fairly good intuition when it comes to what control scheme that performs the best, which is not obvious as the operator only has limited knowledge of what happens on the slave side.In fact, we get more useful information from the interview process than the NASA-TLX.While the operator certainly has a good notion of how well he does in terms of failures, the execution times are in many cases very similar and it is probably fair to say that the operator will not give much thought to the manipulability during the operation.It is therefore not obvious that these results are mirrored in the interview process and the NASA-TLX.On the other hand, we believe that the theoretical metrics such as number of failures, execution time, and manipulability give a better measure of the actual performance; the number of failures, for example, tells us that the control allocation approach clearly outperforms the other methods, but this is not clear from the NASA-TLX test.We see that there is a clear correspondence between the general feedback and the objective metrics, but this is only partially seen from the results of the NASA-TLX test.
The preliminary results give some early predictions regarding the usefulness of the evaluation metrics presented and the discrepancy between these.More importantly it serves as a motivation to investigate this further and shows the importance of being aware of two rather different ways of measuring the performance of different teleoperation control schemes.The number of operators that performed the test in this paper is limited, but sufficient to obtain a preliminary conclusion.Although we need more experiments to get a strong statistical foundation, the results presented give us a clear indication of the performance of the metrics.

Conclusion
In this paper we have compared subjective and objective performance metrics for evaluating the perfor-9 and manipulability give a better measure of the actual performance; the number of failures, for example, tells us that the control allocation approach clearly outperforms the other methods, but this is not clear from the NASA-TLX test.We see that there is a clear correspondence between the general feedback and the objective metrics, but this is only partially seen from the results of the NASA-TLX test.
The preliminary results give some early predictions regarding the usefulness of the evaluation metrics presented and the discrepancy between these.More importantly it serves as a motivation to investigate this further and shows the importance of being aware of two rather different ways of measuring the performance of different teleoperation control schemes.The number of operators that performed the test in this paper is limited, but sufficient to obtain a preliminary conclusion.Although we need more experiments to get a strong statistical foundation, the results presented give us a clear indication of the performance of the metrics.

Conclusion
In this paper we have compared subjective and objective performance metrics for evaluating the performance of different controllers for mechanical systems that are haptically controlled by human operators.We find that even though subjective metrics based on the NASA-TLX and interviews give a fair indication of the performance of a control scheme, the objective and directly measurable evaluation methods represent better metrics of performance evaluation.In fact we find that objective metrics such as execution time, number of failures, and arm manipulability correspond better with the feedback during the interview than the NASA-TLX, which may come as a surprise, as the intention of the NASA-TLX is to capture the mental workload during operation.Even though the feedback from the operators is important to get a good understanding of how a controller performs, the results strongly suggest that objective metrics do better when it comes to evaluating the performance of different control schemes of systems controlled by a human operator.

Figure 1 :
Figure 1: The coordinates of a mobile manipulator with on-board camera

Figure 1 :
Figure 1: The coordinates of a mobile manipulator with on-board camera

Figure 2 :
Figure 2: Determining d 1 and d 2 from the haptic position.

Figure 2 :
Figure 2: Determining d 1 and d 2 from the haptic position.

Figure 5 :
Figure 4: The executing times for 24 inexperienced operators performing the three strategies

Figure 7 :
Figure 6: The number of failures for 24 inexperienced operators performing the three strategies

Figure 4 :Figure 5 :
Figure 4: The executing times for 24 inexperienced operators performing the three strategies

Figure 7 :
Figure 6: The number of failures for 24 inexperienced operators performing the three strategies

Figure 5 :
Figure 5: The median, the maximum, 75th percentile, 25th percentile and minimum values of executing times.

Figure 5 :
Figure 4: The executing times for 24 inexperienced operators performing the three strategies

Figure 7 :
Figure 6: The number of failures for 24 inexperienced operators performing the three strategies

Figure 6 :Figure 5 :
Figure 6: The number of failures for 24 inexperienced operators performing the three strategies

Figure 7 :
Figure 6: The number of failures for 24 inexperienced operators performing the three strategies

Figure 7 :
Figure 7: The median, the maximum, 75th percentile, 25th percentile and minimum values of number of failures.

Figure 9 :
Figure 8: The manipulability in three strategy of 24 inexperienced operators

Figure 11 :
Figure 10: The median, the maximum, 75th percentile, 25th percentile and minimum values of NASA TLX scores.

Figure 8 :Figure 9 :
Figure 8: The manipulability in three strategy of 24 inexperienced operators

Figure 11 :
Figure 10: The median, the maximum, 75th percentile, 25th percentile and minimum values of NASA TLX scores.

Figure 9 :
Figure 8: The manipulability in three strategy of 24 inexperienced operators

Figure 11 :
Figure 10: The median, the maximum, 75th percentile, 25th percentile and minimum values of NASA TLX scores.

Figure 11 :
Figure 10: The median, the maximum, 75th percentile, 25th percentile and minimum values of NASA TLX scores.

Figure 11 :
Figure 11: Six different categories evaluated in measuring workload