Simplified Human-Robot Interaction : Modeling and Evaluation

In this paper a novel concept of human-robot interaction (HRI) modeling is proposed. Including factors like trust in automation, situational awareness, expertise and expectations a new user experience framework is formed for industrial robots. Service Oriented Robot Operation, proposed in a previous paper, creates an abstract level in HRI and it is also included in the framework. This concept is evaluated with exhaustive tests. Results prove that significant improvement in task execution may be achieved and the new system is more usable for operators with less experience with robotics; personnel specific for small and medium enterprises (SMEs).


Introduction
Automation is present in every field of industry.Companies benefit from its reduced costs compared to human labor expenses on long term, in parallel with increased reliability.Quality is more manageable with automated systems while production cycle time may be reduced, thus increasing the overall output quality and quantity as well.Moreover, employees have the advantage of reduced risk and a focus on less monotonous work using creativity.Automation can also enable performing tasks which may be beyond human capabilities.
Robotics, as a subclass of industrial automation, is in the focus because of its modularity and multipurpose characteristics for flexible integrations.However, small and medium-sized enterprises (SMEs) have limited options to benefit from it due to the low number of personnel and the generally lower expertise in robotics.This issue may be overcome by new concepts on how an industrial manipulator may be operated.This paper investigates one possible solution: simplified user interface which displays relevant information for the operator.
Factors that contribute to the importance of flexible robot systems and user interfaces: • as a new trend, SMEs are about to invest into automation, • SMEs often lack employees with high expertise in robotics, • SMEs low output volume and high diversification requires flexible solutions, • current products of robot manufacturers are mainly designed for high volume production, • scientific analysis and design of industrial user interfaces is not a practice.
While studies (Sheridan, 1997;Scholtz, 2003;Steinfeld et al., 2006) explored human-robot interactions (HRI) already, it is rare that these results and methods are strongly connected to industrial applications.Recognizing this fact is important to understand the scientific relevance of the topic.In the next section a new framework for connections between operators and robots will be elaborated.This theory includes many different disciplines mainly from human factors studies.Based upon the framework hypotheses are formalized on the possible improvements for industrial robot user interfaces.The hypotheses were investigated experimentally: a series of usability tests with detailed survey on the user's pre-existing experience, impressions and feelings was conducted.Results show a significant improvement in user performance with the new user interface up against a conservative, on-the-market product.

Extended User Experience Framework
The description of HRI requires systematic analysis of the contributing factors.The most widely cited structure is based on the seven stages of HCI proposed by Norman and Draper (1986).Scholtz (2003) later identified fives roles in interaction for robots: • Supervisor Interaction, • Operator Interaction, • Mechanic Interaction, • Peer Interaction and • Bystander Interaction.
These five roles were extended to seven by Goodrich and Schultz (2007) adding Mentor and Information Consumer.They consider industrial robotics as a low priority application for HRI disciplines and identify the supervisor and peer as primary role.
In case of supervisor role the extent of an industrial interaction is much more restricted than described by Scholtz (2003).In case of a highly automated production line the domain of goals and intentions is narrow and the actions are limited to Start/Stop instructions and basic error handling interventions.The spreading of flexible robot cells is changing this and humans gain more active role in the operation loop.
Including the human in the loop requires special treatment for certain elements in a robot system.Instead of discrete stages (Norman and Draper, 1986) and the iteration through them one might consider a continuous flow of information processing.During the human-robot interaction the user is communicating with the robot system via control (HRi control ) and feedback (HRi feedback ) interfaces resulting in a process similar to a closed-loop feedback control (Figure 1).For security reasons certain methods are implemented that prevent the user to commit dangerous actions.This might be high level fool proofing of the control interface (e.g. it is not possible to change robot program during playback), or low level robot controller safety (e.g.opening the cell door will stop the program playback).
Ideally the feedback created from the raw operation data (artificial perception) for the human should cover all the necessary information considering her role mentioned before.The reality is that there is a secondary channel of perception (natural perception) which is independent from the techniques used for the user interface.One of main research topics in remote control of robots is focusing exactly on the lack of this channel, since in remote operation the users' capability to unconsciously take advantage of all five senses is missing.Recent studies in Cognitive Info-Communication (CogInfoCom) are addressing the integration of this secondary channel into the user interface to transfer extra information adapted the brain's capabilities.

Trust in Automation and Situational Awareness
Two parallel channels of perception raise the question of how the operator will combine the different information to make decisions.Indications on the robot state coming from the user interface are evaluated by their reliability.Comparison between indications and information acquired through other channels defines the credibility of the indicated information and ultimately this is influenced the operator's trust in automation (Hoffman et al., 2013) which determines the importance of the user interface feedback in the overall picture.
The secondary information channel is monitored either closely or superficially.Situational awareness (Endsley, 2000) plays important role in the perception and evaluation of information coming from auxiliary feedback.Although trust in automation and situational awareness are related in a certain degree (e.g. in aviation excessive trust may cause the loss of situational awareness) in this model the two are considered as separate factors which determine user's mental image of the overall events in the robot cell.
Figure 2 depicts the combination of the perceptions for the "big picture" or hybrid perception.The user's tion and situational awareness level of trust in automation determines how much she relies on the information given by the user interface while the user's situational awareness affects how much she comprehends of indicated and hidden variables on the robot cell status.

Experience and Expectations
So far two human factors were identified in the operation process of a robot cell.As it is mentioned in Section 1 for SMEs one issue with the application of robotics is the low experience of personnel with robotics.In order to be able to handle this within the frame of this model, the experience as a factor should be introduced.One theory could be that trust and situational awareness is dependent on experience however this would affect only the perception branch of the model and would not have direct impact on the decision making process.
As perception plays an important role in this framework, Flow model of Csikszentmihalyi (1997) offers the possibility of remaining in this domain.The Flow theory proposes that during an activity the user's involvement depends on the perceived challenge and ability to overcome the challenge (perceived skill).The difficulty of the task is relative to the experience of the user, even a simple problem may seem challenging with no existing experience.
The complete framework for user experience (Figure 3) is based on four factors: (1) trust in automation, (2) situational awareness, (3) existing experience and (4) user expectations.The operator's decisions are affected by the progress in task execution and her mental state.
The goal of the research presented in this paper is to improve the human-robot interaction through the user interface.The intention is to modify the traditional HRi control and the HRi feedback blocks so that the design could support the identification and provides tools to address the four factors directly.

Service Oriented Robot Operation
Service Oriented Robot Operation (SORO) is a concept of user interfaces for industrial robotics proposed by Daniel et al. (2014).It provides an approach where the connection between the robot cell and the operator is abstracted: the user interface offers a variety of services instead of the classical configuration based command inputs.On this abstract level it is simpler to address the users' needs with the help of CogInfoCom channels and icons (Csapo and Baranyi, 2012).Focusing on the human factors in the extended user experience model the authors are looking for answers and proofs for the following questions: Does a SORO based user interface facilitate the use of industrial robots?The assumption is that by applying the SORO approach users will be able to prescind from the technical details of robot controller and focus on the task described with natural language.
Does the service based approach decrease the possibility for errors?The cognitive interpretation of service parameters should provide smaller window for incorrect inputs from the user since the link between the parameters and the actual robot controller variables is pre-programmed and hidden.
Is it possible to create a user interface which is less dependent on the users' existing knowledge?Providing clear and easy to use human-robot interaction may lower the need for expertise with technology and robotics.
Does the use of cognitive icons change the users' feel of challenge?The hypothesis is that the perceived challenge is depending on the level of expertise and the difficulty of the task.To change this it is necessary to increase the experience level of the user by education, or to formalize the task differently which suits the user's existing knowledge.Does the use of cognitive icons change the users' feel of personal skill?The hypothesis is that displaying only relevant information for the operator should decrease the confusion and increase the credibility of the artificial perceptions through the user interface.
Does the use of SORO increase the usability of industrial robots?The ultimate goal of these investigations is to change the quality of HRI with an interface concept which is proven to be easy to learn and requires less technological expertise.This may open new ways for SMEs to adapt to high-tech production appliances.
A new graphical user interface was developed to answer the preceding questions.This program runs in parallel with the original robot controller software and exchange data with it but the focus is laid on creating customized user interfaces which can be easily adapted to the services of the robot cell.
The main goal of the experiments is to verify the effectiveness of the SORO concept by comparing task execution using the original robot controller user interface and the newly developed user interface.The user tests were constructed in a way that each aspect of the extended user experience model is included.Practical operation of a robot cell evaluates the input and output channel from the operator to the abstract layer of SORO (See Figure 4).Since the model handles the output channel of the abstract layer as a feedback in a human-in-the-loop system, the measurement of the artificial perception channel's performance is crucial.Usability test is the standard tool to investigate issues and benefits for user interfaces.Other factors included in the model like trust in automation and influence of user experience were examined in the form of questionnaires.

Experiments
A laboratory experiment was set up in order to verify the improved performance of the SORO approach.A Nachi MC70 robotic manipulator was used with Nachi FD11 controller.Two services were implemented in the robot cell: pick and place operation of three work pieces and a configurable packaging service of bolts and nuts.
The test consisted of two parts, all together four tasks were executed by the participants.In one part the user had to operate the robot through the conservative, traditional user interface shipped with the robot controller (TGUI) executing two tasks, while the other part introduced the new, flexible user interface (FGUI) applying the SORO approach executing the same tasks.
The ultimate goals of the tasks were the same with both user interfaces, namely move two workpieces and pack two bolts and three nuts.The difference occurs in what the aim of individual steps was.
For the pick and place service TGUI steps were very simple: 1. select the appropriate program number from a table provided in the user manual, 2. load this program on the robot controller, 3. start the execution of this program, 4. repeat the sequence to move the second workpiece.
As FGUI hides the actual robot controller procedures from the user, the steps there were even simpler: 1. select the pick and place operation interface (Figure 5), 2. select the destination table for the first workpiece indicated by an image, 3. select the destination table for the second workpiece indicated by an image.
Since in this setup each pick and place movement was pre-programmed in separate robot programs the user had to select and keep in mind the correct program number in the case of TGUI, while in the case of FGUI images helped the identification of the correct inputs on the service's screen.
The packaging service consisted of the following steps in case of TGUI: 1. select the appropriate program number from a table provided in the user manual, 2. open the variable editor window, The FGUI sequence was: 1. select the packaging service interface (Figure 6), 2. select the required number of bolts and nuts from a drop-down list, 3. select one of the three the delivery stations represented by separate buttons.
During task execution audio and video recordings were made and the participant's interaction with the robot controller through the teach pendant was saved in terms of key presses and mouse events.All data was acquired and time stamped on a single computer.
Four questionnaires were filled out during a test session.The pre-test survey gathered census data as age, gender, education and investigated the participant's expertise in mechanical, computer, and robotics technology.A short series of questions were asked about trust in automation in general (based on McBride (2010, pp. 60-62)).After each part of the experiment the participant was asked to answer interview questions (adapted from WAI Site Usability Testing Questions1 ) about the impressions on the current user interface and robot cell operation.A Flow Condition Questionnaire (FCQ, Schaffer (2013, pp. 19)) was filled out and trust in the actual system was tested.The exit poll was a comparison in numerous standard usability features (based also on WAI Site Usability Testing Questions) between TGUI and FGUI.

Pre-test survey results
A total number of 16 participants took part in the experiment, twelve male and four female age ranging from 18 to 77 years.Pilot tests were conducted before, thus the participant ID starts from number six (See Table 1).
Three participants reported no mechanical technology expertise, four rated as average and eight are on advanced level or higher (Table 1).In computer technology six of them are on the average user level, and seven have experience with programming also.In the field of robotics eleven participants reported no or moderate experience and four are on advanced or higher level.
Results of trust in automation in general are presented in Figure 7.The participants had to rate the statements according to their opinion on a scale from 1 to 5 as follows: 1 -Not at all. 2 -Slightly.3 -Moderately.4 -Very much.5 -Extremely.
Last question for the participant was to observe the robot cell and describe the possible operations in it be-

Task Execution Results
Three characteristics were chosen for quantitative data measurement: task execution time, number of performed interactions and ratio between touch screen and keyboard interactions.Task 1 and Task 3 are the pick and place operations in TGUI and FGUI respectively; Task 2 and Task 4 are the packaging services in TGUI and FGUI respectively.Task execution time is measured from the very first interaction with the teach pendant until the robot starts its movement.The necessary time for configuration depends on the experience level of the user and the capabilities of the user interface to simplify interactions.
In case of the pick and place operation task execution time is divided in two parts: Task 1.1 and Task 3.1 represent the setup time of the first workpiece movement, while Task 1.2 and Task 3.2 represent the time elapsed between the robot finishing the previous and starting the next item movement (repeated setup).For packaging service the timer is started at the first interaction and is stopped when the robot starts moving.Results for TGUI are presented in Table 3; for FGUI in Table 4, where SEM is standard error of the sample mean and SD is standard deviation of the sample.
The performed number of interactions shows the quality of the user interface and offers an insight on the possibilities for incorrect data input.The ratio between the interactions with the touch screen and the keys on the teach pendant or the robot controller may indicate the tendencies of attention division caused by the user interface.Results for TGUI and FGUI are shown in Table 5 and Table 6 respectively.Since the ratio of touch to key interactions is expressed in percentage, the unit of standard error of the mean and standard deviation is percentage points (pp).

Post-test Comparison Results
Standpoints of comparison are listed in Figure 12.Participants could choose between three options: TGUI or FGUI performs better or these perform about the same.62% of the answers rate FGUI performance better, 9% of the answers shows TGUI has its advantages while 29% of votes indicate equal opinion on the different user interfaces.The most significant difference in favor of FGUI is present in case of novice users (TGUI: 0, FGUI: 14, Same: 2 votes) and the graphical attractiveness of the user interface.In this survey TGUI seems more adequate for experienced users (TGUI: 8, FGUI: 3, Same: 5 votes).

Discussion of Results
It is difficult to measure the performance of a user interface because of the numerous uncertain parameters appearing because of the human presence in the system.Therefore one has to be cautious analyzing the data statistically and it is a good practice to always look at the results with criticism.

Task Execution Time
The overall picture (from making the comparison between Table 3 and Table 4) shows that the average task execution time and standard deviation in the sample was decreased significantly in the case of the packaging service (249 seconds for Task 2 and 105 seconds for Task 4).While this task was not complicated the shortcuts offered by FGUI reduced the setup time and made the robot cell easier to operate.
In the case of Task 1 and Task 3 the situation is different.To set up the robot controller for the first subtask (83 seconds for Task 1.1 and 80 seconds for Task 3.1) took the same amount of time for the participants in average but, due to the service oriented concept of FGUI, the repetition of similar tasks requires significantly less time (127 seconds for Task 1.2 and 12 seconds for Task 3.2) and effort.While more repetitions may speed up the use of TGUI in long-term these results suggest a faster learning phase in case of FGUI.
Since the dispersion of the data is considerable by means of standard deviation it is possible that comparing mean values between TGUI and FGUI might paint a false image.To check the validity and comparability of the values Figure 8 shows the time spent with TGUI and FGUI for each participant.It is clear that in all cases the use of FGUI required less time and that an average of 50 percent decrease in execution time is consistently present.
Moreover, statistical analysis of data was carried out.Comparison of means is possible with Student's t-test.In this case Group A was composed of individual total execution time with the use of TGUI, while Group B contains the results for FGUI.Paired-samples t-test was executed with alpha level arbitrary set to 0.05 and the null hypothesis was that the means are equal Table 7 shows the statistics details and result is paired, two-tailored t(15) = 4.825, p < 0.001.The null hypothesis is rejected, since p < alpha, resulting in the conclusion that the difference between the mean execution time between TGUI and FGUI is significant.This confirms that user interfaces using the SORO approach in terms of creating simple and clear user interfaces ease the use of industrial robots and reduce setup and operation time significantly.

Effect of Experience
The effect of experience is tested by checking if there is a statistically significant difference between the unexperienced (Group A: no expertise and moderate) and the experienced (Group B: average, advanced and developer/researcher) participants' change in performance.Total mean execution time for each level of expertise in robotics is depicted in Figure 9.
Mean comparison (Table 8) was performed with alpha level set to 0.05 in both groups.The null hypothesis is that there is no significant difference between the mean execution time for TGUI and FGUI in neither of the two groups.
A paired samples, two-tailored t-test resulted the following statistics: As a result the null hypothesis is rejected in first case and kept in second.It is shown that significant difference is present when unexperienced participants were setting up the robot cell, while the difference may not be understood statistically significant for experienced users, although the use of FGUI speeds up the operation by approximately two and a half times for both groups.
These analyses show that SORO approach and careful design of user interfaces make possible to create simplified human-robot interaction.Its advantage is that robot cell operation becomes less reliant on user pre-existing knowledge and experience.

Trust in Automation
During experiments the participants were asked three times about their trust in automation.This offers the possibility to observe the changes in trust in time domain while using the robot cell.
One participant did not answer questions on trust after the second part of the experiment thus one set of data is excluded from analysis.Mean score and standard deviation for fifteen measurements are presented in Table 9.
Paired samples two-tailored t-test with alpha set to 0.05 indicates significant difference in mean scores between pre-test and post-test results: Six participants out of fifteen rated their trust in automation higher after the second part of the experiment.No participant reported a decrease in their trust.No one of the experienced users (Group B in Section 4.2) have reported increase of trust.
This result emphasizes the importance of trust in automation for the model presented in Section 2. As users work with industrial robots their primary information source shifts towards artificial perception thus the responsibility of the user interface to support this is important.

Quality of Interactions
Examination of Table 5 and Table 6 shows that both in case of TGUI and FGUI the performed average number of interaction exceeds the required number.This indicates that most of the participants did not follow the instructions step-by-step and started to use the user interface intuitively.It is confirmed by the video recordings also.
Comparing the difference in excessive number of interactions shows that FGUI performs better.While a total of 53 superfluous inputs were performed in TGUI, only an average of 19 unnecessary interactions were carried out with TGUI.
This result indicates that SORO approach may decrease the possibility for errors however the efficiency of the interactions is not satisfying.
For evaluation of interaction quality success rate of interactions is introduced.This value expresses how much of the performed interactions during a given task were successful compared to the minimum necessary interactions.
Definition: Given n necessary ∈ Z + the minimally required cardinality of discrete events, and n perf ormed ∈ Z + the cardinality of registered discrete events.Assuming n perf ormed n necessary , success rate is defined: If the performed number of interactions is equal to the required number the effectiveness is a value of 1.As the number of required interactions increase the influence of one additional interaction decreases.This number indicates the quality of user interactions with the user interface in regards to the intended interactions.
Introducing n excessive ∈ Z + ∪ {0} the excessive cardinality of registered discrete events, effectiveness can be written: The division by n necessary may be carried out since it is non-zero by definition.As effectiveness depends not only on the excessive cardinality of registered discrete events, but on the minimally required cardinality of discrete events also, it measures the interaction quality by itself for a specific user interface design.
A closer examination of the data and recordings reveals that in case of TGUI the excessive number of interactions are due to the inefficient navigation through menus.Less complicated input methods may increase the quality and usability of this user interface on large scale.
Experimental data shows that FGUI has a more serious issue on the level of interaction.Since this concept is based on touch screen inputs the user experience depends heavily on the quality of the touch screen.Since the teach pendant was not designed originally for this user interface the touch screen performed poorly during tests.The resulting success rate of interactions of the two experiments: As the preceding numbers show in case the efficiency of FGUI than in case of TGUI.This outcome is unexpected and unwanted but has no significant effect on other results.Repeated inputs increase the task execution time and frustration of the user, thus based on the success rate investigation FGUI may experience a larger increase in performance than TGUI if the touch screen was replaced with a more suitable one.
During the development of FGUI it was understood that the touch screen is not the best for this usage but it was acceptable.Participants had hard time to use the teach pendant because they were mostly used to smartphone touch screens which provide better sensitivity.
The hypothesis which was examined through the quality and thus the effectiveness of interactions was that using the SORO approach the possibility of incorrect data input may be decreased since the user may focus on service parameters and not on technical data.With deliberate design to reduce the necessary number of interactions the window for errors is narrowing.

Flow Model Findings
During an activity not only the flow state is possible but according to Csikszentmihalyi (1997), based on the perceived challenge and skill, other mental states may occur also namely: • apathy, • boredom, • relaxation, • control, • flow, • arousal, • anxiety and • worry.Answers are spread throughout the graphs indicating that none of the user interfaces were able to induce a narrow field of emotions.The most frequent answer on both TGUI and FGUI was the feel of control.
The use of TGUI induced boredom in some participants, while after completing the tasks with FGUI some users reported relaxation and flow.This shows the trend of increasing level of perceived skill, while the level of challenge seems unchanged.
The FCQ supports this assumption with an average score of 2.81 (TGUI) and 2.75 (FGUI) on perceived challenge, and 3.19 (TGUI) and 3.56 (TGUI) on perceived skill, but statistically significant difference cannot be proven by t-test.
The tasks executed during the experiment were fairly simple thus these offered no real condition for a deep flow state.Further experiments are necessary to achieve more significant results but as an indication this survey shows a slight decrease in challenge and a slight increase in skills which orients users towards control and relaxation during robot operation.

Conclusions
An extended framework for analyzing human-robot interaction was introduced.A function block based structure of this framework is proposed and several factors are taken into consideration.These factors include trust in automation, situational awareness, experience and user expectations.The approach of Service Ori- ented Robot Operation is merged with this framework and experiments were conducted in order to evaluate the performance of a newly developed user interface for industrial robots which was designed along the SORO concept.
The experimental data proves that robot setup time can be reduced significantly with the new user interface and that unexperienced users gained the most performance.These two features are the most important for robot integration in SMEs since these companies have significantly less human resources.
The abstract task formulation and user interface design prove strong for even simple tasks and it is expected to perform better with more complicated tasks.The possibility for errors is decreased deliberately and experiments confirm the reduction of human errors.
Participants' trust in automation was increased by the experiment which strengthens the importance of it as a factor in human-robot interactions.
Usability survey shows the novel user interface's dominance for novice users, while experienced users reported that the reduced number of accessible options and information may decrease its value for programmers.Participants with less experience were pleased with the simpler user interface as it was less confusing for them.This contradiction is already addressed in this novel user interface as its flexibility lays in freely and run-time editable screens for operation and programming (Daniel et al., 2014).
Figure 1: Simple HRI model

Figure 2 :
Figure 2: HRI model extended with trust in automation and situational awareness

Figure 3 :
Figure 3: Extended user experience framework including trust in automation, situational awareness, and the Flow model

Figure 4 :
Figure 4: Service Oriented Robot Operation (SORO) concept in the extended HRI model

Figure 5 :
Figure 5: FGUI screen for pick and place service

Figure 6 :
Figure 6: FGUI screen for packaging service

Figure 9 :
Figure 9: Correlation between experience in robotics and total time spent using the user interface

Figure 10 and
Figure 10 and Figure 11 depict the flow condition of the participant based on the answer on the mental state where the possible answers were the mental states from

Figure 12 :
Figure 12: Post-test comparison of TGUI and FGUI in usability features

Table 1 :
Pre-test survey on experince Answers are summarized in Table2.Descriptions of the possible operations show that most of the participants were able to identify the pick and place service but the exact purpose of the bolts and nuts is not easy to comprehend at first glance.

Table 3 :
Execution time results of TGUI

Table 5 :
Quantitative results of interactions with TGUI

Table 7 :
Paired samples statistics

Table 8 :
Paired samples statistics for examining experience p > α → null hypothesis kept, for Group B (experienced participants).

Table 9 :
Paired samples statistics for survey on trust in automation