Information Sharing via Projection Function for Coexistence of Robot and Human

Yujin Wakita, Shigeoki Hirai, Takashi Suehiro, Toshio Hori

Autonomous Robots, N0.10, 2001

Summary

                  The authors introduce the concept of safety based on intelligent augmentation of robotic systems. In previous studies the authors introduced the concept of tele-robotic systems (1992,1995,1996), where a robot is operated from another position with no physical contact and monitored through a television, and intelligent monitoring (1992), a system allowing conveyance of only required information through selection of data. The expansion of this last system has been the snapshot function (1995), where a laser pointer helps in teaching mode to estimate the deviation of the position, while the operator can move the robot, teaching the estimated relative deviation. A further implementation is the here proposed projection function (2001), where a robot and human jointly operate through a Digital Desk, a special environment provided with a projector perpendicular to the working table and a speaker. The aim of this research is to achieve intelligent augmentation in order to prevent and avoid undesirable contact, information sharing is a fundamental aspect in cooperative tasks between a person and a robot (Wakita, 1998). The experiment test a human and robot operating in mainly 5 states (initial, approach, grasp, release and final), the main issue is this kind of problem to be solves are: the person does not know the delivery coordinate, the person must keep holding the object until it is released, the person might be frightened by the robot movement.

The projection function consists of projecting on the table the simulated images of the moving robot, so that the human operator knows in real time the robots trajectory and understand the delivery trajectory. Force sensors in the robot’s fingers are used in order to allow the robot understand when the object has been grasped by the operator. A new teaching method also is introduced: the operator activated the teaching mode by touching the robot’s hand, then, instead of physically moving the manipulator, the projected image of the robot follows the operator’s hand to destination, the advantage is that only the model is required and no robot movement; the robot confirm through the speakers that the teaching trajectory has been saved.

The force sensors are an efficient communication method only during grasping, visual monitoring appears to be necessary for the entire delivery task.

It can be observed that humans in cooperation require visual feedback in order to understand that their motion and activity has been understood, each person expects to be observed during their action. So visual information appears to be extremely important by means of perception and it enhance safety in the system.

The digital desks comes to help once again in monitoring and indicating robots and humans in the system, in fact while operating a symbol (in the experiment it is a white rectangle) is projected on the hand of the operator when the robot has detected an action, in this way the human is aware that the robot knows about its presence.

In order to perform the experiment, a CCD camera was used for detection of human’s hand and robot position, and a video projector (SANYO LP-SG60) mounted on the ceiling in parallel with the camera.

The system as programmed, projects a white rectangle on the human’s hand when the CCD and the computer had performed the detection, while stationary hand is recognized a the delivery position.

Key Concepts

Human-Robot Interaction, Human-Robot Cooperation, Team Working

Key Results

The experiment appears to be useful prompting the importance of communication between robots and humans working together, a communication which need also visual feedback in order to ensure safety. A big part of communication is in fact performed not only by direct communication, but also by indirect feedback, showing that the message has been properly received. Future research may require adding information to the system.

Toward a Framework for a Human-Robot Interaction

Sebastian Thrun

Human-Computer Interaction, No.19, 2004

Summary

                  The field of robotics has undergone a considerable change from the time it first appeared as a complete science, robots now perform many assembly and transportation tasks, often equipped with minimal sensing and computing, slaved to perform a repetitive task. The future is more and more seeing the introduction of service robots and this is mainly thanks to reduce in costs of many technologies required and increase in autonomy capabilities.

Robotics appears to be a broad discipline and therefore definitions of this science are not unique, a general definition has been done by the author in a previous paper (Thrun, 2002) a system of robotic sensors, actuators and algorithms. The United Nations has categorized robotics in three fields: industrial robotics, professional service robotics and personal service robotics.

Industrial robotics are the earliest commercial success; an industrial robot operates manipulating its physical environment, it is computer controlled and operates in industrial settings (for example on conveyor belts).

Industrial robotics started in the 60s with the first commercial manipulator, the Unimate, later on in the 70s Nissan Corporation automated an entire assembly line with robots, starting a real “robotic revolution”, simply it can be considered that today the ration human to worker to robots is approximately 10:1 (the automotive industry is definitely the one with biggest application of robotics). However industrial robots are not intended to operate directly with humans.

Professional service robots are the younger kind of robots and are projected to assist people, perhaps in accessible environments or in tasks where speed and precision won’t definitely be met by human operators (as it is becoming more common in surgery).

Personal service robots posses today the highest expected growth rate, they are projected to assist people in domestic tasks and for recreational activities, often these robots are humanoids.

In all three of these fields two are the main drivers: cost and safety, these appear to be the challenges of robotics.

Autonomy refers to the ability the robot has to accommodate variation in the environment, it is a very important factory in human-robot interaction. Industrial robots are not considered to be highly autonomous, they often are called for repetitive tasks and therefore can be programmed, a different scenario appears to be the on of service robots where complexity of the environment brings them to be design to be very autonomous since they have to be able to predict the environment uncertainties, to detect and accommodate people and so on.

Of course there is also a cost issue, which necessitates the personal robots to be low-cost, therefore it they are the most complicated since the need high levels of autonomy and low costs. In human robot interaction extremely important become the interface mechanism, industrial robots are often limited, in fact they hard programmed and programming language and simulation softwares appear to be intermediary between the robot and the human. Service robots of course require richer interfaces and therefore distinguished are indirect and direct interaction methods. Indirect interaction consists of a person operating a robot through a command, while direct interaction consist of a robot taking decision on its on in parallel with a human.

Different technologies exist in order to achieve different method of communication, an interesting example appears to be the Robonaus (Ambrose et al., 2001), a master-slave idea demonstrating how a robot can cooperate with astronaut on a space station. Speech synthetisers and screens also appear to be interesting direct interaction methods.

Investigating humanoids and appearance, together with social aspect of service robots are also important aspect which researched are today investigating for the future of robotics.

Key Concepts

Human Robot Interaction, Human Robot Cooperation

Human – Robot Interaction Through Gesture – Free Spoken Language

Vladimir Kulyukin

Autonomous Robots, No. 16, 2004

Summary

                  Human Robot interaction has become more and more important in relevant activities with challenging tasks, from mining to space walks, therefore tight integration of perception, language and vision are important characteristics for cooperating in such environments, GUI interface is not acceptable any longer. It is believed that everyday language is somehow grounded in visually-guided activities and deep integration must be achieved at this level. The system proposed is a speech method for commanding a robot in performing certain grasping tasks (such as a “pepsi can”). To modes of interaction are possible, local (when the robot and the operator are I each others sight) and remote (which need the use of a camera to allow the human to understand the robots environments). The proposed robot has a standard 3T architecture, there are three tiers of functionality: deliberation, execution and control. The execution ties (which receives from the deliberation tier the inputs to be satisfied) is implemented with the Reactive Action Package (RAP) System, this is a set of methods for achieving a specific goal under different circumstances. Steps needed to execute a method are task nets and the RAP may operate knowing or not where the object is, the RAP becomes a task when its index clause is matching a task description. The control tier contains the robot’s skills, these are enables when the control tier is asked to execute them and is not aware of success or failures, since these are under the execution tier.

Actions are then categorized in internal (for manipulation of robot’s memory) an external (for manipulation of external objects or moving operations), the output of a skill is a set of symbolic assertions that the skill puts in the robot’s memory. Goals, object and actions are part of a Semantic Network, where each not represents a memory organization package (MOP) and they are connected between each other with abstraction links or package links. Nodes are activated by an activation algorithm on the direct memory access parsing (DMAP) done by token sequences. For token T all expectation that present T must be advances, for those activated then the target MOPs must be activated and for each MOP, in presence of callbacks (Kulyukin and Settle, 2001), a run must be performed. Object recognition is performed through GHD (Generalized Hamming Distance), which is a medication of the classical method, being in fact capable of approximate similarity, and color histogram models (CHM). The system needs pieces of a priori knowledge, it needs the semantic network of goals and objects, the library of object models, the context-free command and control grammars for speech recognition, the library of RAP and the library for robotics skills. Interaction is based on passive rarefication of knowledge through goal disambiguation, mutual understand is necessary, denoting this to be a cognitive machinery situation. Voice recognition is performed through Microsoft Speech API (SAPI), which is a middle layer between an application and a speech recognition engine, it includes the Microsoft English SR Engine Version 5, with 6o,000 English words and capable to be provided with other languages. SAPI converts voice input into a token sequence, activating the algorithm discussed above. Goal disambiguation can appear under the form of sensory, mnemonic and linguistic. The robot is capable of asking for clarifying in ambiguous situations. A great advantage of the system is that it allows introspection, this permits operator to ask the robot what it is capable to do and therefore it make the machine easy for non expert operators and helps learning performances both of robot and human.

Key Concepts

Human – Robot Interaction

Key Results

Experiments have been performed proving the importance of introspection and showing how it is mainly used in at the first interaction and abounded with decrease of learning factor. Limitations are still to be overcome, in fact it can’t handle deictic references, quantification and negation and it is capable of interacting with one person.

Adaptive Neural Network Control of Robot Manipulators in Task Space

Shuzhi S. Ge, C.C. Huang, L.C. Woon

IEEE Transactions on Industrial Electronics, VOL.44, NO.6, December 1997

Summary

                  Flexibility is one of the main issues in a production facility and several researches have been done to enable it in control system. Computed Torque Control is an intuitive scheme which has the objective of cancelling non linear dynamics of the manipulator system, but it requires the exact dynamic system which mean that is wouldn’t be flexible enough, so this is why adaptive control methods have been searched to overcome the problem of requiring a priori knowledge. Interesting is the application of Neural Networks, they work well in many systems and if use with parameterized variables, they can be used in different environments. The problem is extended I Task Space (Cartesian Space) and controlled must be the end effector position and it’s force. In order to create the model the GL (Ge-Lee) Matrix and its operator are introduced (page 746-747).

·       Model

In the field of control engineering, neural networks are used for approximating a given non linear function f(y) up to a small error tolerance. The neural network is based on 3 layers: input layer (n nodes), hidden layer (l nodes) and output layer (m nodes). For each hidden layer a Gaussian function is defined: a_i=exp(-(y-μ_i)^T(y-y_i)/σ_i²), with μ being the center vector and σ² being the variance of the gaussian distibution. The output appears to be the Gaussian Functin coming out from the hidden layer weigheted by W. It has been proven that any continuos function can uniformly be approximated by a linear combination of Gaussians.

In modelling a robot’s manipulator the kinematics would be described in the following manner: D(q)q’’+C(q,q’)q’+G(q)=τ, where D os the symmetric positive definite inertia matrix, C is the Coriolis matrix, G is the vector for gravitational forces and τ is the joint torque vector. D(q) and G(q) are function of only q, therefore the are considered as static neural networks and the equation previously introduced can be adapt for their description (page 748), while C is described as a dynamic neural network and therefore q’ is needed to model it, a parameter z=[q^Tq’^T]∈R²ⁿ is used. A general controller can then be constructed demonstrating that no Jacobian matrix is required (for the function please refer to page 749); Kr is introduced to give the proportional derivative type control and k_ssgn(r) is indicating the tracking error. In a closed loop system with positive derivative and tracking signal error, e and e’ tend to be stably 0 with t going to infinite, the presence of sgn(.) function denotates chattering which needs to be minimized.

Key Concepts

Artificial Neural Networks, Control Systems

Key Results

The model of a Neural Network for robot controlling has been applied in simulation in comparison with a basic PD controller (non adaptive control). The simulation involved a two link manipulator and vector M, P and L where introduced. The vector M is: M=P+p_lL, where P is the vector for payloads, p_l is payload at lth link and L is [l²₁ l²₂ l_ll_{2 l1} l₂]^T. For D and G function a 100-node static neural network is used, for C we use a 200-node dynamic neural network.

In the simulation non adaptive control appears to have a significant tracking error and cannot handle changes, using adaptive control allows to reduce a lot the tracking error thanks to the learning mechanism which is provided by the neural network methodology.

Actual D and actual C are shown to not converge to optimal D and C, which actual G converges to its optimal value, this is due to the fact that the desired trajectory is not persistently exciting as for real-world applications.

In conclusion the model allows a good control system without the need of time-consuming computations for obtaining the Jacobian, which describes the necessary inverce kinematics for traditional control systems.

Theory and Application of Neural Networks for Industrial Networks for Industrial Control Systems

Toshio Fukuda, Takanori Shibata

IEEE Transactions on Industrial Electronics, Vol.39,No 6, December 1992

Summary

                  Human’s brain is composed by about 10¹⁰ neurons which allow the brain to be capable of having the following unique characteristics: parallel processing of information, learning function, self-organization capabilities, associative memory and is good for information processing. Ideally researched want to obtain a similar decision making approach and control system for robots, so that artificial neural network basically is a connection of many linear and non linear neuron models and where information is processed in a parallel manner. In the literature there have been approaches in obtaining «mindlike» machines (based on the idea of interconnecting models in the same manner as biological neurons), cybernetics (for which main principles see relationship between engineering principle, feedback and brain function) and the idea of “manufacturing a learning machine”. The literature goes further on in covering recognition system arriving to the Hopfield Net (a series of first-order non-linear differentiable equations that minimize a certain energy function).

·       Models

Each neuron’s output is obtained by summing all the inputs in the same neuron, subtracting the bias and considering a weigh effect on each input. The net is classified in two categories: recurrent net (for which multiple neurons are interconnected) and feed-forward net (which presents a hierarchical structure).

The Hopfield Net is a recurrent net, which has the capability of providing feedback paths, basically with the aim of stabilizing a certain potential field. For the purpose the state of the system and the potential field are used (refer to page 475 for the formulas), the system tends to equilibrium at infinity. This kind of setup allows parallel operation and it’s a case of associative memory (as for the system tends to move to equilibrium points).

Neural Networks are applied in different fields, as for the Travelling Salesman Problem, for which though the only suboptimal solution can be obtained (since the Hopfield transition is based on the least mean algorithm and this may stack to local minimum cases. The Boltzmann Machine is another case of application, for which each neuron is operating with a certain probability, so it also can minimize the energy function as for the Hopfield Net. Further implementations include the Feedforward Neural Network, it’s a back-propagation technique, meaning that is uses gradient search to minimize the error computer as the mean difference between the desired output and the actual one. Once the Back-Propagation is consisting basically the learning face, therefore the overall system first uses the input vector to produce its own output vector, then it computer the difference between the desired output and actual one and adjusts the weights according to the Delta Rule. The initial weights are have to be initialized and random small values are used (for the back propagation algorithm please refer to page 478).

Adaptive Critic appears to be an extended method for learning applications through associative search element and single adaptive critic element (the first being the action network and the latter being the critic network, having as an output a reward or punishment for the first network. Learning method can be offline (carrying unnecessary training), online (problem in initialization), or feedback error learning (which has the issue of lacking of knowledge of the system).

Key Concepts

Artificial Neural Networks, Backpropagation, Delta rule, Robot Learning

Key Results

Neural Networks are applied in vision and speech recognition, design and planning, application control (supervised control, where sensors input information, inverse control, which learns inverse dynamics of a system and neural adaptive control, in order to predict future outputs of a system), knowledge processing (where databases can be use also for initialization and for supervising the net).