Construction Tele-robotic System with Virtual Reality (CG Presentation of Virtual robot and Task Object using Stereo Vision System)

In this study, we investigated a tele-robotics system for a construction machine. The system consists of a servo-controlled construction robot, two joysticks for controlling the robot from a remote place, and a 3-degrees-of-freedom motion base. The operator of the robot sits on the motion base and controls the robot bilaterally from a remote place. The role of the motion base is to realistically simulate the motion of the construction robot. As a tool for assisting the operator, a stereo video of the operation field is projected on a screen. If, in addition to the video, computer graphics (CG) of the robot were to be presented to the operator, the task efficiency could be expected to increase. To confirm this, a CG of a virtual robot was created, and its effectiveness for a task involving carrying an object was determined in this study. This system can present the position and shape of the task object by means of a stereo vision system. The results of the experiment clarified that tasking time was shortened effectively even for amateur operators. Thus, the usefulness of the developed CG system was confirmed.


Introduction
A remote-control robotic system using bilateral control is useful for performing restoration in damaged areas, and also in extreme environments such as space, the seabed, and deep underground. In this study, we investigated a tele-robotics system for a construction machine. The system consists of a servo-controlled construction robot, two joysticks for operating the robot from a remote place, and a 3-degrees-of-freedom motion base. The operator of the robot sits on the motion base and controls the robot bilaterally from a remote place. The role of the motion base is to realistically simulate the motion of the robot.
In order to improve the controllability of the system, we examined (1) the master and slave control method between joysticks and robot arms [1], [2],(2) a presentation method for the motion base[31, [4], and (3) the visual presentation of the task field for an operator[51. Because the visual presentation is the information most essential to the operator, in this study we focused on the presentation method of the operation fi eld of a remote place. In our previous paper, we proposed a presentation method that used a mixed image of stereo video graphics and the CG image of the robot, and clarified that the tasking efficiency was improved. At this stage, however, because the position and the shape of the task object have not presented to the operator, the operator cannot help feeling inconvenienced. In this study, therefore, a CG presentation system, which enables presentation not only of the robot but also of the position and the shape of a task object, was newly developed. Application of the method was expected to increase the tasking efficiency. To confirm this, a CG of a virtual robot was created, and its effectiveness for the task of carrying an object was determined. The results of the experiment clarified that tasking time was shortened effectively even for amateur operators. Thus, the usefulness of the developed CG system was confirmed.

Tele-robotic
System using CG presentation Figure 1 shows a schematic diagram of the telerobotic system that was developed in the course of this research (2). The system is of a bilateral type and is thus divided into two parts; the master system and the slave system. Here, the slave system is a construction robot equipped with a pair of stereo CCD cameras. The master system is controlled by an operator and consists mainly of a manipulator and a screen. The robot has four hydraulic actuators controlled by four servo valves through a computer (PC). Acceleration sensors were attached to the robot for feeding back the robot's movement to the operator.
The manipulator controlled by the operator consists of two joysticks and a motion base on which a seat is set for the operator. The motion base provides 3 degrees of freedom and can move in accordance with the motion of the robot. This means that the operator is able to feel the movement of the robot as if he or she were sitting on the seat of the robot.
The joysticks can be operated in two directions; along the X-and Y-axes. The displacements of the joysticks are detected by position sensors, while the displacements of the actuators are detected by magnetic stroke sensors embedded in the pistons. A stereo video image captured by the CCD cameras is transmitted to a 3D converter then projected onto the screen by a projector. Simultaneously, a signal synchronized with the video image is generated by the 3D converter and transmitted to an infrared unit. This signal enables the operator to alternate between leftand right-hand images of liquid crystal shutter glasses. Thus, the operator's remote vision is stereoscopic.
In the previous paper (5), a CG image of robot motion (without a CG image of the task object) was additionally presented; i.e., with the video image from the CCD cameras. In that case, the operator had to watch both the CG and the video image at the same time, which was tiring.
In this study, we developed a visual presentation system for producing two CG images; one of the robot and one of the task object. As a tool for making a CG image of the task object, we adopted a stereo vision camera named" Triclops" (Fig.2), a product of Point Grey Research, Inc. Triclops is a color-stereo-vision system that provides real-time range images using stereo-computer vision technology. The system consists of a three-calibrated-colors camera module, which is connected to a Pentium PC. Triclops is accurately able to measure the distance to a task object in its field of view at a speed of up to 30 frames/second . In the developed presentation system, the operator can view CG images of the remote robot and the task object from all directions. Figure 3 shows a schematic diagram of the developed tele-robotic system with CG presentation. In the figure, PC1 has the same role as the PC in Fig. 1. The CG images of the robot and the task object are generated by a graphics computer (PC2) according to the signals received from the joysticks and the stereo vision camera" Triclops". Figures 4 and 5 show the arrangement of the experimental setup and a top view of the tele-robotic system, respectively. The robot is set on the left-hand side of the operation site. The operator controls the joysticks, watching the screen in front of him/her. The stereo CCD video cameras are arranged at the back left side of the robot; thus, the operator observes the operation field from a back oblique angle through the screen. When the operator looks directly at the robot,

Experimental results
In the experiment, the operator controls the robot by using the joysticks according to predetermined tasks. In the beginning, the robot is set at the neutral position (Fig.6), and two concrete blocks are placed on a pair of the marked places each other (Fig.7). The operator grasps one of the concrete blocks set in a marked place, then carries it to the center marked place and releases it. Subsequently, and in a similar fashion, the operator grasps and carries the other block.
As control conditions for the operator, three types of visual presentation are set as shown in Table. 1. That is, "3D" corresponds to the stereo vision presentation given by stereo CCD cameras;"CG" corresponds to the presentation of the virtual robot and task objects by Computer Graphics; and "Direct" corresponds to watching the task field directly.
In the experiments, three kinds of CG video images of the virtual robot are simultaneously presented to the operator. The first is a lateral view from the left-hand side; the second a lateral view from the right-hand side; and the third is a top view. These view angles were selected so that the operator could effectively confirm the position of two concrete blocks. Figure 8 shows a projected image presented to the operator.
Thirty-three subjects served as respective operators of the robot, and we measured the time it took each subject to complete the task. Moreover, we counted the number of failed attempts-that is, when a subject could not succeed in completing a task.   Figure 9 shows the average values of the tasking times it took the 33 subjects to complete the assigned tasks. The average tasking time in "3D" was longer than that in "CG" or "Direct". This is thought to be due to the difficulty the operator has in observing the operation field only from a back oblique angle through the screen in the case of "3D". In the case of "CG", however, the operator has access to a VR image of the robot, even when the robot is at a dead angle; thus, the tasking times in this case is considered to nearly coincide with those in "Direct".  Figure 10 shows the ratio of each tasking time to that of direct control. Based on this result, the efficiency in "3D" is approximately 40% , and, in addition, the efficiency in "CG" amounts to nearly 80%. These results confirm the usefulness of the VR image. Figure 11 shows the time required by experts (operators who had operated the tele-robot system several times before) and that by beginners (operators operating the tele-robot system for the first time). In our study, there were 5 experts and 26 beginners. In this figure, it can be seen that the graphs of experts and beginners show nearly the same shape. However, the tasking time of the beginners is longer than that of the experts. The difference in tasking time between the beginners and the experts is the smallest in the case of "CG" , indicating that CG presentation is most effective for beginners. We can see in the figure that the number of failed attempts in" CG" is less than half that in" 3D". This is because the operators could recognize the end position of the robot arm accurately, via the CG image. Figure  13 shows the dispersion of the tasking times with standard deviation. From the figure, it can be seen that the tasking times in "3D" vary relatively widely. In the case of "CG" or "Direct", on the other hand, the dispersion is small, as a result of the stability of the tasks.

Conclusions
In this research, we investigated a tele-robotic construction system developed by us. In our previous study, we developed a system that presents video images transmitted from an operation field. This image was generated by a pair of stereo CCD cameras, allowing a real 3D image to be observed through 3D glasses. However, this system was difficult in that the operator observed the operation field only from a back oblique angle through the screen. We considered that if, instead of the video graphics, CG of the robot were presented to the operator, the tasking efficiency would be expected to increase because the operator would have a multi-angle view of the operation field.
In the present study, we investigated the application of a method that allows the operator to obtain a better sense of the operation field, in order to confirm that this method allowed the operator to control the robot more effectively and stably. To this end, CG images of a virtual robot were generated. It was expected that watching a thus obtained VR robot image in addition to viewing the task object would increase the tasking efficiency. In the experiments, the task of carrying a concrete block was performed by 33 operators, some of whom were amateurs. The results confirmed statistically that the tasking time was shortened by introduction of the VR images. Considering that the 3D glasses are tiring to wear, the overall usefulness of the developed system remains to be assessed.