VR in Human-Robot Collaboration

by Jennifer Wang (2022)


As artificial intelligence (AI) and machine learning (ML) evolve, we expect robots to execute increasingly complex tasks with complete autonomy to sense the environment, respond to it, and execute actions within it. Robots are common in industrial and commercial settings – for complex assembly picking and packing, last-mile delivery, and more. A collaboration between human and robot can implicate many advantages for such complex processes, especially as increased flexibility and adaptability become a key feature of production systems in manufacturing.

The use of virtual reality (VR) systems has the potential to simulate cooperative processes in advance and to include workers and their individual behavior into the simulation. Typical industrial production applications of VR in HRC span from manufacturing process simulation, which are able to provide real time enhanced information used for inspection or with focus on training, to collaborative factory planning during product design when analyzing and evaluating changes prior to implementation.

Existent VR Solutions in Human–Robot Collaboration

Existent solutions can roughly be broken down into 4 categories:

  1. The operator support considers the communication and guidance between two resources of humans and robots.

  2. The simulation category explores solutions upon the utilization of simulation software to enhance the users' understandability of the working ambient.

  3. The instruction section details how the exploitation of virtual and augmented environment assists users in teaching the robot a hierarchy of tasks.

  4. Finally, the teleoperation section aims to examine the existing solutions on how these techniques enable the operator to operate and manipulate the robot remotely.

  1. Operator support

An overarching goal in using VR for HRC is to provide straightforward, bidirectional communication between human and robot teammates. The human is provided information to more clearly understand the robot’s intent and perception capabilities, while the robot is provided information about the human that enables it to build a model. A robot might want to communicate an overall plan, a goal location, or a general intent so that the human collaborator does not duplicate efforts, alter the environment, or put themselves in danger.

One such example is Bolano et al. where the authors used 3D cameras for monitoring the robot, and the GPU-Voxel library to create a point cloud from 3D models to predict robot collision. In this study, audio instruction was used to inform the user regarding the robot's next move.

2. Simulation

The use of VR simulations makes it possible to reduce physical and mental barriers between human and robot. By allowing the user to immerse into the simulation, VR systems create a realistic image of HRC processes that provide an inexpensive and safe method for manufacturers to conduct experiments and train their workers.

De Giorgio et al. demonstrated these benefits in a study where participants in a VR simulation can program the robot by teaching the pose targets and walk around to explore robot movements. In the direct interaction between a human and robot, having the working environment of the robot and the robot itself displayed in a dynamic, 3D virtual environment helped the operator understand how to manage the correspondence of different degrees of freedom when controlling a robotic arm with a human arm.

3. Instruction

In developing an intuitive user interface, the use of VR under the HRC production scheme enables human operators to provide virtual instructions and quality demonstrations to teach the robots a hierarchy of tasks.

To establish a bilateral communication channel for HRC, Kousi et al. proposed an AR-based framework that aids the operator to control the robot in case of unexpected errors and to reprogram the production environment, provide information between robot and operator tasks, and demonstrate real-time execution status feedback. AR applications provided the human operator the mechanisms that allow them to directly instruct the robot in an easy and fast way without any expertise in robotics. It also allowed them to become acutely aware of the limitations faced by the robots and provide feedback and demonstrations accordingly.

4. Teleoperation

VR systems allow users to facilitate intuitive real-time remote teleoperation. By generating specialized spaces in virtual environments, such interfaces establish a common frame of reference for the user and the robot to effectively communicate spatial information for object manipulation such that humans can supervise robots from a distance and control their movements.

One such example is Wang et al. (2019) who integrated a VR interface to perform a welding process. In the process, the operator observed the status of the torch and controls the travel speed, while the robot received a command from the computer and calculated the required pose and moves. The information in the interface is then updated so that the user could supervise and adjust the welding process. The system improved the accuracy and repeatability during the process compared to the manual method and enhanced the consistency of the welding width.