Multi-user swarm robot control system and method using hand gestures

Multi-User Swarm Robot Control System Based on Virtual Reality Hand Tracking

This technology is a multi-user Human-Swarm Interaction (HSI) control system that tracks user hand gestures in virtual reality (VR/AR/MR) environments to simulate swarm robot movement paths, formations, and control commands in real-time, applying them to actual robotic systems.

Existing 2D interface-based swarm control has limitations in 3D spatial manipulation and fails to support complex formation control or simultaneous multi-user operation beyond individual robot control.

This technology integrates head-mounted displays with hand-tracking systems to visualize robot swarms in a 3D virtual space. It enables intuitive control through hand gestures such as pinch-to-zoom for viewpoint manipulation, automatic scaling, waypoint setting, virtual wall creation for herding, and swarm shaping, allowing multiple users to control swarm robots simultaneously. By significantly reducing the operational complexity in fields requiring the simultaneous deployment of multiple robots—such as logistics warehouses, disaster response, and defense surveillance—it provides a practical solution to lower the barriers to the commercialization of swarm robotics.

‍

‍

Key Features:

• An interface unit that provides a control interface for managing swarm robots within a virtual reality environment.
• A hand-gesture recognition unit for inputting control commands via user hand movements into the provided virtual environment.
• A simulation control unit that simulates the behavior of swarm robots in the virtual reality environment based on hand-gesture inputs and controls their actual movements accordingly.
• Execution of control for each mode based on flocking modes, including Basic Mode, Formation Maintenance Mode, and Hand-Gesture Imitation Mode.

‍

This invention was developed with support from the Ministry of Science and ICT’s Human-Centered Soft Robot Technology Research Center and the Ministry of Science and ICT’s project for developing 3D collaborative teleoperation technology for unstructured tasks in harsh environments.