Multi-User Swarm Robot Control System and Method Using Hand Gestures

This technology is a multi-user Human-Swarm Interaction (HSI) control system that tracks user hand gestures in virtual reality (VR/AR/MR) environments to simulate the movement paths, formations, and swarm control commands of robot swarms in real time, and applies them to actual robotic systems.

Existing 2D interface-based swarm control has limitations in manipulating 3D space and fails to support complex formation control or simultaneous control by multiple users beyond individual robot manipulation.

This technology integrates head-mounted displays with hand-tracking systems to visualize robot swarms in a 3D virtual space. It enables functions such as viewpoint manipulation, automatic scaling, waypoint setting, herding via virtual wall creation, and swarm shaping through hand gestures like pinch grips, allowing multiple users to intuitively control robot swarms simultaneously. By significantly reducing the operational difficulty in fields requiring the simultaneous deployment of multiple robots—such as logistics warehouses, disaster response, and defense surveillance—it serves as a solution to lower the practical barriers to the commercialization of swarm robotics.

‍

‍

Key Features:

• An interface unit that provides a control interface for managing robot swarms in a virtual reality environment.
• A hand-gesture recognition unit that inputs user hand movements as control commands within the provided virtual environment.
• A simulation control unit that simulates the movements of the robot swarm in the virtual reality environment based on hand-gesture inputs and provides feedback to the user.
• Implementation of multi-user control by simultaneously managing the actual operations of the robot swarm in tandem with the simulation.