Control the ARmada: LMM Coordination for Multi-Robot AR-HRC

Augmented reality (AR) has shown strong potential to enable more flexible interaction in human-robot collaboration (HRC) by conveying a robot’s state and intent and enabling intuitive control. However, while many systems exist that demonstrate AR’s ability to enhance robot control, they often incorporate control for only a singular robotic collaborator, with interaction techniques that require direct manipulation of virtual content; in scenarios with many robots, these manipulations would become much more frequent, lowering task efficiency through repeated manual interventions. Moreover, increased focus on virtual content could reduce awareness of the real environment, posing safety risks if this environment contains hazards. Motivated by these challenges, we present ARmada, an AR-HRC system leveraging edge scene understanding and an LMM to enhance environmental awareness and enable AR control in multi-robot settings.