RAMPA: Robotic Augmented Reality for Machine Programming by Demonstration

Fatih Doğangün1, Serdar Bahar1, Yiğit Yıldırım1, Bora Toprak Temir1, Emre Uğur1, Mustafa Doğa Doğan2,1,
1 Department of Computer Engineering, Bogazici University        2 Adobe Research, Basel, Switzerland

Paper Code

Abstract

Abstract—As robotics continue to enter various sectors beyond traditional industrial applications, the need for intuitive robot training and interaction systems becomes increasingly more important. This paper introduces Robotic Augmented Reality for Machine Programming (Rampa), a system that utilizes the capabilities of state-of-the-art and commercially available AR headsets, e.g., Meta Quest 3, to facilitate the application of Programming from Demonstration (PbD) approaches on industrial robotic arms, such as Universal Robots UR10. Our approach enables in-situ data recording, visualization, and fine-tuning of skill demonstrations directly within the user’s physical environment. Rampa addresses critical challenges of PbD, such as safety concerns, programming barriers, and the inefficiency of collecting demonstrations on the actual hardware. The performance of our system is evaluated against the traditional method of kinesthetic control in teaching three different robotic manipulation tasks and analyzed with quantitative metrics, measuring task performance and completion time, trajectory smoothness, system usability, user experience, and task load using standardized surveys. Our findings indicate a substantial advancement in how robotic tasks are taught and refined, promising improvements in operational safety, efficiency, and user engagement in robotic programming.

Video

Key Features of Rampa

Data Recording

Users use AR to demonstrate skills, capturing the data directly in the user's environment, using hand pinch, as shown in the figure above. Compared to traditional methods, such as the kinesthetic control, RAMPA simplifies the collection of demonstration trajectories, an integral component of PbD procedures, offering a more efficient experience for robotic programmers.

Data Playback and Adjustment



The system allows users to visualize and modify the movements of a virtual robot before these movements are executed on the actual robotic arm. This step ensures that potential errors can be corrected in a safe, virtual space. Moreover, the ability to adjust any portion of demonstration trajectories increases the operator's overall performance.





ML Training and Preview





Utilizing the collected data, popular PbD models, such as Probabilistic Movement Primitives (ProMPs), can be trained and validated within the AR environment. Users can observe the generated trajectories in real-time, ensuring the robot's behavior aligns with desired outcomes.


BibTeX

@misc{dogangun2024rampa,
      title={RAMPA: Robotic Augmented Reality for Machine Programming and Automation},
      author={Dogangun, Fatih and Bahar, Serdar and Yildirim, Yigit and Temir, Bora Toprak and Ugur, Emre and Dogan, Mustafa Doga},
      howpublished={arXiv:2410.13412, 2024}
}