Robotic Fall Prediction with Egocentric Vision and Proprioception

Abstract

Legged robots have received increasing attention in recent years thanks to ever-improving deep learning techniques, holding promise for programmable robots that can match the capabilities of animals and people. Body pose estimation is a widely used technique, which has been well-developed and integrated with deep learning models for robotic motion challenges, such as running and jumping in unknown environments. Current approaches mainly focus on self-modeling and proprioception. In this paper, a multi-modal model is proposed to predict whether the Poppy^® Humanoid will fall during position-controlled locomotion in real and simulated environments. Our method integrates two modalities: joint trajectories (actual and planned), and egocentric vision. Extensive experiments, performed with both real and simulated data, show that the proposed method can predict falls up to two seconds in advance and outperform the closest baseline by up to 9.11% on the real dataset and 3.49% on the simulated dataset. The real data, and code to regenerate the simulated data, are freely available online at https://github.com/BoruiHe/Robotic-Fall-Prediction-with-Egocentric-Vision-and-Proprioception-SAW.