VB-Com: Learning Vision-Blind Composite Humanoid Locomotion Against Deficient Perception

Junli Ren1,2, Tao Huang1,3, Huayi Wang1,3, Zirui Wang1,4, Qingwei Ben1,5, Jiangmiao Pang1,+ Ping Luo1,2,+
1Shanghai AI Laboratory, 2The University of Hong Kong, 3Shanghai Jiao Tong University,
4Zhejiang University, 5The Chinese University of Hong Kong
Paper arXiv Video X Code (coming soon)

Overview

Abstract

The performance of legged locomotion is closely tied to the accuracy and comprehensiveness of state observations. ``Blind policies", which rely solely on proprioception, are considered highly robust due to the reliability of proprioceptive observations. However, these policies significantly limit locomotion speed and often require collisions with the terrain to adapt. In contrast, ``Vision policies" allows the robot to plan motions in advance and respond proactively to unstructured terrains with an online perception module. However, perception is often compromised by noisy real-world environments, potential sensor failures, and the limitations of current simulations in presenting dynamic or deformable terrains. Humanoid robots, with high degrees of freedom and inherently unstable morphology, are particularly susceptible to misguidance from deficient perception, which can result in falls or termination on challenging dynamic terrains. To leverage the advantages of both vision and blind policies, we propose VB-Com, a composite framework that enables humanoid robots to determine when to rely on the vision policy and when to switch to the blind policy under perceptual deficiency. We demonstrate that VB-Com effectively enables humanoid robots to traverse challenging terrains and obstacles despite perception deficiencies caused by dynamic terrains or perceptual noise.

Interpolate start reference image.

Avoid Dynamic and Static Obstacles

Avoid low speed robots

Recover from high speed collision

Consecutive Avoid Obstacles

H1 vs Dynamic Obstacles

Traverse Hurdles

Deficient Perception

Comprehensive Perception

Sudden Falling Hurdles

Step Recovery on Gaps

G1

G1

H1

h1

Framework

Interpolate start reference image.

VB-Com contributes to the following aspects:

  • We develop a perceptive and a non-perceptive humanoid locomotion policy that can traverse gaps, hurdles and avoid obstacles.
  • We propose a novel hardware-deployable return estimator that predicts future returns achieved by current policy conditioned on proprioceptive states observation.
  • We design a dual-policy composition system that integrates perceptive and non-perceptive policies for robust locomotion through dynamic obstacles and terrains where onboard sensors providedeficient external perception.