ICANEWS

Natural Sciences

Learning Robot Safety from Sparse Human Feedback Using Conformal Prediction

This research introduces a method for learning robot safety preferences from sparse human feedback using conformal prediction. The approach aims to identify reg

Overview

This research introduces a method for learning robot safety preferences from sparse human feedback using conformal prediction. The approach aims to identify regions of states associated with unsafe robot behavior, potentially within a learned latent space. This system functions as a warning mechanism, designed to alert users when a robot enters a suspected unsafe region, thereby mimicking human safety preferences with a guaranteed miss rate.

Research Context

Ensuring robot safety presents several challenges. User-defined constraints may not account for all edge cases, and robot policies, even when trained with initially safe data, can still evolve to exhibit unsafe behaviors. Furthermore, the definition of safety can be subjective and vary across human users. Traditional approaches to safety learning often face difficulties in sample efficiency or require extensive labeled datasets. The presented method addresses these issues by leveraging human feedback on policy trajectories to identify unsafe behaviors.

Approach

The core of the methodology involves presenting policy trajectories to a human observer who provides binary feedback, indicating whether observed robot behavior is safe or unsafe. This sparse binary feedback is then processed using the statistical technique of conformal prediction. Specifically, the method builds upon nearest neighbor classification to identify a region in the state space (or a learned latent space) that is guaranteed to contain a user-specified fraction of future policy errors.

  • The system avoids the common practice in conformal prediction of withholding data, contributing to its sample efficiency.
  • When the robot reaches a state within this identified unsafe region, the system issues an alert.

An additional component of the research is an approach for policy improvement. This involves modifying the robot's policy to actively avoid the suspected unsafe region, potentially enhancing its overall safety performance.

Findings

  • The method successfully identified a region of states guaranteed to contain a user-specified fraction of future policy errors by using sparse human feedback and conformal prediction, building on nearest neighbor classification.
  • From video labeling, the developed system was able to detect instances where a quadcopter visuomotor policy would fail to steer through a designated gate.
  • An approach for policy improvement by avoiding the suspected unsafe region was presented and tested.
  • Experimental testing involved 30 quadcopter flights across 6 navigation tasks, demonstrating an improvement in the safety of a model predictive controller when incorporating the identified unsafe region avoidance.
  • The proposed method exhibited sample efficiency in learning safety preferences.

Why This Matters

The ability to learn robot safety preferences from sparse human feedback with a guaranteed miss rate offers a mechanism for developing more reliable and human-aligned robotic systems. This is particularly relevant in scenarios where explicitly defining all safety constraints is challenging or where human subjectivity plays a significant role in safety perception. The demonstrated policy improvement through avoidance of identified unsafe regions suggests a practical pathway for enhancing the safety performance of autonomous systems in real-world applications.

Key Findings