Overview
SUSD (Structured Unsupervised Skill Discovery) is a framework designed for the autonomous learning of diverse skill sets without reliance on extrinsic rewards. This framework addresses limitations observed in common Unsupervised Skill Discovery (USD) methods, particularly those based on Mutual Information (MI), which often favor static skills, and Distance-Maximizing Skill Discovery (DSD), which can fall short in promoting comprehensive skill sets.
Research Context
Unsupervised Skill Discovery (USD) aims to learn a variety of skills autonomously, without requiring external reward signals. A prevalent strategy in USD involves maximizing the Mutual Information (MI) between skill latent variables and observed states. However, MI-based techniques demonstrate a tendency to discover simple, static skills due to their inherent invariance properties. This characteristic can restrict the range of dynamic, task-relevant behaviors that can be discovered.
A related approach, Distance-Maximizing Skill Discovery (DSD), seeks to encourage more dynamic skills by utilizing state-space distances. Despite this advancement, DSD methods may still not fully promote the discovery of comprehensive skill sets that engage all controllable factors or entities within an environment.
Approach
SUSD introduces a framework that leverages the compositional structure inherent in environments. This is achieved by factorizing the state space into independent components, which can represent elements such as objects or distinct controllable entities. Within this structured approach, SUSD assigns distinct skill variables to these different factors, enabling a more granular level of control over the skill discovery process.
The framework incorporates a dynamic model designed to track learning progress across these factors. This model adaptively directs the agent's focus towards factors that remain underexplored. This structured methodology aims to facilitate the discovery of a broader range of richer and more diverse skills. Additionally, SUSD generates a factorized skill representation, which offers fine-grained and disentangled control over individual entities. This disentangled representation is posited to facilitate efficient training for compositional downstream tasks when integrated with Hierarchical Reinforcement Learning (HRL).
Findings
Experimental evaluations of SUSD were conducted across three distinct environments, encompassing environments with factor counts ranging from 1 to 10. The results indicate that SUSD is capable of discovering diverse and complex skills without the need for supervision. The framework significantly outperformed existing unsupervised skill discovery methods in both factorized and complex environmental settings.
Potential Applications
The factorized skill representation generated by SUSD enables fine-grained and disentangled control over individual entities. This capability facilitates efficient training of compositional downstream tasks through Hierarchical Reinforcement Learning (HRL).