A Hierarchical Goal-Biased Curriculum for Training Reinforcement Learning

Abstract

Hierarchy and curricula are two techniques commonly used to improve training for Reinforcement Learning (RL) agents. Yet few works have examined how to leverage hierarchical planning to generate a curriculum for training RL Options. We formalize a goal skill that extends an RL Option with state-based conditions that must hold during training and execution. We then define a Goal-Skill Network that integrates a Hierarchical Goal Network, a variant of hierarchical planning, with goal skills as the leaves of the network. An automatically generated plan for a Goal-Skill Network correctly orders goal skills such that (1) it is a Goal-Biased Curriculum for training the goal skills, and (2) it can be executed to achieve top-level goals. In a set of six distinct gridworld environments using up to ten goal skills, we demonstrate that these contributions train nearly perfect policies significantly faster than learning a whole policy from scratch.