RLlib Developer Guide

Note

This guide will take you through steps for implementing a new algorithm in RLlib. To apply existing algorithms already implemented in RLlib, please see the user docs.

Recipe for an RLlib algorithm

Here are the steps for implementing a new algorithm in RLlib:

  1. Define an algorithm-specific Policy evaluator class (the core of the algorithm). Evaluators encapsulate framework-specific components such as the policy and loss functions. For an example, see the simple policy gradient evaluator example.
  2. Pick an appropriate Policy optimizer class. Optimizers manage the parallel execution of the algorithm. RLlib provides several built-in optimizers for gradient-based algorithms. Advanced algorithms may find it beneficial to implement their own optimizers.
  3. Wrap the two up in an Agent class. Agents are the user-facing API of RLlib. They provide the necessary “glue” and implement accessory functionality such as statistics reporting and checkpointing.

To help with implementation, RLlib provides common action distributions, preprocessors, and neural network models, found in catalog.py, which are shared by all algorithms. Note that most of these utilities are currently Tensorflow specific.

_images/rllib-api.svg

The Developer API

The following APIs are the building blocks of RLlib algorithms (also take a look at the user components overview).

Agents

Agents implement a particular algorithm and can be used to run some number of iterations of the algorithm, save and load the state of training and evaluate the current policy. All agents inherit from a common base class:

Policy Evaluators and Optimizers

Sample Batches

In order for Optimizers to manipulate sample data, they should be returned from Evaluators in the SampleBatch format (a wrapper around a dict).

Models and Preprocessors

Algorithms share neural network models which inherit from the following class:

Currently we support fully connected and convolutional TensorFlow policies on all algorithms:

A3C also supports a TensorFlow LSTM policy.

Observations are transformed by Preprocessors before used in the model:

Action Distributions

Actions can be sampled from different distributions which have a common base class:

Currently we support the following action distributions:

The Model Catalog

The Model Catalog is the mechanism for algorithms to get canonical preprocessors, models, and action distributions for varying gym environments. It enables easy reuse of these components across different algorithms.