RL-Picker

Please view this page on a large screen

To select appropriate reinforcement-learning algorithms, reply to as many of the following questions as possible:

Hide inappropriate algorithms instead of marking them red

Less preferred algorithms will be marked yellow

Environment dynamics consisting of reward function r and state-transition probability p(s'|s,a)

Not given, but feasible to learn

Primary goal/concern of the learning process

Training stability and/or asymptotic performance Data efficiency, transfer learning, safety, and/or explainability

Neither given nor feasible to learn (e.g. chaotic) Given

Action sequences

Complex or hierarchically divisible into sub-routines Simple or moderately complex

Expert

Not available Available

Computational resources for running several agents in parallel during training

Not available Available

Estimate the "risk" of taking certain actions?

Yes please Not necessary

Which of the following is more important?

Good exploration (e.g. because most actions are bad) and/or efficient use of experience (e.g. because acquiring new experience is expensive) Training stability

Rewards

Non-deceptive and quite dense Sparse, deceptive, or unclear in advance
(in this case, a deterministic target policy shouldn’t be used ... because deterministic policies might not work well if rewards are sparse, even if the optimal policy is deterministic [Matheron et al. 2019], and because randomness increases robustness against deceptive rewards and/or local minima [Mazoure et al. 2019])

Is the optimal policy probably stochastic?

Yes (e.g. important information is not fully observable or a deterministic behavior can be exploited by adversaries)
(in this case, a deterministic target policy shouldn’t be used because it cannot model stochastic behavior) No

Is estimating the benefit (value) of each possible action considerably more difficult than selecting the best action?

Yes, e.g. it is easier to steer a car well than to estimate the exact risks of each possible steering action No

Duration of an episode

Always short Sometimes/always long

Action space

Discrete and very small

State space

Discrete and very small Not very small

Discrete and small Large discrete or low-dimensional continuous Very large discrete or high-dimensional continuous

Important information from past observations

May become unobservable (i.e. past observations need to be taken into account) Remains observable (i.e. past observations do not contain additional important information compared to the current observation)

Unlike the questions above regarding what is dictated by the environment, the following question is about your planned choice of method properties:

What target policy will you choose?

Stochastic Deterministic
(see the red text "deterministic target policy" above regarding why this choice is unavailable)

For selecting a parametric probablity distribution for actions, see Section 3 in the full paper.

Q-learning [Watkins & Dayan 1992] with TD	Model-free	Off-policy	Tabular value-based with exact maximization	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
Q-learning [Watkins & Dayan 1992] with Q(λ)	Model-free	Off-policy	Tabular value-based with exact maximization	Q(λ)	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
SARSA [Rummery et al. 1994] with TD	Model-free	On-policy	Tabular value-based with exact maximization	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
SARSA [Rummery et al. 1994] with SARSA(λ)	Model-free	On-policy	Tabular value-based with exact maximization	SARSA(λ)	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
DQN [Mnih et al. 2015]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
Double DQN (DDQN) [van Hasselt et al. 2016]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
Dueling DQN [Wang et al. 2016]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
DRQN [Hausknecht & Stone 2015]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
Prioritized DQN [Schaul et al. 2016]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
DQfD [Hester et al. 2018] with TD	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Imitation learning
DQfD [Hester et al. 2018] with TD(n)	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD(n)	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Imitation learning
h-DQN [Kulkarni et al. 2016]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Not distributional	Not distributed	Hierarchical	Not imitation learning
Distributional DQN (also called c51) [Bellemare et al. 2017]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Distributional	Not distributed	Not hierarchical	Not imitation learning
QR-DQN [Dabney et al. 2018]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Distributional	Not distributed	Not hierarchical	Not imitation learning
Rainbow [Hessel et al. 2017]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD(n)	No entropy regularization	Distributional	Not distributed	Not hierarchical	Not imitation learning
Sequential DQN [Metz et al. 2017]	Model-free	Off-policy	Non-tabular value-based with approximate maximization and fixed search procedure	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
AQL [Van de Wiele et al. 2020]	Model-free	Off-policy	Non-tabular value-based with approximate maximization and learned search procedure	Q(λ)	Per-state entropy regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
GAN Q-learning [Doan et al. 2018]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Distributional	Not distributed	Not hierarchical	Not imitation learning
Bootstrapped DQN [Osband et al. 2016]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD	No entropy regularization	Distributional	Not distributed	Not hierarchical	Not imitation learning
QT-Opt [Kalashnikov et al. 2018]	Model-free	Off-policy	Non-tabular value-based with approximate maximization and fixed search procedure	TD	No entropy regularization	Distributional	Distributed	Not hierarchical	Not imitation learning
R2D2 [Kapturowski et al. 2019]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD(n)	No entropy regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
Agent57 [Badia et al. 2020]	Model-free	Off-policy	Non-tabular value-based with exact maximization	Retrace(λ)	No entropy regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
Ape-X DQN [Horgan et al. 2018]	Model-free	Off-policy	Non-tabular value-based with exact maximization	TD(n)	No entropy regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
REINFORCE [Williams 1992]	Model-free	On-policy	Policy-based	MC	Per-state entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
PCL [Nachum et al. 2017]	Model-free	Off-policy	Actor-critic	TD(n)	Soft Q-learning	Not distributional	Not distributed	Not hierarchical	Imitation learning
Off-PAC [Degris et al. 2012]	Model-free	Off-policy	Actor-critic	GTD(λ)	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
Reactor [Gruslys et al. 2018]	Model-free	Off-policy	Actor-critic	Retrace(λ)	Per-state entropy regularization	Distributional	Distributed	Not hierarchical	Not imitation learning
A2C-SVPG [Liu et al. 2017]	Model-free	On-policy	Actor-critic	GAE(λ)	Soft Q-learning	Not distributional	Distributed	Not hierarchical	Not imitation learning
MPO [Abdolmaleki et al. 2018]	Model-free	Off-policy	Actor-critic	Retrace(λ)	Kullback–Leibler divergence regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
DSPG [Shi et al. 2019]	Model-free	Off-policy	Actor-critic	TD	Soft Q-learning	Not distributional	Not distributed	Not hierarchical	Not imitation learning
NFP/NBP [Tang & Agrawal 2018]	Model-free	Off-policy	Actor-critic	TD	Per-state entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
AWR [Peng et al. 2019]	Model-free	Off-policy	Actor-critic	TD(λ)	Kullback–Leibler divergence regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
DPG [Silver et al. 2014]	Model-free	Off-policy	Actor-critic	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
DDPG [Lillicrap et al. 2016]	Model-free	Off-policy	Actor-critic	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
D3PG [Barth-Maron et al. 2018]	Model-free	Off-policy	Actor-critic	TD(n)	No entropy regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
D4PG [Barth-Maron et al. 2018]	Model-free	Off-policy	Actor-critic	TD(n)	No entropy regularization	Distributional	Distributed	Not hierarchical	Not imitation learning
TRPO [Schulman et al. 2015a]	Model-free	On-policy	Actor-critic	TD(n)	Kullback–Leibler divergence regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
PPO [Schulman et al. 2017]	Model-free	On-policy	Actor-critic	GAE(λ)	Per-state entropy and Kullback–Leibler divergence regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
HiPPO [Li et al. 2020]	Model-free	On-policy	Actor-critic	GAE(λ)	Kullback–Leibler divergence regularization	Not distributional	Distributed	Hierarchical	Not imitation learning
HiRO [Nachum et al. 2018]	Model-free	Off-policy	Actor-critic	TD	No entropy regularization	Not distributional	Not distributed	Hierarchical	Not imitation learning
SNN4HRL [Florensa et al. 2017]	Model-free	On-policy	Actor-critic	TD(n)	Mutual-information regularization	Not distributional	Not distributed	Hierarchical	Not imitation learning
A2C/A3C [Mnih et al. 2016]	Model-free	On-policy	Actor-critic	TD(n)	Per-state entropy regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
ACER [Wang et al. 2017]	Model-free	Off-policy	Actor-critic	Retrace(λ)	Per-state entropy and Kullback–Leibler divergence regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
NAC [Peters et al. 2005]	Model-free	On-policy	Actor-critic	LSTD-Q(λ)	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
ACKTR [Wu et al. 2017]	Model-free	On-policy	Actor-critic	TD(n)	Per-state entropy and Kullback–Leibler divergence regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
TD3 [Fujimoto et al. 2018]	Model-free	Off-policy	Actor-critic	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
SAC [Haarnoja et al. 2018]	Model-free	Off-policy	Actor-critic	TD	Soft Q-learning	Not distributional	Not distributed	Not hierarchical	Not imitation learning
Normalized actor-critic (sometimes also abbreviated as NAC) [Gao et al. 2018]	Model-free	Off-policy	Actor-critic	TD	Soft Q-learning	Not distributional	Not distributed	Not hierarchical	Imitation learning
Hybrid SAC [Delalleau et al. 2019]	Model-free	Off-policy	Actor-critic	TD	Soft Q-learning	Not distributional	Not distributed	Not hierarchical	Not imitation learning
Distributional SAC [Duan et al. 2021]	Model-free	Off-policy	Actor-critic	TD	Soft Q-learning	Distributional	Distributed	Not hierarchical	Not imitation learning
GAE-TRPO [Schulman et al. 2015b]	Model-free	On-policy	Actor-critic	GAE(λ)	Kullback–Leibler divergence regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
IMPALA [Espeholt et al. 2018]	Model-free	Off-policy	Actor-critic	V-trace(n)	Per-state entropy regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
SEED [Espeholt et al. 2020]	Model-free	Off-policy	Actor-critic	V-trace(n)	Per-state entropy regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
Ape-X DPG [Horgan et al. 2018]	Model-free	Off-policy	Actor-critic	TD(n)	No entropy regularization	Not distributional	Distributed	Not hierarchical	Not imitation learning
Q-Prop [Gu et al. 2017]	Model-free	Off-policy	Actor-critic	GAE(λ)	Kullback–Leibler divergence regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
(RE)LAX [Grathwohl et al. 2018] with MC	Model-free	On-policy	Actor-critic	MC	Per-state entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
(RE)LAX [Grathwohl et al. 2018] with TD	Model-free	On-policy	Actor-critic	TD	Per-state entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
REDQ [Chen et al. 2021] with MC	Model-free	Off-policy	Actor-critic	MC	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
REDQ [Chen et al. 2021] with TD	Model-free	Off-policy	Actor-critic	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
TD-MPC [Hansen et al. 2022]	Model-free	On-policy	Actor-critic	TD	No entropy regularization	Not distributional	Not distributed	Not hierarchical	Not imitation learning
For model-based algorithms, see e.g. the survey papers Moerland et al. (2020b), Moerland et al. (2020a), Wang et al. (2019), Hamrick et al. (2020), Plaat et al. (2020).	Model-based