How do you choose what algorithms to use in a reinforcement learning settings?
The answer can be complicated as it depends on so many factors. As a beginner, it’s usually difficult to answer this question because it’s hard enough to keep track of all these algorithms.
There are many state-of-the-art RL models and systems out there but they might not be a perfect fit to the problem you are trying to solve. Instead, it’s much easier to start by building up some intuition of the kinds of algorithms to use at the fundamental and basic level.
Diagram
Below is a decision tree style diagram that describes how to choose the algorithm for the problem on hand. The goal is to give some guidance and reference in the general direction to choose algorithm but not a definite answer because the field is evolving and many sophisticated algorithms are not mentioned here.
Some explanation of the diagram
While the diagram is mostly self explanatory, there are a few points to take away:
- At the top level, it’s important to first see if the problem can fit in a tabular setting (see tabular-vs-functional-approximation). This largely depends on the size of the problem.
- The node “Control problem?” is showing at multiple places in this diagram because there is a distinction between control-vs-prediction
- The node “Learn on each time step?” is important because it usually distinguish Monte-Carlo based algorithms with other TD based algorithms.
For more details on the terminologies, see Reinforcement Learning Glossary as a reference guide.
Reference: most of the material of this post comes from the book Reinforcement Learning