What is Reinforcement Learning?

Reinforcement learning (RL) is a machine learning (ML) technique that trains models to make decisions to achieve the most optimal results. It mimics the trial-and-error learning process that humans use to achieve their goals. Software actions that work towards your goal are reinforced with positive reinforcement, while actions that detract from the goal are ignored.

RL algorithms use a reward-and-punishment paradigm as they process data. They learn from the feedback of each action and self-discover the best processing paths to achieve outcomes. The algorithms are also capable of delayed gratification, optimizing for future rewards. The best overall strategy may require short-term sacrifices, so the best approach they discover may include some punishments or backtracking along the way. RL is a powerful method to help artificial intelligence (AI) systems achieve optimal outcomes in unseen environments.

There are many benefits to using reinforcement learning (RL) that businesses can take advantage of.

Excels in complex environments

You can use reinforcement learning algorithms in complex environments with many rules and dependencies. A human may not be capable of determining the best path to take in these scenarios, even with superior knowledge of the environment. Instead, RL algorithms adapt to continuously changing environments and find new strategies to optimize results. By using the Markov Decision Process (MDP) mathematical framework, these models can find approximately the best possible path to their intended goal.

Requires less human interaction

In traditional ML algorithms, humans must label data pairs to direct the algorithm. This is what's known as supervised learning. When you use an RL algorithm, this isn't necessary, the model learns by itself, based on reward signals designed by humans. At the same time, RL offers mechanisms to integrate human feedback, allowing for systems that adapt to human preferences, expertise, and corrections.

Optimizes for long-term goals

RL inherently focuses on long-term reward maximization, which makes it appropriate for scenarios where actions have delayed consequences. It is particularly well-suited for real-world situations where immediate reward isn't available for every step, since it can learn from delayed rewards.

For example, decisions about energy consumption or storage might have long-term consequences. RL can be used to optimize long-term energy efficiency and cost.

Discovers the approximately optimal policy

Reinforcement learning can discover an approximately optimal policy through experience, rather than directly through instruction. Through experience, an agent learns what actions lead to the best long-term outcomes. They evaluate feedback from a reward function instead of relying on predefined rules, to find the highest cumulative reward over time.

Using principles from the Markov decision process, many reinforcement learning algorithms estimate future outcomes with a value function. Techniques such as Q-learning support this iterative behavior, resulting in reinforcement learning policies that can discover approximately optimal pathways.

Adapts through interaction

Most benefits within reinforcement learning stem from its ability to improve performance by continuous environment interactions. As the agent learns from trial and error, it refines its behavior based on a reward signal. This adaptive loop lets systems respond to changing conditions.

Reinforcement learning adapts in real time, learning by training agents on live data. However, in some scenarios, you can run offline reinforcement learning to improve policies without needing live interaction. Offline learning typically achieves lower performance.

Reinforcement learning (RL) can be applied to a wide range of real-world use cases. Here are some examples.

Marketing personalization

In applications like recommendation systems, RL can customize suggestions to individual users based on their interactions, leading to more personalized experiences. For example, an application may display products to a user, initially based on some demographic information. With each product interaction, the application learns which products to recommend to the user to optimize product sales, testing known strategies and explorative strategies to compare the effectiveness.

Optimization challenges

Traditional methods of optimization solve problems by evaluating and comparing possible solutions based on certain criteria. In contrast, RL introduces learning from interactions to find the approximately optimal solutions over time.

For example, a cloud spend optimizing system may be too complex to model with traditional optimization techniques. Instead, RL can adjust to fluctuating resource needs and choose optimal instance types, quantities, and configurations. It makes decisions based on factors such as current and available cloud infrastructure, spending, performance, and utilization.

Financial predictions

The dynamics of financial markets are complex, with statistical properties that change over time. RL algorithms can seek to optimize long-term returns by considering transaction costs and adapting to market shifts.

For instance, an algorithm could observe the states and patterns of the stock market before it tests actions and records associated rewards. It dynamically creates a value function and develops a trading strategy. RL in financial markets is challenging and complex, as the number of factors to consider is near-endless.

The learning process of reinforcement learning (RL) algorithms is similar to animal and human reinforcement learning in the field of behavioral psychology. For instance, a child may discover that they receive parental praise when they help a sibling or clean but receive negative reactions when they throw toys or yell. Soon, the child learns which combination of activities results in the end reward.

An RL algorithm mimics a similar learning process. It tries different activities to learn the associated negative and positive values to achieve the cumulative reward outcome.

Key concepts

In reinforcement learning, there are a few key concepts to familiarize yourself with:

• The agent is the ML algorithm
• The environment is the adaptive problem space with attributes such as variables, boundary values, rules, and valid actions
• The action is a step that the RL agent takes to navigate the environment
• The state is the environment at a given point in time
• The reward is the positive, negative, or zero value—in other words, the reward or punishment—for taking an action
• The cumulative reward is the sum of all rewards or the end value
• The policy is the learned strategy

Algorithm basics

Reinforcement learning is based on the Markov decision process, a mathematical modeling of decision-making that often uses discrete time steps. At every step, the reinforcement learning agent takes a new action that results in a new environment state. Similarly, the current state is attributed to the previous action and state.

Through trial and error in moving through the environment, the agent builds a set of rules or a policy. The policy helps it decide which action to take next for optimal expected cumulative reward. The agent must also choose between further environment exploration to learn new state-action rewards or select known high-reward actions from a given state. This is called the exploration-exploitation trade-off.

There are various algorithms used in reinforcement learning (RL)—such as Q-learning, policy gradient methods, Monte Carlo methods, and temporal difference learning. Deep RL is the application of deep neural networks to reinforcement learning. One example of a deep RL algorithm is Trust Region Policy Optimization (TRPO).

All these algorithms can be grouped into two broad categories.

Model-based RL

Model-based reinforcement learning is typically used when environments are well-defined and learnable, so you can model the system, then learn and predict the outcome of an action.

The agent first builds an internal representation (model) of the environment. It uses this process to build the model:

1. It takes actions within the environment and notes the new state and reward value
2. It associates the action-state transition with the reward value

The agent then uses the model to simulate action sequences, to achieve an outcome based on maximizing optimal cumulative rewards. The agent thus learns a policy within the environment to achieve the desired end goal.

Example

Consider a robot learning to navigate a new building to reach a specific room. Initially, the robot explores freely and builds an internal model of the building. For instance, it might learn that it encounters an elevator after moving forward 10 meters from the main entrance. It learns that this elevator has a prolonged wait time versus an elevator 400 meters from the entrance. Once it builds the model, it can build a series of shortest-path, shortest-time, or least obstructive sequences between different locations it visits frequently in the building.

Model-free RL

Model-free RL is best to use when the environment is large, complex, and not easily describable or modelable. It's also ideal when the environment is unknown and changing, and environment-based testing is inexpensive.

The agent doesn't build an internal model of the dynamic environment. Instead, it uses a trial-and-error approach within the environment. It scores and notes state-action pairs and sequences of state-action pairs to develop a policy. This method must interact with the same environment far more than model-based methods.

Example

Consider the robot navigating the building again. Instead of internally building a model of a building, with peak traffic times and elevator behaviors, the robot instead tries different navigation techniques with rewards based on shortest travel time. Each attempt helps to build up a policy for optimal behavior that minimizes travel time to destination. This method takes far more real-world trials, however, it avoids errors from incorrect environmental modelling.

Monte Carlo methods

Monte Carlo methods are a group of model-free reinforcement learning algorithms mainly used for estimating long-term returns. These algorithms observe large sequences of actions and rewards and then average outcomes over these episodes, rather than using the temporal difference (TD) step-based approach. This strategy is useful when an entire episode can be observed.

While supervised and unsupervised learning, and reinforcement learning (RL) are all learning approaches in the field of AI, there are distinctions between the three.

Reinforcement learning vs. supervised learning

In supervised learning, you define both the input and the expected associated output. For instance, you can provide a set of images labeled "dogs" or "cats," and the algorithm is then expected to identify a new animal image as a dog or cat.

Supervised learning algorithms learn patterns and relationships between the input and output pairs. Then, they predict outcomes based on new input data. It requires a labeller, typically a human, to label each data record in a training data set with an output.

Unlike supervised learning, RL has a well-defined end goal in the form of a maximum cumulative reward over a sequence of actions. There is no known dataset or prior knowledge in advance, and the result is time-based, rather than a single prediction. During training, instead of trying to map inputs with known outputs, it maps inputs with possible outcomes. By rewarding desired behaviors, you give weight to the best outcomes.

Reinforcement learning vs. unsupervised learning

Unsupervised learning algorithms receive inputs with no specified outputs during the training process. They find hidden patterns and relationships within the data using statistical means. For instance, you could provide a set of documents, and the algorithm may group them into categories it identifies based on the words in the text.

Conversely, RL is about achieving an optimal outcome over sequential decision making Although it also takes an exploratory approach, the explorations are continuously validated and improved to increase the reward from its value function.

While reinforcement learning (RL) applications can be highly capable, it may not be easy to deploy these algorithms.

Practicality

Experimenting with real-world reward and punishment systems to solve reinforcement learning problems may not be practical. For instance, testing a drone in the real world without testing in a simulator first would lead to significant numbers of broken aircraft. Real-world environments change often, significantly, and with limited warning. It can make it harder for the algorithm to be effective in practice.

Interpretability

Like any field of science, data science looks at conclusive research and findings to establish standards and procedures. Data scientists prefer knowing how a specific conclusion was reached for provability and replication.

With complex RL algorithms, the reasons why a particular sequence of steps was taken may be difficult to ascertain. Which actions in a sequence were the ones that led to the optimal end result? This can be difficult to deduce, which causes implementation challenges.

Amazon Web Services (AWS) has many offerings that help you develop, train, and deploy reinforcement learning (RL) solutions for real-world applications. Explore deep reinforcement learning, deep neural networks, natural language processing, and other artificial intelligence solutions.

With Amazon SageMaker, developers and data scientists can quickly and easily develop scalable RL models. Combine a deep learning framework (such as TensorFlow or PyTorch), an RL toolkit (like RLlib or Ray), and an environment to mimic a real-world scenario. You can use it to create and test your model.

Get started with reinforcement learning on AWS by creating a free account today.