Deep Reinforcement Learning

Offline Reinforcement Learning

Prof. Dr. Sebastian Peitz

Chair of Safe Autonomous Systems, TU Dortmund

Summer term 2026

🚀 by Decker

Content

Learning policies from datasets
Offline reinforcement learning
- Distributional shifts
- Overestimation bias
- Relation to imitation learning
- Algorithmic evolution of methods
Policy constraint methods
- Forward and backward KL divergence
- RL + BC (behavior cloning)
- advantage-weighted regression
- Batch-constrained $Q$-learning
Implicit $Q$-learning
Conservative $Q$-learning
Offline-to-online RL
Model-based offline RL

Where are we?

Lecture contents
Chapter	Topic	Content
	Basics & tabular methods
1-5	Bandits, MDPs, Dynamic Programming, Monte Carlo, TD Learning	RL basics in finite dimensions
	Deep-learning-based methods
6-13	DQN, policy gradients, actor-critic, PPO/SAC, exploration	Deep RL from basics to modern algorithms
	Model-Based Control
14-15	Optimal & feedback control, MPC, planning, model-based RL	Planning & control when we know the model
	Advanced Topics
16	Offline reinforcement learning	RL with a static, given dataset

Learning policies from datasets

Imitation learning

Two of the biggest challenges in RL:
1. Long waiting times until we find a policy that does not fail completely.
2. Designing good reward functions that actually make the agent do what we want it to do.
What can we do to circumvent this?
$\Rightarrow$ “Let’s just show the agent how it’s done?”: learn from an expert!
The concept behind this is imitation learning.
- Given examples from a human expert, try to derive a policy that mimicks this behavior as closely as possible …
- … while generalizing to previously unseen scenarios.

images/16-offline-RL/Imitation-learning.png Source: Medium.com. Source: Innovation campus future mobility.

Some developments in imitation learning (1)

Behavior cloning (BC)

Treats imitation as supervised learning. Inputs: expert states; Outputs/Labels: expert actions as labels
Ignores the underlying MDP.

$\pluspoint$ Simple to implement.

$\pluspoint$ No environment interaction during training.

$\pluspoint$ Very fast.

$\minuspoint$ Compounding errors / covariate shift: A small error at step one puts the agent in a state it has never seen before, causing increasingly poor decisions.

$\minuspoint$ Assumes i.i.d. data distribution, which is false in sequential environments.

Inverse reinforcement learning (IRL)

Instead of copying the actions, IRL tries to figure out the intent by assuming that the expert is optimizing an unobserved reward function.
Extracts hidden reward function from the demonstrations and then runs standard RL on top.

$\pluspoint$ Robust to compounding errors.

$\pluspoint$ Reward function can transfer well even if the environment changes slightly.

$\minuspoint$ Expensive inner-loop problem: RL training in each iteration.

$\minuspoint$ Mathematically underdetermined (many different reward functions can explain the same expert behavior).

Some developments in imitation learning (2)

Interactive / Human-in-the-Loop IL

To fix BC’s compounding errors without the cost of IRL, methods like Dataset Aggregation (DAgger, (Ross, Gordon, and Bagnell 2011)) were introduced.
- Runs current policy in the environment.
- When it makes a mistake and goes off-track, an expert intercepts and provides the correct action labels for those erroneous states.
- This data is added back into the training set.

$\pluspoint$ Directly fixes covariate shift.

$\pluspoint$ Forces the agent to learn how to recover from its own mistakes.

$\minuspoint$ Requires an expert to be constantly available and online during training to label data.

$\minuspoint$ Exhausting for humans and frequently impossible for complex setups.

Generative advarsarial imitation learning (GAIL) (Ho and Ermon 2016)

Two-player game:
- Generator: the agent trying to mimic the expert policy.
- Discriminator: classifier trying to distinguish between agent and expert trajectories.

images/16-offline-RL/GAN.png — Inspired by Generative Adversarial Networks (GANs).

The discriminator penalizes the agent when it does not create expert-like trajectories.

$\pluspoint$ Mimicks entire trajectories, not just single transitions as BC.

$\pluspoint$ No expensive inner loop as in IRL.

$\pluspoint$ Does not require an interactive expert like DAgger.

$\minuspoint$ Inherits GAN instabilities (mode collapse, highly sensitive hyperparameters).

$\minuspoint$ Very data hungry.

What if we want to do better than the data?

The key ideas in imitation learning
Algorithm (class)	Key concept
Behavior cloning (BC)	Copy data (i.e., “expert” as closely as possible).
Inverse RL (IRL)	Identify reward from data $\Rightarrow$ then standard RL.
Dataset aggregation (DAgger)	Expert corrects faulty behavior (meaning there is an online component).
Generative advarsarial imitation learning (GAIL)	The generative version of BC mimicking trajectories.

BUT: What if we cannot get new data, but we still want to be better than the performance of the dataset?

Maybe the data came from

Experts,
semi-experts,
one or more past RL policies,
a mixture of the above?

$\Rightarrow$ Offline reinforcement learning!

Offline Reinforcement Learning

What is offline RL?

Inspired by (Levine et al. 2020).

Data-driven AI is about learning from large datasets.
$\textcolor{green}{\mathbf{+}\text{ Learns about the real world from data.}}$
$\textcolor{red}{\mathbf{-}\text{ Doesn’t try to do better than the data.}}$

Reinforcement learning is about optimization.
$\textcolor{green}{\mathbf{+}\text{ Optimizes a goal with emergent behavior.}}$
$\textcolor{red}{\mathbf{-}\text{ Doesn’t make use of real-world data.}}$

$\Rightarrow$ Data without optimization doesn’t allow us to solve new problems in new ways.
$\Rightarrow$ Optimization without data is hard to apply to the real world outside of simulators.

Offline reinforcement learning: Use prior data for RL training, without any interactions with the environment!
Goal: Given a dataset $\Dc$, learn the best possible policy $\piphi$ that is supported by the dataset.

Can offline RL work?

Find the “good stuff” in a dataset full of good and bad behaviors.
Example, consider a dataset of various drivers:
$\circ$ with imitation learning, we would find the average performance 👎
$\circ$ with offline RL, we hope to extract a policy mimicking the best driver’s performance in each situation 👍
Generalization: good behavior in one place may suggest good behavior in another place.
Driving example: we improve upon the best driver by generalizing to unseen situations.
“Stitching”: parts of good behaviors can be recombined.

Generalization (Source: D4RL).

Stitching (Source: Sergey Levine’s CS285 lecture).

Takeaway: It’s not just imitation learning!
Intuition: Get order from chaos.

On micro scales (e.g., a “straight path” from a bunch of wiggly trajectories).
On macro scales (e.g., $A\to B \to C$ $\Rightarrow$ $A\to C$).

Why is offline RL so hard?

Fundamental problem: counterfactual queries

images/16-offline-RL/car-straight.png — **Training data.**

images/16-offline-RL/car-turn.png — **What the policy wants to explore.**

Is this good or bad?
How can we know if we have never seen it?

Inspired by Sergey Levine’s CS285 lecture.

Online RL algorithms don’t have to handle this.
- They can simply try this action and see what happens.
Offline RL methods have to account for these unseen (OOD: “out of distribution”) actions,
- ideally in a safe way …
- while still making use of generalization to come up with behaviors that are better than the best thing seen in the data!

Example: Q-learning / Bellman update

\[ Q_\theta \gets r + \gamma \E_{s'\sim \psprimesa}\Big[\underbrace{\max_{a'\in\Ac} Q_{\bar{\theta}}(s',a')}_{\textbf{can go OOD}}\Big] \]

💡 out of distribution vs. out of sample.

Out of sample $\Rightarrow$ generalization: Evaluation outside the training dataset, but on data from the same distribution.
Out of distribution $\Rightarrow$ Evaluation on unseen data from a different distribution. This is much harder!

Distributional shifts (1)

images/16-offline-RL/offline-RL.svg — Inspired by (Levine et al. 2020).

Formally: \[\begin{align*} \Dc &= \set{(s_i,a_i,s'_i,r_i)}_{i=1}^N \\ s &\sim \rho_\beta(s) \\ a &\sim \pi_\beta\agivenb{a}{s} \quad \fragment{ \text{(generally unknown)} } \\ s' &\sim \psprimesa \end{align*}\]

Inspired by Sergey Levine’s CS285 lecture.

\[\text{RL objective:}\quad \max_\phi \sum_{t=0}^T \Expsub{\gamma^t r_t}{s\sim\rho_\phi, a\sim \piphi\agivenb{\cdot}{s}}\]

“Model” is valid under $\pi_\beta$…
not under $\piphi$!
- “Model” could be the $Q$-function $Q_{\piphi}$,
- or the dynamics $p_\phi\agivenb{s'}{s,a}$.

Distributional shifts (2)

Example: Empirical risk minimization (maximum likelihood)

\[\theta = \arg\min_{\hat\theta} \Expsub{\cbracket{f_{\hat{\theta}}(x) - y}^2}{x\sim p(x), y\sim p\agivenb{y}{x}}\]

Question: Given some $x^*$, is $f(x^*)$ correct?

\[\Expsub{\cbracket{f_{\hat{\theta}}(x) - y}^2}{x\sim p(x), y\sim p\agivenb{y}{x}} \quad \text{is low}\] $\Rightarrow$ probably yes.
\[\Expsub{\cbracket{f_{\hat{\theta}}(x) - y}^2}{x\sim \bar{p}(x), y\sim p\agivenb{y}{x}} \quad \text{is generally not, if} ~ \bar{p}(x) \neq p(x)\] $\Rightarrow$ probably no.
What if $x^* \sim p(x)$?
- Not necessarily…
- but neural networks tend to generalize well!

What if we pick the maximizing $x^*$?

\[ x^* = \arg\max_x f_\theta(x) \]

Inspired by (Levine et al. 2020).

💡 This is exactly what we’re exploiting in adversarial learning!

images/16-offline-RL/adversarial-panda.png — Adversarial example (Goodfellow, Shlens, and Szegedy 2015).

Overestimation bias in offline RL

$Q$-learning and $Q$-function actor-critic

\[\begin{align*} \phi &= \arg\min_{\hat{\theta}} \Expsub{\norm{Q_\theta(s,a)-y}^2}{s\sim\rho_\beta,a\sim \textcolor{red}{\pi_\beta\agivenb{\cdot}{s}}}, \\ y_i &= r_i + \gamma \Expsub{Q_\theta(s'_i,a')}{s'_i\sim\rho_\beta,a'\sim \textcolor{blue}{\piphi\agivenb{\cdot}{s'_i}}} \end{align*}\]

Trying to find the best policy: we want $\textcolor{blue}{\piphi\agivenb{a}{s}} \neq \textcolor{red}{\pi_\beta\agivenb{a}{s}}$!
But we only have $s'_i$ for $s_i, a_i$ from our offline dataset.
Even worse: We’re actually picking an adversarial example, \[\phi = \arg\max_{\hat{\phi}} \Expsub{Q_\theta(s,a)}{a\sim \pi_{\hat\phi}\agivenb{\cdot}{s}}.\]

Example: SAC offline RL (Kumar et al. 2019)

images/16-offline-RL/offline-RL-return.png — How well the agent performs.

images/16-offline-RL/offline-RL-Q.png — How well it *thinks* it does.

Model-based RL

\[f = \arg\min_{\hat f} \Expsub{\norm{f(s,a)-s'}_2^2}{s\sim\rho_\beta,a\sim\textcolor{red}{\pi_\beta\agivenb{\cdot}{s}}, s'\sim \psprimesa} \]

Probably large: $\Expsub{\norm{f(s,a)-s'}_2^2}{s\sim\rho_\beta,a\sim\textcolor{blue}{\pi_\phi\agivenb{\cdot}{s}}, s'\sim \psprimesa}.$
Even worse: Pick $\piphi$ to maximize the reward under $f$!

Algorithmic evolution of methods

The following strategies (or classes of approaches) are the most common ones today:

Policy constraint methods: Stay close to the data, e.g., \[ \KLdiv{\piphi\agivenb{\cdot}{s}}{\pi_\beta\agivenb{\cdot}{s}}\leq \epsilon. \]

Value-regularization & pessimism: Go anywhere, but assume what you haven’t seen is dangerous $\Rightarrow$ avoid overestimation.

images/16-offline-RL/argmax-distribution.svg — Inspired by (Levine et al. 2020).

Implicit $Q$-learning: Implicitly define constraints to avoid leaving the support of the behavior policy $\pi_\beta$.

Common theme: “Maximize return while staying close to the dataset.

Policy constraint methods

Adding constraints

Simplest approach to avoid overestimation: constrain the distance between behavior policy $\pi_\beta$ and the learned policy $\piphi$.
Remember the KL divergence? \[ \KLdiv{\pi_\beta\agivenb{\cdot}{s}}{\pi_\phi\agivenb{\cdot}{s}} = \Expsub{\log\frac{\pi_\beta\agivenb{a}{s}}{\pi_\phi\agivenb{a}{s}}}{a\sim\pi_\beta\agivenb{\cdot}{s}} \fragment{ = \Expsub{\log\pi_\beta\agivenb{a}{s} - \log\pi_\phi\agivenb{a}{s} }{a\sim\pi_\beta\agivenb{\cdot}{s}}. } \]
Constraining the distance yields a constrained optimization problem: \[\begin{equation} \phi \gets \arg\max_\phi \Expsub{Q(s,a)}{s\sim\Dc,a\sim\pi\agivenb{\cdot}{s}} \qquad\text{s.t.}\qquad \KLdiv{\pi_\beta\agivenb{\cdot}{s}}{\pi_\phi\agivenb{\cdot}{s}}\leq \epsilon. \label{eq:OFF_constrained_policy} \end{equation}\]
- Where have we seen this before? $\Rightarrow$ Natural policy gradient!

Via Lagrange multipliers: additional term in the actor loss function: \[\phi \gets \arg\max_\phi \Expsub{\Expsub{Q(s,a)}{a\sim\pi_\beta\agivenb{\cdot}{s}} - \lambda \KLdiv{\pi_\beta\agivenb{\cdot}{s}}{\pi_\phi\agivenb{\cdot}{s}}}{s\sim\Dc}\] \[\begin{equation} \Rightarrow\quad \phi \gets \arg\max_\phi \Expsub{Q(s,a) + \lambda \log\piphi\agivenb{a}{s} + \mathsf{const}}{s\sim\Dc,a\sim\pi_\beta\agivenb{\cdot}{s}}. \label{eq:OFF_ACBC} \end{equation}\]
- To solve $\eqref{eq:OFF_constrained_policy}$, we need to solve for both $\phi$ and the Lagrange multiplier $\lambda$.
- Alternative: treat $\lambda$ as a hyperparameter.

AC + BC methods

Methods of the type $\eqref{eq:OFF_ACBC}$ are often termed “AC + BC”:
actor-critic plus behavior cloning,
as the term $\log\piphi\agivenb{a}{s}$ is typical in BC methods. Popular examples:
$\bullet$ TD3 + BC, $\quad\bullet$ DDPG + BC,
$\bullet$ SAC + BC.

Forward KL versus reverse KL

Remember: the KL divergence is not a real metric. In particular, it violates the symmetry condition, i.e., \[\begin{align*} \KLdiv{\piphi\agivenb{\cdot}{s}}{\pi_\beta\agivenb{\cdot}{s}} = \Expsub{\log\frac{\piphi\agivenb{a}{s}}{\pi_\beta\agivenb{a}{s}}}{a\sim\piphi\agivenb{\cdot}{s}} \neq \Expsub{\log\frac{\pi_\beta\agivenb{a}{s}}{\piphi\agivenb{a}{s}}}{a\sim\pi_\beta\agivenb{\cdot}{s}} = \KLdiv{\pi_\beta\agivenb{\cdot}{s}}{\pi_\phi\agivenb{\cdot}{s}}. \end{align*}\]

Forward KL a.k.a. “mode covering”: \[\begin{align*} \KLdiv{\pi_\beta\agivenb{\cdot}{s}}{\piphi\agivenb{\cdot}{s}} = \Expsub{\log\pi_\beta\agivenb{a}{s} - \log\piphi\agivenb{a}{s} }{a\sim\pi_\beta\agivenb{\cdot}{s}}. \end{align*}\]
Reverse KL a.k.a. “mode seeking”: \[\begin{align*} \KLdiv{\piphi\agivenb{\cdot}{s}}{\pi_\beta\agivenb{\cdot}{s}} &= \Expsub{\log\piphi\agivenb{a}{s} - \log\pi_\beta\agivenb{a}{s} }{a\sim\piphi\agivenb{\cdot}{s}}\\ &=-\Expsub{\log\pi_\beta\agivenb{a}{s} }{a\sim\piphi\agivenb{\cdot}{s}} - \Hc(\piphi\agivenb{\cdot}{s}). \end{align*}\]

Mode seeking vs. mode covering (inspired by Sergey Levine’s CS285 lecture).

Mode covering: What happens if for some $(s,a)$,
Mode covering: $\pi_\phi\agivenb{a}{s}=0$ while $\pi_\beta\agivenb{a}{s}>0$?
- KL divergence 💥 $\Rightarrow$ Need to cover everything from the dataset!
Mode covering: What happens if for some $(s,a)$,
Mode covering: $\pi_\beta\agivenb{a}{s}=0$ while $\pi_\phi\agivenb{a}{s}>0$?
- KL divergence 💥 $\Rightarrow$ Need to ensure $\pi_\phi$ stays within a subset of $\pi_\beta$.

Which one is desirable?

Mode seeking is what we actually want.
Extract the best behavior from a dataset.
However, it is harder to realize, which is why we often use mode covering in practice, see AC + BC.

How to learn $\pi_\beta$ if we don’t know it

Constraining the KL divergence requires knowledge of the behavior policy $\pi_\beta$.
What can we do if we don’t know it? $\Rightarrow$ extract it from the dataset!

Behavior cloning.

Given a dataset $\Dc=\set{(s_i,a_i)}_{i=1}^N$, we have a probabilistic supervised learning problem.
Find the probability distribution $\pi_\psi\agivenb{a}{s}$ giving rise to $\Dc$.
In other words maximize the likelihood of $a_i$ given $s_i$ over $\Dc$.
Approaches:

$\circ$ Variational autoencoder (VAE) (Kingma and Welling 2013):
images/16-offline-RL/VAE.png Adapted from Wikipedia.

$\circ$ Gaussian policy: $\pi_\psi\agivenb{a}{s} = \Nc(\mu_\psi(s), \sigma^2_\psi(s))$.

$\circ$ Generative modeling
images/16-offline-RL/Diffusion2.png Source: Medium.com.

Alternative to KL divergence

The KL divergence is a natural choice for the policy constraint, but not necessarily the best one:

$\pluspoint$ Mathematically tractable / closed-form solutions.
$\pluspoint$ Direct penalization of OOD actions.
$\pluspoint$ Smooth/soft regularization with tunable hyperparameter $\lambda$.

$\minuspoint$ Asymmetry (forward KL vs. reverse KL).
$\minuspoint$ Overly conservative in unseen regions.
$\minuspoint$ Sensitive to behavior policy estimation.

More generally, we do not even want the two distributions to match!
We just want to find the best behavior supported by the dataset.
Alternative: Minimizing Maximum Mean Discrepancy (MMD), see (Kumar et al. 2019): \[MMD^2(\pi, \beta) = \Exp{k(a_\pi, a_\pi')} - 2\Exp{k(a_\pi, a_\beta)} + \Exp{k(a_\beta, a_\beta')}.\]
- $k$ is a kernel function for implicitly evaluating inner products between feature vectors: $k(a_\pi, a_\beta) = \psi(a_\pi)^\top \psi(a_\beta)$.
- $\mathbb{E}[k(a_\pi, a_\pi')]$: How close to each other are the actions generated by our policy?
- $- 2\mathbb{E}[k(a_\pi, a_\beta)]$: How close are our actions to the dataset actions?
- $\mathbb{E}[k(a_\beta, a_\beta')]$: The internal variance of the dataset actions (a constant baseline).

$\pluspoint$ Bypasses densitiy estimation / works on samples .
$\pluspoint$ No infinite penalty on unsupported actions.
$\pluspoint$ Allows for multimodel generalization.

$\minuspoint$ Computationally expensive.
$\minuspoint$ High sample variance / hard to tune the kernel bandwidth.
$\minuspoint$ Curse of dimensionality.

The BRAC framework

Unification of various approaches in the BRAC framework

In (Wu, Tucker, and Nachum 2019), the authors present the Behavior Regularized Actor-Critic (BRAC).

This can be seen as a unification of many methods.
The constraint can be enforced in two different ways
- In the policy approximation (policy regularization)
- In the value or $Q$ function approximation (value regularization, coming up next)
The constraint is defined as a general distance metric.
- The KL divergence and MMD are then special cases.
Additional design choice: How to approximate the expectation.

Value-regularization & pessimism

Conservative $Q$-learning

We have seen that overestimation of $Q$-values outside the offline dataset $\Dc$ is the main source of failure for offline RL.
Policy constraint methods (previously): Penalize deviations of the policy from the behavior policy $\pi_\beta$.
Alternative: Penalize the $Q$-function outside $\Dc$ $\Rightarrow$ value regularization!
This is what conservative $Q$-learning (CQL, (Kumar et al. 2020)) does:
$\Rightarrow$ introduce a lower bound (a pessimistic approximation) of the actual $Q$-function.
We can then simply use the pessimistic $Q$-function in $Q$-learning or Actor-Critic methods.

images/16-offline-RL/CQL-conservative.png — Source: Berkley AI Research (BAIR) blog.

Question: How can we achieve this?

The idea behind CQL

How do we reduce the $Q$-function estimate outside our dataset?
- Just reduce $Q$ everywhere…
- then increase it again in the region supported by the dataset!
The $Q$-learning step with TD($0$): 💡 We here use an on-policy estimator, as is used in continuous control algorithms such as SAC. A $Q$-learning version were we maximize over actions in the target $y$ is equally possible (Kumar et al. 2020). \[\theta \gets \arg\min_\theta \frac{1}{2}\E_{(s,a,r,s')\sim\Dc} \Big[\Big(\underbrace{r + \gamma \Expsub{Q_{\bar{\theta}}(s,a)}{a'\sim\pi_\phi\agivenb{\cdot}{s'}}}_{y} - Q_\theta(s,a)\Big)^2\Big].\]
Add a conservative penalty term: \[\theta \gets \arg\min_\theta \frac{1}{2}\E_{(s,a,r,s')\sim\Dc} \Big[\Big(y - Q_\theta(s,a)\Big)^2 \Big]+ \alpha \underbrace{\max_\mu \Expsub{Q_\theta(s,a)}{s\sim\Dc,a\sim\mu\agivenb{\cdot}{s}}}_{=\Rc}.\]
- We need to find a policy $\mu$ that maximally reduces the estimate of $Q_\theta$.
- Resulting $Q$-function lower-bounds the true $Q$-function point-wise.
This raises two issues:
1. We don’t want a reduction on the parts supported by $\Dc$.
2. Computation of $\mu$ is an expensive optimization problem in itself!

The CQL($H$) framework (Kumar et al. 2020)

Reduce $Q$ everywhere, boost it on the dataset: \[ \begin{align*} \hat{\Rc} = \Rc - \Expsub{Q_\theta(s,a)}{a\sim\pi_\beta\agivenb{\cdot}{s}}. \end{align*} \]
Add KL-divergence regularizer to $\Rightarrow$ closed-form solution (similar to SAC): \[ \begin{align*} \hat{\Rc} = \E_{s\sim\Dc} \Big[&\log \cbracket{\int_\Ac \exp(Q_\theta(s, a)) \dint{a}} \\ &- \Expsub{Q_\theta(s,a)}{a\sim\pi_\beta\agivenb{\cdot}{s}}\Big]. \end{align*} \]
Lower bound on expected value: \[ \Expsub{Q_\theta(s,a)}{a \sim \pi} \le V^\pi(s).\]

Algorithmic implementation of CQL($H$)

Penalty term on our $Q$-function approximation: \[\theta \gets \arg\min_\theta \frac{1}{2}\E_{(s,a,r,s')\sim\Dc} \Big[\Big(y - Q_\theta(s,a)\Big)^2 \Big]+ \alpha \Expsub{\log \cbracket{\int_\Ac \exp(Q_\theta(s, a)) \dint{a}} - \Expsub{Q_\theta(s,a)}{a\sim\pi_\beta\agivenb{\cdot}{s}}}{s\sim\Dc}.\]
Challenge: Monte Carlo sampling of the integral over $\Ac$!
Uniform sampling will likely miss the large-$Q$ sections $\Rightarrow$ We need a suitable approximation from data.
Proposal in (Kumar et al. 2020): A mixture of three sets
1. Uniform actions ($a \sim U(\Ac)$): ensures exploratino of the entire action space; reduces overestimation peaks in distant regions of the action space.
2. Current policy actions ($a \sim \pi_\phi\agivenb{\cdot}{s}$): exploits the highest peaks in the current $Q$-function $\Rightarrow$ directly targets overestimations $\pi_\phi$ is trying to exploit.
3. Next-state policy actions ($a \sim \pi_\phi\agivenb{\cdot}{s'}$): In the Bellman equation, the target value is calculated using the policy’s action at the next state ($a' \sim \pi(s')$). If $Q(s', a')$ is overestimated, that error propagates back to $Q(s, a)$ via the TD update.
We would have to perform importance sampling to go from $\int \exp(Q_\theta(s, a)) \dint{a} \approx \frac{1}{N} \sum_{i=1}^N \exp(Q_\theta(s, a_i))$ to this sum of three terms. However, we ignore this in practice: \[ \begin{align*} I(s) &= \log\cbracket{\frac{1}{3}\cbracket{\Expsub{\exp{Q_\theta(s, a)}}{a \sim U(\Ac)} + \Expsub{\exp{Q_\theta(s, a)}}{a \sim \pi_\phi\agivenb{\cdot}{s}} + \Expsub{\exp{Q_\theta(s, a)}}{a \sim \pi_\phi\agivenb{\cdot}{s'}}}}, \\ \theta &\gets \arg\min_\theta \frac{1}{2}\E_{(s,a,r,s')\sim\Dc} \Big[\Big(y - Q_\theta(s,a)\Big)^2 \Big]+ \alpha \Expsub{I(s) - \Expsub{Q_\theta(s,a)}{a\sim\pi_\beta\agivenb{\cdot}{s}}}{s\sim\Dc}. \end{align*} \]

Example: Complex gripping tasks

Source: Berkley AI Research (BAIR) blog.

Implicit $Q$-learning

What we have seen until now (and why we need something better)

Explicit Policy Constraints (e.g., BEAR (Kumar et al. 2019) or BRAC (Wu, Tucker, and Nachum 2019))

Restrict the learned policy $\pi\agivenb{a}{s}$ to stay close to the behavior policy $\pi_\beta\agivenb{a}{s}$ that collected the dataset.
Typically using a distance metric like KL-Divergence.

Drawbacks:

Requires explicit modeling of the behavior policy $\pi_\beta$.
If $\Dc$ comes from human demonstrators or multiple mixed policies, $\pi_\beta$ is highly complex and multimodal.
Accurately fitting it is very difficult, and any error in $\pi_\beta$ propagates into the learned policy.

Conservative Q-Learning (CQL, e.g., (Kumar et al. 2020))

Instead of constraining the policy, CQL modifies the objective to learn a conservative lower-bound $Q$-function.
Penalty on the $Q$-values of out-of-distribution actions.

Drawbacks:

CQL requires sampling OOD actions during training to penalize them, which makes it computationally heavy.
Can be excessively conservative, reducing performance on tasks that require stitching together different good sub-trajectories.
Very sensitive to hyperparameter tuning.

The motivating question behind implicit $Q$-learning

What if we never evaluate an out-of-distribution action during training at all?

Instead of constraining the policy explicitly or penalizing unseen actions, IQL constrains the optimization implicitly by querying the $Q$-function only on state-action pairs that actually exist in the dataset.

Benefits of IQL (Kostrikov, Nair, and Levine 2021):

No OOD action evaluation: Avoids the maximization bias because the $Q$-function is never evaluated on unseen actions.
No behavior policy estimation: No need to learn or approximate $\pi_\beta$, eliminating the main source of compounding errors.
Sub-trajectory stitching: Enables the agent to take the best parts of different suboptimal trajectories and combine them into an optimal path.
Computational efficiency: Because it only uses in-sample data, training is significantly faster and more stable than CQL.

Advantage-weighted regression

Let’s consider the chess example

Imagine we are trying to learn how to play chess purely by watching a large database of past games.
- Some games were played by grandmasters, some by mediocre players or even beginners.
- If we copy every move equally (standard Behavior Cloning (BC)), we will become a mediocre player because we are copying the mistakes of the beginners just as much as the great moves of the masters.
Ideally, we want to copy the actions that led to better-than-expected outcomes and ignore (or suppress) the actions that led to poor outcomes.

Advanted-weighted regression (AWR, (Peng et al. 2019)):

Imitation learning on the dataset.
State-action pairs are weighted by an exponential function of the “advantage” (i.e., how much better the action was compared to the average action from that state).
Can be seen as weighted BC.

images/00-introduction/chess.gif — Chess board [Source]

Mathematical derivation of AWR

Remember trust region policy optmization (TRPO)?

Optimize the policy, but don’t let it move too far away from the current one!
The main reason (besides avoiding large update steps): mathematical approximation.
- We needed to take the expectation of our objective function w.r.t. the “wrong” distribution to make it work.
- Instead of sampling from the new policy that we’re optimizing, we took the expectation w.r.t. the old policy.
- Only if these are not too far apart, were we allowed to do this (and capable of deriving a bound for the error).

The starting point of AWR

Maximize the expected advantage $\eta(\pi)$ of our policy $\pi$ over the baseline policy $\pi_\beta$: \[ \eta(\pi) = \Expsub{A^{\pi_\beta}(s,a)}{s\sim\rho^\pi, a\sim\pias} = \Expsub{g - V^{\pi_\beta}(s)}{s\sim\rho^\pi, a\sim\pias}, \]
- where $g\sum_{t=0}^\infty \gamma^t r_t$ is the return from a trajectory following $\pi$,
- and the value $V^{\pi_\beta}(s)$ is the same return, only following the behavior policy $\pi_\beta$.
We cannot sample from the state distribution $\rho^\pi$ $\Rightarrow$ sample from the dataset $\Dc$ and optimize a surrogate objective $\hat\eta(\pi)$: \[ \hat\eta(\pi) = \Expsub{A^{\pi_\beta}(s,a)}{s\sim\rho^{\pi_\beta}, a\sim\pias} = \Expsub{g - V^{\pi_\beta}(s)}{s\sim\textcolor{red}{\Dc}, a\sim\pias}.\]
Similar to TRPO: \[\begin{equation} \pi^* = \arg\max_\pi \hat\eta(\pi) \qquad \text{s.t.}\qquad \KLdivavg{\pi}{\pi_\beta} \leq \epsilon. \label{eq:OFF_expected_advantage} \end{equation}\]

Solution to the AWR problem

Method of Lagrange multipliers $\Rightarrow$ closed-form analytical solution of $\eqref{eq:OFF_expected_advantage}$: \[\pi^*\agivenb{a}{s} \propto \pi_\beta\agivenb{a}{s} \exp\left( \frac{1}{\alpha} A^{\pi_\beta}(s, a) \right),\] with temperature hyperparameter $\alpha$.
- If $A > 0$: $\exp(\frac{1}{\alpha}A) > 1$ $\Rightarrow$ The probability of picking this action increases relative to the dataset.
- If $A < 0$: $\exp(\frac{1}{\alpha}A) < 1$ $\Rightarrow$ The probability of picking this action decreases.
To find a neural network policy $\pi_\phi\agivenb{a}{s}$, we approximate this optimal target policy by minimizing their KL divergence: $\min_\phi \KLdivavg{\pi^*}{\pi_\phi}$.
Insert the analytical solution $\Rightarrow$ $\pi_\beta\agivenb{a}{s}$ cancels out
$\Rightarrow$ weighted maximum likelihood loss: \[\begin{equation} L_\pi(\phi) = -\Expsub{\exp\left( \frac{1}{\alpha} A^{\pi_\beta} \right) \log \pi_\phi\agivenb{a}{s}}{(s, a) \sim \Dc}. \label{eq:OFF_awr_policy} \end{equation}\]
To compute the weight $\exp(\frac{1}{\alpha} A^{\pi_\beta}(s, a))$, we need the advantage.
- Train a standard baseline value function $V_\theta(s)$ via MSE on the dataset $\Dc$: \[\begin{equation} L_V(\theta) = \Expsub{(g - V_\theta(s))^2}{s\sim\Dc}. \label{eq:OFF_awr_value} \end{equation}\]

The AWR algorithm

Fit value function approximation $V_\theta(s)$ to the data set $\Dc$: \[ \min_\theta L_\mathsf{value}(\theta). \]
Optimize policy by approximating the analytically optimal policy: \[ \max_\phi L_\mathsf{policy}(\phi).\]

$\pluspoint$ Simple to implement.

$\minuspoint$ Single-step policy improvement.

$\minuspoint$ Because AWR relies on static dataset returns $g$ rather than propagating future values via Bellman backups ($Q(s,a) = r + \gamma \max V(s')$), it cannot stitch trajectories.

Implicit $Q$-learning (IQL)

Limitations of AWR

It requires entire trajectories to compute Monte Carlo estimates of the value.
It computes the average/expected value via the Monte Carlo approximation.
As AWR relies on learning entire trajectory returns, it can only replicate the best full trajectories already present in the data.

Proposed changes in implicit $Q$-learning (IQL (Kostrikov, Nair, and Levine 2021))

Use TD-learning instead of Monte Carlo sampling, i.e., \[\begin{align*} L_\mathsf{TD}(\theta) &= \Expsub{r + \gamma Q_\bar{\theta}(s',a') - Q_\theta(s,a)}{(s,a,r,s',a')\sim\Dc} \qquad &&\text{(SARSA)} \\ \text{or}\qquad L_\mathsf{TD}(\theta) &= \Expsub{r + \gamma \max_{\hat{a}\in\Ac} Q_\bar{\theta}(s',\hat{a}) - Q_\theta(s,a)}{(s,a,r,s')\sim\Dc}. \qquad &&\text{($Q$-learning)} \end{align*}\]
Do not estimate the mean, but the best possible $Q$-function supported by the data: \[ L_\mathsf{TD}(\theta) = \E_{(s,a,r,s')\sim\Dc} \Big[r + \gamma \max_{\hat{a}\in\Ac \\ \text{s.t.}~\pi_\beta\agivenb{\hat{a}}{s'}>0 } Q_\bar{\theta}(s',\hat{a}) - Q_\theta(s,a)\Big].\]

Approach: Expectile regression plus AWR!

Expectile regression loss

In statistical learning, it can be beneficial to perform asymmetrical regression.
That is, we weight positive devaitions differently from negative deviations: \[ L^\tau_2(u) = \abs{\tau - \mathbb{1}(u<0)}x^2 = \begin{cases} (1-\tau) u^2 & \text{if }u > 0 \\ \tau u^2 & \text{if }u \leq 0 \end{cases}. \]

Left: The asymmetric squared loss used for expectile regression. τ = 0.5 corresponds to the standard mean squared error loss, while τ = 0.9 gives more weight to positive differences. Center: Expectiles of a normal distribution. Right: an example of estimating state conditional expectiles of a two-dimensional random variable. Each x corresponds to a distribution over y. We can approximate a maximum of this random variable with expectile regression: τ = 0.5 correspond to the conditional mean statistics of the distribution, while τ ≈ 1 approximates the maximum operator over in-support values of y. (Source: (Kostrikov, Nair, and Levine 2021, Figure 1)).

The IQL algorithm

Implicit $Q$-learning (IQL (Kostrikov, Nair, and Levine 2021))

Given: A dataset $\Dc$

Estimate the value function by MSE optimization similar to $\eqref{eq:OFF_awr_value}$, but using the expectile loss and a target $Q$ network $Q_\bar{\theta}$: \[\begin{equation} L_V(\psi) = \Expsub{L^\tau_2(Q_\bar{\theta}(s,a) - V_\psi(s))}{(s,a)\sim\Dc}. \label{eq:OFF_iql_value} \end{equation}\]
Train a $Q$-network using the value network and TD learning: \[\begin{equation} L_Q(\theta) = \Expsub{r + \gamma V_\psi(s') - Q_\theta(s,a)}{(s,a,r,s')\sim\Dc}. \label{eq:OFF_iql_q} \end{equation}\]
Train the policy network similar to AWR (Eq. $\eqref{eq:OFF_awr_policy}$): \[\begin{equation} L_\pi(\phi) = -\Expsub{\exp\left( \frac{1}{\alpha} Q_{\bar{\theta}}(s,a) - V_\psi(s) \right) \log \pi_\phi\agivenb{a}{s}}{(s, a) \sim \Dc}. \label{eq:OFF_iql_policy} \end{equation}\]

Note: If we can get access to new data, the above procedure can be repeated iteratively with incoming data, after updating the replay buffer $\Dc$.

IQL on a toy umaze environment (a). When the data is heavily corrupted by suboptimal actions, one-step policy evaluation results in a value function that degrades to zero far from the rewarding states too quickly (c). IQL learns a near-optimal value function, combining the best properties of SARSA-style evaluation with the ability to perform multi-step dynamic programming, leading to value functions that are much closer to optimality (shown in (b)) and producing a much better policy (d). (Source: (Kostrikov, Nair, and Levine 2021, Figure 2)).

💡 In principle, we could also train $\eqref{eq:OFF_iql_q}$ directly, without training a separate value network via $\eqref{eq:OFF_iql_value}$. But the authors of (Kostrikov, Nair, and Levine 2021) argue that this can lead to the exploitation of “lucky samples”, whereas the value averages out such conincidences.

Some results

images/16-offline-RL/IQL-results-table.png — Averaged normalized scores on MuJoCo locomotion and Ant Maze tasks (Source: (Kostrikov, Nair, and Levine 2021, Table 1)).

images/16-offline-RL/IQL-AntMaze-domains.png — Ant maze in various sizes (small, medium, large).

images/16-offline-RL/IQL-AntMaze-results.png — The influence of the expectile loss (Source: (Kostrikov, Nair, and Levine 2021, Figure 3)).

Summary / what we have learned

Offline RL: Learn policies from a fixed, unchangeable dataset $\Dc$
Main challenges:
- Distribution shift between learned policy $\pi$ and dataset policy $\pi_\beta$
- Overestimation of $Q$ outisde the dataset $\Dc$
Approaches to address this:
- Constrain the policy $\Rightarrow$ Explicit policy constraint methods (e.g., AC + BC)
- Pessimistically modify the $Q$-function outside $\Dc$ $\Rightarrow$ Conservative $Q$-learning
- Avoid out-of-distribution actions altogether $\Rightarrow$ Implicit $Q$-learning

References

Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. 2015. “Explaining and Harnessing Adversarial Examples.” In International Conference on Learning Representations (ICLR).

Ho, Jonathan, and Stefano Ermon. 2016. “Generative Adversarial Imitation Learning.” In Advances in Neural Information Processing Systems, edited by D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett. Vol. 29. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2016/file/cc7e2b878868cbae992d1fb743995d8f-Paper.pdf.

Kingma, Diederik P, and Max Welling. 2013. “Auto-Encoding Variational Bayes.” arXiv:1312.6114.

Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. 2021. “Offline Reinforcement Learning with Implicit Q-Learning.” In Deep RL Workshop NeurIPS 2021.

Kumar, Aviral, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. 2019. “Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction.” Advances in Neural Information Processing Systems 32.

Kumar, Aviral, Aurick Zhou, George Tucker, and Sergey Levine. 2020. “Conservative Q-Learning for Offline Reinforcement Learning.” In Advances in Neural Information Processing Systems, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, 33:1179–91. Curran Associates, Inc.

Levine, Sergey, Aviral Kumar, George Tucker, and Justin Fu. 2020. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” arXiv:2005.01643.

Peng, Xue Bin, Aviral Kumar, Grace Zhang, and Sergey Levine. 2019. “Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning.” arXiv: 1910.00177.

Ross, Stephane, Geoffrey Gordon, and Drew Bagnell. 2011. “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning.” In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, edited by Geoffrey Gordon, David Dunson, and Miroslav Dudík, 15:627–35. Proceedings of Machine Learning Research. PMLR.

Wu, Yifan, George Tucker, and Ofir Nachum. 2019. “Behavior Regularized Offline Reinforcement Learning.” arXiv:1911.11361.

Content

Content

Where are we?

Learning policies from datasets

Imitation learning

Some developments in imitation learning (1)

Behavior cloning (BC)

Inverse reinforcement learning (IRL)

Some developments in imitation learning (2)

Interactive / Human-in-the-Loop IL

Generative advarsarial imitation learning (GAIL) (Ho and Ermon 2016)

What if we want to do better than the data?

Offline Reinforcement Learning

What is offline RL?

Offline RL is closely related to supervised learning / data-driven AI!

Can offline RL work?

Why is offline RL so hard?

Example: Q-learning / Bellman update

💡 out of distribution vs. out of sample.

Distributional shifts (1)

Distributional shifts (2)

Example: Empirical risk minimization (maximum likelihood)

What if we pick the maximizing \(x^*\)?

Overestimation bias in offline RL

\(Q\)-learning and \(Q\)-function actor-critic

Example: SAC offline RL (Kumar et al. 2019)

Model-based RL

Algorithmic evolution of methods

Policy constraint methods

Adding constraints

AC + BC methods

Forward KL versus reverse KL

Which one is desirable?

How to learn \(\pi_\beta\) if we don’t know it

Behavior cloning.

Alternative to KL divergence

The BRAC framework

Unification of various approaches in the BRAC framework

Value-regularization & pessimism

Conservative \(Q\)-learning

Question: How can we achieve this?

The idea behind CQL

The CQL(\(H\)) framework (Kumar et al. 2020)

Algorithmic implementation of CQL(\(H\))

Example: Complex gripping tasks

Implicit \(Q\)-learning

What we have seen until now (and why we need something better)

Explicit Policy Constraints (e.g., BEAR (Kumar et al. 2019) or BRAC (Wu, Tucker, and Nachum 2019))

Conservative Q-Learning (CQL, e.g., (Kumar et al. 2020))

The motivating question behind implicit \(Q\)-learning

What if we never evaluate an out-of-distribution action during training at all?

Advantage-weighted regression

Let’s consider the chess example

Advanted-weighted regression (AWR, (Peng et al. 2019)):

Mathematical derivation of AWR

Remember trust region policy optmization (TRPO)?

The starting point of AWR

Solution to the AWR problem

The AWR algorithm

Implicit \(Q\)-learning (IQL)

Limitations of AWR

Proposed changes in implicit \(Q\)-learning (IQL (Kostrikov, Nair, and Levine 2021))

Expectile regression loss

The IQL algorithm

Implicit \(Q\)-learning (IQL (Kostrikov, Nair, and Levine 2021))

Some results

Summary / what we have learned

References