Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration

Mon, 15 Jan 2024 00:00:00 +0000

Mathematical Formulation

Policy Distillation Objective

The core optimization problem is formulated as:

$$ \min_{\pi_{\theta}} \mathbb{E}_{s \sim d_{\text{expert}}} \left[ L_{\text{distill}}(\pi_{\theta}(s), \pi_{\text{expert}}(s)) + \lambda L_{\text{safety}}(\pi_{\theta}) \right] $$

Where:

$\pi_{\theta}$: Online policy network
$\pi_{\text{expert}}$: Offline expert policy
$L_{\text{distill}}$: Policy distillation loss
$L_{\text{safety}}$: Safety constraint loss
$\lambda$: Regularization coefficient

Safe Exploration Constraint

The safety constraint is defined as:

$$ P(\text{safe trajectory}) \geq 1 - \epsilon $$

Technical Approach

Key Components

Two-stage policy extraction
Offline expert policy distillation
Online fine-tuning with safety constraints

Performance Metrics

Reduced collision rate from 30% to <3%
15% improvement in driving task success rates

Research Significance

Addresses critical reinforcement learning challenges:

Mitigates conservative exploration
Enables lightweight policy networks
Improves safety in autonomous systems

Implementation Highlights

Developed versatile driving simulation platform
Implemented online distillation procedure
Extracted lightweight policies from transformer models

Developed at UC Berkeley’s MSC Lab under Prof. Masayoshi Tomizuka

Autonomous Driving | Xinyi Liu's Personal Website