Learning Pessimism for Reinforcement Learning

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

132 Downloads (Pure)

Abstract

Off-policy deep reinforcement learning algorithms commonly compensate for overestimation bias during temporal-difference learning by utilizing pessimistic estimates of the expected target returns. In this work, we propose Generalized Pessimism Learning (GPL), a strategy employing a novel learnable penalty to enact such pessimism. In particular, we propose to learn this penalty alongside the critic with dual TD-learning, a new procedure to estimate and minimize the magnitude of the target returns bias with trivial computational cost. GPL enables us to accurately counteract overestimation bias throughout training without incurring the downsides of overly pessimistic targets. By integrating GPL with popular off-policy algorithms, we achieve state-of-the-art results in both competitive proprioceptive and pixel-based benchmarks.
Original languageEnglish
Title of host publicationThe 37th AAAI Conference on Artificial Intelligence (AAAI-23)
PublisherAAAI Press
Publication statusAccepted/In press - 2023

Fingerprint

Dive into the research topics of 'Learning Pessimism for Reinforcement Learning'. Together they form a unique fingerprint.

Cite this