Linear feature encoding for reinforcement learning

Published

Conference Paper

© 2016 NIPS Foundation - All Rights Reserved. Feature construction is of vital importance in reinforcement learning, as the quality of a value function or policy is largely determined by the corresponding features. The recent successes of deep reinforcement learning (RL) only increase the importance of understanding feature construction. Typical deep RL approaches use a linear output layer, which means that deep RL can be interpreted as a feature construction/encoding network followed by linear value function approximation. This paper develops and evaluates a theory of linear feature encoding. We extend theoretical results on feature quality for linear value function approximation from the uncontrolled case to the controlled case. We then develop a supervised linear feature encoding method that is motivated by insights from linear value function approximation theory, as well as empirical successes from deep RL. The resulting encoder is a surprisingly effective method for linear value function approximation using raw images as inputs.

Duke Authors

Cited Authors

  • Song, Z; Parr, R; Liao, X; Carin, L

Published Date

  • January 1, 2016

Published In

Start / End Page

  • 4231 - 4239

International Standard Serial Number (ISSN)

  • 1049-5258

Citation Source

  • Scopus