An online policy gradient algorithm for Markov decision processes with continuous states and actions

Yao Ma, Tingting Zhao, Kohei Hatano, Masashi Sugiyama

Research output: Contribution to journalLetter

1 Citation (Scopus)

Abstract

We consider the learning problem under an online Markov decision process (MDP) aimed at learning the time-dependent decision-making policy of an agent that minimizes the regret - the difference from the best fixed policy. The difficulty of online MDP learning is that the reward function changes over time. In this letter, we show that a simple online policy gradient algorithm achieves regret for T steps under a certain concavity assumption and under a strong concavity assumption. To the best of our knowledge, this is the first work to present an online MDP algorithm that can handle continuous state, action, and parameter spaces with guarantee. We also illustrate the behavior of the proposed online policy gradient method through experiments.

Original languageEnglish
Pages (from-to)563-593
Number of pages31
JournalNeural Computation
Volume28
Issue number3
DOIs
Publication statusPublished - Mar 1 2016

All Science Journal Classification (ASJC) codes

  • Arts and Humanities (miscellaneous)
  • Cognitive Neuroscience

Fingerprint Dive into the research topics of 'An online policy gradient algorithm for Markov decision processes with continuous states and actions'. Together they form a unique fingerprint.

  • Cite this