Markov Decision Processes

Course Syllabus
October – January 2020
Timings: Tue and Thu 9:30 AM - 11:00 AM
Location: Online

This page contains lecture slides and assignments for ‘CE 273 Markov Decision Processes’. Codes for the programming project and LaTeX resources will also be posted here.


  1. Introduction

  2. Transient Behavior and Classification of States

  3. Limiting Behavior of DTMCs

  4. Finite Horizon MDPs

  5. Applications of Finite Horizon MDPs - Part I

  6. Applications of Finite Horizon MDPs - Part II

  7. Infinite Horizon Discounted MDPs

  8. Value Iteration

  9. Policy Iteration

  10. Linear Programming Methods

  11. Infinite Horizon Total Cost MDPs

  12. Solution Methods for Total Cost MDPs

  13. Infinite Horizon Average Cost MDPs

  14. Optimality Conditions and Classification of Average Cost MDPs

  15. Value Iteration for Average Cost MDPs

  16. Policy Iteration and Linear Programming for Average Cost MDPs

  17. Introduction to Approximate Dynamic Programming

  18. Approximation in Value Space - Part I

  19. Approximation in Value Space - Part II

  20. Q-learning and Approximate Linear Programming

  21. Approximation in Policy Space

  22. Dynamic Discrete Choice Models

  23. Inverse Reinforcement Learning

  24. Semi-Markov Decision Processes

  25. POMDPs and Risk-Sensitive MDPs

  26. Continuous-Time Control



Python Toolbox