Markov Decision Processes

This page contains lecture slides and assignments for ‘CE 273 Markov Decision Processes’. Codes for the programming project and LaTeX resources will also be posted here.

Lectures

1. Introduction
   

2. Transient Behavior and Classification of States
   

3. Limiting Behavior of DTMCs
   

4. Finite Horizon MDPs
   

5. Applications of Finite Horizon MDPs - Part I
   

6. Applications of Finite Horizon MDPs - Part II
   

7. Infinite Horizon Discounted MDPs
   

8. Value Iteration
   

9. Policy Iteration
   

10. Linear Programming Methods
    

11. Infinite Horizon Total Cost MDPs
    

12. Solution Methods for Total Cost MDPs
    

13. Infinite Horizon Average Cost MDPs
    

14. Optimality Conditions and Classification of Average Cost MDPs
    

15. Value Iteration for Average Cost MDPs
    

16. Policy Iteration and Linear Programming for Average Cost MDPs
    

17. Introduction to Approximate Dynamic Programming
    

18. Approximation in Value Space - Part I
    

19. Approximation in Value Space - Part II
    

20. Q-learning and Approximate Linear Programming
    

21. Approximation in Policy Space
    

22. Dynamic Discrete Choice Models
    

23. Inverse Reinforcement Learning
    

24. Semi-Markov Decision Processes
    

25. POMDPs and Risk-Sensitive MDPs
    

26. Continuous-Time Control
    

Resources

LaTeX

https://en.wikibooks.org/wiki/LaTeX
https://www.overleaf.com/

Python Toolbox

http://pymdptoolbox.readthedocs.io/en/latest/api/mdptoolbox.html