IE712: Selected Application of Stochastic Models, July-November 2016
Aim of this course is to learn modelling and anaylsis of engineering problems using theory of stochastic processes.
Lecture Hours
Tuesday 3.35-5pm
Friday 3.35-5pm
Location
LT205
Teaching Assistants
Arko Chatterjee
Shashank Mishra
Syllabus
Markov Decision Process: introduction to Markov chains, finite and infinite horizon models, optimality criteria,
value iteration and policy iteration algorithms, examples from inventory systems, resource allocation, and path planning.
Markov Games: introduction to stochastic Games, discounted and average reward stochastic games, Shapley theorem,
computation of equilibria via linear programming.
Reinforcement learning: approximating cost-to-go function, Temporal Difference learning methods (TD),
Least Square Temporal Difference methods (LSTD), Q-learning.
Queuing Models: birth death process, stability of queues, Little's law, Priority queues,
single class queuing networks,
multi-class queuing networks.
Homeworks
Homework 1
Homework 2
Homework 3
A short project
Course Grades
25 points: Homework and Quizzes
20 points: Midterm exam
15 points: Project
40 points: Final exam
Reference texts
Anurag Kumar, Discrete event stochastic processes and quueing systmes,
available here.
Bruce Hajek, An exploration of random processes for engineers,
available here.
Dimitri Bertsekas, Dynamic Programming and Optimal Control: Vol I and II.
Dimitri Bertsekas, Neuro-dynamic programming
L. Kleinrock, Queueing Systems: Vol I and II.
Jerzy Filar and Koos Vrieze, Competitive Markov Decision Processes.
Moshe Haviv and Refael Hassin, To Queue Or Not to Queue: Equilibrium Behavior in Queueing Systems.
Class notes
Lecture 1 Motivation
Lecture 2 Review of basic probability
Lecture 3 Convergence of stochastic processes
Lecture 4 Markov chains
Lecture 5 Finite horizon MDPs
Lecture 6 Priciple of optimality and Dynamic programming
Lecture 7 Continuous time optimal control
Lecture 8 Infinite horizon MDPs
Lecture 9 Infinite horizon MDPs
Lecture 10Infinite horizon MDPs
Lecture 11 Infinite horizon MDPs
Lecture 12 Competitive MDPs
Lecture 13 Competitive MDPs
Lecture 14 Competitive MDPs
Lecture 15 Reinformcement Learning
Lecture 16 Reinformcement Learning
Lecture 17 Reinformcement Learning
Lecture 18 Reinformcement Learning
Lecture 19 Reinformcement Learning
Lecture 20 Reinformcement Learning
Lecture 21 Reinformcement Learning
Lecture 22 Reinformcement Learning
Lecture 23 Reinformcement Learning
Lecture 24 Continuous chain Markov chains
Lecture 25 Continuous chain Markov chains
Lecture 26 Continuous chain Markov chains