IE712: Selected Application of Stochastic Models, July-November 2016

Aim of this course is to learn modelling and anaylsis of engineering problems using theory of stochastic processes.

Lecture Hours

Tuesday 3.35-5pm
Friday 3.35-5pm



Teaching Assistants

Arko Chatterjee
Shashank Mishra


Markov Decision Process: introduction to Markov chains, finite and infinite horizon models, optimality criteria,
value iteration and policy iteration algorithms, examples from inventory systems, resource allocation, and path planning.

Markov Games: introduction to stochastic Games, discounted and average reward stochastic games, Shapley theorem,
computation of equilibria via linear programming.

Reinforcement learning: approximating cost-to-go function, Temporal Difference learning methods (TD),
Least Square Temporal Difference methods (LSTD), Q-learning.

Queuing Models: birth death process, stability of queues, Little's law, Priority queues,
single class queuing networks, multi-class queuing networks.


Homework 1
Homework 2
Homework 3
A short project

Course Grades

25 points: Homework and Quizzes
20 points: Midterm exam
15 points: Project
40 points: Final exam

Reference texts

  • Anurag Kumar, Discrete event stochastic processes and quueing systmes, available here.
  • Bruce Hajek, An exploration of random processes for engineers, available here.
  • Dimitri Bertsekas, Dynamic Programming and Optimal Control: Vol I and II.
  • Dimitri Bertsekas, Neuro-dynamic programming
  • L. Kleinrock, Queueing Systems: Vol I and II.
  • Jerzy Filar and Koos Vrieze, Competitive Markov Decision Processes.
  • Moshe Haviv and Refael Hassin, To Queue Or Not to Queue: Equilibrium Behavior in Queueing Systems.
  • Class notes

    Lecture 1 Motivation
    Lecture 2 Review of basic probability
    Lecture 3 Convergence of stochastic processes
    Lecture 4 Markov chains
    Lecture 5 Finite horizon MDPs
    Lecture 6 Priciple of optimality and Dynamic programming
    Lecture 7 Continuous time optimal control
    Lecture 8 Infinite horizon MDPs
    Lecture 9 Infinite horizon MDPs
    Lecture 10Infinite horizon MDPs
    Lecture 11 Infinite horizon MDPs
    Lecture 12 Competitive MDPs
    Lecture 13 Competitive MDPs
    Lecture 14 Competitive MDPs
    Lecture 15 Reinformcement Learning
    Lecture 16 Reinformcement Learning
    Lecture 17 Reinformcement Learning
    Lecture 18 Reinformcement Learning
    Lecture 19 Reinformcement Learning
    Lecture 20 Reinformcement Learning
    Lecture 21 Reinformcement Learning
    Lecture 22 Reinformcement Learning
    Lecture 23 Reinformcement Learning
    Lecture 24 Continuous chain Markov chains
    Lecture 25 Continuous chain Markov chains
    Lecture 26 Continuous chain Markov chains