IE 622: Probability and Stochastic Processes II

Prerequisite:  IE 611 or equivalent


Apart from their intrinsic role in the theory of stochastic processes, Markov chains and regenerativeprocesses form an important set of tools for analysis and optimization problems arising in many decision models.

Measure theoretic ideas of probability, expectation, convergence of random variables, limit theorems.

Discrete time countable state Markov chains, hitting times, stopping times and strong Markovproperty. Recurrence and transience. Invariant measures for irreducible chains, ergodic theorem.Convergence in variation and coupling lemma. Absorption probabilities and criterion for transience.Discrete time renewal theory, elementary renewal theorem and renewal reward theorem.Regenerative processes and their time averages.

Jump processes; jump chain and sojourn time construction of continuous time MCs, Poissonprocesses, birth-death processes. Forward and backward equations. Class structure, recurrence andtransience, invariant distributions, convergence to equilibrium. Uniformization and time reversedchains.

Optional: Martingales: conditional expectation, Martingale inequalities, Martingale convergencetheorem, Brownian motion.

  • P. Bremaud, Markov chains, Gibbs fields, Monte Carlo simulation and queues, 2001,Springer-Verlag, New York
  • E. Cinlar, Introduction to stochastic processes, 1975, Prentice Hall Inc., Englewood Cliffs
  • W. Feller, Introduction to probability theory and its applications, Volumes 1 and 2, (1975,1966) John Wiley, New York.
  • G. R. Grimmett and D. R. Stirzaker, Probability and random processes, 3rd edition, 2001,Oxford University Press
  • J. R. Norris, Markov chains, 1999, Cambridge University Press, Cambridge
  • S. M. Ross, Stochastic processes, 2nd Edition, 1996, John Wiley, New York
  • S. Resnick, Adventures in stochastic processes, 1992, Birkhauser Verlag, Boston
  • D. W. Stroock, An introduction to Markov processes, 2005, Springer-Verlag, Berlin
  • R. W. Wolff, Stochastic modelling and the theory of queues, 1989, Prentice Hall Inc.,Englewood Cliffs
  • JeanJacod and Philip Protter,Probability Essentials,Springer,2004.