Skip to main content

IE 622: Probability and Stochastic Processes II

Prerequisite:  IE 621 or equivalent


Apart from their intrinsic role in the theory of stochastic processes, Markov chains and regenerative processes form an important set of tools for analysis and optimization problems arising in many decision models.

Measure theoretic ideas of probability, expectation, convergence of random variables, limit theorems.

Discrete time countable state Markov chains, hitting times, stopping times and strong Markov property. Recurrence and transience. Invariant measures for irreducible chains, ergodic theorem. Convergence in variation and coupling lemma. Absorption probabilities and criterion for transience. Discrete time renewal theory, elementary renewal theorem and renewal reward theorem. Regenerative processes and their time averages.

Jump processes; jump chain and sojourn time construction of continuous time MCs, Poisson processes, birth-death processes. Forward and backward equations. Class structure, recurrence and transience, invariant distributions, convergence to equilibrium. Uniformization and time reversed chains.

Optional: Martingales: conditional expectation, Martingale inequalities, Martingale convergence theorem, Brownian motion.

  • P. Bremaud, Markov chains, Gibbs fields, Monte Carlo simulation and queues, 2001, Springer-Verlag, New York
  • E. Cinlar, Introduction to stochastic processes, 1975, Prentice Hall Inc., Englewood Cliffs
  • W. Feller, Introduction to probability theory and its applications, Volumes 1 and 2, (1975,1966) John Wiley, New York.
  • G. R. Grimmett and D. R. Stirzaker, Probability and random processes, 3rd edition, 2001,Oxford University Press
  • J. R. Norris, Markov chains, 1999, Cambridge University Press, Cambridge
  • S. M. Ross, Stochastic processes, 2nd Edition, 1996, John Wiley, New York
  • S. Resnick, Adventures in stochastic processes, 1992, Birkhauser Verlag, Boston
  • D. W. Stroock, An introduction to Markov processes, 2005, Springer-Verlag, Berlin
  • R. W. Wolff, Stochastic modelling and the theory of queues, 1989, Prentice Hall Inc., Englewood Cliffs
  • Jean Jacod and Philip Protter, Probability Essentials, Springer, 2004.