Prerequisite: IE 611 or equivalent Contents: 

Apart from their intrinsic role in the theory of stochastic processes, Markov chains and regenerativeprocesses form an important set of tools for analysis and optimization problems arising in many decision models. Measure theoretic ideas of probability, expectation, convergence of random variables, limit theorems. Discrete time countable state Markov chains, hitting times, stopping times and strong Markovproperty. Recurrence and transience. Invariant measures for irreducible chains, ergodic theorem.Convergence in variation and coupling lemma. Absorption probabilities and criterion for transience.Discrete time renewal theory, elementary renewal theorem and renewal reward theorem.Regenerative processes and their time averages. Jump processes; jump chain and sojourn time construction of continuous time MCs, Poissonprocesses, birthdeath processes. Forward and backward equations. Class structure, recurrence andtransience, invariant distributions, convergence to equilibrium. Uniformization and time reversedchains. Optional: Martingales: conditional expectation, Martingale inequalities, Martingale convergencetheorem, Brownian motion. 

