Prerequisite: IE 621 or equivalent Contents: |
||
Apart from their intrinsic role in the theory of stochastic processes, Markov chains and regenerative processes form an important set of tools for analysis and optimization problems arising in many decision models. Measure theoretic ideas of probability, expectation, convergence of random variables, limit theorems. Discrete time countable state Markov chains, hitting times, stopping times and strong Markov property. Recurrence and transience. Invariant measures for irreducible chains, ergodic theorem. Convergence in variation and coupling lemma. Absorption probabilities and criterion for transience. Discrete time renewal theory, elementary renewal theorem and renewal reward theorem. Regenerative processes and their time averages. Jump processes; jump chain and sojourn time construction of continuous time MCs, Poisson processes, birth-death processes. Forward and backward equations. Class structure, recurrence and transience, invariant distributions, convergence to equilibrium. Uniformization and time reversed chains. Optional: Martingales: conditional expectation, Martingale inequalities, Martingale convergence theorem, Brownian motion. |
||
|