ABOUT THE COURSE:This course introduces fundamental concepts in stochastic processes which arises in the study of behavior of systems that appear commonly in electrical engineering, computer science, and applied mathematics, and that evolve randomly over time. The course topics include a quick review of foundational probability with a keen focus on conditional probability, discrete and continuous-time Markov chains, discrete and continuous time counting processes like Bernoulli and Poisson processes and their derivatives, renewal theory and the reward theorem, and Brownian motion. The course will emphasize both theory and applications of these concepts. Applications are drawn from a diverse set of areas such as queueing systems, communication networks, and event-driven modeling.Key topics: Probability distributions, conditional expectation, discrete and continuous-time Markov chains, discrete and continuous time counting processes including Bernoulli and Poisson processes, and their derivatives, renewal theory, Brownian motion, applications: stochastic modeling in queues and networks.INTENDED AUDIENCE: Senior undergraduate and postgraduate students in Mathematics, ECE, EE, CSEPREREQUISITES: Introductory course on Probability and Statistics

Syllabus

Week 1: Preliminaries I: Introduction to probability, review of distributions: Bernoulli, Binomial, Poisson, Multinomial, Exponential, Gamma, Gaussian distribution
Week 2:Preliminaries II: Conditional probability, Conditional expectation and variance, Computations with conditioning, Central limit theorem, Software demonstration of simulating discrete and continuous random variables
Week 3:DTMC I: Discrete time stochastic processes, Discrete time Markov chains, transition probabilities, Chapman-Kolmogorov equations, classification of states, Software demonstration of the concepts
Week 4:DTMC II: Limiting probabilities, Connection to Perron Frobenius Theorem, Mean time spent in transient states, Branching processes, Time reversible Markov chains, Applications
Week 5:Discrete time counting processes: Bernoulli random processes, definitions and alternate synthesis approaches via interarrivals, properties, operating on Bernoulli processes like merging and splitting, Applications to simple discrete-time queues
Week 6:Continuous time counting processes: Poisson processes, Interarrival and waiting time distributions, merging and splitting operations, order statistics, conditional distribution on arrival times, marked and compound Poisson processes, Applications
Week 7:CTMC I: Birth and Death processes, Transition probability function.
Week 8:CTMC II : Kolmogorov’s backward and forward equations, Limiting probabilities, Applications
Week 9:Renewal : definitions, examples, Limit theorems, renewal reward, Applications to reliability
Week 10:Applications to Queuing theory: basic definitions of queues and Kendall notation, analysis of M/M/X/X queues
Week 11:Martingale and Brownian motion: definition, connections to other processes, Hitting times, Gambler’s ruin problem, Brownian motion with drift, Geometric Brownian motion, White noise, Gaussian processes, Stationary and weak stationary processes
Week 12:Applications: Option pricing, risk neutral pricing, Arbitrage theorem, Black Scholes option pricing formula