Textbook in PDF format

This text presents selected applications of discrete-time stochastic processes that involve random interactions and algorithms, and revolve around the Markov property. It covers recurrence properties of (excited) random walks, convergence and mixing of Markov chains, distribution modeling using phase-type distributions, applications to search engines and probabilistic automata, and an introduction to the Ising model used in statistical physics. Applications to data science are also considered via hidden Markov models and Markov decision processes. A total of 32 exercises and 17 longer problems are provided with detailed solutions and cover various topics of interest, including statistical learning. Preface List of Figures A Summary of Markov Chains Phase-Type Distributions Synchronizing Automata Random Walks and Recurrence Cookie-Excited Random Walks Convergence to Equilibrium The Ising Model Search Engines Hidden Markov Model Markov Decision Processes Notes Exercises A Probability Generating Functions A.1 Probability Generating Functions Some Properties of Probability Generating Functions B Some Useful Identities References Index Author Index