%{search_type} search results

Normalizing flows, diffusion normalizing flows and variational autoencoders are powerful generative models. This Element provides a unified framework to handle these approaches via Markov chains. The authors consider stochastic normalizing flows as a pair of Markov chains fulfilling some properties, and show how many state-of-the-art models for data generation fit into this framework. Indeed numerical simulations show that including stochastic layers improves the expressivity of the network and allows for generating multimodal distributions from unimodal ones. The Markov chains point of view enables the coupling of both deterministic layers as invertible neural networks and stochastic layers as Metropolis-Hasting layers, Langevin layers, variational autoencoders and diffusion normalizing flows in a mathematically sound way. The authors' framework establishes a useful mathematical tool to combine the various approaches.

• Cover
• Title page
• Copyright page
• Contents
• Chapter Twelve
• Excessive, superharmonic and harmonic functions
• 1. Excessive functions for transition functions
• 2. Excessive functions for Markov processes
• 3. Asymptotic behavior of excessive functionsalong trajectories of a process
• 4. Superharmonie functions
• 5. Harmonie functions
• Chapter Thirteen
• Harmonie and superharmonic functions associated withstrong Feller processes. Probabilistic solution of certainequations
• 1. Some properties of strong Feller processes

• Continuous strong Markov processes on a closed interval
• 1. General properties of one-dimensional continuous strong Markov processes
• 2. Characteristics of regular processes
• 3. Computation of the characteristic and infinitesimal operators
• 4. Superharmonie and harmonic functionsconnected with regular one-dimensional processes
• Chapter Sixteen
• Continuous strong Markov processes on an open interval
• 1. Harmonie functions and behavior of trajectories
• 2. S-functions and character of the motion along a trajectory
• 3. Infinitesimal operators
• Chapter Seventeen

• Construction of one-dimensional continuous strong Markov processes
• 1. Transformations of the state space. Canonical coordinate
• 2. Construction of regular continuous strong Markov processeson an open interval
• 3. Construction of regular continuous strong Markov processeson a closed interval
• 4. Computation of the harmonic functions and resolvents for regular processes
• Appendix
• 1. Measurable spaces and measurable transformations
• 2. Measures and integrals
• 3. Probability spaces. Conditional probabilities and mathematical expectations
• 4. Martingales

• 5. Topological measurable spaces
• 6. Some theorems on partial differential equations
• 7. Measures and countably additive set functions on the line and corresponding point functions
• 8. Convex functions
• Historical
• Bibliographical note
• Bibliography
• Index
• List of symbols

• 2. The Dirichlet problems. Regular points of the boundary
• 3. Harmonie and superharmonic functionsassociated with diffusion processes
• 4. Solutions of the equation ztf- Vf= -g
• 5. Parts of a diffusion process and Green's functions
• Chapter Fourteen
• The multi-dimensional Wiener processand its transformations
• 1. Harmonie and superharmonic functionsrelated to the Wiener process
• 2. The mapping 'l'
• 3. Additive functionals and Green's functions
• 4. Brownian motion with killing measure μ and speed measure v
• 5. q-subprocesses
• Chapter Fifteen

• 1. Introduction to stochastic growth processes
• A. Growth processes in economics, biology and ecology
• B. Stochastic growth with multiplicative noise
• C. Stochastic growth with Markovian dependence
• 2. Stochastic growth processes with exponential growth rates
• D. Exponential growth model driven by a geometric Brownian motion
• E. Exponential growth model with binomial tree growth rates
• F. Exp-Ornstein-Uhlenbeck model
• 3. Lyapunov exponents of the exponential stochastic growth processes
• G. Numerical illustration: the bank account in the Black-Derman-Toy model
• H. Lyapunov exponents and their analyticity
• I.
• Lattice gas analogy
• J.
• Recursion relation for the moments
• 4. One-dimensional lattice gas models with linear attractive interaction
• K. Lattice gases with mutual exclusion
• L. Lattice gas with universal interaction-- mean-field theory
• M. Kac potentials and the Lebowitz-Penrose theory
• N. One-dimensional lattice gas with linear attractive interaction: exact results
• 5. Lattice gas with exponential attractive interactions
• O. Connection to the Black-Karasinski model
• P. Exact result for the Lyapunov exponents
• Q. Limiting case: Kac-Helfand lattice gas and the van der Waals theory
• 6. Applications
• R. Monte Carlo simulation of stochastic volatility models
• S. Asymptotic bond pricing in the Black-Derman-Toy model
• .
• (source: Nielsen Book Data)

The book discusses a class of discrete time stochastic growth processes for which the growth rate is proportional to the exponential of a Gaussian Markov process. These growth processes appear naturally in problems of mathematical finance as discrete time approximations of stochastic volatility models and stochastic interest rates models such as the Black-Derman-Toy and Black-Karasinski models. These processes can be mapped to interacting one-dimensional lattice gases with long-range interactions. The book gives a detailed discussion of these statistical mechanics models, including new results not available in the literature, and their implication for the stochastic growth models. The statistical mechanics analogy is used to understand observed non-analytic dependence of the Lyapunov exponents of the stochastic growth processes considered, which is related to phase transitions in the lattice gas system. The theoretical results are applied to simulations of financial models and are illustrated with Mathematica code. The book includes a general introduction to exponential stochastic growth with examples from biology, population dynamics and finance. The presentation does not assume knowledge of mathematical finance. The new results on lattice gases can be read independently of the rest of the book. The book should be useful to practitioners and academics studying the simulation and application of stochastic growth models.
(source: Nielsen Book Data)

• A non-technical introduction
• 1. A guided tour through the land of operator semigroups
• 2. Generators versus intensity matrices
• 3. Boundary theory: core results
• 4. Boundary theory continued
• 5. The dual perspective
• Solutions and hints to selected exercises
• Commonly used notations
• References
• Index.
• (source: Nielsen Book Data)

Elementary treatments of Markov chains, especially those devoted to discrete-time and finite state-space theory, leave the impression that everything is smooth and easy to understand. This exposition of the works of Kolmogorov, Feller, Chung, Kato, and other mathematical luminaries, which focuses on time-continuous chains but is not so far from being elementary itself, reminds us again that the impression is false: an infinite, but denumerable, state-space is where the fun begins. If you have not heard of Blackwell's example (in which all states are instantaneous), do not understand what the minimal process is, or do not know what happens after explosion, dive right in. But beware lest you are enchanted: 'There are more spells than your commonplace magicians ever dreamed of.'.
(source: Nielsen Book Data)

• Introduction.- Classical Markov chains.- Quantum Markov chains.- Outline.- Preliminaries.- Notation.- Schatten norms.- Functions on Hermitian operators.- Quantum channels.- Entropy measures.- Background and further reading.- Tools for non-commuting operators.- Pinching.- Complex interpolation theory.- Background and further reading.- Multivariate trace inequalities.- Motivation.- Multivariate Araki-Lieb-Thirring inequality.- Multivariate Golden-Thompson inequality.- Multivariate logarithmic trace inequality.- Background and further reading.- Approximate quantum Markov chains.- Quantum Markov chains.- Sufficient criterion for approximate recoverability.- Necessary criterion for approximate recoverability.- Strengthened entropy inequalities.- Background and further reading.- A A large conditional mutual information does not imply bad recovery.- B Example showing the optimality of the Lmax-term.- C Solutions to exercises.- References.- Index.
• (source: Nielsen Book Data)

This book is an introduction to quantum Markov chains and explains how this concept is connected to the question of how well a lost quantum mechanical system can be recovered from a correlated subsystem. To achieve this goal, we strengthen the data-processing inequality such that it reveals a statement about the reconstruction of lost information. The main difficulty in order to understand the behavior of quantum Markov chains arises from the fact that quantum mechanical operators do not commute in general. As a result we start by explaining two techniques of how to deal with non-commuting matrices: the spectral pinching method and complex interpolation theory. Once the reader is familiar with these techniques a novel inequality is presented that extends the celebrated Golden-Thompson inequality to arbitrarily many matrices. This inequality is the key ingredient in understanding approximate quantum Markov chains and it answers a question from matrix analysis that was open since 1973, i.e., if Lieb's triple matrix inequality can be extended to more than three matrices. Finally, we carefully discuss the properties of approximate quantum Markov chains and their implications. The book is aimed to graduate students who want to learn about approximate quantum Markov chains as well as more experienced scientists who want to enter this field. Mathematical majority is necessary, but no prior knowledge of quantum mechanics is required.
(source: Nielsen Book Data)

• Part I Foundations.- Markov Chains: Basic Definitions.- Examples of Markov Chains.- Stopping Times and the Strong Markov Property.- Martingales, Harmonic Functions and Polsson-Dirichlet Problems.- Ergodic Theory for Markov Chains.- Part II Irreducible Chains: Basics.- Atomic Chains.- Markov Chains on a Discrete State Space.- Convergence of Atomic Markov Chains.- Small Sets, Irreducibility and Aperiodicity.- Transience, Recurrence and Harris Recurrence.- Splitting Construction and Invariant Measures.- Feller and T-kernels.- Part III Irreducible Chains: Advanced Topics.- Rates of Convergence for Atomic Markov Chains.- Geometric Recurrence and Regularity.- Geometric Rates of Convergence.- (f, r)-recurrence and Regularity.- Subgeometric Rates of Convergence.- Uniform and V-geometric Ergodicity by Operator Methods.- Coupling for Irreducible Kernels.- Part IV Selected Topics.- Convergence in the Wasserstein Distance.- Central Limit Theorems.- Spectral Theory.- Concentration Inequalities.- Appendices.- A Notations.- B Topology, Measure, and Probability.- C Weak Convergence.- D Total and V-total Variation Distances.- E Martingales.- F Mixing Coefficients.- G Solutions to Selected Exercises.
• (source: Nielsen Book Data)

This book covers the classical theory of Markov chains on general state-spaces as well as many recent developments. The theoretical results are illustrated by simple examples, many of which are taken from Markov Chain Monte Carlo methods. The book is self-contained, while all the results are carefully and concisely proven. Bibliographical notes are added at the end of each chapter to provide an overview of the literature. Part I lays the foundations of the theory of Markov chain on general states-space. Part II covers the basic theory of irreducible Markov chains on general states-space, relying heavily on regeneration techniques. These two parts can serve as a text on general state-space applied Markov chain theory. Although the choice of topics is quite different from what is usually covered, where most of the emphasis is put on countable state space, a graduate student should be able to read almost all these developments without any mathematical background deeper than that needed to study countable state space (very little measure theory is required). Part III covers advanced topics on the theory of irreducible Markov chains. The emphasis is on geometric and subgeometric convergence rates and also on computable bounds. Some results appeared for a first time in a book and others are original. Part IV are selected topics on Markov chains, covering mostly hot recent developments.
(source: Nielsen Book Data)

• Probability Background
• Gambling Problems
• Random Walks
• Discrete-Time Markov Chains
• First Step Analysis
• Classification of States
• Long-Run Behavior of Markov Chains
• Branching Processes
• Continuous-Time Markov Chains
• Discrete-Time Martingales
• Spatial Poisson Processes
• Reliability Theory.

This book provides an undergraduate-level introduction to discrete and continuous-time Markov chains and their applications, with a particular focus on the first step analysis technique and its applications to average hitting times and ruin probabilities. It also discusses classical topics such as recurrence and transience, stationary and limiting distributions, as well as branching processes. It first examines in detail two important examples (gambling processes and random walks) before presenting the general theory itself in the subsequent chapters. It also provides an introduction to discrete-time martingales and their relation to ruin probabilities and mean exit times, together with a chapter on spatial Poisson processes. The concepts presented are illustrated by examples, 138 exercises and 9 problems with their solutions.-- Provided by publisher.

• Laurent Asymptotic Expansions.- Asymptotic Expansions for Moments of Hitting Times for Nonlinearly Perturbed Semi-Markov Processes.- Asymptotic Expansions for Stationary Distributions of Nonlinearly Perturbed Semi-Markov Processes.- Nonlinearly Perturbed Birth-Death-Type Semi-Markov Processes.- Examples and Survey of Applied Perturbed Stochastic Models.- A. Methodological and Bibliographical Remarks.
• (source: Nielsen Book Data)

The book presents new methods of asymptotic analysis for nonlinearly perturbed semi-Markov processes with a finite phase space. These methods are based on special time-space screening procedures for sequential phase space reduction of semi-Markov processes combined with the systematical use of operational calculus for Laurent asymptotic expansions. Effective recurrent algorithms are composed for getting asymptotic expansions, without and with explicit upper bounds for remainders, for power moments of hitting times, stationary and conditional quasi-stationary distributions for nonlinearly perturbed semi-Markov processes. These results are illustrated by asymptotic expansions for birth-death-type semi-Markov processes, which play an important role in various applications. The book will be a useful contribution to the continuing intensive studies in the area. It is an essential reference for theoretical and applied researchers in the field of stochastic processes and their applications that will contribute to continuing extensive studies in the area and remain relevant for years to come. .
(source: Nielsen Book Data)

• 1 Biological Models.- 2 Markov Processes.- 3 Operator Semigroups.- 4 Stochastic Semigroups.- 5 Asymptotic Properties of Stochastic Semigroups - General Results.- 6 Asymptotic Properties of Stochastic Semigroups - Applications.
• (source: Nielsen Book Data)

This book presents a concise introduction to piecewise deterministic Markov processes (PDMPs), with particular emphasis on their applications to biological models. Further, it presents examples of biological phenomena, such as gene activity and population growth, where different types of PDMPs appear: continuous time Markov chains, deterministic processes with jumps, processes with switching dynamics, and point processes. Subsequent chapters present the necessary tools from the theory of stochastic processes and semigroups of linear operators, as well as theoretical results concerning the long-time behaviour of stochastic semigroups induced by PDMPs and their applications to biological models. As such, the book offers a valuable resource for mathematicians and biologists alike. The first group will find new biological models that lead to interesting and often new mathematical questions, while the second can observe how to include seemingly disparate biological proc esses into a unified mathematical theory, and to arrive at revealing biological conclusions. The target audience primarily comprises of researchers in these two fields, but the book will also benefit graduate students.
(source: Nielsen Book Data)

• Review of Probability Short History Review of Basic Probability Definitions Some Common Probability Distributions Properties of a Probability Distribution Properties of the Expected Value Expected Value of a Random Variable with Common Distributions Generating Functions Moment Generating Functions Exercises Discrete-Time, Finite-State Markov Chains Introduction Notation Transition Matrices Directed Graphs: Examples of Markov Chains Random Walk with Reflecting Boundaries Gamblera (TM)s Ruin Ehrenfest Model Central Problem of Markov Chains Condition to Ensure a Unique Equilibrium State Finding the Equilibrium State Transient and Recurrent States Indicator Functions Perron-Frobenius Theorem Absorbing Markov Chains Mean First Passage Time Mean Recurrence Time and the Equilibrium State Fundamental Matrix for Regular Markov Chains Dividing a Markov Chain into Equivalence Classes Periodic Markov Chains Reducible Markov Chains Summary Exercises Discrete-Time, Infinite-State Markov Chains Renewal Processes Delayed Renewal Processes Equilibrium State for Countable Markov Chains Physical Interpretation of the Equilibrium State Null Recurrent versus Positive Recurrent States Difference Equations Branching Processes Random Walk in Exercises Exponential Distribution and Poisson Process Continuous Random Variables Cumulative Distribution Function (Continuous Case) Exponential Distribution o(h) Functions Exponential Distribution as a Model for Arrivals Memoryless Random Variables Poisson Process Poisson Processes with Occurrences of Two Types Exercises Continuous-Time Markov Chains Introduction Generators of Continuous Markov Chains: The Kolmogorov Forward and Backward Equations Connection Between the Steady State of a Continuous Markov Chain and the Steady State of the Embedded Matrix Explosions Birth and Birth-Death Processes Birth and Death Processes Queuing Models Detailed Balance Equations Exercises Reversible Markov Chains Random Walks on Weighted Graphs Discrete-Time Birth-Death Process as a Reversible Markov Chain Continuous-Time Reversible Markov Chains Exercises Bibliography.
• (source: Nielsen Book Data)

Clear, rigorous, and intuitive, Markov Processes provides a bridge from an undergraduate probability course to a course in stochastic processes and also as a reference for those that want to see detailed proofs of the theorems of Markov processes. It contains copious computational examples that motivate and illustrate the theorems. The text is designed to be understandable to students who have taken an undergraduate probability course without needing an instructor to fill in any gaps. The book begins with a review of basic probability, then covers the case of finite state, discrete time Markov processes. Building on this, the text deals with the discrete time, infinite state case and provides background for continuous Markov processes with exponential random variables and Poisson processes. It presents continuous Markov processes which include the basic material of Kolmogorov's equations, infinitesimal generators, and explosions. The book concludes with coverage of both discrete and continuous reversible Markov chains. While Markov processes are touched on in probability courses, this book offers the opportunity to concentrate on the topic when additional study is required. It discusses how Markov processes are applied in a number of fields, including economics, physics, and mathematical biology. The book fills the gap between a calculus based probability course, normally taken as an upper level undergraduate course, and a course in stochastic processes, which is typically a graduate course.
(source: Nielsen Book Data)

• 1. Introduction
• 2. Discrete time Markov chains
• 3. Continuous time Markov chains
• 4. State aggregation of Markov chains
• 5. Sojourn times in subsets of states
• 6. Occupation times
• 7. Performability
• 8. Stationary detection
• 9. Simulation of dependability models
• 10. Bounding techniques.
• (source: Nielsen Book Data)

Dependability metrics are omnipresent in every engineering field, from simple ones through to more complex measures combining performance and dependability aspects of systems. This book presents the mathematical basis of the analysis of these metrics in the most used framework, Markov models, describing both basic results and specialised techniques. The authors first present both discrete and continuous time Markov chains before focusing on dependability measures, which necessitate the study of Markov chains on a subset of states representing different user satisfaction levels for the modelled system. Topics covered include Markovian state lumping, analysis of sojourns on subset of states of Markov chains, analysis of most dependability metrics, fundamentals of performability analysis, and bounding and simulation techniques designed to evaluate dependability measures. The book is of interest to graduate students and researchers in all areas of engineering where the concepts of lifetime, repair duration, availability, reliability and risk are important.
(source: Nielsen Book Data)

• Finite Horizon Models
• Infinite Horizon Models, Expected Total Loss and Discounted Loss
• Long Run Average Loss.
• (source: Nielsen Book Data)

This invaluable book provides approximately eighty examples illustrating the theory of controlled discrete-time Markov processes. Except for applications of the theory to real-life problems like stock exchange, queues, gambling, optimal search etc, the main attention is paid to counter-intuitive, unexpected properties of optimization problems. Such examples illustrate the importance of conditions imposed in the theorems on Markov Decision Processes. Many of the examples are based upon examples published earlier in journal articles or textbooks while several other examples are new. The aim was to collect them together in one reference book which should be considered as a complement to existing monographs on Markov decision processes.The book is self-contained and unified in presentation.The main theoretical statements and constructions are provided, and particular examples can be read independently of others. Examples in Markov Decision Processes is an essential source of reference for mathematicians and all those who apply the optimal control theory to practical purposes. When studying or using mathematical methods, the researcher must understand what can happen if some of the conditions imposed in rigorous theorems are not satisfied. Many examples confirming the importance of such conditions were published in different journal articles which are often difficult to find. This book brings together examples based upon such sources, along with several new ones. In addition, it indicates the areas where Markov decision processes can be used. Active researchers can refer to this book on applicability of mathematical methods and theorems. It is also suitable reading for graduate and research students where they will better understand the theory.
(source: Nielsen Book Data)

• Overview on Latent Markov Modeling Introduction Literature review on latent Markov models Alternative approaches Example datasets
• Background on Latent Variable and Markov Chain Models Introduction Latent variable models Expectation-Maximization algorithm Standard errors Latent class model Selection of the number of latent classes Applications Markov chain model for longitudinal data Applications
• Basic Latent Markov Model Introduction Univariate formulation Multivariate formulation Model identifiability Maximum likelihood estimation Selection of the number of latent states Applications
• Constrained Latent Markov Models Introduction Constraints on the measurement model Constraints on the latent model Maximum likelihood estimation Model selection and hypothesis testing Applications
• Including Individual Covariates and Relaxing Basic Model Assumptions Introduction Notation Covariates in the measurement model Covariates in the latent model Interpretation of the resulting models Maximum likelihood estimation Observed information matrix, identifiability, and standard errors Relaxing local independence Higher order extensions Applications
• Including Random Effects and Extension to Multilevel Data Introduction Random-effects formulation Maximum likelihood estimation Multilevel formulation Application to the student math achievement dataset
• Advanced Topics about Latent Markov Modeling Introduction Dealing with continuous response variables Dealing with missing responses Additional computational issues Decoding and forecasting Selection of the number of latent states
• Bayesian Latent Markov Models Introduction Prior distributions Bayesian inference via reversible jump Alternative sampling Application to the labor market dataset
• Appendix: Software List of Main Symbols Bibliography Index.
• (source: Nielsen Book Data)

Drawing on the authors' extensive research in the analysis of categorical longitudinal data, Latent Markov Models for Longitudinal Data focuses on the formulation of latent Markov models and the practical use of these models. Numerous examples illustrate how latent Markov models are used in economics, education, sociology, and other fields. The R and MATLAB(R) routines used for the examples are available on the authors' website. The book provides you with the essential background on latent variable models, particularly the latent class model. It discusses how the Markov chain model and the latent class model represent a useful paradigm for latent Markov models. The authors illustrate the assumptions of the basic version of the latent Markov model and introduce maximum likelihood estimation through the Expectation-Maximization algorithm. They also cover constrained versions of the basic latent Markov model, describe the inclusion of the individual covariates, and address the random effects and multilevel extensions of the model. After covering advanced topics, the book concludes with a discussion on Bayesian inference as an alternative to maximum likelihood inference. As longitudinal data become increasingly relevant in many fields, researchers must rely on specific statistical and econometric models tailored to their application. A complete overview of latent Markov models, this book demonstrates how to use the models in three types of analysis: transition analysis with measurement errors, analyses that consider unobserved heterogeneity, and finding clusters of units and studying the transition between the clusters.
(source: Nielsen Book Data)

• Preface ix
• Chapter 1. Discrete-Time Markov Chains 1 1.1. Definitions and properties 1 1.2. Strong Markov property 5 1.3. Recurrent and transient states 8 1.4. State classification 12 1.5. Visits to a state 14 1.6. State space decomposition 18 1.7. Irreducible and recurrent Markov chains 22 1.8. Aperiodic Markov chains 30 1.9. Convergence to equilibrium 34 1.10. Ergodic theorem 41 1.11. First passage times and number of visits 53 1.12. Finite Markov chains 68 1.13. Absorbing Markov chains 70 1.14. Examples 76 1.15. Bibliographical notes 87
• Chapter 2. Continuous-Time Markov Chains 89 2.1. Definitions and properties 92 2.2. Transition functions and infinitesimal generator 93 2.3. Kolmogorov's backward equation 108 2.4. Kolmogorov's forward equation 114 2.5. Existence and uniqueness of the solutions 127 2.6. Recurrent and transient states 130 2.7. State classification 137 2.8. Explosion 141 2.9. Irreducible and recurrent Markov chains 148 2.10. Convergence to equilibrium 162 2.11. Ergodic theorem 166 2.12. First passage times 172 2.13. Absorbing Markov chains 184 2.14. Bibliographical notes 190
• Chapter 3. Birth-and-Death Processes 191 3.1. Discrete-time birth-and-death processes 191 3.2. Absorbing discrete-time birth-and-death processes 200 3.3. Periodic discrete-time birth-and-death processes 208 3.4. Continuous-time pure birth processes 209 3.5. Continuous-time birth-and-death processes 213 3.6. Absorbing continuous-time birth-and-death processes 228 3.7. Bibliographical notes 233
• Chapter 4. Uniformization 235 4.1. Introduction 235 4.2. Banach spaces and algebra 237 4.3. Infinite matrices and vectors 243 4.4. Poisson process 249 4.5. Uniformizable Markov chains 263 4.6. First passage time to a subset of states 273 4.7. Finite Markov chains 275 4.8. Transient regime 276 4.9. Bibliographical notes 286
• Chapter 5. Queues 287 5.1. The M/M/1 queue 288 5.2. The M/M/c queue 315 5.3. The M/M/ queue 318 5.4. Phase-type distributions 323 5.5. Markovian arrival processes 326 5.6. Batch Markovian arrival process 342 5.7. Block-structured Markov chains 352 5.8. Applications 370 5.9. Bibliographical notes 380
• Appendix 1 Basic Results 381 Bibliography 387 Index 395.
• (source: Nielsen Book Data)

Markov chains are a fundamental class of stochastic processes. They are widely used to solve problems in a large number of domains such as operational research, computer science, communication networks and manufacturing systems. The success of Markov chains is mainly due to their simplicity of use, the large number of available theoretical results and the quality of algorithms developed for the numerical evaluation of many metrics of interest. The author presents the theory of both discrete-time and continuous-time homogeneous Markov chains. He carefully examines the explosion phenomenon, the Kolmogorov equations, the convergence to equilibrium and the passage time distributions to a state and to a subset of states. These results are applied to birth-and-death processes. He then proposes a detailed study of the uniformization technique by means of Banach algebra. This technique is used for the transient analysis of several queuing systems. Contents 1. Discrete-Time Markov Chains 2. Continuous-Time Markov Chains 3. Birth-and-Death Processes 4. Uniformization 5. Queues About the Authors Bruno Sericola is a Senior Research Scientist at Inria Rennes -- Bretagne Atlantique in France. His main research activity is in performance evaluation of computer and communication systems, dependability analysis of fault-tolerant systems and stochastic models.
(source: Nielsen Book Data)

• Preface ix
• Chapter 1. Discrete-Time Markov Chains 1 1.1. Definitions and properties 1 1.2. Strong Markov property 5 1.3. Recurrent and transient states 8 1.4. State classification 12 1.5. Visits to a state 14 1.6. State space decomposition 18 1.7. Irreducible and recurrent Markov chains 22 1.8. Aperiodic Markov chains 30 1.9. Convergence to equilibrium 34 1.10. Ergodic theorem 41 1.11. First passage times and number of visits 53 1.12. Finite Markov chains 68 1.13. Absorbing Markov chains 70 1.14. Examples 76 1.15. Bibliographical notes 87
• Chapter 2. Continuous-Time Markov Chains 89 2.1. Definitions and properties 92 2.2. Transition functions and infinitesimal generator 93 2.3. Kolmogorov's backward equation 108 2.4. Kolmogorov's forward equation 114 2.5. Existence and uniqueness of the solutions 127 2.6. Recurrent and transient states 130 2.7. State classification 137 2.8. Explosion 141 2.9. Irreducible and recurrent Markov chains 148 2.10. Convergence to equilibrium 162 2.11. Ergodic theorem 166 2.12. First passage times 172 2.13. Absorbing Markov chains 184 2.14. Bibliographical notes 190
• Chapter 3. Birth-and-Death Processes 191 3.1. Discrete-time birth-and-death processes 191 3.2. Absorbing discrete-time birth-and-death processes 200 3.3. Periodic discrete-time birth-and-death processes 208 3.4. Continuous-time pure birth processes 209 3.5. Continuous-time birth-and-death processes 213 3.6. Absorbing continuous-time birth-and-death processes 228 3.7. Bibliographical notes 233
• Chapter 4. Uniformization 235 4.1. Introduction 235 4.2. Banach spaces and algebra 237 4.3. Infinite matrices and vectors 243 4.4. Poisson process 249 4.5. Uniformizable Markov chains 263 4.6. First passage time to a subset of states 273 4.7. Finite Markov chains 275 4.8. Transient regime 276 4.9. Bibliographical notes 286
• Chapter 5. Queues 287 5.1. The M/M/1 queue 288 5.2. The M/M/c queue 315 5.3. The M/M/ queue 318 5.4. Phase-type distributions 323 5.5. Markovian arrival processes 326 5.6. Batch Markovian arrival process 342 5.7. Block-structured Markov chains 352 5.8. Applications 370 5.9. Bibliographical notes 380
• Appendix 1 Basic Results 381 Bibliography 387 Index 395.
• (source: Nielsen Book Data)

Markov chains are a fundamental class of stochastic processes. They are widely used to solve problems in a large number of domains such as operational research, computer science, communication networks and manufacturing systems. The success of Markov chains is mainly due to their simplicity of use, the large number of available theoretical results and the quality of algorithms developed for the numerical evaluation of many metrics of interest. The author presents the theory of both discrete-time and continuous-time homogeneous Markov chains. He carefully examines the explosion phenomenon, the Kolmogorov equations, the convergence to equilibrium and the passage time distributions to a state and to a subset of states. These results are applied to birth-and-death processes. He then proposes a detailed study of the uniformization technique by means of Banach algebra. This technique is used for the transient analysis of several queuing systems. Contents 1. Discrete-Time Markov Chains 2. Continuous-Time Markov Chains 3. Birth-and-Death Processes 4. Uniformization 5. Queues About the Authors Bruno Sericola is a Senior Research Scientist at Inria Rennes -- Bretagne Atlantique in France. His main research activity is in performance evaluation of computer and communication systems, dependability analysis of fault-tolerant systems and stochastic models.
(source: Nielsen Book Data)

• Introduction
• Probability Background
• Gambling Problems
• Random Walks
• Discrete-Time Markov Chains
• First Step Analysis
• Classification of States
• Long-Run Behavior of Markov Chains
• Branching Processes
• Continuous-Time Markov Chains
• Discrete-Time Martingales
• Spatial Poisson Processes
• Reliability Theory.

This book provides an undergraduate introduction to discrete and continuous-time Markov chains and their applications. A large focus is placed on the first step analysis technique and its applications to average hitting times and ruin probabilities. Classical topics such as recurrence and transience, stationary and limiting distributions, as well as branching processes, are also covered. Two major examples (gambling processes and random walks) are treated in detail from the beginning, before the general theory itself is presented in the subsequent chapters. An introduction to discrete-time martingales and their relation to ruin probabilities and mean exit times is also provided, and the book includes a chapter on spatial Poisson processes with some recent results on moment identities and deviation inequalities for Poisson stochastic integrals. The concepts presented are illustrated by examples and by 72 exercises and their complete solutions.
(source: Nielsen Book Data)

• Introduction
• Controlled Markov Chains
• Basic Optimality Criteria
• Policy Iteration and Approximation Theorems
• Overtaking, Bias, and Variance Optimality
• Sensitive Discount Optimality
• Blackwell Optimality
• Constrained Controlled Markov Chains
• Applications
• Zero-Sum Markov Games
• Bias and Overtaking Equilibria for Markov Games.
• (source: Nielsen Book Data)

This book concerns continuous-time controlled Markov chains, also known as continuous-time Markov decision processes. They form a class of stochastic control problems in which a single decision-maker wishes to optimize a given objective function. This book is also concerned with Markov games, where two decision-makers (or players) try to optimize their own objective function. Both decision-making processes appear in a large number of applications in economics, operations research, engineering, and computer science, among other areas.An extensive, self-contained, up-to-date analysis of basic optimality criteria (such as discounted and average reward), and advanced optimality criteria (e.g., bias, overtaking, sensitive discount, and Blackwell optimality) is presented. A particular emphasis is made on the application of the results herein: algorithmic and computational issues are discussed, and applications to population models and epidemic processes are shown.This book is addressed to students and researchers in the fields of stochastic control and stochastic games. Moreover, it could be of interest also to undergraduate and beginning graduate students because the reader is not supposed to have a high mathematical background: a working knowledge of calculus, linear algebra, probability, and continuous-time Markov chains should suffice to understand the contents of the book.
(source: Nielsen Book Data)

• *FrontMatter, pg. i*Contents, pg. vii*Notation, pg. ix*Preface, pg. xi*Chapter One. Symmetric Markovian Semigroups and Dirichlet Forms, pg. 1*Chapter Two. Basic Properties and Examples of Dirichlet Forms, pg. 37*Chapter Three. Symmetric Hunt Processes and Regular Dirichlet Forms, pg. 92*Chapter Four. Additive Functionals of Symmetric Markov Processes, pg. 130*Chapter Five. Time Changes of Symmetric Markov Processes, pg. 166*Chapter Six. Reflected Dirichlet Spaces, pg. 240*Chapter Seven. Boundary Theory for Symmetric Markov Processes, pg. 300*Appendix A. Essentials of Markov Processes, pg. 391*Appendix B. Solutions To Exercises, pg. 443*Notes, pg. 451*Bibliography, pg. 457*Catalogue Of Some Useful Theorems, pg. 467*Index, pg. 473.
• (source: Nielsen Book Data)

This book gives a comprehensive and self-contained introduction to the theory of symmetric Markov processes and symmetric quasi-regular Dirichlet forms. In a detailed and accessible manner, Zhen-Qing Chen and Masatoshi Fukushima cover the essential elements and applications of the theory of symmetric Markov processes, including recurrence/transience criteria, probabilistic potential theory, additive functional theory, and time change theory. The authors develop the theory in a general framework of symmetric quasi-regular Dirichlet forms in a unified manner with that of regular Dirichlet forms, emphasizing the role of extended Dirichlet spaces and the rich interplay between the probabilistic and analytic aspects of the theory. Chen and Fukushima then address the latest advances in the theory, presented here for the first time in any book. Topics include the characterization of time-changed Markov processes in terms of Douglas integrals and a systematic account of reflected Dirichlet spaces, and the important roles such advances play in the boundary theory of symmetric Markov processes. This volume is an ideal resource for researchers and practitioners, and can also serve as a textbook for advanced graduate students. It includes examples, appendixes, and exercises with solutions.
(source: Nielsen Book Data)

• Introduction
• Measure Theory
• Integration
• The Radon-Nikodym Theorem
• A Category of Stochastic Relations
• Probability Theory on Continuous Spaces
• Bisimulation for Labelled Markov Processes
• Metrics for Labelled Markov Processes
• Approximating Labelled Markov Processes.
• (source: Nielsen Book Data)

Labelled Markov processes are probabilistic versions of labelled transition systems with continuous state spaces. The book covers basic probability and measure theory on continuous state spaces and then develops the theory of LMPs.The main topics covered are bisimulation, the logical characterization of bisimulation, metrics and approximation theory. An unusual feature of the book is the connection made with categorical and domain theoretic concepts.
(source: Nielsen Book Data)