This book presents classical Markov Decision Processes (MDP) for real-life applications and optimization. MDP allows users to develop and formally support approximate and simple decision rules, and this book showcases state-of-the-art applications in which MDP was key to the solution approach. The book is divided into six parts. Part 1 is devoted to the state-of-the-art theoretical foundation of MDP, including approximate methods such as policy improvement, successive approximation and infinite state spaces as well as an instructive chapter on Approximate Dynamic Programming. It then continues with five parts of specific and non-exhaustive application areas. Part 2 covers MDP healthcare applications, which includes different screening procedures, appointment scheduling, ambulance scheduling and blood management. Part 3 explores MDP modeling within transportation. This ranges from public to private transportation, from airports and traffic lights to car parking or charging your electric car . Part 4 contains three chapters that illustrates the structure of approximate policies for production or manufacturing structures. In Part 5, communications is highlighted as an important application area for MDP. It includes Gittins indices, down-to-earth call centers and wireless sensor networks. Finally Part 6 is dedicated to financial modeling, offering an instructive review to account for financial portfolios and derivatives under proportional transactional costs. The MDP applications in this book illustrate a variety of both standard and non-standard aspects of MDP modeling and its practical use. This book should appeal to readers for practitioning, academic research and educational purposes, with a background in, among others, operations research, mathematics, computer science, and industrial engineering.

This handbook aims to highlight fundamental, methodological and computational aspects of networks of queues to provide insights and to unify results that can be applied in a more general manner. The handbook is organized into five parts:

Part 1 considers exact analytical results such as of product form type. Topics include characterization of product forms by physical balance concepts and simple traffic flow equations, classes of service and queue disciplines that allow a product form, a unified description of product forms for discrete time queueing networks, insights for insensitivity, and aggregation and decomposition results that allow sub networks to be aggregated into single nodes to reduce computational burden.

""

Part 2 looks at monotonicity and comparison results such as for computational simplification by either of two approaches: stochastic monotonicity and ordering results based on the ordering of the process generators, and comparison results and explicit error bounds based on an underlying Markov reward structure leading to ordering of expectations of performance measures.

""

Part 3 presents diffusion and fluid results. It specifically looks at the fluid regime and the diffusion regime. Both of these are illustrated through fluid limits for the analysis of system stability, diffusion approximations for multi-server systems, and a system fed by Gaussian traffic.

Part 4 illustrates computational and approximate results through the classical MVA (mean value analysis) and QNA (queueing network analyzer) for computing mean and variance of performance measures such as queue lengths and sojourn times; numerical approximation of response time distributions; and approximate decomposition results for large open queueing networks.

""

Part 5 enlightens selected applications as loss networks originating from circuit switched telecommunications applications, capacity sharing originating from packet switching in data networks, and a hospital application that is of growing present day interest.

The book shows that the intertwined progress of theory and practice will remain to be most intriguing and will continue to be the basis of further developments in queueing networks."

This book presents classical Markov Decision Processes (MDP) for real-life applications and optimization. MDP allows users to develop and formally support approximate and simple decision rules, and this book showcases state-of-the-art applications in which MDP was key to the solution approach. The book is divided into six parts. Part 1 is devoted to the state-of-the-art theoretical foundation of MDP, including approximate methods such as policy improvement, successive approximation and infinite state spaces as well as an instructive chapter on Approximate Dynamic Programming. It then continues with five parts of specific and non-exhaustive application areas. Part 2 covers MDP healthcare applications, which includes different screening procedures, appointment scheduling, ambulance scheduling and blood management. Part 3 explores MDP modeling within transportation. This ranges from public to private transportation, from airports and traffic lights to car parking or charging your electric car . Part 4 contains three chapters that illustrates the structure of approximate policies for production or manufacturing structures. In Part 5, communications is highlighted as an important application area for MDP. It includes Gittins indices, down-to-earth call centers and wireless sensor networks. Finally Part 6 is dedicated to financial modeling, offering an instructive review to account for financial portfolios and derivatives under proportional transactional costs. The MDP applications in this book illustrate a variety of both standard and non-standard aspects of MDP modeling and its practical use. This book should appeal to readers for practitioning, academic research and educational purposes, with a background in, among others, operations research, mathematics, computer science, and industrial engineering.

This handbook aims to highlight fundamental, methodological and computational aspects of networks of queues to provide insights and to unify results that can be applied in a more general manner. The handbook is organized into five parts:

Part 1 considers exact analytical results such as of product form type. Topics include characterization of product forms by physical balance concepts and simple traffic flow equations, classes of service and queue disciplines that allow a product form, a unified description of product forms for discrete time queueing networks, insights for insensitivity, and aggregation and decomposition results that allow sub networks to be aggregated into single nodes to reduce computational burden.

""

Part 2 looks at monotonicity and comparison results such as for computational simplification by either of two approaches: stochastic monotonicity and ordering results based on the ordering of the process generators, and comparison results and explicit error bounds based on an underlying Markov reward structure leading to ordering of expectations of performance measures.

""

Part 3 presents diffusion and fluid results. It specifically looks at the fluid regime and the diffusion regime. Both of these are illustrated through fluid limits for the analysis of system stability, diffusion approximations for multi-server systems, and a system fed by Gaussian traffic.

Part 4 illustrates computational and approximate results through the classical MVA (mean value analysis) and QNA (queueing network analyzer) for computing mean and variance of performance measures such as queue lengths and sojourn times; numerical approximation of response time distributions; and approximate decomposition results for large open queueing networks.

""

Part 5 enlightens selected applications as loss networks originating from circuit switched telecommunications applications, capacity sharing originating from packet switching in data networks, and a hospital application that is of growing present day interest.

The book shows that the intertwined progress of theory and practice will remain to be most intriguing and will continue to be the basis of further developments in queueing networks."