Ever since the invention of the computer, users have demanded more and more computational power to tackle increasingly complex problems. A common means of increasing the amount of computational power available for solving a problem is to use parallel computing. Unfortunately, however, creating efficient parallel programs is notoriously difficult. In addition to all of the well-known problems that are associated with constructing a good serial algorithm, there are a number of problems specifically associated with constructing a good parallel algorithm. These mainly revolve around ensuring that all processors are kept busy and that they have timely access to the data that they require. Unfortunately, however, controlling a number of processors operating in parallel can be exponentially more complicated than controlling one processor. Furthermore, unlike data placement in serial programs, where sophisticated compilation techniques that optimise cache behaviour and memory interleaving are common, optimising data placement throughout the vastly more complex memory hierarchy present in parallel computers is often left to the parallel application programmer. All of these problems are compounded by the large number of parallel computing architectures that exist, because they often exhibit vastly different performance characteristics, which makes writing well-optimised, portable code especially difficult. The primary weapon against these problems in a parallel programmer's or parallel computer architect's arsenal is -- or at least should be -- the art of performance prediction. This book provides a historical exposition of over four decades of research into techniques for modelling the performance of computer programs running on parallel computers.

The major research results from the Scalable Input/Output Initiative, exploring software and algorithmic solutions to the I/O imbalance.As we enter the "decade of data," the disparity between the vast amount of data storage capacity (measurable in terabytes and petabytes) and the bandwidth available for accessing it has created an input/output bottleneck that is proving to be a major constraint on the effective use of scientific data for research. Scalable Input/Output is a summary of the major research results of the Scalable I/O Initiative, launched by Paul Messina, then Director of the Center for Advanced Computing Research at the California Institute of Technology, to explore software and algorithmic solutions to the I/O imbalance. The contributors explore techniques for I/O optimization, including: I/O characterization to understand application and system I/O patterns; system checkpointing strategies; collective I/O and parallel database support for scientific applications; parallel I/O libraries and strategies for file striping, prefetching, and write behind; compilation strategies for out-of-core data access; scheduling and shared virtual memory alternatives; network support for low-latency data transfer; and parallel I/O application programming interfaces.

Introduction to Parallel Computing, 2e provides a basic, in-depth look at techniques for the design and analysis of parallel algorithms and for programming them on commercially available parallel platforms. The book discusses principles of parallel algorithms design and different parallel programming models with extensive coverage of MPI, POSIX threads, and Open MP. It provides a broad and balanced coverage of various core topics such as sorting, graph algorithms, discrete optimization techniques, data mining algorithms, and a number of other algorithms used in numerical and scientific computing applications.

This book provides an in-depth study concerning a claqss of problems in the general area of load sharing and balancing in parallel and distributed systems. The authors present the design and analysis of load distribution strategies for arbitrarily divisible loads in multiprocessor/multicomputer systems subjects to the system constraints in the form of communication delays. In particular, two system architecture-single-level tree or star network, and linear network-are thoroughly analyzed.
The text studies two different cases, one of processors with front-ends and the other without. It concentrates on load distribution strategies and performance analysis, and does not cover issues related to implementation of these strategies on a specific system. The book collates research results developed mainly by two groups at the Indian Institute of Science and the State University of New York at Stony Brook. It also covers results by other researchers that have either appeared or are due to appear in computer science literature. The book also provides relevant but easily understandable numerical examples and figures to illustrate important concepts. It is the first book in this area and is intended to spur further research enabling these ideas to be applied to a more general class of loads. The new methodology introduced here allows a close examination of issues involving the integration of communication and computation. In fact, what is presented is a new "calculus" for load sharing problems.

Foreword by Bjarne Stroustrup Software is generally acknowledged to be the single greatest obstacle preventing mainstream adoption of massively-parallel computing. While sequential applications are routinely ported to platforms ranging from PCs to mainframes, most parallel programs only ever run on one type of machine. One reason for this is that most parallel programming systems have failed to insulate their users from the architectures of the machines on which they have run. Those that have been platform-independent have usually also had poor performance.Many researchers now believe that object-oriented languages may offer a solution. By hiding the architecture-specific constructs required for high performance inside platform-independent abstractions, parallel object-oriented programming systems may be able to combine the speed of massively-parallel computing with the comfort of sequential programming.Parallel Programming Using C++ describes fifteen parallel programming systems based on C++, the most popular object-oriented language of today. These systems cover the whole spectrum of parallel programming paradigms, from data parallelism through dataflow and distributed shared memory to message-passing control parallelism.For the parallel programming community, a common parallel application is discussed in each chapter, as part of the description of the system itself. By comparing the implementations of the polygon overlay problem in each system, the reader can get a better sense of their expressiveness and functionality for a common problem. For the systems community, the chapters contain a discussion of the implementation of the various compilers and runtime systems. In addition to discussing the performance of polygon overlay, several of the contributors also discuss the performance of other, more substantial, applications.For the research community, the contributors discuss the motivations for and philosophy of their systems. As well, many of the chapters include critiques that complete the research arc by pointing out possible future research directions. Finally, for the object-oriented community, there are many examples of how encapsulation, inheritance, and polymorphism can be used to control the complexity of developing, debugging, and tuning parallel software.

Parallel distributed processing is transforming the field of cognitive science. Microcognition provides a clear, readable guide to this emerging paradigm from a cognitive philosopher's point of view. It explains and explores the biological basis of PDP, its psychological importance, and its philosophical relevance.

Break into the powerful world of parallel GPU programming with this down-to-earth, practical guide

Designed for professionals across multiple industrial sectors, "Professional CUDA C Programming "presents CUDA -- a parallel computing platform and programming model designed to ease the development of GPU programming -- fundamentals in an easy-to-follow format, and teaches readers how to think in parallel and implement parallel algorithms on GPUs. Each chapter covers a specific topic, and includes workable examples that demonstrate the development process, allowing readers to explore both the "hard" and "soft" aspects of GPU programming.

Computing architectures are experiencing a fundamental shift toward scalable parallel computing motivated by application requirements in industry and science. This book demonstrates the challenges of efficiently utilizing compute resources at peak performance, presents modern techniques for tackling these challenges, while increasing accessibility for professionals who are not necessarily parallel programming experts. The CUDA programming model and tools empower developers to write high-performance applications on a scalable, parallel computing platform: the GPU. However, CUDA itself can be difficult to learn without extensive programming experience. Recognized CUDA authorities John Cheng, Max Grossman, and Ty McKercher guide readers through essential GPU programming skills and best practices in "Professional CUDA C Programming," including: CUDA Programming ModelGPU Execution ModelGPU Memory modelStreams, Event and ConcurrencyMulti-GPU ProgrammingCUDA Domain-Specific LibrariesProfiling and Performance Tuning

The book makes complex CUDA concepts easy to understand for anyone with knowledge of basic software development with exercises designed to be both readable and high-performance. For the professional seeking entrance to parallel computing and the high-performance computing community, "Professional CUDA C Programming "is an invaluable resource, with the most current information available on the market.

In modern computer science, there exists no truly sequential computing system; and most advanced programming is parallel programming. This is particularly evident in modern application domains like scientific computation, data science, machine intelligence, etc. This lucid introductory textbook will be invaluable to students of computer science and technology, acting as a self-contained primer to parallel programming. It takes the reader from introduction to expertise, addressing a broad gamut of issues. It covers different parallel programming styles, describes parallel architecture, includes parallel programming frameworks and techniques, presents algorithmic and analysis techniques and discusses parallel design and performance issues. With its broad coverage, the book can be useful in a wide range of courses; and can also prove useful as a ready reckoner for professionals in the field.

The latest techniques and principles of parallel and grid database processing The growth in grid databases, coupled with the utility of parallel query processing, presents an important opportunity to understand and utilize high-performance parallel database processing within a major database management system (DBMS). This important new book provides readers with a fundamental understanding of parallelism in data-intensive applications, and demonstrates how to develop faster capabilities to support them. It presents a balanced treatment of the theoretical and practical aspects of high-performance databases to demonstrate how parallel query is executed in a DBMS, including concepts, algorithms, analytical models, and grid transactions. High-Performance Parallel Database Processing and Grid Databases serves as a valuable resource for researchers working in parallel databases and for practitioners interested in building a high-performance database. It is also a much-needed, self-contained textbook for database courses at the advanced undergraduate and graduate levels.

With the rise of multi-core architecture, parallel programming is an increasingly important topic for software engineers and computer system designers. Written by well-known researchers Larry Snyder and Calvin Lin, this highly anticipated first edition emphasizes the principles underlying parallel computation, explains the various phenomena, and clarifies why these phenomena represent opportunities or barriers to successful parallel programming. Ideal for an advanced upper-level undergraduate course, Parallel Programming supplies enduring knowledge that will outlive the current hardware and software, aiming to inspire future researchers to build tomorrow's solutions. ...the first basic book on the subject that I've ever seen that seems to have the pulse on the true issues of parallelism that are relevant for students. --Alan Edelman, MIT Principles of Parallel Programming is a wonderful book and I plan to use it in our new parallel programming course...---Peiyi Tang, University of Arkansas, Little Rock I like Principles of Parallel Programming] very much for a few specific reasons: it's concise, covers the most relevant topics but does not take thousand pages to do it, it is hands on and it covers...recent developments with multi-core and GPGPU. --Edin Hodzic, Santa Clara University

The thoroughly updated edition of a guide to parallel programming with MPI, reflecting the latest specifications, with many detailed examples. This book offers a thoroughly updated guide to the MPI (Message-Passing Interface) standard library for writing programs for parallel computers. Since the publication of the previous edition of Using MPI, parallel computing has become mainstream. Today, applications run on computers with millions of processors; multiple processors sharing memory and multicore processors with multiple hardware threads per core are common. The MPI-3 Forum recently brought the MPI standard up to date with respect to developments in hardware capabilities, core language evolution, the needs of applications, and experience gained over the years by vendors, implementers, and users. This third edition of Using MPI reflects these changes in both text and example code. The book takes an informal, tutorial approach, introducing each concept through easy-to-understand examples, including actual code in C and Fortran. Topics include using MPI in simple programs, virtual topologies, MPI datatypes, parallel libraries, and a comparison of MPI with sockets. For the third edition, example code has been brought up to date; applications have been updated; and references reflect the recent attention MPI has received in the literature. A companion volume, Using Advanced MPI, covers more advanced topics, including hybrid programming and coping with large data.

Presenting the main recent advances made in parallel processing, this volume focuses on the design and implementation of systolic algorithms as efficient computational structures that encompass both multiprocessing and pipelining concepts.;While the architecture of present-day parallel supercomputers is largely based on the concept of a shared memory, with its attendant limitations of common access, advances in semiconductor technology have led to the development of highly parallel computer architectures with decentralized storage and limited connections in which each processor possesses high bandwidth local memory connected to a small number of neighbours. Systolic arrays are a typical and highly efficient example of such architectures, enabling cost effective, high-speed parallel processing for large volumes of data, with ultra-high throughput rates. Algorithms suitable for implementation on systolic arrays find applications in areas such as signal and image processing, pattern matching, linear algebra, recurrence algorithms and graph problems.

A comprehensive overview of OpenMP, the standard application programming interface for shared memory parallel computing-a reference for students and professionals. "I hope that readers will learn to use the full expressibility and power of OpenMP. This book should provide an excellent introduction to beginners, and the performance section should help those with some experience who want to push OpenMP to its limits." -from the foreword by David J. Kuck, Intel Fellow, Software and Solutions Group, and Director, Parallel and Distributed Solutions, Intel Corporation OpenMP, a portable programming interface for shared memory parallel computers, was adopted as an informal standard in 1997 by computer scientists who wanted a unified model on which to base programs for shared memory systems. OpenMP is now used by many software developers; it offers significant advantages over both hand-threading and MPI. Using OpenMP offers a comprehensive introduction to parallel programming concepts and a detailed overview of OpenMP. Using OpenMP discusses hardware developments, describes where OpenMP is applicable, and compares OpenMP to other programming interfaces for shared and distributed memory parallel architectures. It introduces the individual features of OpenMP, provides many source code examples that demonstrate the use and functionality of the language constructs, and offers tips on writing an efficient OpenMP program. It describes how to use OpenMP in full-scale applications to achieve high performance on large-scale architectures, discussing several case studies in detail, and offers in-depth troubleshooting advice. It explains how OpenMP is translated into explicitly multithreaded code, providing a valuable behind-the-scenes account of OpenMP program performance. Finally, Using OpenMP considers trends likely to influence OpenMP development, offering a glimpse of the possibilities of a future OpenMP 3.0 from the vantage point of the current OpenMP 2.5. With multicore computer use increasing, the need for a comprehensive introduction and overview of the standard interface is clear. Using OpenMP provides an essential reference not only for students at both undergraduate and graduate levels but also for professionals who intend to parallelize existing codes or develop new parallel programs for shared memory computer architectures.

This volume contains the conference proceedings for the 2001 International Conference on Parallel Processing Workshops.

As per the constant need to solve larger and larger numerical problems, it is not possible to neglect the opportunity that comes from the close adaptation of computational algorithms and their implementations for particular features of computing devices, i.e. the characteristics and performance of available workstations and servers. In the last decade, the advances in hardware manufacturing, the decreasing cost and the spread of GPUs have attracted the attention of researchers for numerical simulations, given that for some problems, GPU-based simulations can significantly outperform the ones based on CPUs. The objective of this book is first to present how to design in a context of GPGPU numerical methods in order to obtain the highest efficiency. A second objective of this book is to propose new auto-tuning techniques to optimize access on GPU. A third objective of this book is to propose new preconditioning techniques for GPGPU. Finally, an original energy consumption model is proposed, leading to a robust and accurate energy consumption prediction model.