Written by high performance computing (HPC) experts, Introduction to High Performance Computing for Scientists and Engineers provides a solid introduction to current mainstream computer architecture, dominant parallel programming models, and useful optimization strategies for scientific HPC. From working in a scientific computing center, the authors gained a unique perspective on the requirements and attitudes of users as well as manufacturers of parallel computers. The text first introduces the architecture of modern cache-based microprocessors and discusses their inherent performance limitations, before describing general optimization strategies for serial code on cache-based architectures. It next covers shared- and distributed-memory parallel computer architectures and the most relevant network topologies. After discussing parallel computing on a theoretical level, the authors show how to avoid or ameliorate typical performance problems connected with OpenMP. They then present cache-coherent non-uniform memory access (ccNUMA) optimization techniques, examine distributed-memory parallel programming with message passing interface (MPI), and explain how to write efficient MPI code. The final chapter focuses on hybrid programming with MPI and OpenMP. Users of high performance computers often have no idea what factors limit time to solution and whether it makes sense to think about optimization at all. This book facilitates an intuitive understanding of performance limitations without relying on heavy computer science knowledge. It also prepares readers for studying more advanced literature. Read about the authors' recent honor: Informatics Europe Curriculum Best Practices Award for Parallelism and Concurrency.

Written by high performance computing (HPC) experts, Introduction to High Performance Computing for Scientists and Engineers provides a solid introduction to current mainstream computer architecture, dominant parallel programming models, and useful optimization strategies for scientific HPC. From working in a scientific computing center, the authors gained a unique perspective on the requirements and attitudes of users as well as manufacturers of parallel computers.

The text first introduces the architecture of modern cache-based microprocessors and discusses their inherent performance limitations, before describing general optimization strategies for serial code on cache-based architectures. It next covers shared- and distributed-memory parallel computer architectures and the most relevant network topologies. After discussing parallel computing on a theoretical level, the authors show how to avoid or ameliorate typical performance problems connected with OpenMP. They then present cache-coherent nonuniform memory access (ccNUMA) optimization techniques, examine distributed-memory parallel programming with message passing interface (MPI), and explain how to write efficient MPI code. The final chapter focuses on hybrid programming with MPI and OpenMP.

Users of high performance computers often have no idea what factors limit time to solution and whether it makes sense to think about optimization at all. This book facilitates an intuitive understanding of performance limitations without relying on heavy computer science knowledge. It also prepares readers for studying more advanced literature.

Read about the authors’ recent honor: Informatics Europe Curriculum Best Practices Award for Parallelism and Concurrency

Table of Contents

Modern Processors

Stored-program computer architecture

General-purpose cache-based microprocessor architecture

Memory hierarchies

Multicore processors

Multithreaded processors

Vector processors

Basic Optimization Techniques for Serial Code

Scalar profiling

Common sense optimizations

Simple measures, large impact

The role of compilers

C++ optimizations

Data Access Optimization

Balance analysis and lightspeed estimates

Storage order

Case study: The Jacobi algorithm

Case study: Dense matrix transpose

Algorithm classification and access optimizations

Case study: Sparse matrix-vector multiply

Parallel Computers

Taxonomy of parallel computing paradigms

Shared-memory computers

Distributed-memory computers

Hierarchical (hybrid) systems

Networks

Basics of Parallelization

Why parallelize?

Parallelism

Parallel scalability

Shared-Memory Parallel Programming with OpenMP

Short introduction to OpenMP

Case study: OpenMP-parallel Jacobi algorithm

Advanced OpenMP: Wavefront parallelization

Efficient OpenMP Programming

Profiling OpenMP programs

Performance pitfalls

Case study: Parallel sparse matrix-vector multiply

Locality Optimizations on ccNUMA Architectures

Locality of access on ccNUMA

Case study: ccNUMA optimization of sparse MVM

Placement pitfalls

ccNUMA issues with C++

Distributed-Memory Parallel Programming with MPI

Message passing

A short introduction to MPI

Example: MPI parallelization of a Jacobi solver

Efficient MPI Programming

MPI performance tools

Communication parameters

Synchronization, serialization, contention

Reducing communication overhead

Understanding intranode point-to-point communication

Hybrid Parallelization with MPI and OpenMP

Basic MPI/OpenMP programming models

MPI taxonomy of thread interoperability

Hybrid decomposition and mapping

Potential benefits and drawbacks of hybrid programming

Appendix A: Topology and Affinity in Multicore Environments

Appendix B: Solutions to the Problems

Bibliography

Index

This is an EXACT reproduction of a book published before 1923. This IS NOT an OCR'd book with strange characters, introduced typographical errors, and jumbled words. This book may have occasional imperfections such as missing or blurred pages, poor pictures, errant marks, etc. that were either part of the original artifact, or were introduced by the scanning process. We believe this work is culturally important, and despite the imperfections, have elected to bring it back into print as part of our continuing commitment to the preservation of printed works worldwide. We appreciate your understanding of the imperfections in the preservation process, and hope you enjoy this valuable book.

"A compelling book on budget-balancing, past and present. Not only is it a useful historical reference work but it is amazingly relevant to today's headlines." --Business Week
Why did it take twenty-five years to balance the federal budget? And why did that goal seem politically impossible up until the last possible moment? Why hasn't the nation achieved a single budget surplus since 1969? Why did the first Republican Congress since the 1950s fail so miserably to redeem its most important promise to voters? In Balancing Act, George Hager and Eric Pianin, two of the country's top political journalists, provide fascinating answers.
The question of the deficit is the reigning obsession of modern politics. Never simply a quarrel over numbers, the deficit has always been a moral issue as well as a fiscal one--a bitter fight to the political death over competing ideas about the right to shape the nation's values and to proclaim publicly what those values are. Bringing key political players like Newt Gingrich, Bill Clinton, Ronald Reagan, and George Mitchell vividly to life, Hager and Pianin provide a gripping and revealing look at what goes on behind the closed doors of Capitol Hill.
"Vivid . . . provocative . . . rich in anecdote and drama, [Balancing Act] is a useful primer on the history of deficit spending"
--Los Angeles Times

This is an EXACT reproduction of a book published before 1923. This IS NOT an OCR'd book with strange characters, introduced typographical errors, and jumbled words. This book may have occasional imperfections such as missing or blurred pages, poor pictures, errant marks, etc. that were either part of the original artifact, or were introduced by the scanning process. We believe this work is culturally important, and despite the imperfections, have elected to bring it back into print as part of our continuing commitment to the preservation of printed works worldwide. We appreciate your understanding of the imperfections in the preservation process, and hope you enjoy this valuable book.

This is an EXACT reproduction of a book published before 1923. This IS NOT an OCR'd book with strange characters, introduced typographical errors, and jumbled words. This book may have occasional imperfections such as missing or blurred pages, poor pictures, errant marks, etc. that were either part of the original artifact, or were introduced by the scanning process. We believe this work is culturally important, and despite the imperfections, have elected to bring it back into print as part of our continuing commitment to the preservation of printed works worldwide. We appreciate your understanding of the imperfections in the preservation process, and hope you enjoy this valuable book.