The combination of high-resolution visual and depth sensing, supported by machine learning, opens up new opportunities to solve real-world problems in computer vision.

This authoritative text/reference presents an interdisciplinary selection of important, cutting-edge research on RGB-D based computer vision. Divided into four sections, the book opens with a detailed survey of the field, followed by a focused examination of RGB-D based 3D reconstruction, mapping and synthesis. The work continues with a section devoted to novel techniques that employ depth data for object detection, segmentation and tracking, and concludes with examples of accurate human action interpretation aided by depth sensors.

Topics and features: discusses the calibration of color and depth cameras, the reduction of noise on depth maps, and methods for capturing human performance in 3D; reviews a selection of applications which use RGB-D information to reconstruct human figures, evaluate energy consumption, and obtain accurate action classification; presents an innovative approach for 3D object retrieval, and for the reconstruction of gas flow from multiple Kinect cameras; describes an RGB-D computer vision system designed to assist the visually impaired, and another for smart-environment sensing to assist elderly and disabled people; examines the effective features that characterize static hand poses, and introduces a unified framework to enforce both temporal and spatial constraints for hand parsing; proposes a new classifier architecture for real-time hand pose recognition, and a novel hand segmentation and gesture recognition system.

Researchers and practitioners working in computer vision, HCI and machine learning will find this to be a must-read text. The book also serves as a useful reference for graduate students studying computer vision, pattern recognition or multimedia.

he problem of analyzing sequences of images to extract three-dimensional T motion and structure has been at the heart of the research in computer vi sion for many years. It is very important since its success or failure will determine whether or not vision can be used as a sensory process in reactive systems. The considerable research interest in this field has been motivated at least by the following two points: 1. The redundancy of information contained in time-varying images can over come several difficulties encountered in interpreting a single image. 2. There are a lot of important applications including automatic vehicle driv ing, traffic control, aerial surveillance, medical inspection and global model construction. However, there are many new problems which should be solved: how to effi ciently process the abundant information contained in time-varying images, how to model the change between images, how to model the uncertainty inherently associated with the imaging system and how to solve inverse problems which are generally ill-posed. There are of course many possibilities for attacking these problems and many more remain to be explored. We discuss a few of them in this book based on work carried out during the last five years in the Computer Vision and Robotics Group at INRIA (Institut National de Recherche en Informatique et en Automatique)."

Appendix 164 3. A 3. A. 1 Approximate Estimation of Fundamental Matrix from General Matrix 164 3. A. 2 Estimation of Affine Transformation 165 4 RECOVERY OF EPIPOLAR GEOMETRY FROM LINE SEGMENTS OR LINES 167 Line Segments or Straight Lines 168 4. 1 4. 2 Solving Motion Using Line Segments Between Two Views 173 4. 2. 1 Overlap of Two Corresponding Line Segments 173 Estimating Motion by Maximizing Overlap 175 4. 2. 2 Implementation Details 4. 2. 3 176 Reconstructing 3D Line Segments 4. 2. 4 179 4. 2. 5 Experimental Results 180 4. 2. 6 Discussions 192 4. 3 Determining Epipolar Geometry of Three Views 194 4. 3. 1 Trifocal Constraints for Point Matches 194 4. 3. 2 Trifocal Constraints for Line Correspondences 199 4. 3. 3 Linear Estimation of K, L, and M Using Points and Lines 200 4. 3. 4 Determining Camera Projection Matrices 201 4. 3. 5 Image Transfer 203 4. 4 Summary 204 5 REDEFINING STEREO, MOTION AND OBJECT RECOGNITION VIA EPIPOLAR GEOMETRY 205 5. 1 Conventional Approaches to Stereo, Motion and Object Recognition 205 5. 1. 1 Stereo 205 5. 1. 2 Motion 206 5. 1. 3 Object Recognition 207 5. 2 Correspondence in Stereo, Motion and Object Recognition as 1D Search 209 5. 2. 1 Stereo Matching 209 xi Contents 5. 2. 2 Motion Correspondence and Segmentation 209 5. 2. 3 3D Object Recognition and Localization 210 Disparity and Spatial Disparity Space 210 5.

Face detection, because of its vast array of applications, is one of the most active research areas in computer vision. In this book, we review various approaches to face detection developed in the past decade, with more emphasis on boosting-based learning algorithms. We then present a series of algorithms that are empowered by the statistical view of boosting and the concept of multiple instance learning. We start by describing a boosting learning framework that is capable to handle billions of training examples. It differs from traditional bootstrapping schemes in that no intermediate thresholds need to be set during training, yet the total number of negative examples used for feature selection remains constant and focused (on the poor performing ones). A multiple instance pruning scheme is then adopted to set the intermediate thresholds after boosting learning. This algorithm generates detectors that are both fast and accurate. We then present two multiple instance learning schemes for face detection, multiple instance learning boosting (MILBoost) and winner-take-all multiple category boosting (WTA-McBoost). MILBoost addresses the uncertainty in accurately pinpointing the location of the object being detected, while WTA-McBoost addresses the uncertainty in determining the most appropriate subcategory label for multiview object detection. Both schemes can resolve the ambiguity of the labeling process and reduce outliers during training, which leads to improved detector performances. In many applications, a detector trained with generic data sets may not perform optimally in a new environment. We propose detection adaption, which is a promising solution for this problem. We present an adaptation scheme based on the Taylor expansion of the boosting learning objective function, and we propose to store the second order statistics of the generic training data for future adaptation. We show that with a small amount of labeled data in the new environment, the detector's performance can be greatly improved. We also present two interesting applications where boosting learning was applied successfully. The first application is face verification for filtering and ranking image/video search results on celebrities. We present boosted multi-task learning (MTL), yet another boosting learning algorithm that extends MILBoost with a graphical model. Since the available number of training images for each celebrity may be limited, learning individual classifiers for each person may cause overfitting. MTL jointly learns classifiers for multiple people by sharing a few boosting classifiers in order to avoid overfitting. The second application addresses the need of speaker detection in conference rooms. The goal is to find who is speaking, given a microphone array and a panoramic video of the room. We show that by combining audio and visual features in a boosting framework, we can determine the speaker's position very accurately. Finally, we offer our thoughts on future directions for face detection. Table of Contents: A Brief Survey of the Face Detection Literature / Cascade-based Real-Time Face Detection / Multiple Instance Learning for Face Detection / Detector Adaptation / Other Applications / Conclusions and Future Work

Appendix 164 3. A 3. A. 1 Approximate Estimation of Fundamental Matrix from General Matrix 164 3. A. 2 Estimation of Affine Transformation 165 4 RECOVERY OF EPIPOLAR GEOMETRY FROM LINE SEGMENTS OR LINES 167 Line Segments or Straight Lines 168 4. 1 4. 2 Solving Motion Using Line Segments Between Two Views 173 4. 2. 1 Overlap of Two Corresponding Line Segments 173 Estimating Motion by Maximizing Overlap 175 4. 2. 2 Implementation Details 4. 2. 3 176 Reconstructing 3D Line Segments 4. 2. 4 179 4. 2. 5 Experimental Results 180 4. 2. 6 Discussions 192 4. 3 Determining Epipolar Geometry of Three Views 194 4. 3. 1 Trifocal Constraints for Point Matches 194 4. 3. 2 Trifocal Constraints for Line Correspondences 199 4. 3. 3 Linear Estimation of K, L, and M Using Points and Lines 200 4. 3. 4 Determining Camera Projection Matrices 201 4. 3. 5 Image Transfer 203 4. 4 Summary 204 5 REDEFINING STEREO, MOTION AND OBJECT RECOGNITION VIA EPIPOLAR GEOMETRY 205 5. 1 Conventional Approaches to Stereo, Motion and Object Recognition 205 5. 1. 1 Stereo 205 5. 1. 2 Motion 206 5. 1. 3 Object Recognition 207 5. 2 Correspondence in Stereo, Motion and Object Recognition as 1D Search 209 5. 2. 1 Stereo Matching 209 xi Contents 5. 2. 2 Motion Correspondence and Segmentation 209 5. 2. 3 3D Object Recognition and Localization 210 Disparity and Spatial Disparity Space 210 5.

This is the first book to treat the analysis of 3D dynamic scenes using a stereovision system. Several approaches are described, for example two different methods for dealing with long and short sequences of images of an unknown environment including an arbitrary number of rigid mobile objects. Results obtained from stereovision systems are found to be superior to those from monocular image systems, which are often very sensitive to noise and therefore of little use in practice. It is shown thatmotion estimation can be further improved by the explicit modeling of uncertainty in geometric objects. The techniques developed in this book have been successfully demonstrated with a large number of real images in the context of visual navigation of a mobile robot.