Regularization, Optimization, Kernels, and Support Vector Machines offers a snapshot of the current state of the art of large-scale machine learning, providing a single multidisciplinary source for the latest research and advances in regularization, sparsity, compressed sensing, convex and large-scale optimization, kernel methods, and support vector machines. Consisting of 21 chapters authored by leading researchers in machine learning, this comprehensive reference: Covers the relationship between support vector machines (SVMs) and the Lasso Discusses multi-layer SVMs Explores nonparametric feature selection, basis pursuit methods, and robust compressive sensing Describes graph-based regularization methods for single- and multi-task learning Considers regularized methods for dictionary learning and portfolio selection Addresses non-negative matrix factorization Examines low-rank matrix and tensor-based models Presents advanced kernel methods for batch and online machine learning, system identification, domain adaptation, and image processing Tackles large-scale algorithms including conditional gradient methods, (non-convex) proximal techniques, and stochastic gradient descent Regularization, Optimization, Kernels, and Support Vector Machines is ideal for researchers in machine learning, pattern recognition, data mining, signal processing, statistical learning, and related areas.

Regularization, Optimization, Kernels, and Support Vector Machines offers a snapshot of the current state of the art of large-scale machine learning, providing a single multidisciplinary source for the latest research and advances in regularization, sparsity, compressed sensing, convex and large-scale optimization, kernel methods, and support vector machines. Consisting of 21 chapters authored by leading researchers in machine learning, this comprehensive reference: Covers the relationship between support vector machines (SVMs) and the Lasso Discusses multi-layer SVMs Explores nonparametric feature selection, basis pursuit methods, and robust compressive sensing Describes graph-based regularization methods for single- and multi-task learning Considers regularized methods for dictionary learning and portfolio selection Addresses non-negative matrix factorization Examines low-rank matrix and tensor-based models Presents advanced kernel methods for batch and online machine learning, system identification, domain adaptation, and image processing Tackles large-scale algorithms including conditional gradient methods, (non-convex) proximal techniques, and stochastic gradient descent Regularization, Optimization, Kernels, and Support Vector Machines is ideal for researchers in machine learning, pattern recognition, data mining, signal processing, statistical learning, and related areas.

Nonlinear Modeling: Advanced Black-Box Techniques discusses methods on Neural nets and related model structures for nonlinear system identification; Enhanced multi-stream Kalman filter training for recurrent networks; The support vector method of function estimation; Parametric density estimation for the classification of acoustic feature vectors in speech recognition; Wavelet-based modeling of nonlinear systems; Nonlinear identification based on fuzzy models; Statistical learning in control and matrix theory; Nonlinear time-series analysis. It also contains the results of the K.U. Leuven time series prediction competition, held within the framework of an international workshop at the K.U. Leuven, Belgium in July 1998.

Artificial neural networks possess several properties that make them particularly attractive for applications to modelling and control of complex non-linear systems. Among these properties are their universal approximation ability, their parallel network structure and the availability of on- and off-line learning methods for the interconnection weights. However, dynamic models that contain neural network architectures might be highly non-linear and difficult to analyse as a result. Artificial Neural Networks for Modelling and Control of Non-Linear Systems investigates the subject from a system theoretical point of view. However the mathematical theory that is required from the reader is limited to matrix calculus, basic analysis, differential equations and basic linear system theory. No preliminary knowledge of neural networks is explicitly required. The book presents both classical and novel network architectures and learning algorithms for modelling and control. Topics include non-linear system identification, neural optimal control, top-down model based neural control design and stability analysis of neural control systems. A major contribution of this book is to introduce NLq Theory as an extension towards modern control theory, in order to analyze and synthesize non-linear systems that contain linear together with static non-linear operators that satisfy a sector condition: neural state space control systems are an example. Moreover, it turns out that NLq Theory is unifying with respect to many problems arising in neural networks, systems and control. Examples show that complex non-linear systems can be modelled and controlled within NLq theory, including mastering chaos. The didactic flavor of this book makes it suitable for use as a text for a course on Neural Networks. In addition, researchers and designers will find many important new techniques, in particular NLq Theory, that have applications in control theory, system theory, circuit theory and Time Series Analysis.

Nonlinear Modeling: Advanced Black-Box Techniques discusses methods on Neural nets and related model structures for nonlinear system identification; Enhanced multi-stream Kalman filter training for recurrent networks; The support vector method of function estimation; Parametric density estimation for the classification of acoustic feature vectors in speech recognition; Wavelet-based modeling of nonlinear systems; Nonlinear identification based on fuzzy models; Statistical learning in control and matrix theory; Nonlinear time-series analysis. It also contains the results of the K.U. Leuven time series prediction competition, held within the framework of an international workshop at the K.U. Leuven, Belgium in July 1998.

Artificial neural networks possess several properties that make them particularly attractive for applications to modelling and control of complex non-linear systems. Among these properties are their universal approximation ability, their parallel network structure and the availability of on- and off-line learning methods for the interconnection weights. However, dynamic models that contain neural network architectures might be highly non-linear and difficult to analyse as a result. Artificial Neural Networks for Modelling and Control of Non-Linear Systems investigates the subject from a system theoretical point of view. However the mathematical theory that is required from the reader is limited to matrix calculus, basic analysis, differential equations and basic linear system theory. No preliminary knowledge of neural networks is explicitly required. The book presents both classical and novel network architectures and learning algorithms for modelling and control. Topics include non-linear system identification, neural optimal control, top-down model based neural control design and stability analysis of neural control systems. A major contribution of this book is to introduce NLq Theory as an extension towards modern control theory, in order to analyze and synthesize non-linear systems that contain linear together with static non-linear operators that satisfy a sector condition: neural state space control systems are an example. Moreover, it turns out that NLq Theory is unifying with respect to many problems arising in neural networks, systems and control. Examples show that complex non-linear systems can be modelled and controlled within NLq theory, including mastering chaos. The didactic flavor of this book makes it suitable for use as a text for a course on Neural Networks. In addition, researchers and designers will find many important new techniques, in particular NLq Theory, that have applications in control theory, system theory, circuit theory and Time Series Analysis.

In recent years, considerable progress has been made in the understanding of problems of learning and generalization. In this context, intelligence basically means the ability to perform well on new data after learning a model on the basis of given data. Such problems arise in many different areas and are becoming increasingly important and crucial towards many applications such as in bioinformatics, multimedia, computer vision and signal processing, internet search and information retrieval, datamining and textmining, finance, fraud detection, measurement systems, process control and several others. Currently, the development of new technologies enables to generate massive amounts of data containing a wealth of information that remains to become explored. Often the dimensionality of the input spaces in these novel applications is huge. This can be seen in the analysis of micro-array data, for example, where expression levels of thousands of genes need to be analyzed given only a limited number of experiments. Without performing dimensionality reduction, the classical statistical paradigms show fundamental shortcomings at this point. Facing these new challenges, there is a need for new mathematical foundations and models in a way that the data can become processed in a reliable way. The subjects in this publication are very interdisciplinary and relate to problems studied in neural networks, machine learning, mathematics and statistics.

This book focuses on Least Squares Support Vector Machines (LS-SVMs) which are reformulations to standard SVMs. LS-SVMs are closely related to regularization networks and Gaussian processes but additionally emphasize and exploit primal-dual interpretations from optimization theory. The authors explain the natural links between LS-SVM classifiers and kernel Fisher discriminant analysis. Bayesian inference of LS-SVM models is discussed, together with methods for imposing spareness and employing robust statistics.

The framework is further extended towards unsupervised learning by considering PCA analysis and its kernel version as a one-class modelling problem. This leads to new primal-dual support vector machine formulations for kernel PCA and kernel CCA analysis. Furthermore, LS-SVM formulations are given for recurrent networks and control. In general, support vector machines may pose heavy computational challenges for large data sets. For this purpose, a method of fixed size LS-SVM is proposed where the estimation is done in the primal space in relation to a Nystrom sampling with active selection of support vectors. The methods are illustrated with several examples.