Search results for: accelerated-optimization-for-machine-learning

Accelerated Optimization for Machine Learning

Author : Zhouchen Lin
File Size : 51.73 MB
Format : PDF, Docs
Download : 763
Read : 906
Download »
This book on optimization includes forewords by Michael I. Jordan, Zongben Xu and Zhi-Quan Luo. Machine learning relies heavily on optimization to solve problems with its learning models, and first-order optimization algorithms are the mainstream approaches. The acceleration of first-order optimization algorithms is crucial for the efficiency of machine learning. Written by leading experts in the field, this book provides a comprehensive introduction to, and state-of-the-art review of accelerated first-order optimization algorithms for machine learning. It discusses a variety of methods, including deterministic and stochastic algorithms, where the algorithms can be synchronous or asynchronous, for unconstrained and constrained problems, which can be convex or non-convex. Offering a rich blend of ideas, theories and proofs, the book is up-to-date and self-contained. It is an excellent reference resource for users who are seeking faster optimization algorithms, as well as for graduate students and researchers wanting to grasp the frontiers of optimization in machine learning in a short time.

First order and Stochastic Optimization Methods for Machine Learning

Author : Guanghui Lan
File Size : 74.53 MB
Format : PDF, Kindle
Download : 835
Read : 801
Download »
This book covers not only foundational materials but also the most recent progresses made during the past few years on the area of machine learning algorithms. In spite of the intensive research and development in this area, there does not exist a systematic treatment to introduce the fundamental concepts and recent progresses on machine learning algorithms, especially on those based on stochastic optimization methods, randomized algorithms, nonconvex optimization, distributed and online learning, and projection free methods. This book will benefit the broad audience in the area of machine learning, artificial intelligence and mathematical programming community by presenting these recent developments in a tutorial style, starting from the basic building blocks to the most carefully designed and complicated algorithms for machine learning.

Machine Learning Accelerated Grid Optimization Tool for Evaluating Carbon Capture Technologies

Author :
File Size : 24.28 MB
Format : PDF, Mobi
Download : 911
Read : 655
Download »

High order Tuners for Convex Optimization

Author : José María Moreu Gamazo
File Size : 86.67 MB
Format : PDF
Download : 276
Read : 225
Download »
Iterative gradient-based algorithms have been increasingly applied for the training of a broad variety of machine learning models including large neural-nets. In particular, momentum-based methods, with accelerated learning guarantees, have received a lot of attention due to their provable guarantees of fast learning in certain classes of problems and multiple algorithms have been derived. However, properties for these methods hold true only for constant regressors. When time-varying regressors occur, which is commonplace in dynamic systems, many of these momentum-based methods cannot guarantee stability. Recently, a new High-order Tuner (HT) was developed and shown to have 1) stability and asymptotic convergence for time-varying regressors and 2) non-asymptotic accelerated learning guarantees for constant regressors. These results were derived for a linear regression framework producing a quadratic loss function. This thesis extends and discuss the results of this same HT for general smooth convex loss functions. Through the exploitation of convexity and smoothness definitions, we establish similar stability and asymptotic convergence guarantees. Additionally we conjecture that the HT has an accelerated convergence rate. Finally, we provide numerical simulations supporting the satisfactory behavior of the HT algorithm as well as the conjecture of accelerated learning.

Convex Optimization

Author : Sébastien Bubeck
File Size : 45.63 MB
Format : PDF, ePub, Docs
Download : 567
Read : 160
Download »
This monograph presents the main complexity theorems in convex optimization and their corresponding algorithms. It begins with the fundamental theory of black-box optimization and proceeds to guide the reader through recent advances in structural optimization and stochastic optimization. The presentation of black-box optimization, strongly influenced by the seminal book by Nesterov, includes the analysis of cutting plane methods, as well as (accelerated) gradient descent schemes. Special attention is also given to non-Euclidean settings (relevant algorithms include Frank-Wolfe, mirror descent, and dual averaging), and discussing their relevance in machine learning. The text provides a gentle introduction to structural optimization with FISTA (to optimize a sum of a smooth and a simple non-smooth term), saddle-point mirror prox (Nemirovski's alternative to Nesterov's smoothing), and a concise description of interior point methods. In stochastic optimization it discusses stochastic gradient descent, mini-batches, random coordinate descent, and sublinear algorithms. It also briefly touches upon convex relaxation of combinatorial problems and the use of randomness to round solutions, as well as random walks based methods.

Robust Accelerated Gradient Methods for Machine Learning

Author : Alireza Fallah
File Size : 51.44 MB
Format : PDF, Kindle
Download : 111
Read : 415
Download »
In this thesis, we study the problem of minimizing a smooth and strongly convex function, which arises in different areas, including regularized regression problems in machine learning. To solve this optimization problem, we consider using first order methods which are popular due to their scalability with large data sets, and we study the case that the exact gradient information is not available. In this setting, a naive implementation of classical first order algorithms need not converge and even accumulate noise. This motivates consideration of robustness of algorithms to noise as another metric in designing fast algorithms. To address this problem, we first propose a definition for the robustness of an algorithm in terms of the asymptotic expected suboptimality of its iterate sequence to input noise power. We focus on Gradient Descent and Accelerated Gradient methods and develop a framework based on a dynamical system representation of these algorithms to characterize their convergence rate and robustness to noise using tools from control theory and optimization. We provide explicit expressions for the convergence rate and robustness of both algorithms for the quadratic case, and also derive tractable and tight upper bounds for general smooth and strongly convex functions. We also develop a computational framework for choosing parameters of these algorithms to achieve a particular trade-off between robustness and rate. As a second contribution, we consider algorithms that can reach optimality (obtaining perfect robustness). The past literature provided lower bounds on the rate of decay of suboptimality in term of initial distance to optimality (in the deterministic case) and error due to gradient noise (in the stochastic case). We design a novel multistage and accelerated universally optimal algorithm that can achieve both of these lower bounds simultaneously without knowledge of initial optimality gap or noise characterization. We finally illustrate the behavior of our algorithm through numerical experiments.

Optimization for Machine Learning

Author : Suvrit Sra
File Size : 75.47 MB
Format : PDF, Mobi
Download : 350
Read : 205
Download »
An up-to-date account of the interplay between optimization and machine learning, accessible to students and researchers in both communities. The interplay between optimization and machine learning is one of the most important developments in modern computational science. Optimization formulations and methods are proving to be vital in designing algorithms to extract essential knowledge from huge volumes of data. Machine learning, however, is not simply a consumer of optimization technology but a rapidly evolving field that is itself generating new optimization ideas. This book captures the state of the art of the interaction between optimization and machine learning in a way that is accessible to researchers in both fields. Optimization approaches have enjoyed prominence in machine learning because of their wide applicability and attractive theoretical properties. The increasing complexity, size, and variety of today's machine learning models call for the reassessment of existing assumptions. This book starts the process of reassessment. It describes the resurgence in novel contexts of established frameworks such as first-order methods, stochastic approximations, convex relaxations, interior-point methods, and proximal methods. It also devotes attention to newer themes such as regularized optimization, robust optimization, gradient and subgradient methods, splitting techniques, and second-order methods. Many of these techniques draw inspiration from other fields, including operations research, theoretical computer science, and subfields of optimization. The book will enrich the ongoing cross-fertilization between the machine learning community and these other fields, and within the broader optimization community.

ICT Innovations 2020 Machine Learning and Applications

Author : Vesna Dimitrova
File Size : 82.96 MB
Format : PDF, ePub, Mobi
Download : 848
Read : 466
Download »

Optinformatics in Evolutionary Learning and Optimization

Author : Liang Feng
File Size : 55.69 MB
Format : PDF, ePub, Mobi
Download : 285
Read : 612
Download »
This book provides readers the recent algorithmic advances towards realizing the notion of optinformatics in evolutionary learning and optimization. The book also provides readers a variety of practical applications, including inter-domain learning in vehicle route planning, data-driven techniques for feature engineering in automated machine learning, as well as evolutionary transfer reinforcement learning. Through reading this book, the readers will understand the concept of optinformatics, recent research progresses in this direction, as well as particular algorithm designs and application of optinformatics. Evolutionary algorithms (EAs) are adaptive search approaches that take inspiration from the principles of natural selection and genetics. Due to their efficacy of global search and ease of usage, EAs have been widely deployed to address complex optimization problems occurring in a plethora of real-world domains, including image processing, automation of machine learning, neural architecture search, urban logistics planning, etc. Despite the success enjoyed by EAs, it is worth noting that most existing EA optimizers conduct the evolutionary search process from scratch, ignoring the data that may have been accumulated from different problems solved in the past. However, today, it is well established that real-world problems seldom exist in isolation, such that harnessing the available data from related problems could yield useful information for more efficient problem-solving. Therefore, in recent years, there is an increasing research trend in conducting knowledge learning and data processing along the course of an optimization process, with the goal of achieving accelerated search in conjunction with better solution quality. To this end, the term optinformatics has been coined in the literature as the incorporation of information processing and data mining (i.e., informatics) techniques into the optimization process. The primary market of this book is researchers from both academia and industry, who are working on computational intelligence methods and their applications. This book is also written to be used as a textbook for a postgraduate course in computational intelligence emphasizing methodologies at the intersection of optimization and machine learning.

An Accelerated Algorithm for Delayed Distributed Convex Optimization

Author : Ioannis Bakagiannis
File Size : 69.4 MB
Format : PDF, ePub, Docs
Download : 102
Read : 420
Download »
"In many large-scale optimization problems arising in the context of machine learning the decision variable is of high-dimension and the objective function decomposes into a sum over a large number of terms (one for each instance in the training data set).In this setting, second-order optimization methods such as Newton or quasi-Newton methods, are not tractable due to the complexity of evaluating and inverting the Hessian, or an approximation thereof. Also, the vast amount of data available is spread around multiple servers making a centralized optimization solution sub optimal or impossible. Therefore we concentrate on first order methods that are scalable in a decentralized setting.In this thesis we provide a framework for distributed delayed convex optimization methods for networks in a master-server setting. Our goal is to optimize a global objective function which is the sum of the local objective functions of the agents in the network.We review Nesterov's accelerated algorithm for centralized optimization since it is the optimal algorithm for the class of convex, and strongly convex functions and to modify it accordingly for decentralized optimization in the master-server setting.It is natural that in an asynchronous setting the current value of the server node is a past value of the master node communicated some time steps ago, and thus gives rise to delays in the analysis. We have proven that a delayed accelerated method maintains the optimality of the algorithm with a convergence rate of O(1/t2). We have also performed simulations and we have verified that the accelerated algorithm performs better that the alternative algorithms for decentralized optimization in a master server setting." --

Learning and Intelligent Optimization

Author : Roberto Battiti
File Size : 62.17 MB
Format : PDF, ePub, Docs
Download : 319
Read : 904
Download »
This book constitutes the thoroughly refereed post-conference proceedings of the 12th International Conference on Learning and Intelligent Optimization, LION 12, held in Kalamata, Greece, in June 2018. The 28 full papers and 12 short papers presented have been carefully reviewed and selected from 62 submissions. The papers explore the advanced research developments in such interconnected fields as mathematical programming, global optimization, machine learning, and artificial intelligence. Special focus is given to advanced ideas, technologies, methods, and applications in optimization and machine learning.

Mathematical Theories of Machine Learning Theory and Applications

Author : Bin Shi
File Size : 37.29 MB
Format : PDF
Download : 405
Read : 972
Download »
This book studies mathematical theories of machine learning. The first part of the book explores the optimality and adaptivity of choosing step sizes of gradient descent for escaping strict saddle points in non-convex optimization problems. In the second part, the authors propose algorithms to find local minima in nonconvex optimization and to obtain global minima in some degree from the Newton Second Law without friction. In the third part, the authors study the problem of subspace clustering with noisy and missing data, which is a problem well-motivated by practical applications data subject to stochastic Gaussian noise and/or incomplete data with uniformly missing entries. In the last part, the authors introduce an novel VAR model with Elastic-Net regularization and its equivalent Bayesian model allowing for both a stable sparsity and a group selection.

Machine Learning and Knowledge Discovery in Databases

Author : Frank Hutter
File Size : 82.1 MB
Format : PDF, ePub, Mobi
Download : 655
Read : 671
Download »

Accelerating Convex Optimization in Machine Learning by Leveraging Functional Growth Conditions

Author : Yi Xu (Algorithm engineer)
File Size : 75.43 MB
Format : PDF, Kindle
Download : 380
Read : 647
Download »
In recent years, unprecedented growths in scale and dimensionality of data raise big computational challenges for traditional optimization algorithms; thus it becomes very important to develop efficient and effective optimization algorithms for solving numerous machine learning problems. Many traditional algorithms (e.g., gradient descent method) are black-box algorithms, which are simple to implement but ignore the underlying geometrical property of the objective function. Recent trend in accelerating these traditional black-box algorithms is to leverage geometrical properties of the objective function such as strong convexity. However, most existing methods rely too much on the knowledge of strong convexity, which makes them not applicable to problems without strong convexity or without knowledge of strong convexity. To bridge the gap between traditional black-box algorithms without knowing the problem's geometrical property and accelerated algorithms under strong convexity, how can we develop adaptive algorithms that can be adaptive to the objective function's underlying geometrical property? To answer this question, in this dissertation we focus on convex optimization problems and propose to explore an error bound condition that characterizes the functional growth condition of the objective function around a global minimum. Under this error bound condition, we develop algorithms that (1) can adapt to the problem's geometrical property to enjoy faster convergence in stochastic optimization; (2) can leverage the problem's structural regularizer to further improve the convergence speed; (3) can address both deterministic and stochastic optimization problems with explicit max-structural loss; (4) can leverage the objective function's smoothness property to improve the convergence rate for stochastic optimization. We first considered stochastic optimization problems with general stochastic loss.

Accelerated Development of Photovoltaics by Physics informed Machine Learning

Author : Juan Felipe Oviedo Perhavec
File Size : 65.69 MB
Format : PDF
Download : 470
Read : 753
Download »
Terawatt-scale deployment of photovoltaics (PV) is required to mitigate the most severe effects of climate change. Despite sustained growth in PV installations, technoeconomic models suggest that further technical advances and cost reduction are required to enable a timely energy transition in the next 10 - 15 years. This limited timeline is incompatible with the historic rate of materials development: solar cell technologies have taken decades to transition from the laboratory to large-scale commercial applications. Recently, the convergence of high-performance computing, high-throughput experimentation and machine learning has shown great promise to accelerate scientific research. In this context, this thesis proposes and demonstrates a comprehensive methodology for accelerated PV development. Machine learning constitutes a key component of the new framework, effectively reconciling the formerly disjoint aspects of first-principles simulation, experimental fabrication and in-depth characterization. This integration is achieved by judiciously formalizing material science problems, and developing and adapting algorithms according to physical principles. Under this interdisciplinary perspective, the physics-informed machine learning approach allows a 3 - 30x acceleration in various aspects of PV development. This work focuses in two particular areas. The first thrust aims to accelerate the screening and optimization of early-stage PV absorbers. The high-dimensionality of the material space, and the sparsity of experimental information, make early-stage material development challenging. First, I address the structural characterization bottleneck in material screening using deep learning techniques and physics-inspired data augmentation. Then, I develop a physics-constrained Bayesian optimization algorithm to efficiently optimize material compositions, fusing experimentation and density functional theory with stochastic constraints. These advancements lead to the discovery of several promising lead-free perovskites, and a 3x more stable multication lead halide perovskite. The second thrust aims to accelerate the industrial transition of more mature PV devices. For this purpose, I reformulate the traditional record-efficiency figure of merit to include probabilistic and manufacturing considerations, allowing industrially-relevant optimization. Then, a scalable physical inference algorithm is developed by a principled combination of Bayesian inference, deep learning and physical models. This inference model efficiently provides physical insights leading to > 3x faster solar cell optimization. Finally, this approach is expanded to solar cell degradation diagnosis. I reduce the characterization time by > 5x using time-series forecasting methods. Then, the scalable inference model is combined with a game-theoretic interpretability algorithm to elucidate physical factors driving degradation. Together, these methodology and results can dramatically accelerate PV technology development, and have a timely impact in climate change. The physics-informed models expand the horizon of applied machine learning, and the fundamental approach of this work is applicable to other energy materials and systems, such as thermoelectrics and batteries.

Machine Learning Optimization and Data Science

Author : Giuseppe Nicosia
File Size : 51.33 MB
Format : PDF, Kindle
Download : 758
Read : 304
Download »
This two-volume set, LNCS 12565 and 12566, constitutes the refereed proceedings of the 6th International Conference on Machine Learning, Optimization, and Data Science, LOD 2020, held in Siena, Italy, in July 2020. The total of 116 full papers presented in this two-volume post-conference proceedings set was carefully reviewed and selected from 209 submissions. These research articles were written by leading scientists in the fields of machine learning, artificial intelligence, reinforcement learning, computational optimization, and data science presenting a substantial array of ideas, technologies, algorithms, methods, and applications.

Advances in Machine Learning and Computational Intelligence

Author : Srikanta Patnaik
File Size : 75.2 MB
Format : PDF, ePub, Docs
Download : 490
Read : 517
Download »
This book gathers selected high-quality papers presented at the International Conference on Machine Learning and Computational Intelligence (ICMLCI-2019), jointly organized by Kunming University of Science and Technology and the Interscience Research Network, Bhubaneswar, India, from April 6 to 7, 2019. Addressing virtually all aspects of intelligent systems, soft computing and machine learning, the topics covered include: prediction; data mining; information retrieval; game playing; robotics; learning methods; pattern visualization; automated knowledge acquisition; fuzzy, stochastic and probabilistic computing; neural computing; big data; social networks and applications of soft computing in various areas.

Optimal Stochastic and Distributed Algorithms for Machine Learning

Author : Hua Ouyang
File Size : 56.55 MB
Format : PDF, Docs
Download : 207
Read : 1098
Download »
Stochastic and data-distributed optimization algorithms have received lots of attention from the machine learning community due to the tremendous demand from the large-scale learning and the big-data related optimization. A lot of stochastic and deterministic learning algorithms are proposed recently under various application scenarios. Nevertheless, many of these algorithms are based on heuristics and their optimality in terms of the generalization error is not sufficiently justified. In this talk, I will explain the concept of an optimal learning algorithm, and show that given a time budget and proper hypothesis space, only those achieving the lower bounds of the estimation error and the optimization error are optimal. Guided by this concept, we investigated the stochastic minimization of nonsmooth convex loss functions, a central problem in machine learning. We proposed a novel algorithm named Accelerated Nonsmooth Stochastic Gradient Descent, which exploits the structure of common nonsmooth loss functions to achieve optimal convergence rates for a class of problems including SVMs. It is the first stochastic algorithm that can achieve the optimal O(1/t) rate for minimizing nonsmooth loss functions. The fast rates are confirmed by empirical comparisons with state-of-the-art algorithms including the averaged SGD. The Alternating Direction Method of Multipliers (ADMM) is another flexible method to explore function structures. In the second part we proposed stochastic ADMM that can be applied to a general class of convex and nonsmooth functions, beyond the smooth and separable least squares loss used in lasso. We also demonstrate the rates of convergence for our algorithm under various structural assumptions of the stochastic function: O(1/sqrt{t}) for convex functions and O(log t/t) for strongly convex functions. A novel application named Graph-Guided SVM is proposed to demonstrate the usefulness of our algorithm. We also extend the scalability of stochastic algorithms to nonlinear kernel machines, where the problem is formulated as a constrained dual quadratic optimization. The simplex constraint can be handled by the classic Frank-Wolfe method. The proposed stochastic Frank-Wolfe methods achieve comparable or even better accuracies than state-of-the-art batch and online kernel SVM solvers, and are significantly faster. The last part investigates the problem of data-distributed learning. We formulate it as a consensus-constrained optimization problem and solve it with ADMM. It turns out that the underlying communication topology is a key factor in achieving a balance between a fast learning rate and computation resource consumption. We analyze the linear convergence behavior of consensus ADMM so as to characterize the interplay between the communication topology and the penalty parameters used in ADMM. We observe that given optimal parameters, the complete bipartite and the master-slave graphs exhibit the fastest convergence, followed by bi-regular graphs.

Mathematical Optimization Theory and Operations Research

Author : Yury Kochetov
File Size : 82.36 MB
Format : PDF, Mobi
Download : 808
Read : 246
Download »
This book constitutes refereed proceedings of the 19th International Conference on Mathematical Optimization Theory and Operations Research, MOTOR 2020, held in July 2020. Due to the COVID-19 pandemic the conference was held online. The 25 full papers and 8 short papers presented in this volume were carefully reviewed and selected from a total of 102 submissions. The papers in the volume are organised according to the following topical headings: ​combinatorial optimization; mathematical programming; global optimization; game theory and mathematical economics; heuristics and metaheuristics; machine learning and data analysis.

Principles and Methods for Accelerated Catalyst Design and Testing

Author : E.G. Derouane
File Size : 72.96 MB
Format : PDF, Mobi
Download : 464
Read : 957
Download »
High throughput experimentation has met great success in drug design but it has, so far, been scarcely used in the field ofcatalysis. We present in this book the outcome of a NATO ASI meeting that was held in Vilamoura, Portugal, between July 15 and 28, 2001, with the objective of delineating and consolidating the principles and methods underpinning accelerated catalyst design, evaluation, and development. There is a need to make the underlying principles of this new methodology more widely understood and to make it available in a coherent and integrated format. The latter objective is particularly important to the young scientists who will constitute the new catalysis researchers generation. Indeed, this field which is at the frontier offundamental science and may be a renaissance for catalysis, is one which is much more complex than classical catalysis itself. It implies a close collaboration between scientists from many disciplines (chemistry, physics, chemical and mechanical engineering, automation, robotics, and scientific computing in general). In addition, this emerging area of science is also of paramount industrial importance, as progress in this area would collapse the time necessary to discover new catalysts or improve existing ones.