Stochastic Optimization ⁸

Optimization Papers in JMLR Volume 26

Optimization Research Papers in JMLR Volume 25

Optimization Research Papers in JMLR Volume 25 (2024) # This document lists papers from JMLR Volume 25 (2024) that focus on optimization research, categorized by their primary themes. Each paper is numbered starting from 1 within its subsection, with a brief description of its key contributions to optimization theory, algorithms, or applications. Convex Optimization # Papers addressing convex optimization problems, including sparse NMF, differential privacy, and sparse regression. Lower Complexity Bounds of Finite-Sum Optimization Problems: The Results and Construction Authors: Yuze Han, Guangzeng Xie, Zhihua Zhang Description: Investigates lower complexity bounds for finite-sum optimization problems in convex settings.

Mathematics - Optimization

Branches of Optimization Research # Convex Optimization # Convex optimization focuses on problems where the objective function and constraints are convex, ensuring a single global optimum. This field is foundational in machine learning, signal processing, and control systems due to its guaranteed convergence and efficient algorithms. Convex Optimization by Boyd and Vandenberghe - PDF Convex Optimization Theory by Dimitri P. Bertsekas - PDF Discrete, Combinatorial, and Integer Optimization # This branch deals with optimization problems involving discrete variables, such as integers or combinatorial structures, often encountered in scheduling, network design, and logistics. Bayesian optimization, a subset, is particularly useful for optimizing expensive black-box functions.

Second-order Stochastic Optimization methods for Machine Learning

Analysis of the Hessian # 1. Empirical Analysis of the Hessian of Over-Parametrized Neural Networks # Year: 2017 Authors: Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, Leon Bottou ArXiv ID: arXiv:1706.04454 URL: https://arxiv.org/abs/1706.04454 Abstract: We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk. We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et al. (2016): Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers. We believe that our observations have striking implications for non-convex optimization in high dimensions. First, the flatness of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading. And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create large connected components at the bottom of the landscape. Second, the dependence of small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs. With this in mind, we may reevaluate the connections within the data-architecture-algorithm framework of a model, hoping that it would shed light into the geometry of high-dimensional and non-convex spaces in modern applications. In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.

Optimization Research Papers in JMLR Volume 24

Optimization Research Papers in JMLR Volume 24 (2023) # This document lists papers from JMLR Volume 24 (2023) that focus on optimization research, categorized by their primary themes. Each paper is numbered starting from 1 within its subsection, with a brief description of its key contributions to optimization theory, algorithms, or applications. Convex Optimization # Papers addressing convex optimization problems, including sparse PCA, L0 regularization, and matrix decomposition. Sparse PCA: A Geometric Approach Authors: Dimitris Bertsimas, Driss Lahlou Kitane Description: Develops a geometric approach for sparse principal component analysis using convex optimization techniques.

Optimization Research Papers in JMLR Volume 23

Optimization Research Papers in JMLR Volume 23 (2022) # This document lists papers from JMLR Volume 23 (2022) that focus on optimization research, categorized by their primary themes. Each paper is numbered starting from 1 within its subsection, with a brief description of its key contributions to optimization theory, algorithms, or applications. Convex Optimization # Papers addressing convex optimization problems, including sparse PCA, L1-regularized SVMs, and metric-constrained problems. Solving Large-Scale Sparse PCA to Certifiable (Near) Optimality Authors: Dimitris Bertsimas, Ryan Cory-Wright, Jean Pauphilet Description: Develops convex optimization techniques for large-scale sparse principal component analysis with certifiable near-optimal solutions.

Optimization Research Papers in JMLR Volume 22

Optimization Research Papers in JMLR Volume 22 (2021) # This document lists papers from JMLR Volume 22 (2021) that focus on optimization research, categorized by their primary themes. Each paper is numbered starting from 1 within its subsection, with a brief description of its key contributions to optimization theory, algorithms, or applications. Convex Optimization # Papers addressing convex optimization problems, including clustering, Wasserstein barycenters, sparse optimization, and bandits. Convex Clustering: Model, Theoretical Guarantee and Efficient Algorithm Authors: Defeng Sun, Kim-Chuan Toh, Yancheng Yuan Description: Proposes a convex clustering model with theoretical guarantees and an efficient algorithm.

Optimization Research Papers in JMLR Volume 21

Optimization Research Papers in JMLR Volume 21 (2020) # This document lists papers from JMLR Volume 21 (2020) that focus on optimization research, categorized by their primary themes. Each paper is numbered starting from 1 within its subsection, with a brief description of its key contributions to optimization theory, algorithms, or applications. Convex Optimization # Papers addressing convex optimization problems, including complexity bounds, convergence analysis, and applications in regression and assortment optimization.