Manfred K. Warmuth
YOU?
Author Swipe
View article: Selective Matching Losses -- Not All Scores Are Created Equal
Selective Matching Losses -- Not All Scores Are Created Equal Open
Learning systems match predicted scores to observations over some domain. Often, it is critical to produce accurate predictions in some subset (or region) of the domain, yet less important to accurately predict in other regions. We constru…
View article: RANK-SMOOTHED PAIRWISE LEARNING IN PERCEPTUAL QUALITY ASSESSMENT
RANK-SMOOTHED PAIRWISE LEARNING IN PERCEPTUAL QUALITY ASSESSMENT Open
Conducting pairwise comparisons is a widely used approach in curating human perceptual preference data. Typically raters are instructed to make their choices according to a specific set of rules that address certain dimensions of image qua…
View article: Optimal Transport with Tempered Exponential Measures
Optimal Transport with Tempered Exponential Measures Open
In the field of optimal transport, two prominent subfields face each other: (i) unregularized optimal transport, ``a-la-Kantorovich'', which leads to extremely sparse plans but with algorithms that scale poorly, and (ii) entropic-regulariz…
View article: Noise misleads rotation invariant algorithms on sparse targets
Noise misleads rotation invariant algorithms on sparse targets Open
It is well known that the class of rotation invariant algorithms are suboptimal even for learning sparse linear problems when the number of examples is below the "dimension" of the problem. This class includes any gradient descent trained …
View article: Tempered Calculus for ML: Application to Hyperbolic Model Embedding
Tempered Calculus for ML: Application to Hyperbolic Model Embedding Open
Most mathematical distortions used in ML are fundamentally integral in nature: $f$-divergences, Bregman divergences, (regularized) optimal transport distances, integral probability metrics, geodesic distances, etc. In this paper, we unveil…
View article: The Tempered Hilbert Simplex Distance and Its Application To Non-linear Embeddings of TEMs
The Tempered Hilbert Simplex Distance and Its Application To Non-linear Embeddings of TEMs Open
Tempered Exponential Measures (TEMs) are a parametric generalization of the exponential family of distributions maximizing the tempered entropy function among positive measures subject to a probability normalization of their power densitie…
View article: Optimal Transport with Tempered Exponential Measures
Optimal Transport with Tempered Exponential Measures Open
In the field of optimal transport, two prominent subfields face each other: (i) unregularized optimal transport, "à-la-Kantorovich", which leads to extremely sparse plans but with algorithms that scale poorly, and (ii) entropic-regularized…
View article: Boosting with Tempered Exponential Measures
Boosting with Tempered Exponential Measures Open
One of the most popular ML algorithms, AdaBoost, can be derived from the dual of a relative entropy minimization problem subject to the fact that the positive weights on the examples sum to one. Essentially, harder examples receive higher …
View article: A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Tasks
A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Tasks Open
We study the phenomenon of \textit{in-context learning} (ICL) exhibited by large language models, where they can adapt to a new learning task, given a handful of labeled examples, without any explicit parameter optimization. Our goal is to…
View article: Clustering above Exponential Families with Tempered Exponential Measures
Clustering above Exponential Families with Tempered Exponential Measures Open
The link with exponential families has allowed $k$-means clustering to be generalized to a wide variety of data generating distributions in exponential families and clustering distortions among Bregman divergences. Getting the framework to…
View article: Layerwise Bregman Representation Learning with Applications to Knowledge Distillation
Layerwise Bregman Representation Learning with Applications to Knowledge Distillation Open
In this work, we propose a novel approach for layerwise representation learning of a trained neural network. In particular, we form a Bregman divergence based on the layer's transfer function and construct an extension of the original Breg…
View article: Learning from Randomly Initialized Neural Network Features
Learning from Randomly Initialized Neural Network Features Open
We present the surprising result that randomly initialized neural networks are good feature extractors in expectation. These random features correspond to finite-sample realizations of what we call Neural Network Prior Kernel (NNPK), which…
View article: Unlabeled sample compression schemes and corner peelings for ample and maximum classes
Unlabeled sample compression schemes and corner peelings for ample and maximum classes Open
View article: Step-size Adaptation Using Exponentiated Gradient Updates
Step-size Adaptation Using Exponentiated Gradient Updates Open
Optimizers like Adam and AdaGrad have been very successful in training large-scale neural networks. Yet, the performance of these methods is heavily dependent on a carefully tuned learning rate schedule. We show that in many large-scale ap…
View article: LocoProp: Enhancing BackProp via Local Loss Optimization
LocoProp: Enhancing BackProp via Local Loss Optimization Open
Second-order methods have shown state-of-the-art performance for optimizing deep neural networks. Nonetheless, their large memory requirement and high computational complexity, compared to first-order methods, hinder their versatility in a…
View article: Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond
Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond Open
Many learning tasks in machine learning can be viewed as taking a gradient step towards minimizing the average loss of a batch of examples in each training iteration. When noise is prevalent in the data, this uniform treatment of examples …
View article: A case where a spindly two-layer linear network whips any neural network with a fully connected input layer
A case where a spindly two-layer linear network whips any neural network with a fully connected input layer Open
It was conjectured that any neural network of any structure and arbitrary differentiable transfer functions at the nodes cannot learn the following problem sample efficiently when trained with gradient descent: The instances are the rows o…
View article: An Implicit Form of Krasulina's k-PCA Update without the Orthonormality Constraint
An Implicit Form of Krasulina's k-PCA Update without the Orthonormality Constraint Open
We shed new insights on the two commonly used updates for the online k-PCA problem, namely, Krasulina's and Oja's updates. We show that Krasulina's update corresponds to a projected gradient descent step on the Stiefel manifold of orthonor…
View article: Reparameterizing Mirror Descent as Gradient Descent
Reparameterizing Mirror Descent as Gradient Descent Open
Most of the recent successful applications of neural networks have been based on training with gradient descent updates. However, for some small networks, other mirror descent updates learn provably more efficiently when the target is spar…
View article: Interpolating Between Gradient Descent and Exponentiated Gradient Using Reparameterized Gradient Descent.
Interpolating Between Gradient Descent and Exponentiated Gradient Using Reparameterized Gradient Descent. Open
Continuous-time mirror descent (CMD) can be seen as the limit case of the discrete-time MD update when the step-size is infinitesimally small. In this paper, we focus on the geometry of the primal and dual CMD updates and introduce a gener…
View article: TriMap: Large-scale Dimensionality Reduction Using Triplets
TriMap: Large-scale Dimensionality Reduction Using Triplets Open
We introduce "TriMap"; a dimensionality reduction technique based on triplet constraints, which preserves the global structure of the data better than the other commonly used methods such as t-SNE, LargeVis, and UMAP. To quantify the globa…
View article: An Implicit Form of Krasulina's k-PCA Update without the Orthonormality\n Constraint
An Implicit Form of Krasulina's k-PCA Update without the Orthonormality\n Constraint Open
We shed new insights on the two commonly used updates for the online $k$-PCA\nproblem, namely, Krasulina's and Oja's updates. We show that Krasulina's update\ncorresponds to a projected gradient descent step on the Stiefel manifold of the\…
View article: Mistake bounds on the noise-free multi-armed bandit game
Mistake bounds on the noise-free multi-armed bandit game Open
View article: Unbiased estimators for random design regression
Unbiased estimators for random design regression Open
In linear regression we wish to estimate the optimum linear least squares predictor for a distribution over $d$-dimensional input points and real-valued responses, based on a small sample. Under standard random design analysis, where the s…
View article: Unbiased estimators for random design regression
Unbiased estimators for random design regression Open
In linear regression we wish to estimate the optimum linear least squares predictor for a distribution over $d$-dimensional input points and real-valued responses, based on a small sample. Under standard random design analysis, where the s…
View article: Robust Bi-Tempered Logistic Loss Based on Bregman Divergences
Robust Bi-Tempered Logistic Loss Based on Bregman Divergences Open
We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperatur…
View article: Adaptive scale-invariant online algorithms for learning linear models
Adaptive scale-invariant online algorithms for learning linear models Open
We consider online learning with linear models, where the algorithm predicts on sequentially revealed instances (feature vectors), and is compared against the best linear function (comparator) in hindsight. Popular algorithms in this frame…
View article: Divergence-Based Motivation for Online EM and Combining Hidden Variable Models
Divergence-Based Motivation for Online EM and Combining Hidden Variable Models Open
Expectation-Maximization (EM) is a prominent approach for parameter estimation of hidden (aka latent) variable models. Given the full batch of data, EM forms an upper-bound of the negative log-likelihood of the model at each iteration and …
View article: Divergence-Based Motivation for Online EM and Combining Hidden Variable\n Models
Divergence-Based Motivation for Online EM and Combining Hidden Variable\n Models Open
Expectation-Maximization (EM) is a prominent approach for parameter\nestimation of hidden (aka latent) variable models. Given the full batch of\ndata, EM forms an upper-bound of the negative log-likelihood of the model at\neach iteration a…
View article: Minimax experimental design: Bridging the gap between statistical and worst-case approaches to least squares regression
Minimax experimental design: Bridging the gap between statistical and worst-case approaches to least squares regression Open
In experimental design, we are given a large collection of vectors, each with a hidden response value that we assume derives from an underlying linear model, and we wish to pick a small subset of the vectors such that querying the correspo…