Ehsan Amid
YOU?
Author Swipe
View article: Restructuring Vector Quantization with the Rotation Trick
Restructuring Vector Quantization with the Rotation Trick Open
Vector Quantized Variational AutoEncoders (VQ-VAEs) are designed to compress a continuous input to a discrete latent space and reconstruct it with minimal distortion. They operate by maintaining a set of vectors -- often referred to as the…
View article: RANK-SMOOTHED PAIRWISE LEARNING IN PERCEPTUAL QUALITY ASSESSMENT
RANK-SMOOTHED PAIRWISE LEARNING IN PERCEPTUAL QUALITY ASSESSMENT Open
Conducting pairwise comparisons is a widely used approach in curating human perceptual preference data. Typically raters are instructed to make their choices according to a specific set of rules that address certain dimensions of image qua…
View article: Optimal Transport with Tempered Exponential Measures
Optimal Transport with Tempered Exponential Measures Open
In the field of optimal transport, two prominent subfields face each other: (i) unregularized optimal transport, ``a-la-Kantorovich'', which leads to extremely sparse plans but with algorithms that scale poorly, and (ii) entropic-regulariz…
View article: Learning from straggler clients in federated learning
Learning from straggler clients in federated learning Open
How well do existing federated learning algorithms learn from client devices that return model updates with a significant time delay? Is it even possible to learn effectively from clients that report back minutes, hours, or days after bein…
View article: Noise misleads rotation invariant algorithms on sparse targets
Noise misleads rotation invariant algorithms on sparse targets Open
It is well known that the class of rotation invariant algorithms are suboptimal even for learning sparse linear problems when the number of examples is below the "dimension" of the problem. This class includes any gradient descent trained …
View article: Tempered Calculus for ML: Application to Hyperbolic Model Embedding
Tempered Calculus for ML: Application to Hyperbolic Model Embedding Open
Most mathematical distortions used in ML are fundamentally integral in nature: $f$-divergences, Bregman divergences, (regularized) optimal transport distances, integral probability metrics, geodesic distances, etc. In this paper, we unveil…
View article: The Tempered Hilbert Simplex Distance and Its Application To Non-linear Embeddings of TEMs
The Tempered Hilbert Simplex Distance and Its Application To Non-linear Embeddings of TEMs Open
Tempered Exponential Measures (TEMs) are a parametric generalization of the exponential family of distributions maximizing the tempered entropy function among positive measures subject to a probability normalization of their power densitie…
View article: Context-Aware Meta-Learning
Context-Aware Meta-Learning Open
Large Language Models like ChatGPT demonstrate a remarkable capacity to learn new concepts during inference without any fine-tuning. However, visual models trained to detect new objects during inference have been unable to replicate this a…
View article: Heterogeneous Federated Learning Using Knowledge Codistillation
Heterogeneous Federated Learning Using Knowledge Codistillation Open
Federated Averaging, and many federated learning algorithm variants which build upon it, have a limitation: all clients must share the same model architecture. This results in unused modeling capacity on many clients, which limits model pe…
View article: Distributionally Robust Post-hoc Classifiers under Prior Shifts
Distributionally Robust Post-hoc Classifiers under Prior Shifts Open
The generalization ability of machine learning models degrades significantly when the test distribution shifts away from the training distribution. We investigate the problem of training models that are robust to shifts caused by changes i…
View article: Optimal Transport with Tempered Exponential Measures
Optimal Transport with Tempered Exponential Measures Open
In the field of optimal transport, two prominent subfields face each other: (i) unregularized optimal transport, "à-la-Kantorovich", which leads to extremely sparse plans but with algorithms that scale poorly, and (ii) entropic-regularized…
View article: To Aggregate or Not? Learning with Separate Noisy Labels
To Aggregate or Not? Learning with Separate Noisy Labels Open
The rawly collected training data often comes with separate noisy labels collected from multiple imperfect annotators (e.g., via crowdsourcing). A typical way of using these separate labels is to first aggregate them into one and apply sta…
View article: Benchmarking Neural Network Training Algorithms
Benchmarking Neural Network Training Algorithms Open
Training algorithms, broadly construed, are an essential part of every deep learning pipeline. Training algorithm improvements that speed up training across a wide variety of workloads (e.g., better update rules, tuning protocols, learning…
View article: Boosting with Tempered Exponential Measures
Boosting with Tempered Exponential Measures Open
One of the most popular ML algorithms, AdaBoost, can be derived from the dual of a relative entropy minimization problem subject to the fact that the positive weights on the examples sum to one. Essentially, harder examples receive higher …
View article: Implicit Geometry and Interaction Embeddings Improve Few-Shot Molecular Property Prediction
Implicit Geometry and Interaction Embeddings Improve Few-Shot Molecular Property Prediction Open
Few-shot learning is a promising approach to molecular property prediction as supervised data is often very limited. However, many important molecular properties depend on complex molecular characteristics -- such as the various 3D geometr…
View article: Clustering above Exponential Families with Tempered Exponential Measures
Clustering above Exponential Families with Tempered Exponential Measures Open
The link with exponential families has allowed $k$-means clustering to be generalized to a wide variety of data generating distributions in exponential families and clustering distortions among Bregman divergences. Getting the framework to…
View article: Layerwise Bregman Representation Learning with Applications to Knowledge Distillation
Layerwise Bregman Representation Learning with Applications to Knowledge Distillation Open
In this work, we propose a novel approach for layerwise representation learning of a trained neural network. In particular, we form a Bregman divergence based on the layer's transfer function and construct an extension of the original Breg…
View article: To Aggregate or Not? Learning with Separate Noisy Labels
To Aggregate or Not? Learning with Separate Noisy Labels Open
The rawly collected training data often comes with separate noisy labels collected from multiple imperfect annotators (e.g., via crowdsourcing). A typical way of using these separate labels is to first aggregate them into one and apply sta…
View article: Extracting Targeted Training Data from ASR Models, and How to Mitigate It
Extracting Targeted Training Data from ASR Models, and How to Mitigate It Open
Recent work has designed methods to demonstrate that model updates in ASR training can leak potentially sensitive attributes of the utterances used in computing the updates. In this work, we design the first method to demonstrate informati…
View article: Learning from Randomly Initialized Neural Network Features
Learning from Randomly Initialized Neural Network Features Open
We present the surprising result that randomly initialized neural networks are good feature extractors in expectation. These random features correspond to finite-sample realizations of what we call Neural Network Prior Kernel (NNPK), which…
View article: Step-size Adaptation Using Exponentiated Gradient Updates
Step-size Adaptation Using Exponentiated Gradient Updates Open
Optimizers like Adam and AdaGrad have been very successful in training large-scale neural networks. Yet, the performance of these methods is heavily dependent on a carefully tuned learning rate schedule. We show that in many large-scale ap…
View article: Public Data-Assisted Mirror Descent for Private Model Training
Public Data-Assisted Mirror Descent for Private Model Training Open
In this paper, we revisit the problem of using in-distribution public data to improve the privacy/utility trade-offs for differentially private (DP) model training. (Here, public data refers to auxiliary data sets that have no privacy conc…
View article: Constrained Instance and Class Reweighting for Robust Learning under Label Noise
Constrained Instance and Class Reweighting for Robust Learning under Label Noise Open
Deep neural networks have shown impressive performance in supervised learning, enabled by their ability to fit well to the provided training data. However, their performance is largely dependent on the quality of the training data and ofte…
View article: Constrained Instance and Class Reweighting for Robust Learning under\n Label Noise
Constrained Instance and Class Reweighting for Robust Learning under\n Label Noise Open
Deep neural networks have shown impressive performance in supervised\nlearning, enabled by their ability to fit well to the provided training data.\nHowever, their performance is largely dependent on the quality of the training\ndata and o…
View article: Privacy-Preserving Wireless Federated Learning Exploiting Inherent Hardware Impairments
Privacy-Preserving Wireless Federated Learning Exploiting Inherent Hardware Impairments Open
We consider a wireless federated learning system where multiple data holder edge devices collaborate to train a global model via sharing their parameter updates with an honest-but-curious parameter server. We demonstrate that the inherent …
View article: Efficiently Identifying Task Groupings for Multi-Task Learning
Efficiently Identifying Task Groupings for Multi-Task Learning Open
Multi-task learning can leverage information learned by one task to benefit the training of other tasks. Despite this capacity, naively training all tasks together in one model often degrades performance, and exhaustively searching through…
View article: LocoProp: Enhancing BackProp via Local Loss Optimization
LocoProp: Enhancing BackProp via Local Loss Optimization Open
Second-order methods have shown state-of-the-art performance for optimizing deep neural networks. Nonetheless, their large memory requirement and high computational complexity, compared to first-order methods, hinder their versatility in a…
View article: Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond
Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond Open
Many learning tasks in machine learning can be viewed as taking a gradient step towards minimizing the average loss of a batch of examples in each training iteration. When noise is prevalent in the data, this uniform treatment of examples …
View article: Privacy-Preserving Wireless Federated Learning Exploiting Inherent Hardware Impairments
Privacy-Preserving Wireless Federated Learning Exploiting Inherent Hardware Impairments Open
We consider a wireless federated learning system where multiple data holder edge devices collaborate to train a global model via sharing their parameter updates with an honest-but-curious parameter server. We demonstrate that the inherent …
View article: Measuring and Harnessing Transference in Multi-Task Learning
Measuring and Harnessing Transference in Multi-Task Learning Open
Multi-task learning can leverage information learned by one task to benefit the training of other tasks. Despite this capacity, naive formulations often degrade performance and in particular, identifying the tasks that would benefit from c…