Michael E. Sander
YOU?
Author Swipe
View article: Loss Functions and Operators Generated by f-Divergences
Loss Functions and Operators Generated by f-Divergences Open
The logistic loss (a.k.a. cross-entropy loss) is one of the most popular loss functions used for multiclass classification. It is also the loss function of choice for next-token prediction in language modeling. It is associated with the Ku…
View article: Joint Learning of Energy-based Models and their Partition Function
Joint Learning of Energy-based Models and their Partition Function Open
Energy-based models (EBMs) offer a flexible framework for parameterizing probability distributions using neural networks. However, learning EBMs by exact maximum likelihood estimation (MLE) is generally intractable, due to the need to comp…
View article: Vers un apprentissage plus profond : réseaux résiduels, équations différentielles neuronales et transformers, en théorie et en pratique
Vers un apprentissage plus profond : réseaux résiduels, équations différentielles neuronales et transformers, en théorie et en pratique Open
This PhD thesis presents contributions to the field of deep learning. From convolutional ResNets to Transformers, residual connections are ubiquitous in state-of-the-art deep learning models. The continuous depth analogues of residual netw…
View article: Towards Understanding the Universality of Transformers for Next-Token Prediction
Towards Understanding the Universality of Transformers for Next-Token Prediction Open
Causal Transformers are trained to predict the next token for a given context. While it is widely accepted that self-attention is crucial for encoding the causal structure of sequences, the precise underlying mechanism behind this in-conte…
View article: How do Transformers perform In-Context Autoregressive Learning?
How do Transformers perform In-Context Autoregressive Learning? Open
Transformers have achieved state-of-the-art performance in language modeling tasks. However, the reasons behind their tremendous success are still unclear. In this paper, towards a better understanding, we train a Transformer model on a si…
View article: Implicit regularization of deep residual networks towards neural ODEs
Implicit regularization of deep residual networks towards neural ODEs Open
Residual neural networks are state-of-the-art deep learning models. Their continuous-depth analog, neural ordinary differential equations (ODEs), are also widely used. Despite their success, the link between the discrete and continuous mod…
View article: Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective
Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective Open
The top-k operator returns a sparse vector, where the non-zero values correspond to the k largest values of the input. Unfortunately, because it is a discontinuous function, it is difficult to incorporate in neural networks trained end-to-…
View article: Vision Transformers provably learn spatial structure
Vision Transformers provably learn spatial structure Open
Vision Transformers (ViTs) have achieved comparable or superior performance than Convolutional Neural Networks (CNNs) in computer vision. This empirical breakthrough is even more remarkable since, in contrast to CNNs, ViTs do not embed any…
View article: Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
Do Residual Neural Networks discretize Neural Ordinary Differential Equations? Open
Neural Ordinary Differential Equations (Neural ODEs) are the continuous analog of Residual Neural Networks (ResNets). We investigate whether the discrete dynamics defined by a ResNet are close to the continuous one of a Neural ODE. We firs…
View article: Sinkformers: Transformers with Doubly Stochastic Attention
Sinkformers: Transformers with Doubly Stochastic Attention Open
Attention based models such as Transformers involve pairwise interactions between data points, modeled with a learnable attention matrix. Importantly, this attention matrix is normalized with the SoftMax operator, which makes it row-wise s…
View article: Momentum Residual Neural Networks
Momentum Residual Neural Networks Open
The training of deep residual neural networks (ResNets) with backpropagation has a memory cost that increases linearly with respect to the depth of the network. A way to circumvent this issue is to use reversible architectures. In this pap…