Behrooz Ghorbani
YOU?
Author Swipe
View article: OpenAI o1 System Card
OpenAI o1 System Card Open
The o1 model series is trained with large-scale reinforcement learning to reason using chain of thought. These advanced reasoning capabilities provide new avenues for improving the safety and robustness of our models. In particular, our mo…
View article: GPT-4o System Card
GPT-4o System Card Open
GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It's trained end-to-end across text, vision, and audio, meaning…
View article: Order Matters in the Presence of Dataset Imbalance for Multilingual Learning
Order Matters in the Presence of Dataset Imbalance for Multilingual Learning Open
In this paper, we empirically study the optimization dynamics of multi-task learning, particularly focusing on those that govern a collection of tasks with significant data imbalance. We present a simple yet effective method of pre-trainin…
View article: Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation
Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation Open
Recent advances in machine translation (MT) have shown that Minimum Bayes Risk (MBR) decoding can be a powerful alternative to beam search decoding, especially when combined with neural-based utility functions. However, the performance of …
View article: Scaling Laws for Multilingual Neural Machine Translation
Scaling Laws for Multilingual Neural Machine Translation Open
In this work, we provide a large-scale empirical study of the scaling properties of multilingual neural machine translation models. We examine how increases in the model size affect the model performance and investigate the role of the tra…
View article: Binarized Neural Machine Translation
Binarized Neural Machine Translation Open
The rapid scaling of language models is motivating research using low-bitwidth quantization. In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its kind. We identify …
View article: Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation
Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation Open
Recent advances in machine translation (MT) have shown that Minimum Bayes Risk (MBR) decoding can be a powerful alternative to beam search decoding, especially when combined with neural-based utility functions. However, the performance of …
View article: Do Current Multi-Task Optimization Methods in Deep Learning Even Help?
Do Current Multi-Task Optimization Methods in Deep Learning Even Help? Open
Recent research has proposed a series of specialized optimization algorithms for deep multi-task models. It is often claimed that these multi-task optimization (MTO) methods yield solutions that are superior to the ones found by simply opt…
View article: Adaptive Gradient Methods at the Edge of Stability
Adaptive Gradient Methods at the Edge of Stability Open
Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning. In this paper, we shed light on the behavior of these algorithms in the full-batch and sufficiently large batch settings. Specificall…
View article: Data Scaling Laws in NMT: The Effect of Noise and Architecture
Data Scaling Laws in NMT: The Effect of Noise and Architecture Open
In this work, we study the effect of varying the architecture and training data quality on the data scaling properties of Neural Machine Translation (NMT). First, we establish that the test loss of encoder-decoder transformer models scales…
View article: Examining Scaling and Transfer of Language Model Architectures for Machine Translation
Examining Scaling and Transfer of Language Model Architectures for Machine Translation Open
Natural language understanding and generation models follow one of the two dominant architectural paradigms: language models (LMs) that process concatenated sequences in a single stack of layers, and encoder-decoder models (EncDec) that ut…
View article: When do neural networks outperform kernel methods?*
When do neural networks outperform kernel methods?* Open
For a certain scaling of the initialization of stochastic gradient descent (SGD), wide neural networks (NN) have been shown to be well approximated by reproducing kernel Hilbert space (RKHS) methods. Recent empirical work showed that, for …
View article: A Loss Curvature Perspective on Training Instability in Deep Learning
A Loss Curvature Perspective on Training Instability in Deep Learning Open
In this work, we study the evolution of the loss Hessian across many classification tasks in order to understand the effect the curvature of the loss has on the training dynamics. Whereas prior work has focused on how different learning ra…
View article: Scaling Laws for Neural Machine Translation
Scaling Laws for Neural Machine Translation Open
We present an empirical study of scaling properties of encoder-decoder Transformer models used in neural machine translation (NMT). We show that cross-entropy loss as a function of model size follows a certain scaling law. Specifically (i)…
View article: Linearized two-layers neural networks in high dimension
Linearized two-layers neural networks in high dimension Open
We consider the problem of learning an unknown function $f_{\star}$ on the $d$-dimensional sphere with respect to the square loss, given i.i.d. samples $\{(y_i,{\boldsymbol x}_i)\}_{i\le n}$ where ${\boldsymbol x}_i$ is a feature vector un…
View article: Discussion of: “Nonparametric regression using deep neural networks with ReLU activation function”
Discussion of: “Nonparametric regression using deep neural networks with ReLU activation function” Open
We congratulate Johannes Schmidt-Hieber for his elegant and thought-provoking results.His article uses deep-learning-inspired methods in the context of nonparametric regression.Schmidt-Hieber defines a rich class of composition-based funct…
View article: Limitations of Lazy Training of Two-layers Neural Networks
Limitations of Lazy Training of Two-layers Neural Networks Open
We study the supervised learning problem under either of the following two models: (1) Feature vectors ${\boldsymbol x}_i$ are $d$-dimensional Gaussians and responses are $y_i = f_*({\boldsymbol x}_i)$ for $f_*$ an unknown quadratic functi…
View article: An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density Open
To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothn…
View article: Optimal Covariance Estimation for Condition Number Loss in the Spiked Model
Optimal Covariance Estimation for Condition Number Loss in the Spiked Model Open
We study estimation of the covariance matrix under relative condition number loss $κ(Σ^{-1/2} \hatΣ Σ^{-1/2})$, where $κ(Δ)$ is the condition number of matrix $Δ$, and $\hatΣ$ and $Σ$ are the estimated and theoretical covariance matrices. …
View article: An Instability in Variational Inference for Topic Models
An Instability in Variational Inference for Topic Models Open
Topic models are Bayesian models that are frequently used to capture the latent structure of certain corpora of documents or images. Each data element in such a corpus (for instance each item in a collection of scientific articles) is rega…