David Grangier
YOU?
Author Swipe
View article: The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining Open
Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to dis…
View article: Pretraining with hierarchical memories: separating long-tail and common knowledge
Pretraining with hierarchical memories: separating long-tail and common knowledge Open
The impressive performance gains of modern language models currently rely on scaling parameters: larger models store more world knowledge and reason better. Yet compressing all world knowledge into parameters is unnecessary, as only a frac…
View article: Compute-Optimal Quantization-Aware Training
Compute-Optimal Quantization-Aware Training Open
Quantization-aware training (QAT) is a leading technique for improving the accuracy of quantized neural networks. Previous work has shown that decomposing training into a full-precision (FP) phase followed by a QAT phase yields superior ac…
View article: Scaling Laws for Optimal Data Mixtures
Scaling Laws for Optimal Data Mixtures Open
Large foundation models are typically trained on data from multiple domains, with the data mixture--the proportion of each domain used--playing a critical role in model performance. The standard approach to selecting this mixture relies on…
View article: Assessing the Role of Data Quality in Training Bilingual Language Models
Assessing the Role of Data Quality in Training Bilingual Language Models Open
Bilingual and multilingual language models offer a promising path toward scaling NLP systems across diverse languages and users. However, their performance often varies wildly between languages as prior works show that adding more language…
View article: Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection Open
A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from that target domain. Finetuning presents two challenges: (i)…
View article: Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging Open
Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-Experts, a novel architecture that can instantiate a model …
View article: Training Bilingual LMs with Data Constraints in the Targeted Language
Training Bilingual LMs with Data Constraints in the Targeted Language Open
Large language models are trained on massive scrapes of the web, as required by current scaling laws. Most progress is made for English, given its abundance of high-quality pretraining data. For most other languages, however, such high qua…
View article: Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP Open
Large pretrained vision-language models like CLIP have shown promising generalization capability, but may struggle in specialized domains (e.g., satellite imagery) or fine-grained classification (e.g., car models) where the visual concepts…
View article: No Need to Talk: Asynchronous Mixture of Language Models
No Need to Talk: Asynchronous Mixture of Language Models Open
We introduce SMALLTALK LM, an innovative method for training a mixture of language models in an almost asynchronous manner. Each model of the mixture specializes in distinct parts of the data distribution, without the need for high-bandwid…
View article: Dynamic Gradient Alignment for Online Data Mixing
Dynamic Gradient Alignment for Online Data Mixing Open
The composition of training data mixtures is critical for effectively training large language models (LLMs), as it directly impacts their performance on downstream tasks. Our goal is to identify an optimal data mixture to specialize an LLM…
View article: Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling Open
Specialist language models (LMs) focus on a specific task or domain on which they often outperform generalist LMs of the same size. However, the specialist data needed to pretrain these models is only available in limited amount for most t…
View article: The AdEMAMix Optimizer: Better, Faster, Older
The AdEMAMix Optimizer: Better, Faster, Older Open
Momentum based optimizers are central to a wide range of machine learning applications. These typically rely on an Exponential Moving Average (EMA) of gradients, which decays exponentially the present contribution of older gradients. This …
View article: Need a Small Specialized Language Model? Plan Early!
Need a Small Specialized Language Model? Plan Early! Open
Large language models are versatile tools but are not suitable for small inference budgets. Small models have more efficient inference, but their lower capacity means that their performance can be good only if one limits their scope to a s…
View article: Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling Open
Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows wi…
View article: Adaptive Training Distributions with Scalable Online Bilevel Optimization
Adaptive Training Distributions with Scalable Online Bilevel Optimization Open
Large neural networks pretrained on web-scale corpora are central to modern machine learning. In this paradigm, the distribution of the large, heterogeneous pretraining data rarely matches that of the application domain. This work consider…
View article: Transfer Learning for Structured Pruning under Limited Task Data
Transfer Learning for Structured Pruning under Limited Task Data Open
Large, pre-trained models are problematic to use in resource constrained applications. Fortunately, task-aware structured pruning methods offer a solution. These approaches reduce model size by dropping structural units like layers and att…
View article: High-Resource Methodological Bias in Low-Resource Investigations
High-Resource Methodological Bias in Low-Resource Investigations Open
The central bottleneck for low-resource NLP is typically regarded to be the quantity of accessible data, overlooking the contribution of data quality. This is particularly seen in the development and evaluation of low-resource systems via …
View article: AudioLM: a Language Modeling Approach to Audio Generation
AudioLM: a Language Modeling Approach to Audio Generation Open
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation spa…
View article: Learning strides in convolutional neural networks
Learning strides in convolutional neural networks Open
Convolutional neural networks typically contain several downsampling operators, such as strided convolutions or pooling layers, that progressively reduce the resolution of intermediate representations. This provides some shift-invariance w…
View article: Learning strides in convolutional neural networks
Learning strides in convolutional neural networks Open
Convolutional neural networks typically contain several downsampling operators, such as strided convolutions or pooling layers, that progressively reduce the resolution of intermediate representations. This provides some shift-invariance w…
View article: The Trade-offs of Domain Adaptation for Neural Language Models
The Trade-offs of Domain Adaptation for Neural Language Models Open
This work connects language model adaptation with concepts of machine learning theory. We consider a training setup with a large out-of-domain set and a small in-domain set. We derive how the benefit of training a model on either set depen…
View article: On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation
On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation Open
Modern unsupervised machine translation (MT) systems reach reasonable translation quality under clean and controlled data conditions. As the performance gap between supervised and unsupervised MT narrows, it is interesting to ask whether t…
View article: A Natural Diet: Towards Improving Naturalness of Machine Translation Output
A Natural Diet: Towards Improving Naturalness of Machine Translation Output Open
Machine translation (MT) evaluation often focuses on accuracy and fluency, without paying much attention to translation style. This means that, even when considered accurate and fluent, MT output can still sound less natural than high qual…
View article: High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics
High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics Open
In Neural Machine Translation, it is typically assumed that the sentence with the highest estimated probability should also be the translation with the highest quality as measured by humans. In this work, we question this assumption and sh…
View article: Dive: End-to-End Speech Diarization Via Iterative Speaker Embedding
Dive: End-to-End Speech Diarization Via Iterative Speaker Embedding Open
We introduce DIVE, an end-to-end speaker diarization algorithm. Our neural algorithm presents the diarization task as an iterative process: it repeatedly builds a representation for each speaker before predicting the voice activity of each…
View article: Minimum Bayes Risk Decoding with Neural Metrics of Translation Quality.
Minimum Bayes Risk Decoding with Neural Metrics of Translation Quality. Open
This work applies Minimum Bayes Risk (MBR) decoding to optimize diverse automated metrics of translation quality. Automatic metrics in machine translation have made tremendous progress recently. In particular, neural metrics, fine-tuned on…
View article: High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics
High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics Open
In Neural Machine Translation, it is typically assumed that the sentence with the highest estimated probability should also be the translation with the highest quality as measured by humans. In this work, we question this assumption and sh…