Maor Ivgi
YOU?
Author Swipe
View article: From Loops to Oops: Fallback Behaviors of Language Models Under Uncertainty
From Loops to Oops: Fallback Behaviors of Language Models Under Uncertainty Open
Large language models (LLMs) often exhibit undesirable behaviors, such as hallucinations and sequence repetitions. We propose to view these behaviors as fallbacks that models exhibit under epistemic uncertainty, and investigate the connect…
View article: DataComp-LM: In search of the next generation of training sets for language models
DataComp-LM: In search of the next generation of training sets for language models Open
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effect…
View article: In-Context Learning with Long-Context Models: An In-Depth Exploration
In-Context Learning with Long-Context Models: An In-Depth Exploration Open
As model context lengths continue to increase, the number of demonstrations that can be provided in-context approaches the size of entire training datasets. We study the behavior of in-context learning (ICL) at this extreme scale on multip…
View article: Accelerated Parameter-Free Stochastic Optimization
Accelerated Parameter-Free Stochastic Optimization Open
We propose a method that achieves near-optimal rates for smooth stochastic convex optimization and requires essentially no prior knowledge of problem parameters. This improves on prior work which requires knowing at least the initial dista…
View article: ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding Open
We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data. We adapt six tasks from the SCROLLS benchmark, and add four new …
View article: DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule
DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule Open
We propose a tuning-free dynamic SGD step size formula, which we call Distance over Gradients (DoG). The DoG step sizes depend on simple empirical quantities (distance from the initial point and norms of gradients) and have no ``learning r…
View article: ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding Open
We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data. We adapt six tasks from the SCROLLS benchmark, and add four new …
View article: Efficient Long-Text Understanding with Short-Text Models
Efficient Long-Text Understanding with Short-Text Models Open
Transformer-based pretrained language models (LMs) are ubiquitous across natural language understanding, but cannot be applied to long sequences such as stories, scientific articles, and long documents due to their quadratic complexity. Wh…
View article: Efficient Long-Text Understanding with Short-Text Models
Efficient Long-Text Understanding with Short-Text Models Open
Transformer-based pretrained language models (LMs) are ubiquitous across natural language understanding, but cannot be applied to long sequences such as stories, scientific articles and long documents, due to their quadratic complexity. Wh…
View article: Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments
Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments Open
Neural scaling laws define a predictable relationship between a model's parameter count and its performance after training in the form of a power law. However, most research to date has not explicitly investigated whether scaling laws can …
View article: SCROLLS: Standardized CompaRison Over Long Language Sequences
SCROLLS: Standardized CompaRison Over Long Language Sequences Open
NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild. We introduce SCROLLS, a suite of tasks that require reasoning over…
View article: SCROLLS: Standardized CompaRison Over Long Language Sequences
SCROLLS: Standardized CompaRison Over Long Language Sequences Open
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.
View article: Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments
Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments Open
Neural scaling laws define a predictable relationship between a model's parameter count and its performance after training in the form of a power law. However, most research to date has not explicitly investigated whether scaling laws can …
View article: Beyond Importance Scores: Interpreting Tabular ML by Visualizing Feature Semantics
Beyond Importance Scores: Interpreting Tabular ML by Visualizing Feature Semantics Open
Interpretability is becoming an active research topic as machine learning (ML) models are more widely used to make critical decisions. Tabular data are one of the most commonly used modes of data in diverse applications such as healthcare …
View article: Beyond Importance Scores: Interpreting Tabular ML by Visualizing Feature\n Semantics
Beyond Importance Scores: Interpreting Tabular ML by Visualizing Feature\n Semantics Open
Interpretability is becoming an active research topic as machine learning\n(ML) models are more widely used to make critical decisions. Tabular data is\none of the most commonly used modes of data in diverse applications such as\nhealthcar…
View article: Beyond Importance Scores: Interpreting Tabular ML by Visualizing Feature Semantics
Beyond Importance Scores: Interpreting Tabular ML by Visualizing Feature Semantics Open
Interpretability is becoming an active research topic as machine learning (ML) models are more widely used to make critical decisions. Tabular data is one of the most commonly used modes of data in diverse applications such as healthcare a…
View article: Scene Graph tO Image Generation with Contextualized Object Layout Refinement
Scene Graph tO Image Generation with Contextualized Object Layout Refinement Open
Generating images from scene graphs is a challenging task that attracted substantial interest recently. Prior works have approached this task by generating an intermediate layout description of the target image. However, the representation…
View article: Achieving Model Robustness through Discrete Adversarial Training
Achieving Model Robustness through Discrete Adversarial Training Open
Discrete adversarial attacks are symbolic perturbations to a language input that preserve the output label but lead to a prediction error. While such attacks have been extensively explored for the purpose of evaluating model robustness, th…
View article: Achieving Model Robustness through Discrete Adversarial Training
Achieving Model Robustness through Discrete Adversarial Training Open
Discrete adversarial attacks are symbolic perturbations to a language input that preserve the output label but lead to a prediction error. While such attacks have been extensively explored for the purpose of evaluating model robustness, th…