Itamar Pres
YOU?
Author Swipe
View article: Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls
Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls Open
Language models are increasingly capable, yet still fail at a seemingly simple task of multi-digit multiplication. In this work, we study why, by reverse-engineering a model that successfully learns multiplication via \emph{implicit chain-…
View article: Competition Dynamics Shape Algorithmic Phases of In-Context Learning
Competition Dynamics Shape Algorithmic Phases of In-Context Learning Open
In-Context Learning (ICL) has significantly expanded the general-purpose nature of large language models, allowing them to adapt to novel tasks using merely the inputted context. This has motivated a series of papers that analyze tractable…
View article: A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity
A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity Open
While alignment algorithms are now commonly used to tune pre-trained language models towards a user's preferences, we lack explanations for the underlying mechanisms in which models become ``aligned'', thus making it difficult to explain p…