Alisa Liu
YOU?
Author Swipe
View article: Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations Open
Modern tokenizers employ deterministic algorithms to map text into a single "canonical" token sequence, yet the same string can be encoded as many non-canonical tokenizations using the tokenizer vocabulary. In this work, we investigate the…
View article: Sampling from Your Language Model One Byte at a Time
Sampling from Your Language Model One Byte at a Time Open
Tokenization is used almost universally by modern language models, enabling efficient text representation using multi-byte or multi-character tokens. However, prior work has shown that tokenization can introduce distortion into the model's…
View article: When One LLM Drools, Multi-LLM Collaboration Rules
When One LLM Drools, Multi-LLM Collaboration Rules Open
This position paper argues that in many realistic (i.e., complex, contextualized, subjective) scenarios, one LLM is not enough to produce a reliable output. We challenge the status quo of relying solely on a single general-purpose LLM and …
View article: Tulu 3: Pushing Frontiers in Open Language Model Post-Training
Tulu 3: Pushing Frontiers in Open Language Model Post-Training Open
Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and r…
View article: Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models
Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models Open
Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. In this paper, we identify and characterize a phenomenon never discussed before, which we call semantic leakage, where models lea…
View article: Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data? Open
The pretraining data of today's strongest language models is opaque; in particular, little is known about the proportions of various domains or languages represented. In this work, we tackle a task which we call data mixture inference, whi…
View article: Decoding-Time Language Model Alignment with Multiple Objectives
Decoding-Time Language Model Alignment with Multiple Objectives Open
Aligning language models (LMs) to human preferences has emerged as a critical pursuit, enabling these models to better serve diverse user needs. Existing methods primarily focus on optimizing LMs for a single reward function, limiting thei…
View article: A Taxonomy of Ambiguity Types for NLP
A Taxonomy of Ambiguity Types for NLP Open
Ambiguity is an critical component of language that allows for more effective communication between speakers, but is often ignored in NLP. Recent work suggests that NLP systems may struggle to grasp certain elements of human language under…
View article: Tuning Language Models by Proxy
Tuning Language Models by Proxy Open
Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors. However, tuning these models has become increasingly resource-intensive, or imposs…
View article: Below the Sea (with the Sharks):Probing Textual Features of Implicit Sentiment in a Literary Case-study
Below the Sea (with the Sharks):Probing Textual Features of Implicit Sentiment in a Literary Case-study Open
Literary language presents an ongoing challenge for Sentiment Analysis (SA) due to its complex, nuanced, and layered form of expression. It is often suggested that effective literary writing is evocative, operates beneath the surface and u…
View article: That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?
That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context? Open
The translation of ambiguous text presents a challenge for translation systems, as it requires using the surrounding context to disambiguate the intended meaning as much as possible. While prior work has studied ambiguities that result fro…
View article: Inverse Scaling: When Bigger Isn't Better
Inverse Scaling: When Bigger Isn't Better Open
Work on scaling laws has found that large language models (LMs) show predictable improvements to overall loss with increased scale (model size, training data, and compute). Here, we present evidence for the claim that LMs may show inverse …
View article: How Language Model Hallucinations Can Snowball
How Language Model Hallucinations Can Snowball Open
A major risk of using language models in practical applications is their tendency to hallucinate incorrect statements. Hallucinations are often attributed to knowledge gaps in LMs, but we hypothesize that in some cases, when justifying pre…
View article: We're Afraid Language Models Aren't Modeling Ambiguity
We're Afraid Language Models Aren't Modeling Ambiguity Open
Ambiguity is an intrinsic feature of natural language. Managing ambiguity is a key part of human language understanding, allowing us to anticipate misunderstanding as communicators and revise our interpretations as listeners. As language m…
View article: Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts
Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts Open
Text detoxification has the potential to mitigate the harms of toxicity by rephrasing text to remove offensive meaning, but subtle toxicity remains challenging to tackle. We introduce MaRCo, a detoxification algorithm that combines control…
View article: That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?
That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context? Open
The translation of ambiguous text presents a challenge for translation systems, as it requires using the surrounding context to disambiguate the intended meaning as much as possible.While prior work has studied ambiguities that result from…
View article: We’re Afraid Language Models Aren’t Modeling Ambiguity
We’re Afraid Language Models Aren’t Modeling Ambiguity Open
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, Yejin Choi. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
View article: Self-Instruct: Aligning Language Models with Self-Generated Instructions
Self-Instruct: Aligning Language Models with Self-Generated Instructions Open
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
View article: Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts
Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts Open
Text detoxification has the potential to mitigate the harms of toxicity by rephrasing text to remove offensive meaning, but subtle toxicity remains challenging to tackle. We introduce MaRCo, a detoxification algorithm that combines control…
View article: Self-Instruct: Aligning Language Models with Self-Generated Instructions
Self-Instruct: Aligning Language Models with Self-Generated Instructions Open
Large "instruction-tuned" language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is…
View article: WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation Open
A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. We introduce a novel approach for dataset creation base…
View article: Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022)
Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022) Open
Knowledge graphs are often used to store common sense information that is useful for various tasks.However, the extraction of contextuallyrelevant knowledge is an unsolved problem, and current approaches are relatively simple.Here we intro…
View article: WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation Open
A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. We introduce a novel approach for dataset creation base…
View article: Generated Knowledge Prompting for Commonsense Reasoning
Generated Knowledge Prompting for Commonsense Reasoning Open
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, Hannaneh Hajishirzi. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.
View article: Generated Knowledge Prompting for Commonsense Reasoning
Generated Knowledge Prompting for Commonsense Reasoning Open
It remains an open question whether incorporating external knowledge benefits commonsense reasoning while maintaining the flexibility of pretrained sequence models. To investigate this question, we develop generated knowledge prompting, wh…
View article: DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts Open
Despite recent advances in natural language generation, it remains challenging to control attributes of generated text. We propose DExperts: Decoding-time Experts, a decoding-time method for controlled text generation that combines a pretr…