Arnold Overwijk
YOU?
Author Swipe
View article: Group-Level Data Selection for Efficient Pretraining
Group-Level Data Selection for Efficient Pretraining Open
In this paper, we introduce Group-MATES, an efficient group-level data selection approach to optimize the speed-quality frontier of language model pretraining. Specifically, Group-MATES parameterizes costly group-level selection with a rel…
View article: Improving Multitask Retrieval by Promoting Task Specialization
Improving Multitask Retrieval by Promoting Task Specialization Open
In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks. Despite its practical appeal, naive multitask retrieval lags behind task-specific retrieval in which a separate retriever is trained fo…
View article: Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories
Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories Open
In this paper we improve the zero-shot generalization ability of language models via Mixture-Of-Memory Augmentation (MoMA), a mechanism that retrieves augmentation documents from multiple information corpora ("external memories"), with the…
View article: Improving Multitask Retrieval by Promoting Task Specialization
Improving Multitask Retrieval by Promoting Task Specialization Open
In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks. Despite its practical appeal, naive multitask retrieval lags behind task-specific retrieval, in which a separate retriever is trained f…
View article: Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories
Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories Open
In this paper we improve the zero-shot generalization ability of language models via Mixture-Of-Memory Augmentation (MoMA), a mechanism that retrieves augmentation documents from multiple information corpora (external memories), with the o…
View article: ClueWeb22: 10 Billion Web Documents with Visual and Semantic Information
ClueWeb22: 10 Billion Web Documents with Visual and Semantic Information Open
ClueWeb22, the newest iteration of the ClueWeb line of datasets, provides 10 billion web pages affiliated with rich information. Its design was influenced by the need for a high quality, large scale web corpus to support a range of academi…
View article: Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives Open
In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained model. We show the catastrophic forgetting phenomena behind t…
View article: COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning
COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning Open
We present a new zero-shot dense retrieval (ZeroDR) method, COCO-DR, to improve the generalization ability of dense retrieval by combating the distribution shifts between source training tasks and target scenarios. To mitigate the impact o…
View article: COCO-DR: Combating Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning
COCO-DR: Combating Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning Open
We present a new zero-shot dense retrieval (ZeroDR) method, COCO-DR, to improve the generalization ability of dense retrieval by combating the distribution shifts between source training tasks and target scenarios. To mitigate the impact o…
View article: Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives Open
In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained model. We show the catastrophic forgetting phenomena behind t…
View article: Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder
Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder Open
Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space. Autoencoder-based language models are appealing in dense retrieval as they train the encoder to output high-quality emb…
View article: Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder.
Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder. Open
Many real-world applications use Siamese networks to efficiently match text sequences at scale, which require high-quality sequence encodings. This paper pre-trains language models dedicated to sequence matching in Siamese architectures. W…
View article: Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder
Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder Open
Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, Arnold Overwijk. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.
View article: Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval Open
Conducting text retrieval in a dense learned representation space has many intriguing advantages over sparse retrieval. Yet the effectiveness of dense retrieval (DR) often requires combination with sparse retrieval. In this paper, we ident…
View article: Open Domain Web Keyphrase Extraction Beyond Language Modeling
Open Domain Web Keyphrase Extraction Beyond Language Modeling Open
This paper studies keyphrase extraction in real-world scenarios where documents are from diverse domains and have variant content quality. We curate and release OpenKP, a large scale open domain keyphrase extraction dataset with near one h…
View article: Open Domain Web Keyphrase Extraction Beyond Language Modeling
Open Domain Web Keyphrase Extraction Beyond Language Modeling Open
Lee Xiong, Chuan Hu, Chenyan Xiong, Daniel Campos, Arnold Overwijk. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC…