Hiroaki Yamagiwa
YOU?
Author Swipe
View article: Revealing Language Model Trajectories via Kullback-Leibler Divergence
Revealing Language Model Trajectories via Kullback-Leibler Divergence Open
A recently proposed method enables efficient estimation of the KL divergence between language models, including models with different architectures, by assigning coordinates based on log-likelihood vectors. To better understand the behavio…
View article: Likelihood Variance as Text Importance for Resampling Texts to Map Language Models
Likelihood Variance as Text Importance for Resampling Texts to Map Language Models Open
We address the computational cost of constructing a model map, which embeds diverse language models into a common space for comparison via KL divergence. The map relies on log-likelihoods over a large text set, making the cost proportional…
View article: Predicting drug–gene relations via analogy tasks with word embeddings
Predicting drug–gene relations via analogy tasks with word embeddings Open
View article: Mapping 1,000+ Language Models via the Log-Likelihood Vector
Mapping 1,000+ Language Models via the Log-Likelihood Vector Open
To compare autoregressive language models at scale, we propose using log-likelihood vectors computed on a predefined text set as model features. This approach has a solid theoretical basis: when treated as model coordinates, their squared …
View article: Mapping 1,000+ Language Models via the Log-Likelihood Vector
Mapping 1,000+ Language Models via the Log-Likelihood Vector Open
View article: Likelihood Variance as Text Importance for Resampling Texts to Map Language Models
Likelihood Variance as Text Importance for Resampling Texts to Map Language Models Open
View article: Quantifying Lexical Semantic Shift via Unbalanced Optimal Transport
Quantifying Lexical Semantic Shift via Unbalanced Optimal Transport Open
Lexical semantic change detection aims to identify shifts in word meanings over time. While existing methods using embeddings from a diachronic corpus pair estimate the degree of change for target words, they offer limited insight into cha…
View article: Understanding Higher-Order Correlations Among Semantic Components in Embeddings
Understanding Higher-Order Correlations Among Semantic Components in Embeddings Open
Independent Component Analysis (ICA) offers interpretable semantic components of embeddings. While ICA theory assumes that embeddings can be linearly decomposed into independent components, real-world data often do not satisfy this assumpt…
View article: Norm of Mean Contextualized Embeddings Determines their Variance
Norm of Mean Contextualized Embeddings Determines their Variance Open
Contextualized embeddings vary by context, even for the same token, and form a distribution in the embedding space. To analyze this distribution, we focus on the norm of the mean embedding and the variance of the embeddings. In this study,…
View article: Shimo Lab at "Discharge Me!": Discharge Summarization by Prompt-Driven Concatenation of Electronic Health Record Sections
Shimo Lab at "Discharge Me!": Discharge Summarization by Prompt-Driven Concatenation of Electronic Health Record Sections Open
In this paper, we present our approach to the shared task "Discharge Me!" at the BioNLP Workshop 2024. The primary goal of this task is to reduce the time and effort clinicians spend on writing detailed notes in the electronic health recor…
View article: Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings
Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings Open
Cosine similarity is widely used to measure the similarity between two embeddings, while interpretations based on angle and correlation coefficient are common. In this study, we focus on the interpretable axes of embeddings transformed by …
View article: Predicting drug-gene relations via analogy tasks with word embeddings
Predicting drug-gene relations via analogy tasks with word embeddings Open
Natural language processing (NLP) is utilized in a wide range of fields, where words in text are typically transformed into feature vectors called embeddings. BioConceptVec is a specific example of embeddings tailored for biology, trained …
View article: Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings
Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings Open
Word embedding is one of the most important components in natural language processing, but interpreting high-dimensional embeddings remains a challenging problem. To address this problem, Independent Component Analysis (ICA) is identified …
View article: Zero-Shot Edge Detection with SCESAME: Spectral Clustering-based Ensemble for Segment Anything Model Estimation
Zero-Shot Edge Detection with SCESAME: Spectral Clustering-based Ensemble for Segment Anything Model Estimation Open
This paper proposes a novel zero-shot edge detection with SCESAME, which stands for Spectral Clustering-based Ensemble for Segment Anything Model Estimation, based on the recently proposed Segment Anything Model (SAM). SAM is a foundation …
View article: Discovering Universal Geometry in Embeddings with ICA
Discovering Universal Geometry in Embeddings with ICA Open
This study utilizes Independent Component Analysis (ICA) to unveil a consistent semantic structure within embeddings of words or images. Our approach extracts independent semantic components from the embeddings of a pre-trained model by le…
View article: Discovering Universal Geometry in Embeddings with ICA
Discovering Universal Geometry in Embeddings with ICA Open
This study utilizes Independent Component Analysis (ICA) to unveil a consistent semantic structure within embeddings of words or images. Our approach extracts independent semantic components from the embeddings of a pre-trained model by le…
View article: Improving word mover’s distance by leveraging self-attention matrix
Improving word mover’s distance by leveraging self-attention matrix Open
Measuring the semantic similarity between two sentences is still an important task. The word mover’s distance (WMD) computes the similarity via the optimal alignment between the sets of word embeddings. However, WMD does not utilize word o…
View article: Improving word mover's distance by leveraging self-attention matrix
Improving word mover's distance by leveraging self-attention matrix Open
Measuring the semantic similarity between two sentences is still an important task. The word mover's distance (WMD) computes the similarity via the optimal alignment between the sets of word embeddings. However, WMD does not utilize word o…