Taiji Suzuki
YOU?
Author Swipe
View article: Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning
Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning Open
Mamba, a recently proposed linear-time sequence model, has attracted significant attention for its computational efficiency and strong empirical performance. However, a rigorous theoretical understanding of its underlying mechanisms remain…
View article: Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression
Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression Open
State-space models (SSMs), particularly Mamba, emerge as an efficient Transformer alternative with linear complexity for long-sequence modeling. Recent empirical works demonstrate Mamba's in-context learning (ICL) capabilities competitive …
View article: Hessian-guided Perturbed Wasserstein Gradient Flows for Escaping Saddle Points
Hessian-guided Perturbed Wasserstein Gradient Flows for Escaping Saddle Points Open
Wasserstein gradient flow (WGF) is a common method to perform optimization over the space of probability measures. While WGF is guaranteed to converge to a first-order stationary point, for nonconvex functionals the converged solution does…
View article: Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel
Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel Open
Gradient-based optimization methods have shown remarkable empirical success, yet their theoretical generalization properties remain only partially understood. In this paper, we establish a generalization bound for gradient flow that aligns…
View article: Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning
Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning Open
Mixture of Experts (MoE), an ensemble of specialized models equipped with a router that dynamically distributes each input to appropriate experts, has achieved successful results in the field of machine learning. However, theoretical under…
View article: On the Role of Label Noise in the Feature Learning Process
On the Role of Label Noise in the Feature Learning Process Open
Deep learning with noisy labels presents significant challenges. In this work, we theoretically characterize the role of label noise from a feature learning perspective. Specifically, we consider a signal-noise data distribution, where eac…
View article: Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models
Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models Open
Aligning large language models (LLMs) with human preferences is crucial for safe deployment, yet existing methods assume specific preference models like Bradley-Terry model. This assumption leads to statistical inconsistency, where more da…
View article: When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars
When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars Open
The ability to acquire latent semantics is one of the key properties that determines the performance of language models. One convenient approach to invoke this ability is to prepend metadata (e.g. URLs, domains, and styles) at the beginnin…
View article: Propagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemble
Propagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemble Open
Mean-field Langevin dynamics (MFLD) is an optimization method derived by taking the mean-field limit of noisy gradient descent for two-layer neural networks in the mean-field regime. Recently, the propagation of chaos (PoC) for MFLD has ga…
View article: Direct Distributional Optimization for Provable Alignment of Diffusion Models
Direct Distributional Optimization for Provable Alignment of Diffusion Models Open
We introduce a novel alignment method for diffusion models from distribution optimization perspectives while providing rigorous convergence guarantees. We first formulate the problem as a generic regularized loss minimization over probabil…
View article: Metastable Dynamics of Chain-of-Thought Reasoning: Provable Benefits of Search, RL and Distillation
Metastable Dynamics of Chain-of-Thought Reasoning: Provable Benefits of Search, RL and Distillation Open
A key paradigm to improve the reasoning capabilities of large language models (LLMs) is to allocate more inference-time compute to search against a verifier or reward model. This process can then be utilized to refine the pretrained model …
View article: On the Comparison between Multi-modal and Single-modal Contrastive Learning
On the Comparison between Multi-modal and Single-modal Contrastive Learning Open
Multi-modal contrastive learning with language supervision has presented a paradigm shift in modern machine learning. By pre-training on a web-scale dataset, multi-modal contrastive learning can learn high-quality representations that exhi…
View article: Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning
Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning Open
Transformer-based large language models (LLMs) have displayed remarkable creative prowess and emergence capabilities. Existing empirical studies have revealed a strong connection between these LLMs' impressive emergence abilities and their…
View article: Pretrained transformer efficiently learns low-dimensional target functions in-context
Pretrained transformer efficiently learns low-dimensional target functions in-context Open
Transformers can efficiently learn in-context from example demonstrations. Most existing theoretical analyses studied the in-context learning (ICL) ability of transformers for linear function classes, where it is typically shown that the m…
View article: Transformers Provably Solve Parity Efficiently with Chain of Thought
Transformers Provably Solve Parity Efficiently with Chain of Thought Open
This work provides the first theoretical analysis of training transformers to solve complex problems by recursively generating intermediate states, analogous to fine-tuning for chain-of-thought (CoT) reasoning. We consider training a one-l…
View article: On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent
On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent Open
The Adam optimizer is widely used for transformer optimization in practice, which makes understanding the underlying optimization mechanisms an important problem. However, due to the Adam's complexity, theoretical analysis of how it optimi…
View article: Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization
Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization Open
Transformers have demonstrated great power in the recent development of large foundational models. In particular, the Vision Transformer (ViT) has brought revolutionary changes to the field of vision, achieving significant accomplishments …
View article: Transformers are Minimax Optimal Nonparametric In-Context Learners
Transformers are Minimax Optimal Nonparametric In-Context Learners Open
In-context learning (ICL) of large language models has proven to be a surprisingly effective method of learning a new task from only a few demonstrative examples. In this paper, we study the efficacy of ICL from the viewpoint of statistica…
View article: Convergence error analysis of reflected gradient Langevin dynamics for non-convex constrained optimization
Convergence error analysis of reflected gradient Langevin dynamics for non-convex constrained optimization Open
Gradient Langevin dynamics and a variety of its variants have attracted increasing attention owing to their convergence towards the global optimal solution, initially in the unconstrained convex framework while recently even in convex cons…
View article: Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinations
Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinations Open
We study the computational and sample complexity of learning a target function $f_*:\mathbb{R}^d\to\mathbb{R}$ with additive structure, that is, $f_*(x) = \frac{1}{\sqrt{M}}\sum_{m=1}^M f_m(\langle x, v_m\rangle)$, where $f_1,f_2,...,f_M:\…
View article: Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples
Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples Open
Neural Network-based active learning (NAL) is a cost-effective data selection technique that utilizes neural networks to select and train on a small subset of samples. While existing work successfully develops various effective or theory-j…
View article: Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit
Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit Open
We study the problem of gradient descent learning of a single-index target function $f_*(\boldsymbol{x}) = \textstyleσ_*\left(\langle\boldsymbol{x},\boldsymbolθ\rangle\right)$ under isotropic Gaussian data in $\mathbb{R}^d$, where the unkn…
View article: Flow matching achieves almost minimax optimal convergence
Flow matching achieves almost minimax optimal convergence Open
Flow matching (FM) has gained significant attention as a simulation-free generative model. Unlike diffusion models, which are based on stochastic differential equations, FM employs a simpler approach by solving an ordinary differential equ…
View article: State Space Models are Comparable to Transformers in Estimating Functions with Dynamic Smoothness
State Space Models are Comparable to Transformers in Estimating Functions with Dynamic Smoothness Open
Deep neural networks based on state space models (SSMs) are attracting much attention in sequence modeling since their computational cost is significantly smaller than that of Transformers. While the capabilities of SSMs have been primaril…
View article: State-Free Inference of State-Space Models: The Transfer Function Approach
State-Free Inference of State-Space Models: The Transfer Function Approach Open
We approach designing a state-space model for deep learning applications through its dual representation, the transfer function, and uncover a highly efficient sequence parallel inference algorithm that is state-free: unlike other proposed…
View article: Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric
Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric Open
In typical multimodal contrastive learning, such as CLIP, encoders produce one point in the latent representation space for each input. However, one-point representation has difficulty in capturing the relationship and the similarity struc…
View article: Mechanistic Design and Scaling of Hybrid Architectures
Mechanistic Design and Scaling of Hybrid Architectures Open
The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale model training and evaluation. We set out to simplify this …
View article: Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective
Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective Open
In this paper, we study the feature learning ability of two-layer neural networks in the mean-field regime through the lens of kernel methods. To focus on the dynamics of the kernel induced by the first layer, we utilize a two-timescale li…
View article: How do Transformers perform In-Context Autoregressive Learning?
How do Transformers perform In-Context Autoregressive Learning? Open
Transformers have achieved state-of-the-art performance in language modeling tasks. However, the reasons behind their tremendous success are still unclear. In this paper, towards a better understanding, we train a Transformer model on a si…
View article: Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape
Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape Open
Large language models based on the Transformer architecture have demonstrated impressive capabilities to learn in context. However, existing theoretical studies on how this phenomenon arises are limited to the dynamics of a single layer of…