Yangjun Ruan
YOU?
Author Swipe
View article: LM Agents May Fail to Act on Their Own Risk Knowledge
LM Agents May Fail to Act on Their Own Risk Knowledge Open
Language model (LM) agents have demonstrated significant potential for automating real-world tasks, yet they pose a diverse array of potential, severe risks in safety-critical scenarios. In this work, we identify a significant gap between …
View article: Reasoning to Learn from Latent Thoughts
Reasoning to Learn from Latent Thoughts Open
Compute scaling for language model (LM) pretraining has outpaced the growth of human-written texts, leading to concerns that data will become the bottleneck to LM scaling. To continue scaling pretraining in this data-constrained regime, we…
View article: MixMin: Finding Data Mixtures via Convex Minimization
MixMin: Finding Data Mixtures via Convex Minimization Open
Modern machine learning pipelines are increasingly combining and mixing data from diverse and disparate sources, e.g., pre-training large language models. Yet, finding the optimal data mixture is a challenging and open problem. We formaliz…
View article: APPL: A Prompt Programming Language for Harmonious Integration of Programs and Large Language Model Prompts
APPL: A Prompt Programming Language for Harmonious Integration of Programs and Large Language Model Prompts Open
View article: Graph-based Uncertainty Metrics for Long-form Language Model Outputs
Graph-based Uncertainty Metrics for Long-form Language Model Outputs Open
Recent advancements in Large Language Models (LLMs) have significantly improved text generation capabilities, but these systems are still known to hallucinate, and granular uncertainty estimation for long-form LLM generations remains chall…
View article: APPL: A Prompt Programming Language for Harmonious Integration of Programs and Large Language Model Prompts
APPL: A Prompt Programming Language for Harmonious Integration of Programs and Large Language Model Prompts Open
Large Language Models (LLMs) have become increasingly capable of handling diverse tasks with the aid of well-crafted prompts and integration of external tools, but as task complexity rises, the workflow involving LLMs can be complicated an…
View article: Observational Scaling Laws and the Predictability of Language Model Performance
Observational Scaling Laws and the Predictability of Language Model Performance Open
Understanding how language model performance varies with scale is critical to benchmark and algorithm development. Scaling laws are one approach to building this understanding, but the requirement of training models across many different s…
View article: FastSpeech: Fast, Robust and Controllable Text to Speech
FastSpeech: Fast, Robust and Controllable Text to Speech Open
Neural network based end-to-end text to speech (TTS) has significantly improved the quality of synthesized speech. Prominent methods (e.g., Tacotron 2) usually first generate mel-spectrogram from text, and then synthesize speech from the m…
View article: Identifying the Risks of LM Agents with an LM-Emulated Sandbox
Identifying the Risks of LM Agents with an LM-Emulated Sandbox Open
Recent advances in Language Model (LM) agents and tool use, exemplified by applications like ChatGPT Plugins, enable a rich set of capabilities but also amplify potential risks - such as leaking private data or causing financial losses. Id…
View article: Weighted Ensemble Self-Supervised Learning
Weighted Ensemble Self-Supervised Learning Open
Ensembling has proven to be a powerful technique for boosting model performance, uncertainty estimation, and robustness in supervised learning. Advances in self-supervised learning (SSL) enable leveraging large unlabeled corpora for state-…
View article: Augment with Care: Contrastive Learning for Combinatorial Problems
Augment with Care: Contrastive Learning for Combinatorial Problems Open
Supervised learning can improve the design of state-of-the-art solvers for combinatorial problems, but labelling large numbers of combinatorial instances is often impractical due to exponential worst-case complexity. Inspired by the recent…
View article: Optimal Representations for Covariate Shift
Optimal Representations for Covariate Shift Open
Machine learning systems often experience a distribution shift between training and testing. In this paper, we introduce a simple variational objective whose optima are exactly the set of all representations on which risk minimizers are gu…
View article: Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding Open
Latent variable models have been successfully applied in lossless compression with the bits-back coding algorithm. However, bits-back suffers from an increase in the bitrate equal to the KL divergence between the approximate posterior and …
View article: Learning to Learn by Zeroth-Order Oracle
Learning to Learn by Zeroth-Order Oracle Open
In the learning to learn (L2L) framework, we cast the design of optimization algorithms as a machine learning problem and use deep neural networks to learn the update rules. In this paper, we extend the L2L framework to zeroth-order (ZO) o…
View article: Learning to Learn by Zeroth-Order Oracle
Learning to Learn by Zeroth-Order Oracle Open
In the learning to learn (L2L) framework, we cast the design of optimization algorithms as a machine learning problem and use deep neural networks to learn the update rules. In this paper, we extend the L2L framework to zeroth-order (ZO) o…
View article: FastSpeech: Fast, Robust and Controllable Text to Speech
FastSpeech: Fast, Robust and Controllable Text to Speech Open
Neural network based end-to-end text to speech (TTS) has significantly improved the quality of synthesized speech. Prominent methods (e.g., Tacotron 2) usually first generate mel-spectrogram from text, and then synthesize speech from the m…