Learning Dynamics of VLM Finetuning Article Swipe

MAKE IMPACT

J. Q. Zhang , Kaitong Cai , Jing Yang , Keze Wang · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2510.11978

Preference-based finetuning of vision--language models (VLMs) is brittle: trivially wrong negatives inject uninformative gradients that destabilize training. We recast alignment as \textbf{learning-dynamics--aware optimization} and introduce \textbf{Cooling-Weighted DPO (CW-DPO)}, a two-stage recipe that explicitly models and exploits the training trajectory. \textbf{Stage 1} performs supervised finetuning with \textbf{gentle negatives}: \textbf{low-weight smoothed supervision} that regularizes the base policy and curbs overconfidence without explicit penalties. \textbf{Stage 2} applies a DPO objective in which the \textbf{negative term is scaled by a cooling weight} computed from the model's \textbf{average token log-probability} on each negative, suppressing uninformative gradients from easy or off-distribution samples while preserving signal from hard negatives. In practice, we emphasize \textbf{on-policy negatives} and allow \textbf{mixed negatives} by blending a controllable fraction of dataset negatives to maintain contrast freshness. Throughout, we instrument training with $Δ\!\log p$ probes on positives and negatives as first-class signals for early stopping, curriculum design, and failure diagnosis. Across diverse VLM tasks, CW-DPO yields \textbf{more stable optimization}, \textbf{better calibration}, and \textbf{higher pairwise win-rates} than SFT-only and vanilla DPO, while \textbf{converging in fewer steps}. Ablations isolate the \textbf{cooling-weight mechanism} as the primary driver of these gains and show complementary benefits from mixing on-policy and dataset negatives. Taken together, our results show that \textbf{smoothing learning dynamics before cooling preferences} is a simple, general principle for robust VLM alignment.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2510.11978

PDF

https://arxiv.org/pdf/2510.11978

OA Status

green

OpenAlex ID

https://openalex.org/W4415257533

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4415257533

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2510.11978

Digital Object Identifier

Title

Learning Dynamics of VLM Finetuning

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-10-13

Full publication date if available

Authors

J. Q. Zhang, Kaitong Cai, Jing Yang, Keze Wang

List of authors in order

Landing page

https://arxiv.org/abs/2510.11978

Publisher landing page

PDF URL

https://arxiv.org/pdf/2510.11978

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2510.11978

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload