We introduce Conservative Supervised Fine-Tuning (ConSFT). Operating under a prior-data-free constraint, ConSFT resolves the catastrophic forgetting dilemma in Flow-Matching Vision-Language-Action (VLA) models through dynamic confidence-based gradient modulation, improving foundational capability retention by ~20% without architectural overhead.
Standard Supervised Fine-Tuning (SFT) minimizes the negative log-likelihood across all target demonstrations. This unconstrained optimization drives a dense, global perturbation of network parameters. In Vision-Language-Action (VLA) models, such dense updates disrupt foundational competencies, directly degrading the model's pre-trained capabilities and overall generalization.
To isolate the optimization mechanics, we empirically compared SFT with Trust-Region Reinforcement Learning (RL) baselines. Our experiments demonstrate that RL consistently exhibits significantly less catastrophic forgetting. Crucially, we identify a strong correlation between this capability retention and the extreme parameter update sparsity (often >99%) induced by the clipping mechanism.
When the policy ratio \(r_t(\theta)\) deviates beyond the \((1 \pm \epsilon)\) threshold, the gradient with respect to \(\theta\) is strictly truncated to zero. By explicitly clamping the updates for samples that diverge from the prior, the optimization step is bounded. This zero-gradient truncation generates highly sparse parameter updates, naturally leaving the majority of pre-trained foundational weights undisturbed.
Inspired by the trust-region clipping mechanisms in Reinforcement Learning (RL), we integrate a dynamic learning mechanism into Supervised Fine-Tuning (SFT). In RL, optimization is bounded by an importance sampling ratio to ensure stable updates—a mechanism empirically shown to effectively mitigate catastrophic forgetting. For continuous-time flow-matching architectures where exact likelihoods are intractable, recent developments like Flow Policy Optimization (FPO) demonstrate that this policy ratio can be effectively approximated using the exponential loss difference: \(\exp(\mathcal{L}_{\text{behavior}} - \mathcal{L}_{\theta})\).
We adapt this concept directly into the SFT paradigm. Since standard SFT training operates purely on static offline demonstrations, an explicit behavior policy does not exist. To resolve this, we treat the demonstration data as being generated by an ideal expert policy whose theoretical flow-matching loss is strictly zero (\(\mathcal{L}_{\text{behavior}} = \mathcal{L}_{\text{expert}} = 0\)).
By linking the trust-region ratio surrogate to the SFT framework, setting \(\mathcal{L}_{\theta} = \mathcal{L}_{\text{SFT}}\), and introducing a temperature parameter \(\tau\) to regulate the optimization boundary, we derive the Conservative Importance Weight (\(\omega_{\text{ConSFT}}\)):
This formulation successfully translates the hard clipping boundaries of RL into a dynamic, soft weighting mechanism for SFT. It automatically suppresses excessive gradients from high-loss transitions, enforcing the necessary parameter update sparsity to preserve pre-trained foundational capabilities without requiring a parallel reference network.
ConSFT requires no architectural modifications and zero historical data. The optimization objective is formulated as:
1. Forgetting Risk Attenuation: The soft weighting term modulates the gradient step size during training. By applying an exponentially decaying weight to high-loss transitions, ConSFT limits the magnitude of parameter updates. This bounding mechanism prevents dense overwriting of pre-trained weights, attenuating the risk of catastrophic forgetting to preserve foundational capabilities.
2. Progressive Assimilation Dynamic: ConSFT does not discard high-loss data. Instead, it establishes a progressive learning dynamic. During early adaptation, the model learns from low-loss samples that align with its prior. As the network adapts, initially difficult transitions become better modeled, their losses decrease, and their optimization weights increase. This feedback loop allows the model to learn complex, out-of-distribution transitions without sudden parameter shifts.
To evaluate capability retention, we benchmarked ConSFT against standard unconstrained SFT and established continual learning paradigms, including KL-Regularization (LwF), Low-Rank Adaptation (LoRA), and Experience Replay (ER). While vanilla SFT rapidly overwrites pre-trained skills and structural constraints like LoRA hinder target-task convergence, ConSFT effectively balances new adaptation with foundational preservation. Strikingly, ConSFT outperforms even the data-heavy Experience Replay baseline in retaining prior capabilities across multiple evaluation suites. By dynamically attenuating the forgetting risk, our method achieves state-of-the-art capability retention without requiring external data buffers or parallel reference networks, proving to be the most robust and efficient optimization strategy for flow-matching VLAs.
Physical deployments confirm that ConSFT prevents spatial drift during high-precision adaptation and maintains robust execution against dense visual distractors, preserving pre-trained physical intuitions.
Fine-tuning on a high-precision, long-horizon insertion task: the policy sequentially grasps, transfers, and inserts four test tubes. This centimeter-level continuous control imposes severe spatial biases, serving as a rigorous adaptation stress test to evaluate the retention of foundational capabilities.