paper / thom_wolf / Jun 7
Asynchronous DiLoCo-style training improves hardware utilization by eliminating synchronization barriers, but produces stale pseudo-gradients that diverge from the current global optimization trajectory — a problem compounded by non-IID data distributions across workers. HeLoCo addresses this by using outer momentum as a reference signal to selectively correct incoming pseudo-gradients: aligned updates are preserved while conflicting directional components are adjusted before the outer update. Evaluated on multilingual language model training with heterogeneous workers and non-IID data, HeLoCo consistently outperforms both synchronous and asynchronous baselines across token and wall-clock budgets.
distributed-trainingfederated-learningasynchronous-optimizationlow-communicationnon-iid-datallm-traininggradient-correction
“HeLoCo outperforms existing asynchronous DiLoCo-based baselines by up to 7.5% on validation loss at a fixed token budget.”
paper / thom_wolf / Jun 7
Bootstrap Your Generator (ByG) is a framework that enables training flow matching models for image and video editing without paired examples, instead leveraging the base model's own frozen knowledge as the supervisory signal. It combines instruction-following cues extracted from the frozen model with cycle-consistency constraints to preserve structure, and introduces gradient routing over clean predictions back to noisy training states to close the train-inference gap. ByG achieves state-of-the-art results on data-scarce editing tasks and outperforms supervised baselines trained on millions of paired samples, eliminating the need for external reward models or paired datasets entirely.
image-editingvideo-editingflow-matchinggenerative-modelsunpaired-trainingcycle-consistencycomputer-vision
“ByG outperforms supervised baselines trained on millions of paired samples for image and video editing tasks.”
paper / thom_wolf / May 12 / failed
paper / thom_wolf / May 12 / failed