Dataset Engineering Dominates Cost in Training a 22B-Parameter Video Foundation Model from Scratch
Summer-22B is a video foundation model trained from scratch on ~50 million clips, with the authors reporting that dataset engineering — not architecture or optimization — consumed the majority of effort. The team built a custom data management system (Lavender Data) using metadata-driven curation and multi-stage filtering, and adopted μP (Maximal Update Parameterization) with hypersphere-constrained optimization for stable scaling. A key counterintuitive finding is that architectural variants produced smaller performance differences than expected, while μP hyperparameter transfer remained effective even under geometric constraints.