Reproducing OLMo 3 7B Pre-training in MaxText: case study of large scale training on TPUs

T1Covered by 1 source · first reported by Google — The Keyword / AI / Research / DeepMind / Developers at 24 Sept, 15:11 UTC

The MaxText team successfully reproduced AI2’s OLMo 3 7B language model from scratch on Google Cloud TPUs using JAX/XLA, precisely matching the original PyTorch-on-GPU reference across pre-training and mid-training stages on all held-out evaluations. The implementation achieved up to 57.4% Model…

cloudhardware

Sources

• T1Reproducing OLMo 3 7B Pre-training in MaxText: case study of large scale training on TPUsGoogle — The Keyword / AI / Research / DeepMind / Developers24 Sept, 15:11