Pre-training
Training models from scratch
- Focus
- Transformer architecture, data quality, and training efficiency.
- Method
- We design the architecture and data, then train at scale with 5D parallelism.
- Signal
- Models built for the task, not borrowed and forced to fit.