Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed-source or limited in their capability for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan 2 excels in vertical domains such as medicine and law. We will release all pre-training model checkpoints to benefit the research community in better understanding the training dynamics of Baichuan 2.
View on arXiv@article{yang2025_2309.10305,
title={ Baichuan 2: Open Large-scale Language Models },
author={ Aiyuan Yang and Bin Xiao and Bingning Wang and Borong Zhang and Ce Bian and Chao Yin and Chenxu Lv and Da Pan and Dian Wang and Dong Yan and Fan Yang and Fei Deng and Feng Wang and Feng Liu and Guangwei Ai and Guosheng Dong and Haizhou Zhao and Hang Xu and Haoze Sun and Hongda Zhang and Hui Liu and Jiaming Ji and Jian Xie and JunTao Dai and Kun Fang and Lei Su and Liang Song and Lifeng Liu and Liyun Ru and Luyao Ma and Mang Wang and Mickel Liu and MingAn Lin and Nuolan Nie and Peidong Guo and Ruiyang Sun and Tao Zhang and Tianpeng Li and Tianyu Li and Wei Cheng and Weipeng Chen and Xiangrong Zeng and Xiaochuan Wang and Xiaoxi Chen and Xin Men and Xin Yu and Xuehai Pan and Yanjun Shen and Yiding Wang and Yiyu Li and Youxin Jiang and Yuchen Gao and Yupeng Zhang and Zenan Zhou and Zhiying Wu },
journal={arXiv preprint arXiv:2309.10305},
year={ 2025 }
}