Pretrained Hybrids with MAD Skills

2 June 2024

Papers citing "Pretrained Hybrids with MAD Skills"

3 / 3 papers shown

Title
Mechanistic Design and Scaling of Hybrid Architectures Michael Poli Armin W. Thomas Eric N. D. Nguyen Pragaash Ponnusamy Bjorn Deiseroth ... Brian Hie Stefano Ermon Christopher Ré Ce Zhang Stefano Massaroli MoE 29 19 0 26 Mar 2024
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models Soham De Samuel L. Smith Anushan Fernando Aleksandar Botev George-Christian Muraru ... David Budden Yee Whye Teh Razvan Pascanu Nando de Freitas Çağlar Gülçehre Mamba 22 50 0 29 Feb 2024
Simple linear attention language models balance the recall-throughput tradeoff Simran Arora Sabri Eyuboglu Michael Zhang Aman Timalsina Silas Alberti Dylan Zinsley James Zou Atri Rudra Christopher Ré 20 18 0 28 Feb 2024