Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task

24 October 2022

Papers citing "Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task"

50 / 200 papers shown

Title
From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers Shaoxiong Duan Yining Shi Wei Xu 28 8 0 18 Oct 2023
Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT D. Hazineh Zechen Zhang Jeffery Chiu 30 6 0 11 Oct 2023
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets Samuel Marks Max Tegmark HILM 102 181 0 10 Oct 2023
From task structures to world models: What do LLMs know? ilker. yildirim L. A. Paul 29 41 0 06 Oct 2023
Language Models Represent Space and Time Wes Gurnee Max Tegmark 54 142 0 03 Oct 2023
Conceptual Framework for Autonomous Cognitive Entities David Shapiro Wangfan Li Manuel Delaflor Carlos Toxtli 46 1 0 03 Oct 2023
Towards Causal Foundation Model: on Duality between Causal Inference and Attention Jiaqi Zhang Joel Jennings Agrin Hilmkil Nick Pawlowski Cheng Zhang Chao Ma CML 72 13 0 01 Oct 2023
SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation Matthias Lindemann Alexander Koller Ivan Titov AI4CE 24 2 0 01 Oct 2023
Improving Length-Generalization in Transformers via Task Hinting Pranjal Awasthi Anupam Gupta 41 8 0 01 Oct 2023
Towards Best Practices of Activation Patching in Language Models: Metrics and Methods Fred Zhang Neel Nanda LLMSV 43 101 0 27 Sep 2023
Generative AI vs. AGI: The Cognitive Strengths and Weaknesses of Modern LLMs Ben Goertzel 38 14 0 19 Sep 2023
Breaking through the learning plateaus of in-context learning in Transformer Jingwen Fu Tao Yang Yuwang Wang Yan Lu Nanning Zheng 32 1 0 12 Sep 2023
Explaining grokking through circuit efficiency Vikrant Varma Rohin Shah Zachary Kenton János Kramár Ramana Kumar 26 49 0 05 Sep 2023
Emergent Linear Representations in World Models of Self-Supervised Sequence Models Neel Nanda Andrew Lee Martin Wattenberg FAtt MILM 55 149 0 02 Sep 2023
Introducing ChatSQC: Enhancing Statistical Quality Control with Augmented AI F. Megahed Ying-Ju Chen Inez M. Zwetsloot S. Knoth D. Montgomery L. A. Jones‐Farmer 25 3 0 22 Aug 2023
Contrasting Linguistic Patterns in Human and LLM-Generated Text Alberto Muñoz-Ortiz Carlos Gómez-Rodríguez David Vilares DeLMO 30 2 0 17 Aug 2023
Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operation Xinshuo Hu Dongfang Li Baotian Hu Zihao Zheng Zhenyu Liu Hao Fei KELM MU 40 26 0 16 Aug 2023
Multimodal Neurons in Pretrained Text-Only Transformers Sarah Schwettmann Neil Chowdhury Samuel J. Klein David Bau Antonio Torralba MILM 40 27 0 03 Aug 2023
Learning to Model the World with Language Jessy Lin Yuqing Du Olivia Watkins Danijar Hafner Pieter Abbeel Dan Klein Anca Dragan LM&Ro SyDa 49 51 0 31 Jul 2023
Large Language Models Michael R Douglas LLMAG LM&MA 59 568 0 11 Jul 2023
Substance or Style: What Does Your Image Embedding Know? Cyrus Rashtchian Charles Herrmann Chun-Sung Ferng Ayan Chakrabarti Dilip Krishnan Deqing Sun Da-Cheng Juan Andrew Tomkins 36 6 0 10 Jul 2023
Discovering Variable Binding Circuitry with Desiderata Xander Davies Max Nadeau Nikhil Prakash Tamar Rott Shaham David Bau 36 13 0 07 Jul 2023
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks Zhaofeng Wu Linlu Qiu Alexis Ross Ekin Akyürek Boyuan Chen Bailin Wang Najoung Kim Jacob Andreas Yoon Kim LRM ReLM 63 197 0 05 Jul 2023
Domain-specific ChatBots for Science using Embeddings Kevin G. Yager 53 8 0 15 Jun 2023
Opportunities for Large Language Models and Discourse in Engineering Design Jan Göpfert J. Weinand Patrick Kuckertz D. Stolten AI4CE 47 4 0 15 Jun 2023
Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model Yida Chen Fernanda Viégas Martin Wattenberg DiffM 14 22 0 09 Jun 2023
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model Kenneth Li Oam Patel Fernanda Viégas Hanspeter Pfister Martin Wattenberg KELM HILM 58 495 0 06 Jun 2023
The Hidden Language of Diffusion Models Hila Chefer Oran Lang Mor Geva Volodymyr Polosukhin Assaf Shocher Michal Irani Inbar Mosseri Lior Wolf DiffM 30 26 0 01 Jun 2023
Passive learning of active causal strategies in agents and language models Andrew Kyle Lampinen Stephanie C. Y. Chan Ishita Dasgupta A. Nam Jane X. Wang 34 15 0 25 May 2023
Language Models Implement Simple Word2Vec-style Vector Arithmetic Jack Merullo Carsten Eickhoff Ellie Pavlick KELM 36 54 0 25 May 2023
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning L. Guan Karthik Valmeekam S. Sreedharan Subbarao Kambhampati LLMAG 24 162 0 24 May 2023
Training Transitive and Commutative Multimodal Transformers with LoReTTa Manuel Tran Yashin Dicente Cid Amal Lahiani Fabian J. Theis Tingying Peng Eldad Klaiman 26 2 0 23 May 2023
Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models Oana Ignat Zhijing Jin Artem Abzaliev Laura Biester Santiago Castro ... Verónica Pérez-Rosas Siqi Shen Zekun Wang Winston Wu Rada Mihalcea LRM 46 6 0 21 May 2023
A Glimpse in ChatGPT Capabilities and its impact for AI research Frank Joublin Antonello Ceravola Joerg Deigmoeller Michael Gienger M. Franzius Julian Eggert SILM AI4MH ALM ELM 30 15 0 10 May 2023
The System Model and the User Model: Exploring AI Dashboard Design Fernanda Viégas Martin Wattenberg 28 6 0 04 May 2023
Entity Tracking in Language Models Najoung Kim Sebastian Schuster 60 19 0 03 May 2023
Finding Neurons in a Haystack: Case Studies with Sparse Probing Wes Gurnee Neel Nanda Matthew Pauly Katherine Harvey Dmitrii Troitskii Dimitris Bertsimas MILM 165 192 0 02 May 2023
How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model Michael Hanna Ollie Liu Alexandre Variengien LRM 212 123 0 30 Apr 2023
The Vector Grounding Problem Dimitri Coelho Mollo Raphael Milliere 46 26 0 04 Apr 2023
Eight Things to Know about Large Language Models Sam Bowman ALM 32 114 0 02 Apr 2023
The Quantization Model of Neural Scaling Eric J. Michaud Ziming Liu Uzay Girit Max Tegmark MILM 32 77 0 23 Mar 2023
Eliciting Latent Predictions from Transformers with the Tuned Lens Nora Belrose Zach Furman Logan Smith Danny Halawi Igor V. Ostrovsky Lev McKinney Stella Biderman Jacob Steinhardt 27 196 0 14 Mar 2023
Could a Large Language Model be Conscious? D. Chalmers LRM AI4CE ELM 29 84 0 04 Mar 2023
A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations Bilal Chughtai Lawrence Chan Neel Nanda 21 96 0 06 Feb 2023
Towards Reliable Neural Specifications Chuqin Geng Nham Le Xiaojie Xu Zhaoyue Wang A. Gurfinkel X. Si AAML 36 10 0 28 Oct 2022
Formal Semantic Geometry over Transformer-based Variational AutoEncoder Yingji Zhang Danilo S. Carvalho Ian Pratt-Hartmann André Freitas 36 4 0 12 Oct 2022
Diffusion-LM Improves Controllable Text Generation Xiang Lisa Li John Thickstun Ishaan Gulrajani Percy Liang Tatsunori B. Hashimoto AI4CE 173 781 0 27 May 2022
Probing Classifiers: Promises, Shortcomings, and Advances Yonatan Belinkov 229 409 0 24 Feb 2021
What you can cram into a single vector: Probing sentence embeddings for linguistic properties Alexis Conneau Germán Kruszewski Guillaume Lample Loïc Barrault Marco Baroni 201 883 0 03 May 2018
Simpler Context-Dependent Logical Forms via Model Projections R. Long Panupong Pasupat Percy Liang 210 101 0 16 Jun 2016