Position Information in Transformers: An Overview

22 February 2021

Papers citing "Position Information in Transformers: An Overview"

50 / 66 papers shown

Title
Dual Filter: A Mathematical Framework for Inference using Transformer-like Architectures Heng-Sheng Chang P. Mehta 44 0 0 01 May 2025
The effect of the number of parameters and the number of local feature patches on loss landscapes in distributed quantum neural networks Yoshiaki Kawase 76 0 0 27 Apr 2025
Distilling semantically aware orders for autoregressive image generation Rishav Pramanik Antoine Poupon Juan A. Rodriguez Masih Aminbeidokhti David Vazquez Christopher Pal Zhaozheng Yin M. Pedersoli 31 0 0 23 Apr 2025
Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models Jianing Qi Jiawei Liu Hao Tang Zhigang Zhu 114 1 0 21 Mar 2025
Are Transformers Truly Foundational for Robotics? James A. R. Marshall Andrew B. Barron AI4CE 78 0 0 25 Nov 2024
Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecasting Jingjing Xu Caesar Wu Yuan-Fang Li Grégoire Danoy Pascal Bouvry AI4TS 45 1 0 29 Jul 2024
Shared Imagination: LLMs Hallucinate Alike Yilun Zhou Caiming Xiong Silvio Savarese Chien-Sheng Wu HILM 32 1 0 23 Jul 2024
Transformers with Stochastic Competition for Tabular Data Modelling Andreas Voskou Charalambos Christoforou S. Chatzis LMTD 37 1 0 18 Jul 2024
An Effective-Efficient Approach for Dense Multi-Label Action Detection Faegheh Sardari Armin Mustafa Philip J. B. Jackson Adrian Hilton 37 0 0 10 Jun 2024
Contextual Position Encoding: Learning to Count What's Important O. Yu. Golovneva Tianlu Wang Jason Weston Sainbayar Sukhbaatar 53 25 0 29 May 2024
Reference Neural Operators: Learning the Smooth Dependence of Solutions of PDEs on Geometric Deformations Ze Cheng Zhongkai Hao Xiaoqiang Wang Jianing Huang Youjia Wu Xudan Liu Yiru Zhao Songming Liu Hang Su AI4CE 42 2 0 27 May 2024
Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learning Junfeng Chen Kailiang Wu 42 3 0 15 May 2024
Test-Time Augmentation for Traveling Salesperson Problem Ryo Ishiyama Takahiro Shirakawa Seiichi Uchida Shinnosuke Matsuo 54 0 0 08 May 2024
Learning with 3D rotations, a hitchhiker's guide to SO(3) A. R. Geist Jonas Frey Mikel Zobro Anna Levina Georg Martius 3DH SSL 43 18 0 17 Apr 2024
Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda Johannes Schneider 83 26 0 15 Apr 2024
A Morphology-Based Investigation of Positional Encodings Poulami Ghosh Shikhar Vashishth Raj Dabre Pushpak Bhattacharyya 34 1 0 06 Apr 2024
MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation Weiguo Gao 42 1 0 26 Mar 2024
Materials science in the era of large language models: a perspective Ge Lei Ronan Docherty Samuel J. Cooper 45 18 0 11 Mar 2024
Temporal Cross-Attention for Dynamic Embedding and Tokenization of Multimodal Electronic Health Records Yingbo Ma Suraj Kolla Dhruv Kaliraman Victoria Nolan Zhenhong Hu ... T. Ozrazgat-Baslanti Tyler J. Loftus Parisa Rashidi A. Bihorac B. Shickel AI4TS 34 1 0 06 Mar 2024
Knowledge of Pretrained Language Models on Surface Information of Tokens Tatsuya Hiraoka Naoaki Okazaki 32 1 0 15 Feb 2024
Accelerating Material Property Prediction using Generically Complete Isometry Invariants Jonathan Balasingham Viktor Zamaraev V. Kurlin 16 5 0 22 Jan 2024
SymTC: A Symbiotic Transformer-CNN Net for Instance Segmentation of Lumbar Spine MRI Jiasong Chen Linchen Qian Linhai Ma Timur Urakov Weiyong Gu Liang Liang MedIm 39 4 0 17 Jan 2024
Code Simulation Challenges for Large Language Models Emanuele La Malfa Christoph Weinhuber Orazio Torre Fangru Lin Samuele Marro Anthony Cohn Nigel Shadbolt Michael Wooldridge LLMAG LRM 22 8 0 17 Jan 2024
Graph Language Models Moritz Plenz Anette Frank KELM AI4CE 28 6 0 13 Jan 2024
Algebraic Positional Encodings Konstantinos Kogkalidis Jean-Philippe Bernardy Vikas K. Garg 24 1 0 26 Dec 2023
Graph Neural Networks with Diverse Spectral Filtering Jingwei Guo Kaizhu Huang Xinping Yi Rui Zhang 72 12 0 14 Dec 2023
Prompt Cache: Modular Attention Reuse for Low-Latency Inference In Gim Guojun Chen Seung-seob Lee Nikhil Sarda Anurag Khandelwal Lin Zhong 42 77 0 07 Nov 2023
Transformers as Graph-to-Graph Models James Henderson Alireza Mohammadshahi Andrei Catalin Coman Lesly Miculicich GNN 35 6 0 27 Oct 2023
The Locality and Symmetry of Positional Encodings Lihu Chen Gaël Varoquaux Fabian M. Suchanek 44 0 0 19 Oct 2023
From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers Shaoxiong Duan Yining Shi Wei Xu 28 8 0 18 Oct 2023
DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions Haochen Wang Junsong Fan Yuxi Wang Kaiyou Song Tong Wang Zhaoxiang Zhang 29 19 0 07 Sep 2023
PAT: Position-Aware Transformer for Dense Multi-Label Action Detection Faegheh Sardari A. Mustafa Philip J. B. Jackson A. Hilton ViT 27 6 0 09 Aug 2023
A Survey of Techniques for Optimizing Transformer Inference Krishna Teja Chitty-Venkata Sparsh Mittal M. Emani V. Vishwanath Arun Somani 48 63 0 16 Jul 2023
Pseudo-rigid body networks: learning interpretable deformable object dynamics from partial observations Shamil Mamedov A. R. Geist Jan Swevers Sebastian Trimpe AI4CE 21 2 0 16 Jul 2023
Monotonic Location Attention for Length Generalization Jishnu Ray Chowdhury Cornelia Caragea LLMAG 24 8 0 31 May 2023
Improving Position Encoding of Transformers for Multivariate Time Series Classification Navid Mohammadi Foumani Chang Wei Tan Geoffrey I. Webb Mahsa Salehi AI4TS 30 74 0 26 May 2023
Causal Decision Transformer for Recommender Systems via Offline Reinforcement Learning Siyu Wang Xiaocong Chen Dietmar Jannach Lina Yao CML OffRL 24 27 0 17 Apr 2023
Language Model Behavior: A Comprehensive Survey Tyler A. Chang Benjamin Bergen VLM LRM LM&MA 29 103 0 20 Mar 2023
Universal Morphology Control via Contextual Modulation Zheng Xiong Jacob Beck Shimon Whiteson 33 13 0 22 Feb 2023
Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE Qihuang Zhong Liang Ding Keqin Peng Juhua Liu Bo Du Li Shen Yibing Zhan Dacheng Tao VLM 42 13 0 18 Feb 2023
Enhancing Multivariate Time Series Classifiers through Self-Attention and Relative Positioning Infusion Mehryar Abbasi Parvaneh Saeedi AI4TS 32 6 0 13 Feb 2023
Investigating the Effect of Relative Positional Embeddings on AMR-to-Text Generation with Structural Adapters Sébastien Montella Alexis Nasr Johannes Heinecke Frédéric Béchet L. Rojas-Barahona 29 2 0 12 Feb 2023
Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames Ondrej Biza Sjoerd van Steenkiste Mehdi S. M. Sajjadi Gamaleldin F. Elsayed Aravindh Mahendran Thomas Kipf OCL 50 32 0 09 Feb 2023
It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformers Ana-Maria Bucur Adrian Cosma Paolo Rosso Liviu P. Dinu 33 34 0 13 Jan 2023
A Length-Extrapolatable Transformer Yutao Sun Li Dong Barun Patra Shuming Ma Shaohan Huang Alon Benhaim Vishrav Chaudhary Xia Song Furu Wei 30 115 0 20 Dec 2022
Explainability of Text Processing and Retrieval Methods: A Critical Survey Sourav Saha Debapriyo Majumdar Mandar Mitra 18 5 0 14 Dec 2022
Position Embedding Needs an Independent Layer Normalization Runyi Yu Zhennan Wang Yinhuai Wang Kehan Li Yian Zhao Jian Zhang Guoli Song Jie Chen 31 1 0 10 Dec 2022
Word Order Matters when you Increase Masking Karim Lasri Alessandro Lenci Thierry Poibeau 38 7 0 08 Nov 2022
Generalized Attention Mechanism and Relative Position for Transformer R. Pandya ViT 11 1 0 24 Jul 2022
Position Prediction as an Effective Pretraining Strategy Shuangfei Zhai Navdeep Jaitly Jason Ramapuram Dan Busbridge Tatiana Likhomanenko Joseph Y. Cheng Walter A. Talbott Chen Huang Hanlin Goh J. Susskind ViT 51 23 0 15 Jul 2022