ResearchTrend.AI
  • Papers
  • Communities
  • Events
  • Blog
  • Pricing
Papers
Communities
Social Events
Terms and Conditions
Pricing
Parameter LabParameter LabTwitterGitHubLinkedInBlueskyYoutube

© 2025 ResearchTrend.AI, All rights reserved.

  1. Home
  2. Papers
  3. 2309.12927
22
5

Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks

22 September 2023
Sina Khajehabdollahi
Roxana Zeraati
E. Giannakakis
T. Schafer
Georg Martius
Anna Levina
    LRM
ArXivPDFHTML
Abstract

Recurrent neural networks (RNNs) in the brain and in silico excel at solving tasks with intricate temporal dependencies. Long timescales required for solving such tasks can arise from properties of individual neurons (single-neuron timescale, τ\tauτ, e.g., membrane time constant in biological neurons) or recurrent interactions among them (network-mediated timescale). However, the contribution of each mechanism for optimally solving memory-dependent tasks remains poorly understood. Here, we train RNNs to solve NNN-parity and NNN-delayed match-to-sample tasks with increasing memory requirements controlled by NNN by simultaneously optimizing recurrent weights and τ\tauτs. We find that for both tasks RNNs develop longer timescales with increasing NNN, but depending on the learning objective, they use different mechanisms. Two distinct curricula define learning objectives: sequential learning of a single-NNN (single-head) or simultaneous learning of multiple NNNs (multi-head). Single-head networks increase their τ\tauτ with NNN and are able to solve tasks for large NNN, but they suffer from catastrophic forgetting. However, multi-head networks, which are explicitly required to hold multiple concurrent memories, keep τ\tauτ constant and develop longer timescales through recurrent connectivity. Moreover, we show that the multi-head curriculum increases training speed and network stability to ablations and perturbations, and allows RNNs to generalize better to tasks beyond their training regime. This curriculum also significantly improves training GRUs and LSTMs for large-NNN tasks. Our results suggest that adapting timescales to task requirements via recurrent interactions allows learning more complex objectives and improves the RNN's performance.

View on arXiv
Comments on this paper