Learning Expressive Random Feature Models via Parametrized Activations

Random feature (RF) method is a powerful kernel approximation technique, but is typically equipped with fixed activation functions, limiting its adaptability across diverse tasks. To overcome this limitation, we introduce the Random Feature Model with Learnable Activation Functions (RFLAF), which enhances the model expressivity by parameterizing activation functions as weighted sums of basis functions. Specifically, we propose to use radial basis functions (RBFs) as bases. We first analyze the RF model with a single RBF activation, deriving a novel kernel and presenting its theoretical properties. Extending this to multiple RBFs, we show that RFLAF significantly expands the function space of RF models while maintaining parameter efficiency. Experimental results across multiple tasks demonstrate that RFLAF consistently outperforms standard RF models with minimal extra computational cost. Furthermore, RFLAF showcases the ability of recovering the optimal activation function directly from data. Our work provides a deeper understanding of the component of learnable activation functions within modern neural networks architectures.