Symbolic Synthesis of Neural Networks

Neural networks adapt very well to distributed and continuous representations, but struggle to learn and generalize from small amounts of data. Symbolic systems commonly achieve data efficient generalization by exploiting modularity to benefit from local and discrete features of a representation. These features allow symbolic programs to be improved one module at a time and to experience combinatorial growth in the values they can successfully process. However, it is difficult to design components that can be used to form symbolic abstractions and which are highly-overparametrized like neural networks, as the adjustment of parameters makes the semantics of modules unstable. I present Graph-based Symbolically Synthesized Neural Networks (G-SSNNs), a form of neural network whose topology and parameters are informed by the output of a symbolic program. I demonstrate that by developing symbolic abstractions at a population level, and applying gradient-based optimization to such neural models at an individual level, I can elicit reliable patterns of improved generalization with small quantities of data known to contain local and discrete features. The paradigm embodied by G-SSNNs offers a route towards the communal development of compact and composable abstractions which can be flexibly repurposed for a variety of tasks and high-dimensional media. In future work, I hope to pursue these benefits by exploring more ambitious G-SSNN designs based on more complex classes of symbolic programs. The code and data associated with the reported results are publicly available at https://github.com/shlomenu/symbolically_synthesized_networks .