Which scaling rule applies to Artificial Neural Networks

Although an ANN is a biology-mimicking system, it is built from components designed/fabricated for use in conventional computing, and it is created by experts trained in conventional computing. Making computing systems, actively cooperating and communicating, using segregated single processors, however, has severe performance limitations. The achievable payload computing performance sensitively depends on workload type, and this effect is only poorly known. The type of workload, AI-based systems generate, has an exceptionally low payload computational performance. Unfortunately, initial successes of demo systems that comprise only a few "neurons" and solve simple tasks are misleading: scaling of processor-based ANN systems is strongly nonlinear. The paper discusses some major limiting factors that affect their performance. It points out that for building biology-mimicking large systems, it is inevitable to perform drastic changes in the present computing paradigm (mainly to comprehend temporal behavior of components) and technical implementation of architectures.