Visual Learning of Arithmetic Operations

Many learning tasks combine perceptual and cognitive (e.g. logical, symbolic) sub-tasks. In some cases end-to-end learning of the perceptual and cognitive sub-tasks can be more difficult than learning each sub-task individually. In other cases such end-to-end learning can be very efficient. As end-to-end learning is becoming more prevalent in AI, it is important to understand when this approach can work efficiently. A toy example is presented: end-to-end visual learning of arithmetic operations from pictures of numbers. The perceptual and cognitive sub-tasks are OCR and arithmetic respectively. The input consists of two pictures, each showing a 7-digit number. The output is the picture of the result of the arithmetic operation (e.g. addition) on the two input numbers. The concept of a number, or of an operator, is not explicitly introduced. Learning is carried out by a deep neural network. In our experiments, a simple network was successful at visual end-to-end learning of addition and subtraction. Other operations, e.g. multiplication, were not learnable by our network, even when each separate sub-task was easily learnable.