The Mechanism of Additive Composition

We prove an upper bound for the bias of additive composition (Foltz et al., 1998; Landauer and Dutnais, 1997; Mitchell and Lapata, 2010), a widely used method for computing meanings of phrases by averaging the vector representations of their constituent words. The result endorses additive composition as a reasonable operation for calculating meanings of phrases, which is the first theoretical analysis on compositional frameworks from a machine learning point of view. The theory also suggests ways to improve additive compositionality, including: transforming entries of distributional word vectors by a function that meets a specific condition, constructing a novel type of vector representations to make additive composition sensitive to word order, and utilizing singular value decomposition to train word vectors.