Counterfactual Learning from Logs for Improved Ranking of E-Commerce Products

Improved search quality enhances users' satisfaction, which directly impacts sales growth of an E-Commerce (E-Com) platform. Learning to Rank (LTR) algorithms require relevance judgments (labels) on products for learning. In commercial scenarios, getting such judgments poses an immense challenge in application of LTR algorithms. In the literature, it is proposed to employ user feedback signals such as clicks, orders etc to generate relevance judgments. It is done by aggregating the logged data and calculating click rate etc of products for the queries in the logs. In this paper, we advocate counterfactual risk minimization (CRM) approach which circumvents the need of such data pre-processing and is better suited for learning from logged data, i.e. contextual bandit feedback. Due to unavailability of public E-Com LTR dataset, we provide \textit{Commercial dataset} from our E-Com platform. It contains more than 10 million click log entries and 1 million order logs from a catalogue of about 3.5 million products and 3000 queries. To the best of our knowledge, this is the first work which examines effectiveness of CRM approach in learning ranking model from real-world logged data. Our empirical evaluation shows that CRM approach not only learns effectively from logged contextual-bandit feedback but also that our CRM based method outperforms full-information (e.g. cross-entropy) loss on various deep neural network models as well as traditional models like LambdaMART. These findings show that by adopting CRM learning approach, E-Com platforms can get better product search quality compared to full-information approach, without artificially mending the data.