Octopus: A Framework for Cost-Quality-Time Optimization in Crowdsourcing

Managing micro-tasks on crowdsourcing marketplaces involves balancing conflicting objectives -- the quality of work, total cost incurred and time to completion. Previous agents have focused on cost-quality, or cost-time tradeoffs, limiting their real-world applicability. As a step towards this goal we present Octopus, the first AI agent that jointly manages all three objectives in tandem. Octopus is based on a computationally tractable, multi-agent formulation consisting of three components; one that sets the price per ballot to adjust the rate of completion of tasks, another that optimizes each task for quality and a third that performs task selection. We demonstrate that Octopus outperforms existing state-of-the-art approaches in simulation and experiments with real data, demonstrating its superior performance. We also deploy Octopus on Amazon Mechanical Turk to establish its ability to manage tasks in a real-world, dynamic setting.