Speculative Decoding for Multi-Sample Inference

We propose a novel speculative decoding method tailored for multi-sample reasoning scenarios, such as self-consistency and Best-of-N sampling. Our method exploits the intrinsic consensus of parallel generation paths to synthesize high-quality draft tokens without requiring auxiliary models or external databases. By dynamically analyzing structural patterns across parallel reasoning paths through a probabilistic aggregation mechanism, it identifies consensus token sequences that align with the decoding distribution. Evaluations on mathematical reasoning benchmarks demonstrate a substantial improvement in draft acceptance rates over baselines, while reducing the latency in draft token construction. This work establishes a paradigm shift for efficient multi-sample inference, enabling seamless integration of speculative decoding with sampling-based reasoning techniques.

@article{li2025_2503.05330,
  title={ Speculative Decoding for Multi-Sample Inference },
  author={ Yiwei Li and Jiayi Shi and Shaoxiong Feng and Peiwen Yuan and Xinglin Wang and Yueqi Zhang and Ji Zhang and Chuyi Tan and Boyuan Pan and Yao Hu and Kan Li },
  journal={arXiv preprint arXiv:2503.05330},
  year={ 2025 }
}