COBias and Debias: Balancing Class Accuracies for Language Models in Inference Time via Nonlinear Integer Programming

Large language models (LLMs) are good knowledge bases but struggle to perform equally well for all classes in text classification tasks. This paper investigates a fundamental inference-time problem in language models: imbalanced class accuracies. We find what's underneath the issue is a tendency to over-predict some classes while under-predicting some others. This class accuracy imbalance is difficult to solve from the root via better pre-training or fine-tuning strategies, but we show it can be effectively mitigated via inference-time combinatorial optimization. To this end, we conceptualize and quantify the over- and under-prediction issue as the Contextual Oddity Bias (COBias), and propose the Debiasing as Nonlinear Integer Programming (DNIP) model to correct in-context learned class probabilities based on minimizing COBias and maximizing overall accuracy, without LLM parameter update. Considering that the DNIP model implicitly contains non-differentiable elements, we therefore use the simulated annealing algorithm to solve it. Extensive evaluations on three LLMs across seven NLP classification tasks in different prompting settings show that DNIP simultaneously achieves significant COBias reduction (-27%) and accuracy improvement (+12%) over the conventional ICL approach, suggesting that inference-time mitigation of class accuracy imbalance is a promising direction to push forward LLM performances.