A hybrid ensemble method with negative correlation learning for regression

Hybrid ensemble, an essential branch of ensembles, has flourished in numerous machine learning problems, especially regression. Several studies have confirmed the importance of diversity; however, previous ensembles only consider diversity in the sub-model training stage, with limited improvement compared to single models. In contrast, this study focuses on the sub-model combination stage of the ensemble. It solves a non-convex optimization problem using an interior-point filtering linear-search algorithm to select and weight sub-models from a heterogeneous model pool automatically. This optimization problem innovatively incorporates negative correlation learning as a penalty term. Thus, a diverse model subset can be selected. Experimental results show that the approach outperforms single model and overcomes the instability of the models and parameters. Compared to bagging and stacking without model diversity, our method stands out more and confirms the importance of diversity in the ensemble. Additionally, the performance of our proposed method is better than that of simple and weighted averages, and the variance of the weights is lower and more stable than that of a linear model. Finally, the prediction accuracy can be further improved by fine-tuning the weights using the error inverse weights. In conclusion, the value of this study lies in its ease of use and effectiveness, allowing the hybrid ensemble to embrace both diversity and accuracy.