Infinite Ensemble Learning with Support Vector Machines

Citation

Lin, Hsuan-Tien (2005) Infinite Ensemble Learning with Support Vector Machines. Master's thesis, California Institute of Technology. doi:10.7907/E03R-EN93. https://resolver.caltech.edu/CaltechETD:etd-05262005-030549

Abstract

Ensemble learning algorithms such as boosting can achieve better performance by averaging over the predictions of base learners. However, existing algorithms are limited to combining only a finite number of base learners, and the generated ensemble is usually sparse. It is not clear whether we should construct an ensemble classifier with a larger or even an infinite number of base learners.

In addition, constructing an infinite ensemble itself is a challenging task. In this paper, we formulate an infinite ensemble learning framework based on SVM. The framework could output an infinite and nonsparse ensemble, and can be applied to construct new kernels for SVM as well as to interpret existing ones. We demonstrate the framework with a concrete application, the stump kernel, which embodies infinitely many decision stumps. The stump kernel is simple, yet powerful.

Experimental results show that SVM with the stump kernel usually achieves better performance than boosting, even with noisy data.

Thesis Files