关键词: 模型选择, 泛化误差, 组块3×, 2交叉验证, 方差正则化

Abstract: In traditional machine learning, model selection is always directly performed based on the estimation of one performance measure index, without considering the variance of the estimation. However, this neglection may probably lead to the selection of a wrong model. Therefore, a method of adding the information of variance into the study of classification model selection is considered in order to improve the generalization ability of the selected model, that is, the variance estimation of the block 3×2 cross-validation estimation of the generalization error is added as a regularization term into the traditional model selection criterion, and a new variance-regularized classification model selection criterion is proposed. The simulated and real data experiments show that the proposed model selection criterion has a higher probability to select the correct classification model in the classification model selection problem compared to the traditional methods. The importance of variance in model selection and the effectiveness of the   proposed model selection criteria are also validated. Furthermore, the consistency in selection of the proposed criterions is theoretically proven in the model selection task of two-class classification problem.

Key words: model selection, generalization error, blocked 3×2 cross-validation, variance-regularized