关键词: F1度量, 交叉验证, 方差, 分类算法, 模拟实验

Abstract: In the research on statistical machine learning, researchers often perform quantitative experiments to compare F1 measure of classification algorithms based on cross validation. In order to obtain statistically convincing conclusion, it is very important to estimate the uncertainty of F1 measure. In particular, the blocked 3×2 cross validation is demonstrated that its performance is superior to other cross validation methods such as the standard K-fold cross validation by theory and experiments. Thus, this paper studies theoretically the variance of F1 measure based on blocked 3×2 cross validation. The structure of variance shows that it is composed of three parts: block variance, within-block covariance and between-blocks covariance, which also implies that the commonly used sample variance may grossly underestimate or overestimate the real variance. The above theoretical results are validated by the experiments in simulated and real data sets through bar chart method. The experimental results show that the within-block covariance and between-blocks covariance are of same order as the block variance. The within-block and between-blocks correlations can not be neglected.

Key words: F1 measure, cross validation, variance, classification algorithm, simulated experiment