Bias of Error Rates in Linear Discriminant Analysis Caused by Feature Selection and Sample Size

The holdout and leave-one-out error estimates for a two-class problem with multivariate normal distributions and common covariance are derived as a function of the number of feature candidates, classifier dimensionality, sample size and Mahalanobis distance, using Monte Carlo simulations. It is demonstrated that the leave-one-out error rate is a highly biased estimate of the true error if feature selection is performed on the same data before error estimation. This problem is especially pronounced when analyzing many features on a small data set. The holdout error is an almost unbiased estimate of the true error independent of the number of feature candidates.