Why Feature Scaling Works: Condition Number as the Key to Gradient Descent Convergence | RISE Research

This paper provides a geometric explanation for why feature scaling accelerates gradient descent convergence, an effect that is universally recommended in machine learning practice but rarely given rigorous theoretical justification. The author uses the condition number kappa of the cost function's Hessian matrix as the unifying diagnostic quantity linking feature variance disparity to optimization difficulty, first establishing and empirically validating this link on synthetic functions with controlled condition numbers. The paper then compares four normalization methods (standardisation, min-max scaling, robust scaling, and PCA whitening) across varying dataset sizes and feature scale imbalances. Whitening is found to force kappa to its theoretical minimum of 1 regardless of the original scale ratio on clean data, but this advantage collapses under outlier contamination since it relies on an accurate covariance estimate; standardisation and min-max scaling offer more practical trade-offs but leave residual ill-conditioning when features are correlated, and min-max scaling is shown to be particularly sensitive to outliers at high variance ratios. To address this outlier sensitivity, the paper proposes and validates a two-stage pipeline combining robust scaling (which uses the outlier-resistant interquartile range) with whitening, achieving near-optimal conditioning even on noisy datasets. The paper also studies a complementary family of 'pure factor scaling' methods (KRCS, MRF, and a composite method) that leave the underlying data untouched and instead rescale the gradient itself using running second moments, showing these can outperform raw gradient descent without altering the data's condition number, demonstrating that ill-conditioning is only one of several sources of slow convergence. Finally, the paper connects these static preprocessing strategies to the dynamic normalization methods used inside neural networks (batch normalization and layer normalization), arguing both operate by continuously reducing the effective condition number during training rather than as a one-time preprocessing step.