Singular Value Decomposition (SVD) and its close relative, Principal Component Analysis (PCA), are well-knownlinear matrix decomposition techniques that are widely used in applications such as dimension reduction andclustering. However, an important limitation of SVD/PCA is its sensitivity to noise in the input data. Inthis paper, we take another look at the problem of regularisation and show that different formulations of theminimisation problem lead to qualitatively different solutions.