In this talk we present both numerical analysis and experiments to study a few basic computational issues in practice: (1) the numerical error one can achieve given a finite machine precision, (2) the learning dynamics and computation cost to achieve a given accuracy, and (3) stability with respect to perturbations. These issues are addressed for both approximation and optimization in asymptotic and non-asymptotic regimes.