The resurging interest in deep learning is commonly attributed to advances in hardware and growing data sizes and less so to new algorithmic improvements. However, cutting-edge numerical methods are needed to tackle ever larger and more complex learning problems. In this talk, I will illustrate numerical analysis tools for improving the effectiveness of deep learning algorithms. With a focus on deep neural networks that can be modeled as differential equations, I will highlight the importance of choosing an adequate time integrator. I will also compare, using a numerical example,  the difference between the first-discretize-then-optimize and the first-optimize-then-discretize paradigms for training residual neural networks. Finally, I show that exploiting the separable structure of most learning problems can increase the efficiency and the accuracy of training.

Zoom: Contact A. Saibaba for link.