- This event has passed.
Numerical Analysis: Bao Wang, The University of Utah, Implicit Methods for Deep Learning
October 18, 2022 | 3:30 pm - 4:30 pm EDT
At the heart of modern deep learning is the deep neural nets accompanied by stochastic optimization and sampling algorithms. These architectures and algorithms have an explicit flavor. In particular, traditional deep nets compose affine and simple nonlinear transformations, defining an explicit function for representation learning. The explicit deep nets lack flexibility and adaptivity for modeling irregularly sampled and complex data; training these models is expensive in computational time and memory costs and is often unstable. Implicit neural nets have emerged as new innovative architectures for deep learning. Explicit neural nets transform the input into features using an explicit function. In contrast, implicit nets learn representations in an implicit manner, e.g., 1) solving an ordinary differential equation (ODE) with input data as the initial condition, leading to a neural ODE model; 2) finding the fixed point of an algebraic equation, resulting in a deep equilibrium model (DEQ); or 3) solving an optimization problem, resulting in a differentiable optimization layer. These implicit models enable flexible and diverse neural architecture design, adaptive computation, interpretable prediction, and memory-efficient training.
In this talk, I will present a few recent results on PDE and monotone operator theory-based deep learning approaches for learning time series and graphs, ranging from neural ODE to fixed-point networks. The new approaches are guaranteed to learn long-range dependencies in the underlying data, which is crucial for learning intrinsic patterns from complex data. The presented new approaches are substantially more expressive and require much less computational cost than the existing implicit neural networks. If time permits, I will also talk about implicit stochastic optimization and Langevin-based MCMC sampling algorithms — together with implicit deep neural nets — serving as the pillar of implicit deep learning.