Proving that shallow tanh networks, once the obvious redundancies are removed, form a smooth manifold — the geometry behind natural gradient methods.
Natural Gradient Descent has become a method of choice for training physics-informed neural networks. A tempting way to explain its success is to view it as Riemannian gradient descent on the set of functions that the network can represent — which would immediately bring the convergence theory of Riemannian optimization into play.
The problem is that this set is generally not a smooth manifold: the parametrization is far from injective, and its image is neither a manifold nor closed.
Some redundancies are obvious: permuting neurons or flipping their signs does not change the network, and neurons whose outgoing weights vanish contribute nothing. We prove that, once these are excluded, the remaining irreducible shallow tanh networks form a smooth embedded submanifold of a Sobolev space, with the expected dimension.
The proof combines group actions, complex analysis, and an asymptotic analysis of how neurons can behave in the limit.
The results were presented at the Analysis Junior Seminars at SISSA (Trieste) and will be presented in the Numerical Analysis Seminars at the Politecnico di Torino.