Virvel i färger
Seminarium i matematik

Integral representations of shallow neural network

Välkommen till föreläsningen i seminarieserien i matematik.

Abstrakt:

A theoretical understanding of which functions can be well approximated by neural networks has been studied extensively in the field of approximation theory with neural networks. In par- ticular, integral representation techniques for shallow neural networks have received increasing attention. Beside the fact that shallow neural networks are simple to treat and analyze, they can represent certain high-dimension functions, which is one of the key reasons behind their success.

We begin our talk by introducing artificial neural networks from mathematical point of view. We then delve deeper into the mathematical theory behind the integral representation of shallow neural networks. More precisely, we provide a comprehensive study on how the representational cost, for a given function, depends on the chosen activation function.

Mainly, we characterize the set of functions that can be represented by infinite width neural networks with RePU activation function max(0,x)p, when the network coefficients are regular- ized by an l2/p (quasi)norm. Compared to the more well-known ReLU activation function (which corresponds to p = 1), the RePU activation functions exhibit a greater degree of smoothness which makes them preferable in several applications. Our main result shows that such repre- sentations are possible for a given function if and only if the function is κ-order Lipschitz and its R-norm is finite. This extends earlier work on this topic that has been restricted to the case of the ReLU activation function and coefficient bounds with respect to the l2 norm. Since for q < 2, lq regularizations are known to promote sparsity, our results also shed light on the ability to obtain sparse neural network representations.

B1006 Ahmed Abdeljawad Lägg till i din kalender