Summary:
Convolutional Neural Networks have caused a revolution in the field of computer vision in recent years by continually breaking many state-of-the-art records. CNNs are mathematical models that consist of layers of convolutional operators followed by non-linear activation functions. The non-linear activation functions improve the model's expressive ability, by allowing it to adapt to a wide range of data variations. Another method for increasing the non-linearity of models is the employment of numerous convolutional layers and the formulation of a complex structure between them.
Throughout these years, research has focused on improving these non-linear techniques so that the model can generalize with increasing flexibility on the data. However, there has been a minimal study on improving the nature of the convolution process itself. To tackle this issue, in this bachelor's thesis, we seek to replace the linear convolutional operators with non-linear ones, namely Volterra convolutions.
Volterra convolutions are polynomial approximation functions and are, in fact, the most well-known models for analyzing complex dynamic systems found in nature. As a result, they are deemed appropriate for enhancing the expressive capacity of the linear convolution operator, as well as introducing additional search spaces and dimensions for our estimation functions that are more susceptible to data variations.
In this study, we implement and evaluate the non-linear Volterra convolutions by using the CIFAR10 and CIFAR100 datasets. We demonstrate that they outperform their linear counterparts with just minor changes to our model design. Moreover, we cast light on how the information is interpreted and the higher-order relations that arise in the receptive fields. Also, we identify a resemblance between the non-linear terms of this method and the modern self-attention models that have contributed significantly to the field of computer vision recently. Finally, we show relationships between the non-linear convolutions and the deeper layers of our network, revealing a resemblance to polynomial functions.
The implementations of the non-linear convolutions are provided in this link: https://github.com/AGiannoutsos/Volterra-Convolutions
Keywords:
Volterra Convolutions, Non-Linear Convolutions, Convolutional Neural Networks, Quadratic Convolutions
