Before the popularity of CNN, many feature extraction methods were proposed. One of the promising concepts is Independent Component Analysis (ICA). ICA resembles the receptive fields of simple cells in the visual cortex. ICA on natural images produces phase and frequency-sensitive decorrelated filters, which resemble oriented Gabor functions (Figure 15). ICA is different from PCA and cannot be calculated analytically. ICA requires the minimum mutual information among the output vectors and is achieved by choosing a nonlinear activation function as the cumulative density function of the underlying independent components. ICA filters were observed to have more sparsely distributed outputs on natural images.
FIGURE 15 filters trained through ZCA-whitened natural images, which are visually the same as the ICA filters
In 2007, Thomas used a biologically plausible method, a feed-forward path of object detection. It used the Gabor Function to tune the filter size. It is claimed that the Gabor function is more biologically inclined and has been proven to be an effective model for simple cortical cell receptive fields. The model is divided into simple and complex units. The filter sizes were calculated from 7x7 to 37x37@16 for simple units and 8x8 for complex units.
Sobel and Prewitt filters are popular gradient-based methods for Edge detection in the frequency domain. Mathilde et al. proposed ’deep-cluster,’ a novel K-means-based clustering approach for large-scale, end-to-end training of convolution layers. The paper was focused on optimization based on K-means, but the K-means clustering was not a part of the main convolutional architecture. AlexNet was arbitrarily selected as the backbone architecture and can be replaced with any similar architecture. The first layer used the Sobel filter as a feature extractor. The Vertical and Horizontal weight matrices for Sobel filters are:
\begin{equation} W_{\text{ver}}=\ \begin{bmatrix}-1&0&1\\ -2&0&2\\ -1&0&1\\ \end{bmatrix}\text{\ and\ }W_{\text{Hor}}=\ \begin{bmatrix}1&2&1\\ 0&0&0\\ -1&-2&-1\\ \end{bmatrix}\ \nonumber \\ \end{equation}
Most filters cannot get trained over raw images with colors, which is the primary motivation for applying the Sobel filter. The Sobel filter is pre-defined, and the values are fixed for edge detection in the vertical and horizontal directions. As no learning happens for the Sobel filter, it remains independent of further training. The Prewitt filters share a similar property to Sobel and could generate a similar result.