Lab 2: Signals in Time: Audio Processing and Classification¶

Audio is mathematically modeled as a function $x(t)$ in which $t$ represents time and $x(t)$ is an electric signal that is generated by transforming air pressure waves with a microphone. The same pressure waves can be reconstructed from the electrical signal using a speaker. In this lab we want to use what we learned in Lab 1 to process audio. In particular, we want to clean up an audio signal (Labs 2A and 2B) and we want to recognize spoken words (Lab 2C).

Lab 2C: Classification¶

0. Environment setup¶

1. Classification Losses¶

A classification is a partition of reality. Given realizations $x$ of some object we ascribe to each of them a unique label $y_c$ that marks them as members of class $c$. In a classification task we want to train an artificial intelligence that when presented with an input $x$ predicts a class label $\hat{y}$ that matches the label $y_c$ ascribed by the real world.

To model classification as empirical risk minimization (ERM) we just need to define a proper loss. The easiest is to define a hit loss that takes the value $\ell(y, \hat{y}) = 1$ when $\hat{y} \neq y$ and the value $\ell(y, \hat{y}) = 0$ when $\hat{y} = y$. To write this loss formally we define the indicator function $I ( y = y_c )$ which takes the values

$$\tag{3} I ( y = y_c ) = 1 \text{~when~} y = y_c, \quad \\ I ( y = y_c ) = 0 \text{~when~} y \neq y_c. $$

1.2 Cross Entropy Loss¶

The loss (3) is a reasonable choice but it is not differentiable with respect to the score variable $\hat{s}$. This is a drawback for implementing gradient descent. To solve this problem we define the soft maximum function

$$ \text{softmax}(\hat{s}) = \log \sum_{d=1}^{C} e^{\hat{s}_d}. $$

The soft maximum is a differentiable approximation of the maximum in the sense that $\text{softmax}(\hat{s}) \approx \max_d \hat{s}_d$. We can therefore replace the maximum $\max_d \hat{s}_d$ in \eqref{eqn_max_loss} by the soft maximum function to define the loss

$$ \ell(y, \hat{s}) = \sum_{c=1}^{C} I [ y = y_c ] \times \Big[ \log \sum_{d=1}^{C} e^{\hat{s}_d} – \hat{s}_c \Big]. $$

This is the cross entropy loss. Its use in classification tasks is standard.

2. Readout Layers¶

In Lab 1B we processed audio with a convolutional neural network (CNN). Our CNN was such that if the input signal contains $N$ entries, the output signal contains $N$ entries as well. This is unsuitable for a classification task in which we want the output to be a vector of scores $s$ containing as many entries as classes.

To sort this out we add a readout layer to match the dimensionalities of the input and output. A readout layer is just a fully connected layer at the output. Instead of a convolutional filter, the last layer is the fully connected layer,

$$ \begin{equation} \Phi(x; \mathcal{H}) ~=~ x_L ~=~ \sigma \Big(\, A_L x_{L-1} \,\Big) . \end{equation} $$ (eqn_readout)

If we have input signals $x=x_0$ of dimension $N$ and we have $C$ classes, the matrix $A_L$ is of dimension $C \times N$. This results in an output $\Phi(x; \mathcal{H}) = x_L$ containing $C$ entries that we can equate to the predicted score $s$.

In (1) we consider a CNN with layers that have a single channel. In practice, we have already seen that we have to use CNNs with multiple channels. This is not a problem but it requires that we reshape the matrix $x_{L-1}$ to turn it into a vector. We do that by stacking the columns of $x_{L-1}$ on top of each other,

$$ \text{vec}(x_{L-1}) = \Big[\, x_{L-1}^1;\, x_{L-1}^2;\, \ldots;\, x_{L-1}^{F_{l-1}} \,\Big] . $$ (2)

This operation has no conceptual significance. It is just a reshaping of data. We need to do this to write the readout layer as the matrix multiplication with $x_{L-1} = \text{vec}(x_{L-1})$.I

Task 1¶

Modify the CNN code of Lab2B to endow it with a method that implements a single convolutional layer. Add another method to implement a readout layer. Use these two methods to modify the CNN class to implement a CNN with readout.

This class receives as initialization parameters the number of layers $L$ and the dimensions $N$ and $C$ of the input features $x$ and the output scores $s$. To specify the $L-1$ convolutional layers we also accept the vectors $[K_1, \ldots, K_{L-1}]$ and $[F_0, F_1, \ldots, F_{L-1}]$ containing the number of taps $K_\ell$ of the filters used at each layer and the number of features $F_\ell$ at the output of each layer.

The forward method of this class takes a matrix $x$ of dimension $N\times F_0$ as an input and returns the output $\Phi(x; \mathcal{H})$ of a CNN with $L-1$ convolutional layers and a readout layer. The dimension of this output vector is $C$.

Use Relu nonliniearities in all layers.

Note: We will use PyTorch’s implementation of 1-D convolutions: nn.Conv1D(). This function takes the same arguments and performs the same operation as the 1-D convolution function we created in lab 1B. However, PyTorch’s implementation is faster and more memory-efficient since it uses a low-level programming language. You can read more about this implemenation in the following link: https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html.

2.1 Classification of Time Signals¶

The dimension of the output of a CNN with a readout layer can be chosen to have dimension $C$ equal to the number of classes. We can then train a CNN with readout to classify time signals by solving the empirical risk minimization problem

$$ \begin{equation} \tag{11} \mathcal{H}^* ~=~ \argmin_\mathcal{H} ~ \frac{1}{Q} \, \sum_{q=1}^{Q} \, \ell \Big( y_q, \mathbb{\Phi}(\mathbb{x_q}; \mathcal{H}) \Big) ~. \end{equation} $$

In (11) , the tensor $\mathcal{H}$ contains the convolutional filters in Layers $\ell=1$ through $\ell=L-1$ as well as the matrix $\mathbf{A}_L$ that defines the readout layer. The function $\ell ( y_q, \mathbb{\Phi}(\mathbb{x_q}; \mathcal{H}) )$ is the cross entropy loss between the available class label $y_q$ and the vector of scores $\mathbf{s}_q = \mathbb{\Phi}(\mathbb{x_q}; \mathcal{H})$.

When this CNN is deployed to make operational classifications, the output is a vector of scores $\mathbf{s} = \mathbb{\Phi}(\mathbb{x}; \mathcal{H})$. Entry $s_c$ in this vector is the likelihood we assign to class $c$. This likelihood can be converted to a hard class decision by selecting the class with the largest score

$$ \begin{equation}\tag{12} c(\mathbf{x}) ~=~ \argmax_d ~ \Big [ \mathbb{\Phi}(\mathbf{x}; \mathcal{H}) \Big]_d ~. \end{equation} $$

When using CNNs in classification problems we train them to minimize cross entropy losses. However, it is customary to evaluate their performances by counting correct classifications. In particular, if we have a test set with $\tilde{Q}$ entries we evaluate the CNN with the error rate,

$$ \begin{equation}\tag{13} e(\mathcal{H}) ~=~ \frac{1}{\tilde{Q}} \, \sum_{q=1}^{\tilde{Q}} \, \mathbb{I} \Big( y_q \neq \mathbb{\Phi}(\mathbb{x_q}; \mathcal{H}) \Big) ~. \end{equation} $$

This is the percentage of erroneous classifications in the test set.

Observe that we can also convert scores $\mathbb{s}$ to probabilities using (6). This is important when we want to gauge the confidence of a classification. In (12) we assign classes to the highest score. Thus, we predict with the same confidence whether the second highest score is close to the highest score or not. Reporting probabilities distinguishes these two situations by saying that the probability of the most likely class is, say, $90%$ or $60%$.

We must say that since we are training with a cross entropy loss this is the interpretation of the output that is most reasonable. The use of (12) is nevertheless more common.

Task 2¶

Load the data. This data contains audio samples in which people are speaking the digits 0, 1 and 2. Use the class of Task 1 to train a CNN with a readout layer to classify the audio signals into the different possible spoken digits. The CNN should have the following parameters: 2 layers, number of channels per layer: 8, kernel size (number of taps) at each layer: 80 and 3 respectively, relu non-linearities, learning rate: 0.03.

Evaluate the training cross entropy loss and the test accuracy in terms of the relative number of incorrect classifications.

Loading and exploring the data¶

Evaluate function¶

Training the CNN¶

The CNN used in Task 2 has failed at class prediction. This is not unexpected. We introduced CNNs to overcome the limitations of fully connected neural networks. In adding the pooling operation we have rendered the architecture closer to a fully connected neural network than to a CNN. We have lost the locality and equivariance that the CNN inherits from the use of convolutions.

This challenge looks insurmountable, because we have an inherent mismatch between the nature of the input — a time signal — and the nature of the output — a class. At some point we need to abandon the time domain. The solution to this challenge is to reduce the time dimension while retaining the structure of time. This is accomplished by pooling, which we explain in the next section.

3. Pooling¶

Pooling operators reduce the dimension of the input signal while retaining the structure of time. They do so by introducing a reduction factor $\Delta$, dividing the time line in windows of width $\Delta$, and extracting information from every window.

The simplest form of pooling is sampling. Let $w=[w_0; w_1;\ldots;w_{N-1}]$ be a signal with $N$ components. We define the sampled signal as the signal $x=[x_0; x_1;\ldots;x_{N_s}]$ in which the component $x_m$ is given by $$ \tag{14} x_m ~=~ w_{m\Delta} ~. $$ Thus, the sampled signal $x$ copies one out of $\Delta$ entries of $w$ and ignores the rest. The rationale for ignoring entries is that the values of entries in a window, say $w_{m\Delta}$ and $w_{m\Delta+1}$, are not very different and we loose little information by discarding some of them.

In general, computing some aggregate summary of the entries in a pooling window is more effective than plain sampling. Average pooling replaces the entries in each pooling window by their average,

$$ \tag{15} x_m ~=~ \frac{1}{\Delta} \Big( \, w_{m\Delta} + w_{m\Delta+1} + \ldots + w_{m\Delta+(\Delta-1)} \,\Big) ~. $$

Another common approach to summarizing entries in a pooling window is to compute the maximum value, $$ \tag{16} x_m ~=~ \max \Big( \, w_{m\Delta} ,\, w_{m\Delta+1} ,\, \ldots ,\, w_{m\Delta+(\Delta-1)} \,\Big) ~. $$ This is called max pooling. Max pooling is more common than average pooling and both are more common than sampling. In any case, pooling is effective when the signal $w$ does not change much within the pooling window. When this is the case, sampling, average pooling, and max pooling all produce similar summary signals $x$.

The pooling operations in (14)-(16) apply to individual vectors. In CNNs we want to pool multiple features. This is done by pooling each feature individually. Given the matrix feature $W=[w^1; w^2;\ldots;w^{F}]$ the pooled matrix feature $X=[x^1; x^2;\ldots;x^{F}]$ contains the same number of features $F$ and is such that each of the individual vector features $x^f$ is pooled separately. Thus, Component $m$ of Feature $f$ is given by $$ \tag{17} x_m^f ~=~ \text{pool} \Big( \, w^f_{k\Delta} ,\, w^f_{k\Delta+1} ,\, \ldots ,\, w^f_{k\Delta+(\Delta-1)} \,\Big) ~, $$ where the operation $\text{pool}(\cdot)$ stands in for either sampling, average pooling or max pooling.

3.1 Locality and Equivariance¶

Pooling operators retain the locality and equivariance of time. The pooled signal $\mathbf{x}$ is also a time signal except that its components are more spaced out. If we recall the definition of the sampling time in Lab 2A, the signal $\mathbf{w}$ has entries spaced by the sampling time $T_s$ and the signal $\mathbf{x}$ has entries spaced by $\Delta T_s$. This is important because it means that the pooled signal $\mathbf{x}$ can be processed with a convolutional filter. In a layered architecture, this fact implies that signals can be pooled at Layer $\ell-1$ and processed with a convolutional filter at Layer $\ell$ (Section 4).

Although somewhat obvious it is worth remarking that the summary signal contains a smaller number of entries. If $N$ is a multiple of $\Delta$, the number of entries of $\mathbf{x}$ is $N_s = N / \Delta$. Otherwise, the number of entries of $\mathbf{x}$ is given by the integer division $N_s = N \div K + 1$.

This is important in a classification task because it is almost always the case that the number of classes $C$ is much smaller than the number of entries in the input signal $\mathbf{X}$. Thus, the use of pooling at intermediate layers ends up reducing the complexity of the readout layer. This is a eureka moment. Fully connected neural networks do not work for large dimensional signals but they do work for low dimensional signals. We can then use pooling to progressively reduce the dimension of the input signal. All the while we retain the structure of time and the ability of processing with convolutional layers. Once we reach a point at which signals are of sufficiently low complexity we can train an effective readout layer. This is the CNN architecture used in classification tasks that we introduce in the following section.

4. CNNs for Classification Tasks¶

Combining pooling and readout yields the CNN architecture that is used for classification tasks. In this architecture we have $L-1$ convolutional layers and one readout layer. Each of the convolutional layers involves a convolutional filter, a pointwise nonlinearity, and a pooling operation,

$$ \tag{18} \begin{equation} \mathbf{Z}_\ell = \sum_{k=0}^{K_1} \mathcal{S}^k \mathbf{X}_{\ell-1} \mathbf{H}_{\ell k}, \qquad \mathbf{W}_\ell = \sigma \Big(\, \mathbf{Z}_\ell \, \Big), \qquad \mathbf{X}_\ell = \text{pool} \Big(\, \mathbf{W}_\ell \, \Big) . \qquad \end{equation} $$

The multiple feature layer input $\mathbf{X}_{\ell-1}$ is processed with a MIMO convolutional filter with coefficients $\mathbf{H}_{\ell k}$. The filter output $\mathbf{Z}_\ell$ is processed with the pointwise nonlinearity $\sigma(\cdot)$. The output of this operation is a matrix of features $\mathbf{W}_\ell$ whose dimension is reduced by the pooling operator $\text{pool}(\cdot)$ to yield the Layer $\ell$ output $\mathbf{X}_\ell$.

The input $\mathbf{X}_{\ell-1}$ is of dimension $N_{\ell-1} \times F_{\ell-1}$. The output $\mathbf{X}_\ell$ is of dimension $N_{\ell} \times F_{\ell}$. The change in the length of the features from $N_{\ell-1}$ to $N_{\ell}$ is determined by the pooling operator $\text{pool}(\cdot)$. The change in the number of features from $F_{\ell-1}$ to $F_{\ell}$ is determined by the filters of the MIMO filter. The coefficients $\mathbf{H}_{\ell k}$ are of dimension $F_{\ell-1} \times F_{\ell}$.

We follow the $L-1$ convolutional layers with a readout layer,

$$ \tag{19} \begin{equation} \mathbf{Z}_{L} = \mathbf{A}_{L} \text{vec} \Big(\,\mathbf{X}_{L-1} \,\Big), \qquad \mathbf{x}_{L} = \sigma \Big(\, \mathbf{x}_{L} \, \Big) . \end{equation} $$

The input to Layer $L$ is $\text{vec}(\mathbf{X}_{L-1})$. This is the output of Layer $L-1$ rearranged in vector form. This input is multiplied by the matrix $\mathbf{A}_{L}$ to produce the intermediate feature $\mathbf{Z}_{L}$. This is then passed through the pointwise nonlinearity $\sigma(\cdot)$ to produce the layer’s output $\mathbf{X}_{L}$.

The input vector $\text{vec}(\mathbf{X}_{L-1})$ is of dimension $N_{L-1} \times F_{L-1}$. The output vector is of dimension $N_L$. This implies that the matrix $\mathbf{A}_{L}$ has $N_{L-1} \times F_{L-1}$ columns and $N_L$ rows.

The output of Layer $L$ is also the output of the CNN. We write this for reference as

$$ \tag{20} \begin{equation} \mathbf{\Phi} (\mathbf{x}; \mathcal{H}) = \mathbf{x}_L, \end{equation} $$

where the tensor $\mathcal{H}$ contains the filters of the convolutional filters used in Layers 1 through $L-1$ and the coefficients $\mathbf{A}_{L}$ of the fully connected layer. The output $\mathbf{\Phi} (\mathbf{x}; \mathcal{H})$ can be interpreted as a vector of classification scores and used in the classification empirical risk minimization problem shown in (11).

A CNN with pooling and readout resembles a CNN with readout only. Both include $L-1$ convolutional layers followed by a fully connected readout layer. The difference is that in the CNN with pooling the convolutional layers process signals of progressively lower dimension. Thus, the readout matrix $\mathbf{A}_{L}$ is of lower dimension too. This is expected to avoid the failure we observed in Task 2.

Task 3¶

Modify the CNN to incorporate pooling. This can be done by modifying the method that implements convolutional layers to incorporate the pooling operation.

As in any CNN the initialization parameters include the number of layers $L$ along with vectors $[K_1, \ldots, K_{L-1}]$ and $[F_0, F_1, \ldots, F_{L-1}]$. These vectors contain the number of taps $K_\ell$ of the filters used at each layer and the number of features $F_\ell$ at the output of each layer.

Since we are incorporating pooling the initialization parameters must also include the vecor $[N_0, N_1, \ldots, N_{L}]$ containing the dimension $N_\ell$ of the features at the output of each layer. Notice that $N_0$ matches the dimension of the input signal and $N_L=C$ matches the number of classes.

The forward method of this class takes a matrix $x$ of dimension $N_0\times F_0$ as an input and returns the output $\Phi(x; \mathcal{H})$ of a CNN with $L-1$ convolutional layers and a readout layer. The dimension of this output is $N_L \times 1$.

Use relu nonlinearities in all layers. Use max pooling in all convolutional layers where pooling is implemented.

Naive pooling implementations¶

Here are two naive implementations of pooling. If your pooling implementation looks like this, it is fine. Their computations are correct, but they are slow because they use for loops. But if you want to make your training faster, you can use the vectorized implementation of max pooling in the next section.

Implementing Max Pooling and CNN with Pooling¶

Now we will implement a vectorized MaxPool1D layer, and a CNN with pooling layers.

Task 4¶

Load the same data used in Task 2. Use the class of Task 3 to train a CNN with a readout layer to classify the audio signals into the different possible spoken digits. The CNN should have the following parameters: 2 layers, number of channels per layer: 8, kernel size (number of taps) at each layer: 80 and 3 respectively, learning rate: 0.05, pooling layers: max pooling.

Evaluate the training cross entropy loss and the test accuracy in terms of the relative number of incorrect classifications.

Training CNN with Pooling¶