Hello world, welcome back to my page! Here I wanna show you another project that I just done, **A Deep Autoencoder**. So autoencoder is essentially just a kind of neural network architecture, yet this one is more special thanks to its ability to generate new data based on given sample represented in lower dimension. Here I am going to be using MNIST Handwritten Digit dataset in which each of its image samples has the size of 28 by 28 pixels. This size is then going to be flattened, hence we will have 784 values to represent each of those images.

As usual, I also include all code required for this project in the end of this article.

Before we jump into the code, let me explain first about the structure of a deep autoencoder. Look at the figure below.

What you are seeing in the picture above is a structure of the deep autoencoder that we are going to construct in this project. An autoencoder has two main parts, namely **encoder **and **decoder**. The encoder part, which covers the first half of the entire network, has a purpose to map a sample into its lower dimensional representation. In this case, the encoder consists of an input layer which takes 784 features. Next, it is connected to a hidden layer of 32 neurons and then followed by 2-neurons layer. The encoder part ends at this 2-neurons layer, which is usually called as **latent space**.** **Since this latent space has exactly two dimensions, then we are able to represent all the data in a simple cartesian coordinate system in order to find out where the location of those digit numbers are encoded.

The next half of an autoencoder is called **decoder**. The architecture of a decoder is nearly the same as the encoder part. However, instead of lowering the dimensionality of data, it maps back a value in latent space to the original image shape. In this project, the decoder takes two input values, in which it should be two coordinate numbers that represent a location in a latent space. Then it is attached to a hidden layer and output (original shape) layer of size 32 and 784 respectively.

I think that’s all of the explanation about autoencoder, so now let’s start to implement this!

As usual, the first thing to do is to import all required modules, namely NumPy, Matplotlib, and Keras. The MNIST Handwritten Digit dataset that we will use is available from Keras datasets, so we can load it directly through the code.

import numpy as np

import matplotlib.pyplot as plt

from keras.datasets import mnist

from keras.models import Model

from keras.layers import Dense, Input# Loading MNIST Digit dataset

(X_train, y_train), (X_test, y_test) = mnist.load_data()

Now that we already have 4 variables, where *X_train *and *y_train* consist of 60000 data-label pairs while the test variables consist of 10000 pairs. Spoiler alert: we do not use use both *y_train *and *y_test *for training. I will explain the reason later.

1. Microsoft Azure Machine Learning x Udacity — Lesson 4 Notes

2. Fundamentals of AI, ML and Deep Learning for Product Managers

3. Roadmap to Data Science

4. Work on Artificial Intelligence Projects

After loading the dataset, the next thing to do is to preprocess those data. Fortunately, the preprocessing steps is very simple for this case because the shape of the images are already uniform (28 by 28). So now, what we need to do is to flatten out both *X_train *and *X_test*, then keep those flattened array to *X_train_flat *and *X_test_flat*.

# Convert 2D arrays into 1D (flattening)

X_train_flat =

X_train.reshape(60000, X_train.shape[1]*X_train.shape[2])X_test_flat =

X_test.reshape(10000, X_test.shape[1]*X_test.shape[2])

If we check the shape of both flattened variables you will get (60000, 784) and (10000, 784) for train and test data respectively.

The next preprocessing step is array values normalization. We know that pixel brightness in images are represented with values ranging between 0 and 255. In order for neural network to work best, we need those numbers to lie between 0 and 1, Even though in some other cases this step might not affect much. The normalization process can be done like this:

`# Normalize values`

X_train_flat = X_train_flat/255

X_test_flat = X_test_flat/255

In addition, we do not convert both labels (*y_train *and *y_test*) into **one-hot **encoding representation because, as I said earlier, those data are literally not used to train the neural network model.

After all preprocessing steps done, now we are able to construct the autoencoder. The structure of this deep autoencoder is already shown in the figure that I put in the early part of this writing. Below is the code implementation of the architecture.

`input_1 = Input(shape=(X_train_flat.shape[1],))`

hidden_1 = Dense(32, activation='relu')(input_1)

latent_space = Dense(2, activation='relu')(hidden_1)

hidden_2 = Dense(32, activation='relu')(latent_space)

output_1 = Dense(X_train_flat.shape[1], activation='sigmoid')(hidden_2)

Technically speaking, this deep autoencoder takes an array of size 784 as the input value (the flattened image array). Next, those values are delivered to the next layers, namely *hidden_1, latent_space *and *hidden_2 *respectively before eventually reach the last layer called *output_1*. Note that we call this a **deep autoencoder** due to the existence of *hidden_1* and *hidden_2* layer. If those two layers do not exist, we can simply call it as an **autoencoder**.

Next, we need to define the segments of the network (**encoder**, **decoder**, and the entire model). Here I will use a variable called *autoencoder* to store the entire neural network model and use *encoder* variable to store the first half of the network.

`autoencoder = Model(inputs=input_1, outputs=output_1)`

encoder = Model(inputs=input_1, outputs=latent_space)

Notice the way I define the model variables. The *autoencoder* takes the very first layer (*input_1*) as the input and the very last layer (*output_1*) as the output in order to take all the 5 layers of the network. The encoder part, however, stops at the *latent_space* because we want to take the value from this layer to get the lower dimension representation of an image data.

The decoder part is kinda tricky though. Below is the code creating the decoder part:

decoder_input = Input(shape=(2,))

decoder_layer_1 = autoencoder.layers[-2](decoder_input)

decoder_output = autoencoder.layers[-1](decoder_layer_1)decoder = Model(inputs=decoder_input, outputs=decoder_output)

First we need to create a placeholder called *decoder_input*. This is done because essentially we want to give a particular value as the input of the *latent_space*, while the *latent_space *itself is actually not an input layer. So we can say that *decoder_input *and *latent_space* are actually representing the same layer, but *decoder_input *takes a value from user while the *latent_space* takes a value from the previous layer of the network.

Next, I define more decoder layers which are also basically taken from the layers of the model stored in *autoencoder *variable. *decoder_layer_1* is exactly the same as the second last layer of the entire network, while *decoder_output* is the same as the output of *autoencoder*. Lastly, we need to define the *decoder* variable itself which is the second half of the entire network.

We may check the entire structure of this deep autoencoder using *autoencoder.summary()* just to check whether we have already constructed the model exactly like what is displayed in the picture I shown earlier. Below is the model summary.

`Model: "model_1"`

_________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

input_1 (InputLayer) (None, 784) 0

_________________________________________________________________

dense_1 (Dense) (None, 32) 25120

_________________________________________________________________

dense_2 (Dense) (None, 2) 66

_________________________________________________________________

dense_3 (Dense) (None, 32) 96

_________________________________________________________________

dense_4 (Dense) (None, 784) 25872

=================================================================

Total params: 51,154

Trainable params: 51,154

Non-trainable params: 0

_________________________________________________________________

You may also run *encoder.summary() *or *decoder.summary()* if you want.

A neural network can not be trained before we define the loss function and the optimizer. In this case, I decided to go with binary cross entropy loss function and *adam *optimizer. You may change this loss function to something like *mse *(Mean Squared Error), while other optimizers like *adagrad *or *adadelta *are also applicable. Below is how I compile the model:

`autoencoder.compile(loss='binary_crossentropy', optimizer='adam')`

Now, our deep autoencoder is ready to train. The training of such generative model is quite different to model for performing classification.

`autoencoder.fit(X_train_flat, X_train_flat, epochs=10, validation_data=(X_test_flat, X_test_flat))`

Notice that when fitting (a.k.a. training) the neural network model, the first and second argument are the same variable (both are *X_train_flat*). If you are familiar with classification task, usually we set the first argument as the sample (X) while the second one is used to pass the ground truth (y). The reason why in autoencoder we pass both *X* variables is because we want the output of the model to be as similar as possible with the input data. Therefore, as I have mentioned in the earlier part of this writing, actually loading *y_train *and *y_test* is not necessary for the training process.

Anyway, below is the output of the model fitting after 10 epochs. We can see here that the loss value decreases as the epoch goes. Theoretically, this loss value can still go lower as we increase the number of epochs. Note that I removed the result of epoch 2 to 9 for simplicity.

`Train on 60000 samples, validate on 10000 samples`

Epoch 1/10

60000/60000 [==============================] - 11s 186us/step - loss: 0.2133 - val_loss: 0.2098

.

.

.

.

.

Epoch 10/10

60000/60000 [==============================] - 10s 171us/step - loss: 0.1983 - val_loss: 0.1985

Up to this point, our deep autoencoder has just been trained well. Now that we are able to find out the lower dimension representation of all images and draw the distribution in a simple scatter plot using **encoder**. Then we can also use the **decoder **to perform digit image reconstruction.

After training the entire deep autoencoder model, we can perform mapping from 784-dimension flattened image to 2-dimension latent space. Now we are going to try to map all those training data into latent space using only the **encoder **part of the model, which can be achieved using the following code:

`encoded_values = encoder.predict(X_train_flat)`

Here the shape of *encoded_values* variable is (60000, 2), where it represents the number of samples and its dimension respectively. We can think of this 2-dimensional shape as a value in x-y coordinate system for each sample. Hence, we are able to put all those data into a scatter plot. Notice that the *y_train* is used to color-code the samples. Below is the code to do so:

`plt.figure(figsize=(13,10))`

plt.scatter(encoded_values[:,0], encoded_values[:,1], s=4, c=y_train, cmap='hsv')

plt.colorbar()

plt.show()

Which displays the following image:

And there it is! The figure above shows the handwritten digit distribution in two-dimensional latent space. Previously, each of the images in the dataset are represented in 784 dimensions, which is absolutely impossible to visualize the label distribution. However, now it is a lot easier to see the distribution of all images because we have encode those high data dimension to only 2 dimensions.

Furthermore, the scatter plot above tells some interesting facts. First, we can see that the picture with label 1 (orange) looks very far from most of the other images. Here we can say that images with label 1 are very different with most other numbers. Next, when we pay more attention to data points with label 4 and 9 (those in between pink and purple), we can say that these two handwritten digits are quite similar to each other due to the fact that these data points are literally spread in a same cluster.

That’s all of the **encoder**, now let’s jump into **decoder**.

Now, what if we are given a pair of x-y coordinate value representing a point in a latent space? Can we reconstruct the handwriting image from that point? Yes we can! It can simply be achieved by performing prediction using the **decoder** model that we defined earlier.

`decoded_values = decoder.predict(encoded_values)`

Remember, the shape of *encoded_values* variable is (60000, 2), meaning that it contains 60000 data points in our latent space in which each of those samples are represented using two values. Now we use this variable as the argument of *predict()* method on the decoder, in which its return value is a 60000 flattened images where each of those images are having 784 values representing the brightness of each pixel. Since the MNIST image should have the size of 28 by 28, then we still need to reshape this output value. Below is the code for it.

`decoded_values = decoded_values.reshape(60000, 28, 28)`

Up to this point, we already got the reconstructed images stored in *decoded_values* variable. Now we can compare each of the sample value stored in the variable with the actual handwritten digit image stored in *X_train* variable. Here I decided to print out 10 images of index 110 to 119 (out of 60000).

Below is the code to display the actual images taken from *X_train *and its output along with the labels:

`# Display some images`

fig, axes = plt.subplots(ncols=10, sharex=False,

sharey=True, figsize=(20, 7))

counter = 0

for i in range(110, 120):

axes[counter].set_title(y_train[i])

axes[counter].imshow(X_train[i], cmap='gray')

axes[counter].get_xaxis().set_visible(False)

axes[counter].get_yaxis().set_visible(False)

counter += 1

plt.show()