`pygad.cnn`

Module¶

This section of the PyGAD’s library documentation discusses the
**pygad.cnn** module.

Using the **pygad.cnn** module, convolutional neural networks (CNNs) are
created. The purpose of this module is to only implement the **forward
pass** of a convolutional neural network without using a training
algorithm. The **pygad.cnn** module builds the network layers,
implements the activations functions, trains the network, makes
predictions, and more.

Later, the **pygad.gacnn** module is used to train the **pygad.cnn**
network using the genetic algorithm built in the **pygad** module.

# Supported Layers¶

Each layer supported by the **pygad.cnn** module has a corresponding
class. The layers and their classes are:

**Input**: Implemented using the`pygad.cnn.Input2D`

class.**Convolution**: Implemented using the`pygad.cnn.Conv2D`

class.**Max Pooling**: Implemented using the`pygad.cnn.MaxPooling2D`

class.**Average Pooling**: Implemented using the`pygad.cnn.AveragePooling2D`

class.**Flatten**: Implemented using the`pygad.cnn.Flatten`

class.**ReLU**: Implemented using the`pygad.cnn.ReLU`

class.**Sigmoid**: Implemented using the`pygad.cnn.Sigmoid`

class.**Dense**(Fully Connected): Implemented using the`pygad.cnn.Dense`

class.

In the future, more layers will be added.

Except for the input layer, all of listed layers has 4 instance attributes that do the same function which are:

`previous_layer`

: A reference to the previous layer in the CNN architecture.`layer_input_size`

: The size of the input to the layer.`layer_output_size`

: The size of the output from the layer.`layer_output`

: The latest output generated from the layer. It default to`None`

.

In addition to such attributes, the layers may have some additional attributes. The next subsections discuss such layers.

`pygad.cnn.Input2D`

Class¶

The `pygad.cnn.Input2D`

class creates the input layer for the
convolutional neural network. For each network, there is only a single
input layer. The network architecture must start with an input layer.

This class has no methods or class attributes. All it has is a
constructor that accepts a parameter named `input_shape`

representing
the shape of the input.

The instances from the `Input2D`

class has the following attributes:

`input_shape`

: The shape of the input to the pygad.cnn.`layer_output_size`

Here is an example of building an input layer with shape
`(50, 50, 3)`

.

```
input_layer = pygad.cnn.Input2D(input_shape=(50, 50, 3))
```

Here is how to access the attributes within the instance of the
`pygad.cnn.Input2D`

class.

```
input_shape = input_layer.input_shape
layer_output_size = input_layer.layer_output_size
print("Input2D Input shape =", input_shape)
print("Input2D Output shape =", layer_output_size)
```

This is everything about the input layer.

`pygad.cnn.Conv2D`

Class¶

Using the `pygad.cnn.Conv2D`

class, convolution (conv) layers can be
created. To create a convolution layer, just create a new instance of
the class. The constructor accepts the following parameters:

`num_filters`

: Number of filters.`kernel_size`

: Filter kernel size.`previous_layer`

: A reference to the previous layer. Using the`previous_layer`

attribute, a linked list is created that connects all network layers. For more information about this attribute, please check the**previous_layer**attribute section of the`pygad.nn`

module documentation.`activation_function=None`

: A string representing the activation function to be used in this layer. Defaults to`None`

which means no activation function is applied while applying the convolution layer. An activation layer can be added separately in this case. The supported activation functions in the conv layer are`relu`

and`sigmoid`

.

Within the constructor, the accepted parameters are used as instance attributes. Besides the parameters, some new instance attributes are created which are:

`filter_bank_size`

: Size of the filter bank in this layer.`initial_weights`

: The initial weights for the conv layer.`trained_weights`

: The trained weights of the conv layer. This attribute is initialized by the value in the`initial_weights`

attribute.`layer_input_size`

`layer_output_size`

`layer_output`

Here is an example for creating a conv layer with 2 filters and a kernel
size of 3. Note that the `previous_layer`

parameter is assigned to the
input layer `input_layer`

.

```
conv_layer = pygad.cnn.Conv2D(num_filters=2,
kernel_size=3,
previous_layer=input_layer,
activation_function=None)
```

Here is how to access some attributes in the dense layer:

```
filter_bank_size = conv_layer.filter_bank_size
conv_initail_weights = conv_layer.initial_weights
print("Filter bank size attributes =", filter_bank_size)
print("Initial weights of the conv layer :", conv_initail_weights)
```

Because `conv_layer`

holds a reference to the input layer, then the
number of input neurons can be accessed.

```
input_layer = conv_layer.previous_layer
input_shape = input_layer.num_neurons
print("Input shape =", input_shape)
```

Here is another conv layer where its `previous_layer`

attribute points
to the previously created conv layer and it uses the `ReLU`

activation
function.

```
conv_layer2 = pygad.cnn.Conv2D(num_filters=2,
kernel_size=3,
previous_layer=conv_layer,
activation_function="relu")
```

Because `conv_layer2`

holds a reference to `conv_layer`

in its
`previous_layer`

attribute, then the attributes in `conv_layer`

can
be accessed.

```
conv_layer = conv_layer2.previous_layer
filter_bank_size = conv_layer.filter_bank_size
print("Filter bank size attributes =", filter_bank_size)
```

After getting the reference to `conv_layer`

, we can use it to access
the number of input neurons.

```
conv_layer = conv_layer2.previous_layer
input_layer = conv_layer.previous_layer
input_shape = input_layer.num_neurons
print("Input shape =", input_shape)
```

`pygad.cnn.MaxPooling2D`

Class¶

The `pygad.cnn.MaxPooling2D`

class builds a max pooling layer for the
CNN architecture. The constructor of this class accepts the following
parameter:

`pool_size`

: Size of the window.`previous_layer`

: A reference to the previous layer in the CNN architecture.`stride=2`

: A stride that default to 2.

Within the constructor, the accepted parameters are used as instance attributes. Besides the parameters, some new instance attributes are created which are:

`layer_input_size`

`layer_output_size`

`layer_output`

`pygad.cnn.AveragePooling2D`

Class¶

The `pygad.cnn.AveragePooling2D`

class is similar to the
`pygad.cnn.MaxPooling2D`

class except that it applies the max pooling
operation rather than average pooling.

`pygad.cnn.Flatten`

Class¶

The `pygad.cnn.Flatten`

class implements the flatten layer which
converts the output of the previous layer into a 1D vector. The
constructor accepts only the `previous_layer`

parameter.

The following instance attributes exist:

`previous_layer`

`layer_input_size`

`layer_output_size`

`layer_output`

`pygad.cnn.ReLU`

Class¶

The `pygad.cnn.ReLU`

class implements the ReLU layer which applies the
ReLU activation function to the output of the previous layer.

The constructor accepts only the `previous_layer`

parameter.

The following instance attributes exist:

`previous_layer`

`layer_input_size`

`layer_output_size`

`layer_output`

`pygad.cnn.Sigmoid`

Class¶

The `pygad.cnn.Sigmoid`

class is similar to the `pygad.cnn.ReLU`

class except that it applies the sigmoid function rather than the ReLU
function.

`pygad.cnn.Dense`

Class¶

The `pygad.cnn.Dense`

class implement the dense layer. Its constructor
accepts the following parameters:

`num_neurons`

: Number of neurons in the dense layer.`previous_layer`

: A reference to the previous layer.`activation_function`

: A string representing the activation function to be used in this layer. Defaults to`"sigmoid"`

. Currently, the supported activation functions in the dense layer are`"sigmoid"`

,`"relu"`

, and`softmax`

.

Within the constructor, the accepted parameters are used as instance attributes. Besides the parameters, some new instance attributes are created which are:

`initial_weights`

: The initial weights for the dense layer.`trained_weights`

: The trained weights of the dense layer. This attribute is initialized by the value in the`initial_weights`

attribute.`layer_input_size`

`layer_output_size`

`layer_output`

`pygad.cnn.Model`

Class¶

An instance of the `pygad.cnn.Model`

class represents a CNN model. The
constructor of this class accepts the following parameters:

`last_layer`

: A reference to the last layer in the CNN architecture (i.e. dense layer).`epochs=10`

: Number of epochs.`learning_rate=0.01`

: Learning rate.

Within the constructor, the accepted parameters are used as instance
attributes. Besides the parameters, a new instance attribute named
`network_layers`

is created which holds a list with references to the
CNN layers. Such a list is returned using the `get_layers()`

method in
the `pygad.cnn.Model`

class.

There are a number of methods in the `pygad.cnn.Model`

class which
serves in training, testing, and retrieving information about the model.
These methods are discussed in the next subsections.

`get_layers()`

¶

Creates a list of all layers in the CNN model. It accepts no parameters.

`train()`

¶

Trains the CNN model.

Accepts the following parameters:

`train_inputs`

: Training data inputs.`train_outputs`

: Training data outputs.

This method trains the CNN model according to the number of epochs
specified in the constructor of the `pygad.cnn.Model`

class.

It is important to note that no learning algorithm is used for training the pygad.cnn. Just the learning rate is used for making some changes which is better than leaving the weights unchanged.

`feed_sample()`

¶

Feeds a sample in the CNN layers and returns results of the last layer in the pygad.cnn.

`update_weights()`

¶

Updates the CNN weights using the learning rate. It is important to note that no learning algorithm is used for training the pygad.cnn. Just the learning rate is used for making some changes which is better than leaving the weights unchanged.

`predict()`

¶

Uses the trained CNN for making predictions.

Accepts the following parameter:

`data_inputs`

: The inputs to predict their label.

It returns a list holding the samples predictions.

`summary()`

¶

Prints a summary of the CNN architecture.

# Supported Activation Functions¶

The supported activation functions in the convolution layer are:

Sigmoid: Implemented using the

`pygad.cnn.sigmoid()`

function.Rectified Linear Unit (ReLU): Implemented using the

`pygad.cnn.relu()`

function.

The dense layer supports these functions besides the `softmax`

function implemented in the `pygad.cnn.softmax()`

function.

# Steps to Build a Neural Network¶

This section discusses how to use the `pygad.cnn`

module for building
a neural network. The summary of the steps are as follows:

Reading the Data

Building the CNN Architecture

Building Model

Model Summary

Training the CNN

Making Predictions

Calculating Some Statistics

## Reading the Data¶

Before building the network architecture, the first thing to do is to prepare the data that will be used for training the network.

In this example, 4 classes of the **Fruits360** dataset are used for
preparing the training data. The 4 classes are:

Apple Braeburn: This class’s data is available at https://github.com/ahmedfgad/NumPyANN/tree/master/apple

Lemon Meyer: This class’s data is available at https://github.com/ahmedfgad/NumPyANN/tree/master/lemon

Mango: This class’s data is available at https://github.com/ahmedfgad/NumPyANN/tree/master/mango

Raspberry: This class’s data is available at https://github.com/ahmedfgad/NumPyANN/tree/master/raspberry

Just 20 samples from each of the 4 classes are saved into a NumPy array available in the dataset_inputs.npy file: https://github.com/ahmedfgad/NumPyCNN/blob/master/dataset_inputs.npy

The shape of this array is `(80, 100, 100, 3)`

where the shape of the
single image is `(100, 100, 3)`

.

The dataset_outputs.npy file (https://github.com/ahmedfgad/NumPyCNN/blob/master/dataset_outputs.npy) has the class labels for the 4 classes:

Apple Braeburn: Class label is

**0**Lemon Meyer: Class label is

**1**Mango: Class label is

**2**Raspberry: Class label is

**3**

Simply, download and reach the 2 files to return the NumPy arrays according to the next 2 lines:

```
train_inputs = numpy.load("dataset_inputs.npy")
train_outputs = numpy.load("dataset_outputs.npy")
```

After the data is prepared, next is to create the network architecture.

## Building the Network Architecture¶

The input layer is created by instantiating the `pygad.cnn.Input2D`

class according to the next code. A network can only have a single input
layer.

```
import pygad.cnn
sample_shape = train_inputs.shape[1:]
input_layer = pygad.cnn.Input2D(input_shape=sample_shape)
```

After the input layer is created, next is to create a number of layers layers according to the next code. Normally, the last dense layer is regarded as the output layer. Note that the output layer has a number of neurons equal to the number of classes in the dataset which is 4.

```
conv_layer1 = pygad.cnn.Conv2D(num_filters=2,
kernel_size=3,
previous_layer=input_layer,
activation_function=None)
relu_layer1 = pygad.cnn.Sigmoid(previous_layer=conv_layer1)
average_pooling_layer = pygad.cnn.AveragePooling2D(pool_size=2,
previous_layer=relu_layer1,
stride=2)
conv_layer2 = pygad.cnn.Conv2D(num_filters=3,
kernel_size=3,
previous_layer=average_pooling_layer,
activation_function=None)
relu_layer2 = pygad.cnn.ReLU(previous_layer=conv_layer2)
max_pooling_layer = pygad.cnn.MaxPooling2D(pool_size=2,
previous_layer=relu_layer2,
stride=2)
conv_layer3 = pygad.cnn.Conv2D(num_filters=1,
kernel_size=3,
previous_layer=max_pooling_layer,
activation_function=None)
relu_layer3 = pygad.cnn.ReLU(previous_layer=conv_layer3)
pooling_layer = pygad.cnn.AveragePooling2D(pool_size=2,
previous_layer=relu_layer3,
stride=2)
flatten_layer = pygad.cnn.Flatten(previous_layer=pooling_layer)
dense_layer1 = pygad.cnn.Dense(num_neurons=100,
previous_layer=flatten_layer,
activation_function="relu")
dense_layer2 = pygad.cnn.Dense(num_neurons=4,
previous_layer=dense_layer1,
activation_function="softmax")
```

After the network architecture is prepared, the next step is to create a CNN model.

## Building Model¶

The CNN model is created as an instance of the `pygad.cnn.Model`

class. Here is an example.

```
model = pygad.cnn.Model(last_layer=dense_layer2,
epochs=5,
learning_rate=0.01)
```

After the model is created, a summary of the model architecture can be printed.

## Model Summary¶

The `summary()`

method in the `pygad.cnn.Model`

class prints a
summary of the CNN model.

```
model.summary()
```

```
----------Network Architecture----------
<class 'pygad.cnn.Conv2D'>
<class 'pygad.cnn.Sigmoid'>
<class 'pygad.cnn.AveragePooling2D'>
<class 'pygad.cnn.Conv2D'>
<class 'pygad.cnn.ReLU'>
<class 'pygad.cnn.MaxPooling2D'>
<class 'pygad.cnn.Conv2D'>
<class 'pygad.cnn.ReLU'>
<class 'pygad.cnn.AveragePooling2D'>
<class 'pygad.cnn.Flatten'>
<class 'pygad.cnn.Dense'>
<class 'pygad.cnn.Dense'>
----------------------------------------
```

## Training the Network¶

After the model and the data are prepared, then the model can be trained
using the the `pygad.cnn.train()`

method.

```
model.train(train_inputs=train_inputs,
train_outputs=train_outputs)
```

After training the network, the next step is to make predictions.

## Making Predictions¶

The `pygad.cnn.predict()`

method uses the trained network for making
predictions. Here is an example.

```
predictions = model.predict(data_inputs=train_inputs)
```

It is not expected to have high accuracy in the predictions because no training algorithm is used.

## Calculating Some Statistics¶

Based on the predictions the network made, some statistics can be calculated such as the number of correct and wrong predictions in addition to the classification accuracy.

```
num_wrong = numpy.where(predictions != train_outputs)[0]
num_correct = train_outputs.size - num_wrong.size
accuracy = 100 * (num_correct/train_outputs.size)
print(f"Number of correct classifications : {num_correct}.")
print(f"Number of wrong classifications : {num_wrong.size}.")
print(f"Classification accuracy : {accuracy}.")
```

It is very important to note that it is not expected that the
classification accuracy is high because no training algorithm is used.
Please check the documentation of the `pygad.gacnn`

module for
training the CNN using the genetic algorithm.

# Examples¶

This section gives the complete code of some examples that build neural
networks using `pygad.cnn`

. Each subsection builds a different
network.

## Image Classification¶

This example is discussed in the **Steps to Build a Convolutional Neural
Network** section and its complete code is listed below.

Remember to either download or create the dataset_features.npy and dataset_outputs.npy files before running this code.

```
import numpy
import pygad.cnn
"""
Convolutional neural network implementation using NumPy
A tutorial that helps to get started (Building Convolutional Neural Network using NumPy from Scratch) available in these links:
https://www.linkedin.com/pulse/building-convolutional-neural-network-using-numpy-from-ahmed-gad
https://towardsdatascience.com/building-convolutional-neural-network-using-numpy-from-scratch-b30aac50e50a
https://www.kdnuggets.com/2018/04/building-convolutional-neural-network-numpy-scratch.html
It is also translated into Chinese: http://m.aliyun.com/yunqi/articles/585741
"""
train_inputs = numpy.load("dataset_inputs.npy")
train_outputs = numpy.load("dataset_outputs.npy")
sample_shape = train_inputs.shape[1:]
num_classes = 4
input_layer = pygad.cnn.Input2D(input_shape=sample_shape)
conv_layer1 = pygad.cnn.Conv2D(num_filters=2,
kernel_size=3,
previous_layer=input_layer,
activation_function=None)
relu_layer1 = pygad.cnn.Sigmoid(previous_layer=conv_layer1)
average_pooling_layer = pygad.cnn.AveragePooling2D(pool_size=2,
previous_layer=relu_layer1,
stride=2)
conv_layer2 = pygad.cnn.Conv2D(num_filters=3,
kernel_size=3,
previous_layer=average_pooling_layer,
activation_function=None)
relu_layer2 = pygad.cnn.ReLU(previous_layer=conv_layer2)
max_pooling_layer = pygad.cnn.MaxPooling2D(pool_size=2,
previous_layer=relu_layer2,
stride=2)
conv_layer3 = pygad.cnn.Conv2D(num_filters=1,
kernel_size=3,
previous_layer=max_pooling_layer,
activation_function=None)
relu_layer3 = pygad.cnn.ReLU(previous_layer=conv_layer3)
pooling_layer = pygad.cnn.AveragePooling2D(pool_size=2,
previous_layer=relu_layer3,
stride=2)
flatten_layer = pygad.cnn.Flatten(previous_layer=pooling_layer)
dense_layer1 = pygad.cnn.Dense(num_neurons=100,
previous_layer=flatten_layer,
activation_function="relu")
dense_layer2 = pygad.cnn.Dense(num_neurons=num_classes,
previous_layer=dense_layer1,
activation_function="softmax")
model = pygad.cnn.Model(last_layer=dense_layer2,
epochs=1,
learning_rate=0.01)
model.summary()
model.train(train_inputs=train_inputs,
train_outputs=train_outputs)
predictions = model.predict(data_inputs=train_inputs)
print(predictions)
num_wrong = numpy.where(predictions != train_outputs)[0]
num_correct = train_outputs.size - num_wrong.size
accuracy = 100 * (num_correct/train_outputs.size)
print(f"Number of correct classifications : {num_correct}.")
print(f"Number of wrong classifications : {num_wrong.size}.")
print(f"Classification accuracy : {accuracy}.")
```