Datasets

Thanks to the ADNI, we could access the magnetic resonance imaging (MRI) dataset which is publicly available at the adni.loni.usc.edu. We downloaded one of their created collections: ADNI1—2 years, 1.5 T MRI scans, totally consisting of 2,042 images, which were collected from 2,042 subjects (1,205 males and 837 females) age ranging 55–92 years old. MRI images were classified into three groups: CN—567 images, MCI—1,206 images, and AD—269 images. We performed the analysis in five main steps: 1) Access the ADNI database and download MRI collection; 2) image pre-processing; 3) dataset division into training, validation, and testing sets; 4) deploying the artificial neural networks; 5) provide performance metrics. We provide a scheme of our work in Figure 1.

Display full size

Figure 1. 

The overall diagram of the steps of the analysis provided in this work. Brain images represented in the figure are created with Python library Nilearn, which is specifically designed for manipulation as well as visualisation of neuroimaging data, thus, the reason for them here is only to depict the process. MNI: Montreal Neurological Institute

Pre-processing stage: normalization, resampling and skull-stripping

We do not apply any image transformation to be submitted to the network as in some previous works [7, 11]. However, we perform some necessary pre-processing steps such as normalization, resampling, standardisation, and skull-stripping to prepare the images for training. These steps are inevitable since the purpose is to ensure that the structure of all images is of the same shape so that the classification would not be biased by irrelevant to the disease features.

Train, validation, and test sets

In order to verify the performance of the network, these tests usually are performed with unseen model data. In this particular case, the dataset was divided into three sets: training set consisted of a total of 1,476 images, the validation set of 260 images, and the test set of 306 images. The split was implemented with a ratio of 70/15/15 (%) for train/test/validation sets respectively. The networks were trained on the training set, the validation set was used for the optimization of the network parameters after each training epoch. And the test set was used to evaluate the success of the model.

Both train/test/validation split and k-fold cross-validation are commonly used techniques for evaluating the performance of ML models. The choice between the two methods largely depends on the size of the dataset and the specific goals of the analysis. Many authors claim that k-fold cross-validation or leave-one-out validation is a plausible method to evaluate the model [22, 23]. However, there are some general advantages of train/test/validation over k-fold cross-validation. Few of them are that train/test/validation split is easier to interpret than the cross-validation and that the latter has higher computational complexity [24]. Another pitfall of cross-validation is its variability and that a repeated procedure might be needed [24, 25]. Moreover, train/validation/test is more often applied when there is a sufficiently high number of data [24]. We selected train/test/validation split method mostly relying on the reasoning that we had enough large datasets to train and validate the network.

Model deployment and evaluation

In this work, a pipeline was implemented to classify MRI images deploying two DNN models, 3D-VGG-16 and ResNet3D. To implement the networks, Keras and Tensorflow libraries in Python 3.9 were used. To calculate the loss of the network, a cross-entropy function was deployed. In order to adjust the weights correctly while training the network, Adam optimizer function [26] was used for the ResNet3D model, and a root mean square (RMS) error as an optimizer function [27] for 3D-VGG-16, both networks were trained with a learning rate of 0.0001. The purpose of loss function and optimizer is to reduce the error in classification, so that the features of each group are associated and learned correctly. An early stopping [28] parameter was used based on validation accuracy and convergence speed while training both models for 20 epochs. Eventually, evaluation of specificity (precision), sensitivity (recall), F1 score and area under the receiver operating characteristic (ROC) curve (AUC), which is also an accuracy, was performed. AUC is calculated as the average of the micro F1 score for all the classes. It evaluates the ratio of how many correct predictions were made out of the total available in the class.

3D-VGG-16 architecture

The architecture of Very DCNNs was proposed by Simonyan and Zisserman [29]. It is based on the CNN model; however, the novelty of 3D-VGG-16 lies in the evaluation of increasing the depth of the layers from 16 to 19 in the network and using very small (3 × 3 × 3) convolution filters.

For the MRI images of ADNI dataset classification, we implemented a stack of 16 convolutional layers followed by 13 convolutional layers and three fully-connected layers. Each layer contained the rectified linear unit (ReLU) non-linear activation function, except the final layer was the soft-max activation layer. The model was trained for 20 epochs. A schematic image of 3D-VGG-16 network is provided in Figure 2, and in Figure S1 we provide a summary of the model with the number of parameters indicated.

Display full size

Figure 2. 

Architecture of 3D-VGG-16, a schematic visualisation of the 3D-VGG-16 neural network. (A) The input to the first Conv3D is of a fixed size, red-green-blue (RGB) format image. The image is passed through the stack of convolutional layers, where the first numbers inside the brackets indicate the number of filters, and the second number the size of the filter, in this case, 3 stands for 3 × 3 × 3 size filter passes through the image with one-pixel stride, this allows preserving the spatial resolution after the convolution. After convolution layers follow the spatial 3D pooling layer. Pooling is performed over the 2 × 2 × 2 pixel window, with stride 2. This layer preserves the shape and edges of the image; (B) flattening layer turns the 2D feature maps into a 1D feature vector, which is submitted to fully-connected layers; (C) three fully-connected layers follow the stack of convolutional layers of 4,096 channels each. All hidden layers are equipped with ReLU function, which allows for overcoming the problem of vanishing gradient descent, thus enhancing the learning process. The last layer contains the nonlinear softmax function, that gives the probability distribution of classes. The class with the highest probability will be selected as the predicted class. Conv3D: 3D convolutional layer

ResNet3D architecture

The ResNet3D is based on ResNets proposed by He et al. [30]. In general, ResNets are advantageous, because they introduce shortcut connections that bypass a signal from one layer to the next, which reduces to a minimum the issue of vanishing g gradient descent. We developed a ResNet3D network following the work of Hara et al. [31], where they used ResNet3D to identify human movements from videos. There is no temporal component in the structural MRI images, however, we used the spatiotemporal identification capability of ResNet to enhance the spatiospatial features [14].

We implemented a 34-layer ResNet3D model (Figure 3). The model consists of one initial convolutional layer, 16 residual units, and 3 fully connected layers. We train the model with 20 epochs and then extracted testing and training metrics. We also visualize the number of parameters of the model providing its’ summary in Figure S2.

Display full size

Figure 3. 

ResNet3D-34 architecture. The created network starts with the initial Conv3D, with 64 filters and a size of 7 × 7 × 7. Then 4 residual blocks follow. Each of these blocks has a certain number of residual units, indicated as a number below the block 1) 3 units with 64 filters; 2) 4 units with 128 filters; 3) 6 units with 256 filters; 4) 3 units with 512 filters, each filter of size 3 × 3 × 3, with the stride of 1 pixel, each unit contains two convolutional layers. Each convolutional layer is batch normalized, that is every time a feature map is created it is normalized and then submitted to the following block, this improves the learning efficiency. The last block is three fully connected dense layers with a dropout, that excludes inefficient connections. Each hidden layer is activated by the ReLU function and the for the classification we use the soft-max activation function

Comparison of 3D-VGG-16 and ResNet3D models

The ResNet3D model performed better, with 85% validation accuracy, while the 3D-VGG-16 reached barely 61% (Table 1). Moreover, due to the set-up parameter of early stopping, the network stopped after 6 epochs due to lack of convergence, which might occur due to failure to minimize the error and being sensitive to inhomogeneous amount of data in each category. It is important to note that using the 3D-VGG-16 network, we also experienced several out-of-memory problems, which indicates the computational power consumption of the network and inefficacy of trainability. While the ResNet3D was trained faster, more efficiently, and with higher accuracy.

Here we provide a short summary of the two models’ performance in classifying ADNI MRI brain slice images. It is seen that ResNet3D model has superiority over 3D-VGG-16 due to the higher training and validation accuracy. Furthermore, the time required for one epoch training in ResNet3D model is shorter. Due to the different optimizers used for both networks, the loss function units are different and thus, cannot be compared. However, for 3D-VGG-16 model loss is extremely high.

Performance of the ResNet3D model

Since we are interested in creating a potential system that is able to analyse MRI brain images with high accuracy, we are concentrating on the more detailed investigation of the performance of the ResNet3D model giving the classification report. Not only the validation accuracy of the ResNet3D model was high (Figure 4), but also the precision, recall, F1 score, and support metrics for each group were promising (Table 2).

Display full size

Figure 4. 

Accuracy and loss curves for training and validation sets. Left: the dotted curve describes the training accuracy, which fluctuates around 1, demonstrating the effective learning process. The solid curve stands for validation accuracy, where more fluctuations are observed, however in an early stage, after 8 epochs, it reaches 85% of classification accuracy. The values on the y-axis are given as the ratio and not in percentage. Right: similar to the left panel, here we show the loss function for training (dotted line) and validation (solid line). After 8 epochs the training and validation error decreases to a minimum. The network was trained for 20 epochs, but here the curves are represented only up to 8 epochs because the convergence of accuracy and loss functions saturated at the 8th epoch

Why necessarily ResNet3D has potential in AD detection?

The appearance of CNN provided a great prospective in supervised learning image classification. However, developing DNNs, after a certain number of layers the model diminishes its learning process, this is due to that during back-propagation process, the gradients decrease and vanish while propagating to the lower layers. This prevents weight learning and updating in the layers, which blocks the network from converging [36]. Where the 3D-VGG-16 networks have similar architecture to CNN, only that here, the number of hyperparameters is reduced, always using a 3 × 3 filter with a 1-step stride to convolve it with an image analysed. However, this is still a large network of 16 layers and also does not solve the encountered problems of vanishing gradient descent as well as smaller dimension features from learning process. Thus, one needs a better solution how to improve the learning process of the network, instead of increasing its volume, which often ends up as a clumsy, with thousands of parameters, and stagnant network without convergence in accuracy.

He et al. [30] proposed a different framework, which is based on a residual learning formulation of the shortcut connections where the output of a previous layer can be mapped to the connecting layer. The ResNet3D architecture typically consists of several residual blocks, each of which represents a sum of identity mapping and a stack of convolutions. The convolutional layers from the standard ResNet are replaced with 3D convolution blocks [37]. An advantage of residual learning is that it does not increase the complexity of the network, while optimizing its performance, unlike in visual geometry-group (VGG) networks, which is crucial when one has to analyse enough large amount of high-resolution brain images. Another benefit of 3D network models is that 3D convolutional kernels are invariant to the tissue discrimination in all dimensions, thus it can more easily learn fractal features of the cortex voxels [14]. Eventually, ResNets have the ability for embedding learning, where discrete features of the analysed image can be mapped to a lower dimensional space and meaningfully represent the same properties in a transformed way. These features further can be deployed with transfer learning by combining any ML classifier to advance the neurodegenerative process detection in the MRI image [18] and move towards an expert system for AD detection.

Limitations of the study

Despite achieving a promising result with ResNet3D model, we encountered some issues and uncertainties. One of them was the elimination of the skull from the MRI images. During the brain extraction procedure, the main problem was to find an appropriate parameter for the fractional intensity threshold. In practice, this parameter measures the aggressiveness of the algorithm in removing image components that do not represent brain tissue. A very small value would leave some irrelevant information in the image, while a very large value might remove some brain tissues. The main difficulty was that not all MRI data is obtained and preprocessed in the same way before being stored in the ADNI database, and instead, we apply the same filter to all datasets. Thus, it would not be possible to find an ideal fractional intensity threshold for the entire dataset. Nevertheless, before applying the filter for skull removal we performed several tests, and a threshold of 0.2 was used, as it was able to maintain a correct balance for most images. It is needed to mention that besides FSL-BET there are other methods, such as the high-definition brain extraction tool (HD-BET), based on artificial neural networks that provide a robust result in skull-stripping [38]. Nevertheless, HD-BET is a novel tool for brain extraction, we opted for a more traditional FMRIB Software Library (FSL) technique due to its versatility, that is, we could perform more functions with one toolkit.

It is also important to specify that training a 3D architecture is a much slower learning process and increases the need for more images. In general, the more complex the network and the more information it has to process, the more observations it has to learn from, therefore efficiency of the network might decrease.

Furthermore, the inhomogeneous amounts of data in different classes for training and for validation might strongly impact the DNN performance, it can be simply biased by the features learnt from the group with larger contents. Thus, tests could be done by varying the amount of data in different classes and tracking how the performance of the DNN changes.

Future developments

We have illustrated in this project, in a practical way, how to use neural networks (3D-VGG-16 and ResNet3D) to support the diagnosis of AD. However, the achieved results are not yet cutting-edge and further developments are needed. In the paper, we have addressed some problems that we still encounter today in the construction of a DNN, especially in the field of medical diagnosis. The use of various open libraries, such as SimpleElastix, and open databases such as ADNI have allowed us to read, process, and contribute to the development of systems for neurodegenerative brain disease diagnosis. Nevertheless, a more generalized DNN infrastructure, with less computational and time complexity requirements as well as higher interpretability [39, 40] is needed. Therefore, following the first step, to have deployed the ResNet3D network on ADNI database, we further aim to improve this pipeline with transfer learning, by deriving the relevant maps from ResNet models. Such work has been implemented by Dyrba et al. [40] in 2021, where they tackle the CNNs interpretability problem with layer-wise relevance propagation (LRP) and deep Taylor decomposition to inspect the relevant feature in the MRI scan images feature maps. However, these maps do not hold the ground truth yet. Further developments are needed for a more precise prediction of the neurodegenerative processes in the brain from MRI images to move towards an expert system in clinical application for AD.