This talk will focus on improving the efficacy and effectiveness of deep self-supervised CNN
models for image classification. With the recent advancements of deep learning-based methods
in image classification, the requirement of a huge amount of training data is inevitable to avoid
overfitting problems. Moreover, supervised deep learning models require labeled datasets for
training. Preparing such a huge amount of labeled data takes a lot of human effort and time. In
this scenario, self-supervised models are becoming popular because of their ability to learn
even from unlabeled datasets. However, the efficient transfer of knowledge learned by
self-supervised models into a target task is an unsolved problem.
First, this talk will discuss a method for the efficient transfer of knowledge learned by a
self-supervised model, into a target task. The hyperparameters such as the number of layers,
number of units in each layer, learning rate, and dropout are automatically tuned in the Fully
Connected (FC) layers, using a Bayesian optimization technique called the Tree-structured
Parzen Estimator Approach (TPE) algorithm.
In the second problem, we extended the concept of automatically tuning (autotuning) the
hyperparameters (as proposed in the first problem), to apply on CNN layers. This work uses a
pre-trained self-supervised model to transfer knowledge for image classification. We propose an
efficient Bayesian optimization-based method for autotuning the hyperparameters of the
self-supervised model during the knowledge transfer. The proposed autotuned image classifier
consists of a few CNN layers followed by an FC layer. Finally, we use a softmax layer to obtain
the probability scores of classes for the images.
In the third problem, in addition to the challenges mentioned in the second problem, we further
focus on parameter overhead and GPU usage for hours. This work proposes a method to
address the three challenges for an image classification task, namely the limited availability of
labeled datasets for training, parameter overhead, and GPU usage for extended periods during
training. We introduce an improved transfer learning approach for a target data set, in which we
take the learned features from a self-supervised model after minimizing its parameters by
removing the final layer. The learned features are then fed into a CNN classifier, followed by a
multi-layer perceptron (MLP), where the hyperparameters of both the CNN and MLP are
automatically tuned (autotuned) using a Bayesian optimization based technique. Further, we
reduce the GFLOPs measure by limiting the search space for the hyperparameters, not
compromising with the performance.
Our ongoing work aims to propose an approach for transferring knowledge from a larger
self-supervised model to a smaller model through a knowledge distillation task. The approach
includes hyperparameter tuning of the MLP classifier in the student model, as well as the
balancing factor and temperature, which play a crucial role in the knowledge distillation process.
To enhance the effectiveness of the knowledge distillation, the ongoing work aims to introduce a
loss function which is a linear combination of the hard target loss function, soft target loss
function, and Barlow Twins loss. This approach is expected to improve the accuracy and
efficiency of the knowledge transfer process.
In the future, we plan to prune CNN layers and filters from the self-supervised model which are
not contributing to the training process.
