Publication Date

Fall 2022

Degree Type

Master's Project

Degree Name

Master of Science (MS)

Department

Computer Science

First Advisor

Mark Stamp

Second Advisor

William Andreopoulos

Third Advisor

Jianwei Li

Keywords

Batch Normalization, ResNet

Abstract

Batch Normalization (BatchNorm) is a technique that enables the training of deep neural networks, especially Convolutional Neural Networks (CNN) for computer vision tasks. It has been empirically demonstrated that BatchNorm increases per- formance, stability, and accuracy, although the reasons for these improvements are unclear. BatchNorm consists of a normalization step with trainable shift and scale parameters. In this paper, we examine the role of normalization and the shift and scale parameters in BatchNorm. We implement two new optimizers in PyTorch: a version of BatchNorm that we refer to as AffineLayer, which includes the shift and scale transform without normalization, and a version with just the normalization step, which we call BatchNorm-minus. We compare the performance of our AffineLayer and BatchNorm-minus implementations to standard BatchNorm, and we also compare these to the case where no batch normalization is used. We experiment with the ResNet18 and ResNet50 models over various batch sizes. Among other findings, we provide empirical evidence that the success of BatchNorm may be primarily due to improved weight initialization.

Download

Share

COinS