Publication Date

Spring 2026

Degree Type

Master's Project

Degree Name

Master of Science in Computer Science (MSCS)

Department

Computer Science

First Advisor

Navrati Saxena

Second Advisor

Robert Chun

Third Advisor

Rashmi Bhat

Keywords

LoRA, PEFT, Activation Norms, Gradient Norms, LLM

Abstract

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning (PEFT) technique for large language models (LLMs). LoRA’s benefits are derived from its light weight and modular adapters. Standard LoRA applies adapters uniformly across all Transformer layers, implicitly assuming that each layer contributes equally to task adaptation. Nonetheless, it is discovered that, LLMs have internal substructures that contribute in a disproportionate manner. In this study, we present a theoretical analysis of how the activation amplitude of a layer affects LoRA weight updates. For selective Low-Rank Adaptation, we suggest Act-LoRA, a straightforward activation-guided layer selection technique. We evaluate this strategy for both encoder-only and decoder-only architectures using the GLUE benchmark. Our method achieved a 20% GPUh saving with a 1% drop in GLUE score using DeBERTaV3-Base on a single-instance GPU with 50% less LoRA parameters. It also achieved 2% GPUh savings with a less than 0.15% drop in GLUE score with the Llama-3.1-8B model in Distributed Data Parallel mode with 25% fewer LoRA parameters. Our experiments and analysis show that the compute and memory requirements of LoRA adapters increase linearly with the number of selected layers. We further compare activation-guided selection against gradient-guided importance metrics and show that activation norms yield more stable and reproducible layer rankings across seeds and datasets. Our findings show that activation-guided layer selection is a useful and efficient method to enhance the LoRA benefits. Which also makes it instantly compatible with distributed training.

Download

Available for download on Thursday, May 20, 2027

Share

COinS