Publication Date
Spring 2026
Degree Type
Thesis
Degree Name
Master of Science (MS)
Department
Computer Engineering
Advisor
Kaikai Liu; Bernardo Flores; Mahima Agumbe Suresh
Abstract
Large Language Models (LLMs) are expensive to deploy on long inputs, as transformer inference complexity and memory overhead grow with sequence length, causing slowdowns and length-extrapolation failures. Key-Value (KV) cache techniques avoid recomputation during autoregressive generation but introduce a memory bottleneck that scales linearly with sequence length. State-of-the-art approaches address this either by bounding the KV cache to enable streaming over long sequences, or by pruning redundant attention layers to reduce model memory footprint — but no prior work has combined both techniques. We conduct ablation studies on Mistral-7B-v0.1 and Llama 3-8B using streaming, attention layer pruning, and their combination across various pruning levels and window sizes, all on a single NVIDIA A100 40GB GPU. We evaluate stream-aware calibration effects on pruning decisions, downstream task performance, language modeling on long sequences, KV cache utilization, and the memory-perplexity tradeoff. Our results show that combining pruning with streaming reduces memory overhead while maintaining stable perplexities over long sequences. Without streaming, baseline Llama 3-8B’s memory grows unboundedly and exhausts GPU resources before completing generation. With streaming enabled, the model achieves stable perplexities at 512 MiB cache utilization. Further dropping 8 redundant attention layers with a 4K window achieves an additional 25% reduction in cache overhead with only a 1.21-point perplexity increase. Code available at: https://github.com/AshleyEmily/Streaming-LLM-Drop
Recommended Citation
Irawan, Ashley, "Lightweight Streaming LLMs: Attention Layer Pruning for Memory-Efficient Long-Sequence Inference" (2026). Master's Theses. 5754.
DOI: https://doi.org/10.31979/etd.t6uv-qd3q
https://scholarworks.sjsu.edu/etd_theses/5754
