Off-campus SJSU users: To download campus access theses, please use the following link to log into our proxy server with your SJSU library user name and PIN.

Publication Date

Spring 2026

Degree Type

Thesis - Campus Access Only

Degree Name

Master of Science (MS)

Department

Computer Engineering

Advisor

Gheorghi Guzun; KaiKai Liu; Wencen Wu

Abstract

Large language model (LLM) inference is dominated by repeated linear projections. As the model size increases, both compute cost and model memory grow rapidly. This thesis evaluates whether bit-sliced vectors (BSVs) can serve as a compressed format representation for GPU-based LLM inference while keeping the computation in native bit-wise form instead of converting it back to dense low-precision matrix multiplication. The study first started with evaluating the BSV and it’s vector algebra on CPU work load to understand the impact of using different bit-budget on storage and accuracy. The results were encouraging on the storage side but linear operators designed could not handle transformer workloads. And we pushed further to redesign the pipeline to work on GPU and leverage it’s parallelism. So, the redesigned inference pipeline includes CPU and CUDA BSV builders, a PyTorch replacement for dense linear layers, kernel-only microbenchmarks, and end-to-end next-token evaluation on OPT-family models using the LAMBADA dataset. The principal operating point includes 7-bit query budget, a 6-bit key budget, chunk-local rescaling, and a tensor-core-oriented tile geometry. The full replacement on facebook/opt-6.7b reduced the static model size from 12700.031 MB in dense form to 5022.281 MB. On the accuracy side, it achieved top1 = 0.741, top5 = 0.896, and 66.452 ms per-sample dot latency, compared with the dense baseline at top1 = 0.798, top5 = 0.929, and 15.493 ms. At smaller scale, the dot path already shows speedup. Under full replacement, opt-125m reduced per-sample dot latency from 3.044 ms to 1.913 ms. These results show that bit-sliced execution can already improve inference speed in some settings. The main difficulty is extending those gains to larger models.

Download

Share

COinS