Publication Date

Spring 2026

Degree Type

Master's Project

Degree Name

Master of Science in Computer Science (MSCS)

Department

Computer Science

First Advisor

Maryam Khazaei

Second Advisor

Ethel Tshukudu

Third Advisor

Matthew Morozov

Keywords

language model reasoning, accuracy-cost frontier, token budgets, preference optimization, supervised fine-tuning

Abstract

Direct Preference Optimization (DPO) is considered to improve accuracy cost frontier in language models. The accuracy cost frontier means creating an optimal trade-off between the accuracy and the cost in terms of monetary value of tokens. This project evaluates DPO in language models, especially in the smaller to mid size models such as 3B to 14B parameter range, when forcing the model to arrive at an answer in a token budget. This is done for a range of 64, 128, 256, 512 and 1024. To help support the claims we are making, we have performed multiple training and evaluation experiments on supervised fine-tuning, direct preference optimization, shuffled direct preference optimization which is where labels are shuffled. Preference dataset was generated from the SFT training data (called in-domain preference dataset). Additionally, a set of experiments was also performed on data from a dataset that looks different from MathQA called Math-Step-DPO-10K [2, 3]. This set of experiments was called cross domain. The evaluation pipeline consists of extraction and verification based pipeline on the MathQA dataset. Accuracy@Budget (model accuracy achieved at a certain budget) and the normalized area under the curve (AUCnorm) is calculated and collected. The findings in this work are that preference based training does not consistently increase the accuracy of a language model when a strict token budget is applied, especially when the base model used has SFT training done already. And across all models, SFT comes out as the top performer. DPO might be close when the data used for training is from the same source as the data used in SFT training, but it does not provide any significant improvements especially considering how costly generation ofpreference data can be. To make sure these effects are real, we also shuffled the labels on the Step DPO data and trained a group of models which confirmed the results. Additionally, multi-seed experiments were run, to make sure the result is not random. The metric AUCnorm spread of less than or equal to 0.03 was achieved which confirms this. Practically this helps users carefully assign resources and effort required for training by identifying when SFT might be sufficient on its own.

Download

Available for download on Wednesday, May 26, 2027

Share

COinS