Publication Date
Fall 2024
Degree Type
Master's Project
Degree Name
Master of Science in Computer Science (MSCS)
Department
Computer Science
First Advisor
William Andreopoulos
Second Advisor
Robert Chun
Third Advisor
Chung Wen Tsao
Keywords
AI-generated code, AI-driven software development, Generative AI
Abstract
With the advancements in the stream of AI in the recent time and the evolution of Generative AI, it is a given that there is a need to effectively integrate AI into daily tasks, including Coding. When talking about Generative AI, one important thing to consider is prompting, which is that way to talk to the AI. Depending on specific needs and tasks the way we need to prompt AI can vary. With rapid development in the field, there are a lot of new benchmarks that evaluate the AI coders on correctness, but to effectively adapt AI into actual coding tasks, we also need to know how the prompts we provide affect the metrics. This project evaluates the impact of 4 different prompting methods on state of the art AI code generator called Qwen 2.5 - 7b Coder. The impact of prompting strategies like Zero Shot prompting, Reflective prompting, Mixture of Experts (MOE) prompting and Few Shot prompting, on critical software quality metrics, including cognitive complexity, code legibility, and cyclomatic complexity is under study to understand what prompting strategies might be of use for what kind of tasks. The study used a dataset of coding problems from Codeforces platform to study the impact of various prompts can have on the production of code that is maintainable, less complex, and simple to read by conducting a systematic comparison. The results can help show how different prompt types affect the non-functional software metrics of AI-generated code and can help close the gap between benchmarks that prioritize pure correctness. These findings contribute to a deeper understanding of prompt engineering’s role in shaping AI-driven software development.
Recommended Citation
Murthy, Sirisha Krishna, "Code Quality Enhancement: Evaluating AI Code Generation with Software Metrics" (2024). Master's Projects. 1442.
https://scholarworks.sjsu.edu/etd_projects/1442