Publication Date

Spring 2026

Degree Type

Master's Project

Degree Name

Master of Science in Computer Science (MSCS)

Department

Computer Science

First Advisor

William Andreopoulos

Second Advisor

Sayma Akther

Third Advisor

Gargi Sheguri

Keywords

API hallucination, large language models, code generation, structured grounding, selective augmentation, program repair, cloud APIs

Abstract

Large Language Models (LLMs) have shown impressive capabilities in code generation, but they still experience API hallucination. API hallucination can be defined as the incorrectly generated code which fails to follow the schema of the external APIs. This is particularly true for cloud platforms such as AWS, Azure, whose APIs are continuously evolving, and because of the same reason have limited contribution to the LLM training data. This results in making the generated code unusable even when there might be only minor errors in parameter usage. This work presents a systematic study of API hallucination. It proposes a unified framework combining structured grounding, selective augmentation, and validator-guided repair via thorough experimentation. We construct a benchmark of 7,784 tasks spanning 1,946 AWS APIs using a botocore-derived API index and evaluate correctness using a schema-aware validator based on AST analysis. Most of the failures observed during the experimentation are at the schema-level, e.g. invalid or missing parameters. The implementation of compact API contracts via structured grounding improves the pass rate from 84.8% to 98.4% but entails a far higher cost. By implementing a learned routing model to perform selective augmentation, unnecessary grounding steps are reduced and a pass rate of 97.97% is achieved. However, the model tends to perform excessive augmentation. Validator-guided repair, final step of the implementation, provides an efficient post-generation correction mechanism, achieving 96.6% pass rate with minimal overhead. Combining routing and repair yields the best overall performance achieving 98.63% pass rate with only 1.16× cost and near-baseline latency. These results demonstrate that API hallucination is best addressed through a combination of grounding, prediction, and correction, enabling reliable and cost-efficient code generation for real-world APIs.

Download

Available for download on Saturday, May 22, 2027

Share

COinS