Publication Date

Spring 2026

Degree Type

Master's Project

Degree Name

Master of Science in Computer Science (MSCS)

Department

Computer Science

First Advisor

William Andreopoulos

Second Advisor

Navrati Saxena

Third Advisor

Pruthviraj Urankar

Keywords

Multimodal Large Language Models, Hallucination Correction, Visual Grounding, Scene Graph Generation, Relational Reasoning, Woodpecker, Training-Free, Post-Hoc Correction

Abstract

Multimodal large language models (MLLMs), such as LLaVA, may be able to create rich descriptions of an image; however they suffer from the hallucination problem, in which their output does not reflect what was visually presented. The main goal of this project is to address limitations of Woodpecker, a state-of-the-art method of correcting hallucinations in multimodal language models through post-hoc methods. Woodpecker uses objects and attributes in relation to an image to ground it. We propose RelCheck, a training-free technique that adds a relational component to the object-level visual grounding. To do so, we built a multi-layered visual knowledge base using object-level detection with GroundingDINO, learned relational triples with RelTR (a scene graph generation model), and deterministic spatial relations derived from bounding box coordinates. This evidence then helps guide a LLM in its reasoning to correct the hallucination. We test on the two main hallucination benchmarks, POPE, and the hallucination subtasks of MME, with our target model being LLaVA v1 13B. We compare our system against the raw MLLM and Woodpecker-style pipeline. Our results show promising improvement across the benchmarks, as well as a breakdown across different types of queries. These demonstrate the usefulness of relational knowledge in correcting hallucinations in MLLMs.

Download

Available for download on Saturday, May 22, 2027

Share

COinS