Publication Date
Fall 2024
Degree Type
Master's Project
Degree Name
Master of Science in Computer Science (MSCS)
Department
Computer Science
First Advisor
William Andreopoulos
Second Advisor
Katerina Potika
Third Advisor
Navrati Saxena
Keywords
Retrieval-Augmented Generation (RAG), AI Companions, Multimodal RAG, Large Language Models (LLMs), Generative AI, Vector Databases, Graph RAG, Prompt Engineering, Embedding Techniques, Knowledge Graphs, Contextual Retrieval, CLIP
Abstract
The rapid advancement in generative AI and large language models have forever revolutionized how we synthesize data. This project explores and experiments with the potential of a multimodal Retrieval-Augmented Generation (RAG) framework for processing text, tabular and image data. Starting with prompt engineering techniques, we address their limitations in dynamic and domain-specific real world applications by building a multimodal RAG pipeline and evaluating it against human-generated ground truth. The project culminates in BrightMind.ai, a full-stack educational platform featuring novel personalized AI companions for context-aware and adaptive response generation. Its innovative capabilities extend to music, video, and code generation, setting it apart from existing AI assistants. BrightMind.ai sets a benchmark for interactive, multimodal AI applications and opens pathways for enhanced user engagement with generative AI technologies.
Recommended Citation
Rathore, Charul, "Multimodal Retrieval-Augmented Generation: Design and Application" (2024). Master's Projects. 1434.
https://scholarworks.sjsu.edu/etd_projects/1434
