Human-AI Collaborative Content Analysis: Investigating the Efficacy and Challenges of LLM-Assisted Content Analysis for TikTok Videos on Palliative Care
Abstract
Palliative care is frequently misunderstood, yet short videos on social media can help disseminate useful information and build supportive communities. One major challenge is that manually analyzing such content is labor-intensive and time-consuming. Meanwhile, large language models (LLMs) show promise for automated content analysis, but their domain-specific accuracy in this sensitive area remains uncertain. In this study, we propose an iterative LLM-LLM agentic conversational approach to identify palliative care themes from 56 TikTok videos. We collected video transcripts, metadata, visual labels, and on-screen text to build a multimodal dataset. Through iterative dialogues between two LLMs, we generated initial themes and refined them via human feedback to address missed dimensions. Our approach identified themes such as Policy, Advocacy, and Access, as well as Emotional Support and Coping while highlighting omissions like Humor and Saying Goodbye, underlining the need for human oversight. Our findings reveal that LLM-driven automation can reduce annotation workload, but it has limitations in capturing emotional content. The contributions of this work include a new annotated dataset of 242 TikTok videos, a validated LLM-based thematic analysis pipeline, and evidence that combining automated and human-in-the-loop methods enhances reliability and accuracy in short-form video analysis.
