Human-AI Collaborative Content Analysis: Investigating the Efficacy and Challenges of LLM-Assisted Content Analysis for TikTok Videos on Palliative Care

Publication Date

10-1-2025

Document Type

Article

Publication Title

Proceedings of the Association for Information Science and Technology

Volume

62

Issue

1

DOI

10.1002/pra2.1251

First Page

229

Last Page

240

Abstract

Palliative care is frequently misunderstood, yet short videos on social media can help disseminate useful information and build supportive communities. One major challenge is that manually analyzing such content is labor-intensive and time-consuming. Meanwhile, large language models (LLMs) show promise for automated content analysis, but their domain-specific accuracy in this sensitive area remains uncertain. In this study, we propose an iterative LLM-LLM agentic conversational approach to identify palliative care themes from 56 TikTok videos. We collected video transcripts, metadata, visual labels, and on-screen text to build a multimodal dataset. Through iterative dialogues between two LLMs, we generated initial themes and refined them via human feedback to address missed dimensions. Our approach identified themes such as Policy, Advocacy, and Access, as well as Emotional Support and Coping while highlighting omissions like Humor and Saying Goodbye, underlining the need for human oversight. Our findings reveal that LLM-driven automation can reduce annotation workload, but it has limitations in capturing emotional content. The contributions of this work include a new annotated dataset of 242 TikTok videos, a validated LLM-based thematic analysis pipeline, and evidence that combining automated and human-in-the-loop methods enhances reliability and accuracy in short-form video analysis.

Keywords

Artificial Intelligence (AI), Content Analysis, Health Information Seeking, Human-AI Collaboration, Large Language Models (LLMs)

Department

Information

Link to Full Text

Share

COinS