Evaluating Lip Synchronization Techniques in LLM-Based Virtual Humans for Healthcare and Computer Science Applications

Abstract

Virtual Humans (VHs) deliver interventions in health and education through accessible, personalized messaging in human-similar design. Recent advancements in large language models (LLMs) enable sophisticated personalization through real-time generation of contextually relevant responses. However, current VH technology relies on pre-scripted conversations with carefully produced lip synchronization, creating a barrier to leveraging LLM capabilities. Real-time lip synchronization remains technically challenging, often resulting in misalignment between speech and mouth movements that diminishes perception of realistic human conversation. Anthropomorphism is that perception of a virtual human as realistic based on characteristics and behavior. This paper explores looped pre-recorded mouth movements played alongside real-time generated audio, which the current studies refer to as "pseudo lip-synchronization." Data from three studies comparing full and pseudo lip-synchronization techniques reveals contextual factors and interface design may outweigh lip synchronization quality in user perception. This strategy can maintain anthropomorphism in VHs when generating real-time audio, enabling dynamic and personalized interactions without sacrificing perceived human-likeness.