CultIcon-Bench

A Pilot Benchmark for Cultural Interpretation of Visual Icons

Abstract

Visual icons are widely used in user interfaces and multimodal AI systems, yet their interpretation often varies across cultural contexts. Symbols that appear universal may convey different meanings depending on social norms and cultural conventions.

We introduce CultIcon-Bench, a pilot benchmark designed to study culturally grounded interpretation of visual icons. The benchmark pairs icon-like visual symbols with short textual contexts and cultural identifiers, enabling controlled evaluation of whether a model correctly interprets the intended meaning under different cultural settings.

The dataset is organized around a taxonomy of cultural conflict classes, including gestures, politeness norms, privacy expectations, religion, holidays, rituals, dress codes, and culturally dependent humor. It is constructed using a prompt-seeded generation pipeline followed by manual filtering to retain culturally ambiguous scenarios.

Preliminary baseline experiments using mBERT and a multimodal CLIP zero-shot model demonstrate that culturally conditioned evaluation reveals performance differences across cultural groups that are not visible through aggregate metrics.