Probing Knowledge Graph Reliability and Semantic Coherence with Language Models

Abstract

Knowledge graphs (KGs) are widely used as structured representations that support reasoning, inference, and integration across heterogeneous data sources. Yet, despite their central role in modern AI systems, the extent to which KGs preserve consistent and coherent relational structure remains insufficiently examined. This paper evaluates how well KGs maintain semantic coherence and whether they are sufficiently expressive and complete under realistic constraints on representation formats and available resources. We propose a systematic probing framework that leverages language models in two complementary ways: (1) an embedding-based analysis that measures the stability of relational semantics across alternative verbalizations, and (2) a ranking-based evaluation that tests the consistency of relational interpretations under controlled prompts. Together, these methods provide an empirical assessment of the robustness of KG semantics. Our results highlight both the strengths and the limitations of KGs as practical semantic representations and offer suggestions for future work on KG evaluation.