Evaluating Personalized Content Using Large Language Models

Abstract

Educational content is typically designed for a broad audience, often failing to address the specific needs, contexts, and backgrounds of individual learners. While certain educational campaigns (e.g., public health outreach) develop multiple targeted versions of learning content, historically it has been infeasible to do at scale. However, Large Language Models (LLMs) offer the potential to adapt content based on detailed descriptions of learner characteristics, such as demographic information, situational context, resources availability, and risk factors. To explore techniques to generate such content, we developed Generative AI for Micro-Tailored Adaptation (GAIMA), a multi-agent LLM framework designed to personalize educational and training documents. GAIMA employs a feedback-driven pipeline architecture where a content modification agent generates personalized adaptations and a feedback moderator agent evaluates quality, safety, and educational value. This iterative process refines content through multiple cycles until it meets detailed standards for personalization depth and learner appropriateness. We evaluate the system using a composite framework that includes ROUGE-L and BERTScore, style transfer ratio, expertise recall, NLI-based faithfulness, and separate LLM-as-judge scores for relevance and grounding. We present a comparative analysis against a zero-shot LLM baseline, quantifying the value of iterative feedback. Our results demonstrate that GAIMA achieves 23.1% improvement over zero-shot baselines while generating more personalized, context-aware content that maintains educational integrity and safety standards.