Generative Models for Multiple-Choice Question Answering in Portuguese: A Monolingual and Multilingual Experimental Study

Abstract

Multiple-choice questions are commonly used to assess knowledge through a set of possible answers to a given question. Determining the correct answer relies on the balance between understanding the question’s content and the associated logic. Generative models are widely applied in Multiple-Choice Question Answering (MCQA) tasks, as they can process the context and predict the correct answer based on the provided input. In this regard, the language used in the question is a critical factor, as comprehension may require understanding linguistic nuances. This work investigates the performance of transformer-based generative models in the MCQA task for Portuguese, under both zero-shot and one-shot scenarios. We compare monolingual (Sabiá-7B and Tucano-2B4) and multilingual (LLaMA-8B and LLaMA-3B) models on MCQA datasets focused on college entrance exams, aiming to evaluate the influence of prior knowledge and the model's adaptation to complex languages. Our results demonstrate that, although LLaMA-8B was not specifically trained for Portuguese, it outperforms the Sabiá-7B model on the ENEM-Challenge and BLUEX datasets. Finally, we show that multilingual models with more recent architectures outperform monolingual models.