Classifying the Emotional Polarity of Digital Communications Using Large Language Models

Abstract

Large Language Models (LLMs) employ deep learning algorithms to generalize patterns in data. Applying these LLMs to classification tasks can reduce the required labor and time. The research aims to fine-tune the LLM Llama 3.1 to correctly identify whether a chosen text message exhibits a positive or negative emotion. The goal of this procedure is to apply the fine-tuned LLM to large databases of text messages and locate users whose recent texts contain a large proportion of negative samples. This way, I can alert the users and direct them to help very early on. I chose the Stanford Sentiment Treebank v2 (SST-2) dataset. It mimics the emotional polarity of real texts with its even positive-negative sample distribution and its contextless format. I used the Unsloth framework and LoRa to significantly reduce the resources required during the fine-tuning process. I tested the model by taking SST-2’s train split and inputting them individually into the trained model. Using this method, I found the Llama model to be highly accurate, with an accuracy of 94.8%. Interestingly, it had a high average Binary Cross-Entropy (BCE) Loss of 0.782 but achieved high accuracy. The testing against other models shows that the BCE Loss for sentiment analysis is not correlated to the actual accuracy of the model. From the results, I determined Llama 3.1 was the most suitable LLM for the sentiment analysis of large text databases.