Public Opinion Classification on Government Policy Using Social Media: An Exploration of ChatGPT’s Capabilities and Limitations

Abstract

Gauging the public’s sentiments and opinions toward policies is a critical task for policy makers. Social media posts offer a wealth of information for such a task but also pose unique challenges for achieving reasonable accuracy and transparency. While a well-trained machine learning model can offer accurate classifications of public perspectives, it cannot offer its reasoning for that classification or any further abstraction, which makes it difficult to use such solutions in practice. ChatGPT offers a possible solution; if the LLM has the requisite accuracy, the ability to explain the reasoning behind its decisions is built in. In this poster, we demonstrate ChatGPT’s potential for such a task by documenting its accuracy and reasoning on opinion classification for posts regarding government policy in zero-shot and few-shot scenarios and compare the results to those of well-respected Natural Language Inference (NLI) models and ground truth human labels.