Evaluating the Effectiveness of Persona Simulation in Opinion Prediction with GPT-4.1

Abstract

Persona simulation involves utilizing large language models to anticipate human choices or interactions based on specific characteristic information. These can be used as additional data points in surveys, addressing the challenges that come with data collection, such as imbalanced sampling or non-response bias. Thus, to further understand current limitations and future directions, we tested persona simulation in opinion prediction using GPT-4.1. Personas with demographic and personality data were taken from Columbia University’s Personas dataset to be used in election forecasting. Using personas from nine U.S. states, GPT-4.1 accurately predicted 2024 election outcomes in eight out of the nine states, only failing in one of the swing states. Yet additional analysis of the American National Election Studies dataset revealed accuracies of 0.648 and 0.610 using logistic regression and GPT-4.1, respectively, indicating great room for voting prediction improvement. We then utilized the American Trends Panel Wave 123 dataset from Pew Research Center, which focused on opinions related to medicine and technologies. GPT-4.1 was able to anticipate beliefs about childhood vaccines with an accuracy of up to 0.94. Furthermore, we applied GPT-4.1 to generate conversations among personas and observed that the simulated dialogues and opinions adhered well to personas' personalities and backgrounds, albeit lacking natural human-like flow. Persona simulation proves to be a promising application of artificial intelligence as long as biases are addressed. In the near future, it will be beneficial to apply it to opinion analysis and reaction prediction in diverse fields ranging from public health to lawmaking to economics.