Do YouTubers Promote Bullshiting using ChatGPT? Exploring the Use of Large-Language Models in YouTube Videos and Their Risk Landscapes

Abstract

Large language models (LLMs), such as ChatGPT, are by nature “bullshit” generators, as they are not recognizant of “truths.” As per Frankfurt’s theory, bullshit is defined as utterances made without acknowledgment of the truth. LLMs generate responses by predicting the next most probable word in a sequence without recognition of the truth. As ChatGPT peaks in the hype cycle as of early 2023, there have been many media contents that cover ChatGPT and similar LLMs. At best, these media contents can be beneficial in helping people understand the technology; but at worse, they could facilitate the generation of bullshit on the Internet, which is already filled with human-generated misinformation. This makes it necessary to understand how much the media contents have potential risk (or provide opportunities) to the media environment on the Internet. To understand the use of ChatGTP and their potential risk of bullshit generation, we analyzed YouTube videos that have been published since the introduction of ChatGTP in December 2023. Among the entire videos collected from YouTube APIs using keywords, “ChatGPT, GPT-3.5, and GPT-4,” we randomly sampled about 400 Youtube videos and analyzed their potential risk of bullshit, manually categorizing videos as low risk, high risk, reducing risk, or bullshit itself. A majority of high-risk videos were about how to make money through automatically generating media content, whereas experimental videos that tested ChatGPT’s capabilities tended to reduce the risk of bullshit by explaining the shortfalls of ChatGPT. In addition, sentiment analysis of the videos’ comments, ANOVA, and Fisher’s exact test were conducted to examine the relationships between video features, type of bullshit risk, and video performance. Results show that high-risk YouTube videos may be incentivized more in spreading bullshit, as they promote monetary success using ChatGPT. Also, based on the baseline understanding of YouTube videos, we developed a machine learning model that can predict videos’ bullshit risks by leveraging channel statistics such as the number of views, likes, comments, and subscribers. Our study provides an implication for social media designers that, as there is a high potential of AI-generated content to spread bullshit, there is a need to develop more effective content moderation strategies.