The following article was originally published by the Annenberg School for Communication.
About a quarter of Americans get their news on YouTube. With its billions of users and hours upon hours of content, YouTube is one the largest online media platforms in the world.
In recent years, there has been a popular narrative in the media that videos from highly partisan, conspiracy theory-driven YouTube channels radicalize young Americans and that YouTube’s recommendation algorithm leads users down a path of increasingly radical content.
However, a new study from the Computational Social Science Lab (CSSLab) at the University of Pennsylvania finds that users’ own political interests and preferences play the primary role in what they choose to watch. In fact, if the recommendation features have any impact on users’ media diets, it is a moderating one.
“On average, relying exclusively on the recommender results in less partisan consumption,” says lead author Homa Hosseinmardi, associate research scientist at the CSSLab.
YouTube Bots
To determine the true effect of YouTube’s recommendation algorithm on what users watch, the researchers created bots that either followed the recommendation engines or completely ignored them. To do this, the researchers created bots trained on the YouTube watch history from a set of 87,988 real life users collected from October 2021 to December 2022.
Hosseinmardi and co-authors Amir Ghasemian, Miguel Rivera-Lanas, Manoel Horta Ribeiro, Robert West, and Duncan J. Watts aimed to untangle the complex relationship between user preferences and the recommendation algorithm, a relationship that evolves with each video watched.
These bots were assigned individualized YouTube accounts so that their viewing history could be tracked, and the partisanship of what they watched was estimated using the metadata associated with each video.
User Behavior vs. Algorithm Influence
During two experiments, the bots, each with its own YouTube account, went through a “learning phase” — they watched the same sequence of videos to ensure that they all presented the same preferences to YouTube’s algorithm.
Next, bots were placed into groups. Some bots continued to follow the watching history of the real life user it was trained on; others were assigned to be experimental “counterfactual bots” — bots following specific rules designed to separate user behavior from algorithmic influence.
In experiment one, after the learning phases, the control bot continued to watch videos from user’s history while counterfactual bots deviated from users’ real-life behavior and only selected videos from the list of recommended videos, without taking the user preferences into account.
“On average, relying exclusively on the recommender results in less partisan consumption.”— Homa Hosseinmardi
Some counterfactual bots always selected the first (“up next”) video from the sidebar recommendations; others randomly selected one of the top 30 videos listed in the sidebar recommendations; and others randomly selected a video from the top 15 videos in the homepage recommendations.
The researchers found that the counterfactual bots, on average, consumed less partisan content than the corresponding real user — a result that is stronger for heavier consumers of partisan content.
“This gap corresponds to an intrinsic preference of users for such content relative to what the algorithm recommends,” Hosseinmardi says. “The study exhibits similar moderating effects on bots consuming far-left content, or when bots are subscribed to channels on the extreme side of the political partisan spectrum.”
‘Forgetting Time’ of Recommendation Algorithms
In experiment two, researchers aimed to estimate the “forgetting time” of the YouTube recommender.
“Recommendation algorithms have been criticized for continuing to recommend problematic content to previously interested users long after they have lost interest in it themselves,” Hosseinmardi says.
During this experiment, researchers calculated the recommender’s forgetting time for a user with a long (120 video) history of far-right video consumption who changes their diet to moderate news for the next 60 videos.
While the control bots continued watching a far-right diet for the whole experiment, counterfactual bots simulated a user “switching” from one set of preferences (watching far-right videos) to another (watching moderate videos). As the counterfactual bots changed their media preferences, the researchers tracked the average partisanship of recommended videos in the sidebar and homepage.
“On average, the recommended videos on the sidebar shifted toward moderate content after about 30 videos,” Hosseinmardi says, “while homepage recommendations tended to adjust less rapidly, showing homepage recommendations cater more to one’s preferences and sidebar recommendations are more related to the nature of the video currently being watched.”
What’s Next
“The YouTube recommendation algorithm has been accused of leading its users toward conspiratorial beliefs. While these accusations hold some merit, we must not overlook that users have a significant agency over their actions and may have viewed the same content, or worse, even without any recommendations,” Hosseinmardi says.
Moving forward, the researchers hope that others can adopt their method for studying AI-mediated platforms where user preferences and algorithms interact, in order to better understand the role that algorithmic content recommendation engines play in our daily lives.
“Causally Estimating the Effect of YouTube’s Recommender System Using Counterfactual Bots” was authored by Homa Hosseinmardi, Amir Ghasemian, Miguel Rivera-Lanas, and Duncan J. Watts and published in Proceedings of the National Academy of Sciences. The Computational Social Science Lab is part of the School of Engineering and Applied Science, Annenberg School for Communication and the Wharton School at the University of Pennsylvania.
