Self-driving cars lack social intelligence in traffic

Should I go or give way? It is one of the most basic questions in traffic, whether merging in on a motorway or at the door of the metro. The decision is one that humans typically make quickly and intuitively, because doing so relies on social interactions trained from the time we begin to walk.

Self-driving cars on the other hand, which are already on the road in several parts of the world, still struggle when navigating these social interactions in traffic. This has been demonstrated in new research conducted at the University of Copenhagen’s Department of Computer Science. Researchers analyzed an array of videos uploaded by YouTube users of self-driving cars in various traffic situations. The results show that self-driving cars have a particularly tough time understanding when to ‘yield’ - when to give way and when to drive on.

"The ability to navigate in traffic is based on much more than traffic rules. Social interactions, including body language, play a major role when we signal each other in traffic. This is where the programming of self-driving cars still falls short. That is why it is difficult for them to consistently understand when to stop and when someone is stopping for them, which can be both annoying and dangerous," says Professor Barry Brown, who has studied the evolution of self-driving car road behavior for the past five years.

Sorry, it’s a self-driving car!
Companies like Waymo and Cruise have launched taxi services with self-driving cars in parts of the United States. Tesla has rolled out their FSD model (full self-driving) to about 100,000 volunteer drivers in the US and Canada. And the media is brimming with stories about how good self-driving cars perform. But according to Professor Brown and his team, their actual road performance is a well-kept trade secret that very few have insight into. Therefore, the researchers performed in-depth analyses using 18 hours of YouTube footage filmed by enthusiasts testing cars from the back seat.

One of their video examples shows a family of four standing by the curb of a residential street in the United States. There is no pedestrian crossing, but the family would like to cross the road. As the driverless car approaches, it slows, causing the two adults in the family to wave their hands as a sign for the car to drive on. Instead, the car stops right next to them for 11 seconds. Then, as the family begins walking across the road, the car starts moving again, causing them to jump back onto the sidewalk, whereupon the person in the back seat rolls down the window and yells, "Sorry, self-driving car!".

"The situation is similar to the main problem we found in our analysis and demonstrates the inability of self-driving cars to understand social interactions in traffic. The driverless vehicle stops so as to not hit pedestrians, but ends up driving into them anyway because it doesn't understand the signals. Besides creating confusion and wasted time in traffic, it can also be downright dangerous," says Professor Brown.

A drive in foggy Frisco
In tech centric San Francisco, the performance of self-driving cars can be judged up close. Here, driverless cars have been unleashed in several parts of the city as buses and taxis, navigating the hilly streets among people and other natural phenomena. And according to the researcher, this has created plenty of resistance among the city’s residents:

"Self-driving cars are causing traffic jams and problems in San Francisco because they react inappropriately to other road users. Recently, the city’s media wrote of a chaotic traffic event caused by self-driving cars due to fog. Fog caused the self-driving cars to overreact, stop and block traffic, even though fog is extremely common in the city," says Professor Brown.

Robotic cars have been in the works for 10 years and the industry behind them has spent over DKK 40 billion to push their development. Yet the outcome has been cars that still drive with many mistakes, blocking other drivers and disrupting the smooth flow of traffic.

Why do you think it's so difficult to program self-driving cars to understand social interactions in traffic?

"I think that part of the answer is that we take the social element for granted. We don't think about it when we get into a car and drive – we just do it automatically. But when it comes to designing systems, you need to describe everything we take for granted and incorporate it into the design. The car industry could learn from having a more sociological approach. Understanding social interactions that are part of traffic should be used to design self-driving cars’ interactions with other road users, similar to how research has helped improve the usability of mobile phones and technology more broadly."

About the study:

The researchers analyzed 18 hours of video footage of self-driving cars from 70 different YouTube videos.
Using different video analysis techniques the researchers studied the video sequences in depth, rather than making a broader superficial analysis.
The study is called: "The Halting Problem: Video analysis of self-driving cars in traffic" has just been presented at the 2023 CHI Conference on Human Factors in Computing Systems, where it won the conference’s best paper award.
The study was conducted by Barry Brown of the University of Copenhagen and Stockholm University, Mathias Broth of Linköping University, and Erik Vinkhuyzen of Kings College, London.