Nature cover: Humans lose to AI again, this time in a racing game

Written by: Cooper

Editor: Kou Jianchao

Typesetting: Kou Jianchao

Many potential applications of artificial intelligence (AI) involve making more optimized real-time decisions when interacting with humans, and competitive or gambling games are the best stage for display.

Recently, a cover article published in the journal Nature reported that AI defeated world champion human players in the racing game Gran Turismo. The AI ​​program, called "Gran Turismo (GT) Sophy", is a neural network driven program that exhibits extraordinary speed, control ability and driving strategy while complying with the rules of racing.

(Source: Nature)

The core team that completed the development of this AI program is from Sony AI. The "GT Racing" series of games was developed by Japan's Polyphony Digital. It faithfully reproduces the non-linear control challenges of real racing and encapsulates complex multi-agent interactions. The game has been released on game console platforms such as Sony PlayStation and PSP. It is a popular racing game with a very realistic control experience.

If this AI program is used, human players will probably never be able to outrun the enhanced stand-alone program again, right?

Figure｜Game screenshot (Source: GT Racing)

The researchers believe that this achievement may make racing games more interesting and provide high-level competitions for training professional racing drivers and discovering new racing skills. This method is also expected to be applied to real-world systems such as robots, drones and self-driving cars.

Speed ​​and passion on the track

Driving a racing car requires tremendous skill. Modern Formula 1 cars display amazing engineering precision, yet the sport's popularity has less to do with the performance of the cars than with the skill and courage of the top drivers in pushing their cars to the limit. For more than a century, success on the track has been about speed and excitement.

Figure｜F1 Formula One Racing Competition (Source: GNEWS)

The goal of a car race is simple: if you can complete the race in less time than your competitors, you win. However, achieving this goal requires an extremely complex physics battle, and racing on the track requires careful use of the friction between the tires and the road, which has a limited amount of friction.

To win the race, drivers must choose the trajectory that keeps the car on track within the ever-changing friction limits. Brake too early into a turn and your car will slow down, wasting time. Brake too late and you won't have enough turning force to maintain your desired line as you approach the tightest part of the turn. Brake too hard and you may end up spinning your car.

Therefore, professional race car drivers are very good at finding and maintaining the limits of their car lap after lap throughout the entire race.

Although the handling limits of a race car are complex, they are well described physically, so it stands to reason that they can be calculated or learned.

In recent years, deep reinforcement learning (DRL) has become a key component of AI research milestones in areas such as Atari, StarCraft, and Dota. In order for AI to have an impact on robotics and automation, researchers must demonstrate the ability to successfully control complex physical systems. In addition, many potential applications of AI technology require interaction in close-to-human situations while respecting imprecise human specifications. Car racing is a typical domain full of these challenges.

Figure｜Comparison of game competition data (Source: Nature)

Research into autonomous racing has accelerated in recent years using full-scale, large-scale, and simulated vehicles. A common approach is to pre-compute trajectories and execute them using model predictive control. However, when driving at the absolute limit of friction, small modeling errors can be catastrophic.

Competing with other drivers places higher demands on the accuracy of AI modeling and introduces complex aerodynamic interactions, further prompting engineers to improve control schemes to continuously predict and adapt to the optimal trajectory of the track. One day, it will not be an empty talk for driverless cars to compete with human drivers on the track.

The Making of an “AI Racer”

During the development of GT Sophy, the researchers explored various ways to use machine learning to avoid modeling complexity, including using supervised learning to model vehicle dynamics and using imitation learning, evolutionary methods, or reinforcement learning to learn driving policies.

To be successful, a racing driver must be highly skilled in four areas: (1) car control, (2) racing tactics, (3) racing etiquette, and (4) racing strategy.

To control the car, drivers have a detailed knowledge of their vehicle's dynamics and the characteristics of the track. On this basis, the driver builds the tactical skills required to execute precise maneuvers by defending against opponents. At the same time, the driver must abide by highly refined but imprecise rules of sportsmanship. Finally, the driver uses strategic thinking when simulating opponents and deciding when and how to try to overtake.

The success of GT Sophy in simulation racing, a field that requires real-time, continuous control in an environment with highly realistic, complex physics, shows for the first time that it is possible to train an AI agent to perform better than top human racing drivers across a range of cars and track types.

The result can be seen as another important step in the continued advancement of computers in competitive tasks such as chess, Go, Adventure, poker, and StarCraft.

Figure｜GT Sophy's training (Source: Nature)

Remarkably, GT Sophy learned to navigate the lane in just a few hours and outperformed 95% of the human competitors in the dataset. It trained for another nine days, accumulating more than 45,000 hours of driving time and reducing its lap times by one-tenth of a second until its lap times stopped improving.

Progress rewards alone are not enough to motivate the AI ​​program to win the race. If the human opponent is fast enough, the AI ​​program will learn to follow and try to accumulate more rewards and overtake without risking a potentially catastrophic collision.

To evaluate the GT Sophy, the researchers pitted it against top GT drivers in two events. The GT Sophy achieved superhuman timing performances on all three tracks tested, being able to execute several types of turns, effectively use drifts, disrupt vehicles behind, intercept opponents and perform other emergency maneuvers.

Although GT Sophy showed enough tactical skills, there are still many areas that need to be improved, especially in terms of strategic decision-making. For example, GT Sophy sometimes leaves enough space on the same track for the opponent to take advantage.

Figure | AI drivers surpass human players (Source: Nature)

More attention outside of competitive games

When it comes to e-sports and gambling games, it is no longer a rare thing for AI to defeat humans, and it is certain that AI will become stronger and stronger, and even the top human players will have to admit defeat. However, there is not much suspense and significance in winning e-sports competitions. The key is to see how these AI programs that surpass humans can actually overcome industry bottlenecks and truly benefit human life.

On February 10, 1996, the supercomputer Deep Blue challenged the chess world champion Kasparov for the first time and lost 2:4. In May 1997, Deep Blue challenged Kasparov again and finally defeated Kasparov 3.5:2.5, becoming the first computer system to defeat the chess world champion within the standard game time limit.

However, the shortcoming of Deep Blue is that it has no intuition and does not possess a true "intelligent soul". It can only rely on its super computing power to make up for its shortcomings in analytical thinking. Deep Blue, which won the competition, soon retired.

In March 2016, Google AI's AlphaGo defeated Go world champion Lee Sedol in four games, which was considered a true milestone in AI. AlphaGo used a method that combined Monte Carlo tree search with two deep neural networks. With this design, the computer can spontaneously learn and perform analytical training like the human brain, and continuously learn to improve its chess skills.

Since then, various new AI programs have emerged one after another. On December 10, 2018, AlphaStar, an artificial intelligence developed by DeepMind for the real-time strategy game StarCraft, was able to defeat 99.8% of human professional players in the world.

Undoubtedly, the current GT Sophy is another continuation of AI's victory.

J. Christian Gerdes, a professor of mechanical engineering at Stanford University, believes that the impact of GT Sophy's research may go far beyond the scope of video games. As many companies work to perfect fully autonomous vehicles that transport goods or passengers, it is worth further exploring how much of the software should use neural networks and how much should be based solely on physics.

Overall, neural networks are the undisputed champions when it comes to sensing and identifying objects in the surrounding environment. However, trajectory planning is still the domain of physics and optimization, and GT Sophy's success on the gaming track suggests that neural networks may one day play a larger role in the software of automated vehicles than they do today.

Even more challenging might be the lap-to-lap variation. In reality, the tire conditions of a car change between laps, and a human driver must adapt to that variation throughout the race. Can GT Sophy do the same with more data? Where does that data come from? That would give the AI ​​more room to evolve.

References:

https://www.nature.com/articles/s41586-021-04357-7

https://www.nature.com/articles/d41586-022-00304-2