Implication
of different AI models
in autonomous vehicles

It is also possible to visualize the evolution of the amounts invested by the different nations/continents since 2017. To do this, visit the Our World in Data website, where you can consult the interactive chart from which the above graphs are extracted.

Now that we've covered the financial aspects, let's turn to the software. To begin with, there are several levels of vehicle autonomy.

The source is the U.S. Department of Transportation. More precisely, it's the official website of the National Highway Traffic Safety Administration, a government agency.

To conclude on the subject of vehicle autonomy levels, the presence of a driver is still mandatory in vehicles equipped with an embedded computer.

By the end of 2024, the sale of autonomous vehicles with level 4 and level 5 automation will be prohibited for private customers.

It is also important to note that in vehicles with level 1 to level 3 automation, the presence of a vigilant driver is mandatory.

Having looked at the different levels of autonomous vehicle automation, let's move on to the different layers that make up the software of autonomous vehicles.

In the software section, we can distinguish 3 main layers:

• Layer 1: scene perception and localization

o Detection

o Classification

o Tracking

o Segmentation

• Layer 2: Motion planning and decision-making

o Behavioral prediction

o Behavioral decision-making

o Path planning

• Layer 3: Simulator and scenario generation

Fig: Autonomous driving system hierarchy | Chen, S. and co, A Review of Decision-Making and Planning for Autonomous Vehicles in Intersection Environments, World Electric Vehicle Journal, 2024.

Layer 1: Perception of scene and localization

Thanks to the combined use of cameras, located all around the vehicle, radar and telemetry, with LIDAR (Laser Imaging Detection and Ranging). After training the algorithms, the embedded computer is able to visualize, classify, detect and estimate the distance of objects surrounding the autonomous vehicle. These objects may include road signs, pedestrians, vehicles or obstacles.

The embedded computer breaks down this first layer into 4 tasks: detection, classification, tracking and segmentation.

        1.  Detection

Recognizing and determining the objects surrounding the vehicle. This task is performed using deep learning (DL) and, more specifically, convolutional neural networks (CNN).

Fig.: A typical architecture of convolutional neural network.

        2.  Classification

The task of accurately associating the nature of an object. Machine learning (ML) is used for this task.

Image credit: Mindy Support

        3.  Tracking

In this task, the embedded computer tracks an object. It also calculates the speed of movement of objects in the environment and the speed of the autonomous vehicle.

        4.  Segmentation

This last task is often performed in parallel with layer 1 – “Detection. Segmentation” is based on the same neural networks (CNN). The segmentation process is divided into 2 sections. One section is called the “encoder” and the other section is called the “decoder”.

Video: Semantic Segmentation for Autonomous Driving | TensorFlow Implementation of ENet
Source: youtube.com

To conclude this 1st layer, let's evoke the notion of “fusion-based perception”. Based on the human concept of trying to catch a falling object. A concept that works thanks to the simultaneous sensory perception of hearing and vision.

This human concept, known as “sensor fusion”, is applied to autonomous vehicles with the joint and redundantuse of all sensors (cameras, radar, LIDAR) and algorithmic processing of all this information.

Layer 2: Motion planning and decision-making

In this second layer, once the embedded computer is “aware” of the objects and environment around it, it plans a safe action. This layer contains information on the guidelines to be followed by the autonomous vehicle.

These include speed limits, laws and regulations in force in different countries. But also, all the information linked to road navigation.

This layer comprises 3 sub-layers: Behavioral prediction, Behavioral decision-making and Path planning.

1.    Behavioral prediction

Using automatic learning algorithms, this enables prediction of the behavior of other vehicles and pedestrians at intersections.

2.    Behavioral decision-making

Based on perception results and traffic rules, this enables the development of behavioral decision-making strategies for autonomous vehicles. Contextual factors such as traffic flow and regulations are taken into account.

3.  Path planning

This part involves calculating the best possible trajectory for the vehicle to travel safely for its passengers, as well as for people in the environment around the vehicle (cyclists, pedestrians and other vehicles).

Layer 3: Simulator & scenario generation

The Simulator & scenario generation layer creates a virtual universe of the environment surrounding the autonomous vehicle.

Using the information provided by the sensors, this creates a kind of digital twin of the real environment. With this, the embedded computer will create a multitude of scenarios with the various elements that make up the real environment.

Environmental conditions include elements such as weather conditions, the presence of pedestrians, cyclists, other vehicles and traffic conditions (fluid, congested).

For example, it will predict the behavior of a pedestrian in the vicinity of the autonomous vehicle. Will the pedestrian cross in front of the autonomous vehicle or stop?

All these simulations are carried out in real time. Information on these conditions is retrieved using the “Scene perception and localization” layer.

Fig.: Relationship representations of roads and junctions

To conclude

First of all, this information shows that the operation of an autonomous vehicle is far from being as simple as it seems. While it's easy enough for a human being to drive, it's different for machines. Driving a vehicle is proof that the human brain is complex. Even if a human being is driving in a different region (city, country) from his or her region of origin, he or she has the ability to adapt more or less quickly to a new road environment.

As Carlo van de Weijer points out in the documentary “Les dilemmes éthiques de la voiture autonome - Une ‘Moral machine’ pour les trancher? ” (Arté, 2024), the fact that there are tacit rules between different road users: motorists, pedestrians and cyclists. It is not possible to transcribe these tacit rules into the computer language of an autonomous vehicle.

And then there are the different traffic laws in the world, and even in each city of the same country. And let's not forget that different cultures have different ways of driving. Even on this last point, it would be a step forward in terms of respect for human life in certain parts of the world.

And let's not forget that we haven't even touched on the notion of the ethics of autonomous vehicles. A notion that also varies depending on the individual - potentially 8 billion people who think differently.

As shown by this study by a research team at the Massachusetts Institute of Technology - MIT. Based on the thought experiment known as the streetcar dilemma, this study imagines an initial scenario in which the main protagonist is an autonomous vehicle (with or without a human being on board). The study proposes 2 solutions and leaves it up to the participants to choose the solution they consider fairest.

Thinking experiment: Example of a choice you have to make.
Source: moralmachine.net

If you're interested, you can take part by visiting the Moral Machine website (https://www.moralmachine.net/).

In this issue, we saw an estimate of how much they are investing in the development of autonomous vehicles. In this section, we found that the USA and China are the biggest investors, far ahead of the European continent.

In the technical section, we discovered the different layers that make up the functioning of an autonomous vehicle. And the different levels of automation in autonomous vehicles.

After learning about the different levels of automation, we went on to discover how autonomous driving works.

The autonomous driving software consists of 3 layers:

- Understanding and processing the information that makes up the autonomous vehicle's environment.

With the help of different types of sensors and deep learning and machine learning algorithms, the embedded computer takes cognizance of the information that makes up the autonomous vehicle's physical environment.

- Analysis, planning and decision-making by the autonomous vehicle's embedded computers.

Using information from the environment surrounding the autonomous vehicle and predictive models, the embedded computer plans and executes actions to guarantee enhanced safety for its passengers, as well as for the vehicles (cars and cyclists) and pedestrians surrounding the autonomous vehicle.

- Simulator & scenario generation is a critical layer.

This is where the autonomous vehicle's embedded computer will simulate a multitude of scenarios based on the conditions of the environment in which it finds itself. The autonomous vehicle's embedded computer will create a digital twin of its environment.

Even if the level of automation is high, scientific studies such as that conducted at the California Institute of Technology (CalTech) (Karena X. Cai and co., Rules of the Road: Safety and Liveness Guarantees for Autonomous Vehicles, Caltech, 2021) or the development of the use of large language models (LLM), generative artificial intelligence, with this scientific study from Shanghai University (Yang & co., LLM4Drive: A Survey of Large Language Models for Autonomous Driving, Shanghai Jiao Tong University, 2024).

The use of LLMs in autonomous driving would enable better consideration of the environment surrounding the autonomous vehicle. These studies are regularly carried out to increase the safety and autonomy of autonomous vehicles.

Finally, according to van de Weijer, it is technically possible to achieve a vehicle autonomy of 90-95% in the city. It is also possible to have an autonomous vehicle with 99% autonomy on the freeway. But this high level of autonomy makes the autonomous vehicle too “cautious” and, therefore, dangerous. This excessive caution would lead to congestion on freeways and motorway interchanges. This excessive caution is due to the strict application of the Highway Code and laws when engaging in the main flow or when overtaking other vehicles.

In reality, when overtaking, motorists do not respect safety distances to the nearest centimeter or even meter. An autonomous vehicle, on the other hand, is programmed to respect safety distances to the nearest millimeter. In practice, this is rarely (if ever) the case for human beings.

List of articles and video used in this article:

• James Darley, Waymo Tops the Podium in the Autonomous Vehicle Revolution, Sustainability Magazine, December 10, 2024.

• Chen, S., Hu, X., Zhao, J., Wang, R., & Qiao, M. (2024). A Review of Decision-Making and Planning for Autonomous Vehicles in Intersection Environments. World Electric Vehicle Journal, 15(3), 99. doi.org/10.3390/wevj15030099.

• Cai, Karena Xin (2021) Safe and Interpretable Autonomous Systems Design: Behavioral Contracts and Semantic-Based Perception. Dissertation (Ph.D.), California Institute of Technology (Caltech), rev.2, March 2021. doi:10.7907/w3m8-es32.

• Zhenjie Yang, Xiaosong Jia, Hongyang Li and Junchi Yan, LLM4Drive: A Survey of Large Language Models for Autonomous Driving, School of AI and Department of CSE, Shanghai Jiao Tong University, OpenDriveLab, rev.4, August 2024. doi.org/10.48550/arXiv.2311.01043.

• Medrano-Berumen, Christopher & Akbaş, Mustafa. (2019). Abstract Simulation Scenario Generation for Autonomous Vehicle Verification.

• Gao, L., Zhou, R., & Zhang, K. (2023). Scenario Generation for Autonomous Vehicles with Deep-Learning-Based Heterogeneous Driver Models: Implementation and Verification. Sensors, 23(9), 4570. doi.org/10.3390/s23094570.

• E. Awad, S. Dsouza, A. Shariff, I. Rahwan, J.-F. Bonnefon, Universals and variations in moral decisions made in 42 countries by 70,000 participants. Proceedings of the National Academy of Sciences, 2020.

• Wen, M., Park, J. & Cho, K. A scenario generation pipeline for autonomous vehicle simulators. Hum. Cent. Comput. Inf. Sci. 10, 24 (2020). doi.org/10.1186/s13673-020-00231-z.

• Moorissa Tjokro, How Perception Stack Works in Autonomous Driving Systems, A General Framework for Perception — an Introduction to Self-Driving Cars (Part 5), Medium, 2022.

• Zhao, Jingyuan & Zhao, Wenyi & Deng, Bo & Wang, Zhenghong & Zhang, Feng & Zheng, Wenxiang & Cao, Wanke & Nan, Jinrui & Lian, Yubo & Burke, Andrew. (2023). Autonomous driving system: A comprehensive survey. Expert Systems with Applications. 242. 122836. 10.1016/j.eswa.2023.122836.

• Annual private investment in artificial intelligence, Our World in Data, 2023.

• The Road to Full Automation, National Highway Traffic Safety Administration, U.S. Department of Transportation, date: n/a.

• Les dilemmes éthiques de la voiture autonome – Une "Moral machine" pour les trancher ? Arté VOD, 2020. Réalisation : Geert Rozinga. arte.tv/fr/videos/118632-000-A/les-dilemmes-ethiques-de-la-voiture-autonome/.