Deep learning is a method of machine learning. It allows an in-car AI to be trained to predict outputs based on a selected set of inputs. Hidden layers placed between the input layer and the output layer create numerous correlations with possible scenarios. The more layers, the more correlations are possible. Together, these layers form a neural network

How does a neural network work?

• Each image from sensors is labeled based on the similarity of closely located pixels
• Labeled pixels are assigned with features (shape, distance, location, movement)
• Based on these features, a car’s AI detects objects
• These objects (with known features) become inputs for the neural network
• Thousands of objects and millions of features are used to train the network how to act

Further actions largely depend on how a neural network perceives each object. If a car relies only on strict categorization of objects, the algorithm will work like this:

• Receive an image with detected objects
• Analyze features of objects
• Recognize a car based on its features
• Act based on the knowledge that a car is driving ahead

The same goes for other objects such as infrastructure, pedestrians, and cyclists. This approach almost eliminates the chance of adequately reacting to objects that haven’t been categorized based on features, however. Another option is direct perception. Here’s what that looks like:

• Receive an image with detected objects
• Analyze object features
• Act as you should based on these features

As you can see, direct perception excludes the necessity of recognizing a car as a car in order to react to it accordingly. This type of detection is much more practical because of its ability to adapt to new environments based on previous training and at the same time isn’t limited by a list of objects the car can react to.

Why did the Tesla hit the fire truck?

Returning to the Tesla autopilot and fire truck incident, it’s quite possible that the car was driving on cruise control mode, which relies on ADAS subsystems that are not the same that autonomous driving at its common perception. Perhaps Tesla used the first option of object categorization. That could explain why the car didn’t recognize the fire truck. The truck isn’t a typical vehicle and it was stopped at the moment of the crash, so two main factors of detecting a car were missing. It turns out that the Tesla car didn’t have enough features to categorize the fire truck and respond to it. For this reason, the Tesla just drove straight ahead.