吳恩達深度學習4.3練習_Convolutional Neural Networks_Car detection
轉載自吳恩達老師深度學習課程作業notebook
Autonomous driving - Car detection
Welcome to your week 3 programming assignment. You will learn about object detection using the very powerful YOLO model. Many of the ideas in this notebook are described in the two YOLO papers: Redmon et al., 2016 (https://arxiv.org/abs/1506.02640
You will learn to:
- Use object detection on a car detection dataset
- Deal with bounding boxes
Run the following cell to load the packages and dependencies that are going to be useful for your journey!
import argparse
import Using TensorFlow backend.
Important Note: As you can see, we import Keras’s backend as K. This means that to use a Keras function in this notebook, you will need to write: K.function(...).
1 - Problem Statement
You are working on a self-driving car. As a critical component of this project, you’d like to first build a car detection system. To collect data, you’ve mounted a camera to the hood (meaning the front) of the car, which takes pictures of the road ahead every few seconds while you drive around.
We would like to especially thank [drive.ai](https://www.drive.ai/) for providing this dataset! Drive.ai is a company building the brains of self-driving vehicles.
You’ve gathered all these images into a folder and have labelled them by drawing bounding boxes around every car you found. Here’s an example of what your bounding boxes look like.
If you have 80 classes that you want YOLO to recognize, you can represent the class label either as an integer from 1 to 80, or as an 80-dimensional vector (with 80 numbers) one component of which is 1 and the rest of which are 0. The video lectures had used the latter representation; in this notebook, we will use both representations, depending on which is more convenient for a particular step.
In this exercise, you will learn how YOLO works, then apply it to car detection. Because the YOLO model is very computationally expensive to train, we will load pre-trained weights for you to use.
2 - YOLO
YOLO (“you only look once”) is a popular algoritm because it achieves high accuracy while also being able to run in real-time. This algorithm “only looks once” at the image in the sense that it requires only one forward propagation pass through the network to make predictions. After non-max suppression, it then outputs recognized objects together with the bounding boxes.
2.1 - Model details
First things to know:
- The input is a batch of images of shape (m, 608, 608, 3)
- The output is a list of bounding boxes along with the recognized classes. Each bounding box is represented by 6 numbers as explained above. If you expand into an 80-dimensional vector, each bounding box is then represented by 85 numbers.
We will use 5 anchor boxes. So you can think of the YOLO architecture as the following: IMAGE (m, 608, 608, 3) -> DEEP CNN -> ENCODING (m, 19, 19, 5, 85).
Lets look in greater detail at what this encoding represents.
If the center/midpoint of an object falls into a grid cell, that grid cell is responsible for detecting that object.
Since we are using 5 anchor boxes, each of the 19 x19 cells thus encodes information about 5 boxes. Anchor boxes are defined only by their width and height.
For simplicity, we will flatten the last two last dimensions of the shape (19, 19, 5, 85) encoding. So the output of the Deep CNN is (19, 19, 425).
Now, for each box (of each cell) we will compute the following elementwise product and extract a probability that the box contains a certain class.
Here’s one way to visualize what YOLO is predicting on an image:
- For each of the 19x19 grid cells, find the maximum of the probability scores (taking a max across both the 5 anchor boxes and across different classes).
- Color that grid cell according to what object that grid cell considers the most likely.
Doing this results in this picture:
Note that this visualization isn’t a core part of the YOLO algorithm itself for making predictions; it’s just a nice way of visualizing an intermediate result of the algorithm.
Another way to visualize YOLO’s output is to plot the bounding boxes that it outputs. Doing that results in a visualization like this:
In the figure above, we plotted only boxes that the model had assigned a high probability to, but this is still too many boxes. You’d like to filter the algorithm’s output down to a much smaller number of detected objects. To do so, you’ll use non-max suppression. Specifically, you’ll carry out these steps:
- Get rid of boxes with a low score (meaning, the box is not very confident about detecting a class)
- Select only one box when several boxes overlap with each other and detect the same object.
2.2 - Filtering with a threshold on class scores
You are going to apply a first filter by thresholding. You would like to get rid of any box for which the class “score” is less than a chosen threshold.
The model gives you a total of 19x19x5x85 numbers, with each box described by 85 numbers. It’ll be convenient to rearrange the (19,19,5,85) (or (19,19,425)) dimensional tensor into the following variables:
box_confidence: tensor of shape containing (confidence probability that there’s some object) for each of the 5 boxes predicted in each of the 19x19 cells.boxes: tensor of shape containing for each of the 5 boxes per cell.box_class_probs: tensor of shape containing the detection probabilities for each of the 80 classes for each of the 5 boxes per cell.
Exercise: Implement yolo_filter_boxes().
- Compute box scores by doing the elementwise product as described in Figure 4. The following code may help you choose the right operator:
a = np.random.randn(19*19, 5, 1)
b = np.random.randn(19*19, 5, 80)
c = a * b # shape of c will be (19*19, 5, 80)
- For each box, find:
- Create a mask by using a threshold. As a reminder:
([0.9, 0.3, 0.4, 0.5, 0.1] < 0.4)returns:[False, True, False, False, True]. The mask should be True for the boxes you want to keep. - Use TensorFlow to apply the mask to box_class_scores, boxes and box_classes to filter out the boxes we don’t want. You should be left with just the subset of boxes you want to keep. (Hint)
Reminder: to call a Keras function, you should use K.function(...).
np.random.seed(0)
a = np.random.randint(2,5,24).reshape((2,2,2,3))
print ( a )
print ('- '*10)
b = np.random.randint(2,5,8).reshape((2,2,2,1))
print ( b )
print ('- '*10)
c = a*b
print ( c )
print ('- '*10)
d = (a > 3 )
# e = (a > 2 )
# print ( d,'\n '*3,e)
f = tf.boolean_mask( c, d )
g = K.argmax(a, axis=-1)
with tf.Session() as sess:
print ( sess.run(f) )
print ('* '*10)
print ( sess.run(g) )
print (f.shape)
[[[[2 3 2]
[3 3 4]]
[[2 4 2]
[2 2 4]]]
[[[3 4 4]
[2 3 3]]
[[3 3 2]
[3 2 2]]]]
- - - - - - - - - -
[[[[3]
[4]]
[[2]
[4]]]
[[[2]
[3]]
[[3]
[4]]]]
- - - - - - - - - -
[[[[ 6 9 6]
[12 12 16]]
[[ 4 8 4]
[ 8 8 16]]]
[[[ 6 8 8]
[ 6 9 9]]
[[ 9 9 6]
[12 8 8]]]]
- - - - - - - - - -
[16 8 16 8 8]
* * * * * * * * * *
[[[1 2]
[1 2]]
[[1 1]
[0 0]]]
(?,)
# GRADED FUNCTION: yolo_filter_boxes
def yolo_filter_boxes(box_confidence, boxes, box_class_probs, threshold = .6):
"""Filters YOLO boxes by thresholding on object and class confidence.
Arguments:
box_confidence -- tensor of shape (19, 19, 5, 1)
boxes -- tensor of shape (19, 19, 5, 4)
box_class_probs -- tensor of shape (19, 19, 5, 80)
threshold -- real value, if [ highest class probability score < threshold], then get rid of the corresponding box
Returns:
scores -- tensor of shape (None,), containing the class probability score for selected boxes
boxes -- tensor of shape (None, 4), containing (b_x, b_y, b_h, b_w) coordinates of selected boxes
classes -- tensor of shape (None,), containing the index of the class detected by the selected boxes
Note: "None" is here because you don't know the exact number of selected boxes, as it depends on the threshold.
For example, the actual output size of scores would be (10,) if there are 10 boxes.
輸入：
1、box_confidence,標註每一個box裡面 是否有檢測物件，取值0 or 1
2、boxes，每一個box的邊框資料，每一個cell裡面有5個boxes
3、box_class_probs，每一個box裡面每個分類對應的概率大小
4、threshold，
演算法操作：
1、計算box_scores,每個box每個分類的分數，可以直接把box_confidence=0的資料幹掉，加快後面計算，
如果classes很大，可以直接把box_confidence幾乎不會取0。
2、box_classes=K.argmax(box_scores, axis=-1)一個box裡面只放一個類，找出19*19*5個boxes裡面分值最大類的index
3、box_class_scores=K.max(box_scores, axis=-1, keepdims=False)找出19*19*5個boxes裡面最大分值
4、(box_class_scores >= threshold )返回19*19*5個boxes裡面的最大類分值大於threshold的布林矩陣，
如果某個box裡面的最大值都小於閾值threshold，就認為裡面沒有我們要找的物件。
輸出：
scores、boxes、classes根據布林矩陣保留有效資料
"""
# Step 1: Compute box scores
### START CODE HERE ### (≈ 1 line)
box_scores = box_confidence*box_class_probs
# 維度（19,19,5,80）
### END CODE HERE ###
# Step 2: Find the box_classes thanks to the max box_scores, keep track of the corresponding score
### START CODE HERE ### (≈ 2 lines)
box_classes = K.argmax(box_scores, axis=-1)
# Returns the index of the maximum value along an axis.
# 找出19*19*5個boxes裡面分值最大類的index
box_class_scores = K.max(box_scores, axis=-1, keepdims=False)
# 維度（19,19,5,1） 80個分數裡面保留最大值
# 找出19*19*5個boxes裡面最大分值
### END CODE HERE ###
# Step 3: Create a filtering mask based on "box_class_scores" by using "threshold". The mask should have the
# same dimension as box_class_scores, and be True for the boxes you want to keep (with probability >= threshold)
### START CODE HERE ### (≈ 1 line)
filtering_mask = (box_class_scores >= threshold )
# 返回19*19*5個boxes裡面的最大類分值大於threshold的布林矩陣
### END CODE HERE ###
# Step 4: Apply the mask to scores, boxes and classes
### START CODE HERE ### (≈ 3 lines)
boxes = tf.boolean_mask( boxes, filtering_mask )
# 邊框篩選：維度（19,19,5,4）根據mask篩選，輸出維度不確定（x，4）
classes = tf.boolean_mask( box_classes, filtering_mask )
# 類別篩選：維度（19,19,5）根據mask篩選，輸出維度不確定（x，1）,3處x值相同
scores = tf.boolean_mask( box_class_scores, filtering_mask )
'''
函式原型：tf.boolean_mask(tensor,mask,name='boolean_mask',axis=None)
引數：tensor是N維度的tensor（實數或者布林矩陣都可以），mask是K維度的，注意K小於等於N，name可選項也就是這個操作的名字，axis是一個0維度的int型tensor，
表示的是從引數tensor的哪個axis開始mask，預設的情況下，axis=0表示從第一維度進行mask，因此K+axis小於等於N。
返回的是N-K+1維度的tensor，也就是mask為True的地方儲存下來。
一般來說，0<K=dim(mask)<=N=dim(tensor),mask的大小必須匹配引數tensor的shape的前K維度
'''
# 分數篩選：維度（19,19,5,1）根據mask篩選，輸出維度不確定（x，1）
### END CODE HERE ###
return scores, boxes, classes
with tf.Session() as test_a:
box_confidence = tf.random_normal([19, 19, 5, 1], mean=1, stddev=4, seed = 1)
boxes = tf.random_normal([19, 19, 5, 4], mean=1, stddev=4, seed = 1)
box_class_probs = tf.random_normal([19, 19, 5, 80], mean=1, stddev=4, seed = 1)
scores, boxes, classes = yolo_filter_boxes(box_confidence, boxes, box_class_probs, threshold = 0.5)
# test_a.run([scores, boxes, classes])
print("scores[2] = " + str(scores[2].eval()))
print("boxes[2] = " + str(boxes[2].eval()))
print("classes[2] = " + str(classes[2].eval()))
print("scores.shape = " + str(scores.shape))
print("boxes.shape = " + str(boxes.shape))
print("classes.shape = " + str(classes.shape))
scores[2] = 10.750582
boxes[2] = [ 8.426533 3.2713668 -0.5313436 -4.9413733]
classes[2] = 7
scores.shape = (?,)
boxes.shape = (?, 4)
classes.shape = (?,)
Expected Output:
scores[2]
10.7506boxes[2]
[ 8.42653275 3.27136683 -0.5313437 -4.94137383]classes[2]
7scores.shape
(?,)boxes.shape
(?, 4)classes.shape
(?,)2.3 - Non-max suppression
Even after filtering by thresholding over the classes scores, you still end up a lot of overlapping boxes. A second filter for selecting the right boxes is called non-maximum suppression (NMS).
Non-max suppression uses the very important function called “Intersection over Union”, or IoU.
Exercise: Implement iou(). Some hints:
- In this exercise only, we define a box using its two corners (upper left and lower right): (x1, y1, x2, y2) rather than the midpoint and height/width.
- To calculate the area of a rectangle you need to multiply its height (y2 - y1) by its width (x2 - x1)
- You’ll also need to find the coordinates (xi1, yi1, xi2, yi2) of the intersection of two boxes. Remember that:
- xi1 = maximum of the x1 coordinates of the two boxes
- yi1 = maximum of the y1 coordinates of the two boxes
- xi2 = minimum of the x2 coordinates of the two boxes
- yi2 = minimum of the y2 coordinates of the two boxes
In this code, we use the convention that (0,0) is the top-left corner of an image, (1,0) is the upper-right corner, and (1,1) the lower-right corner.
# GRADED FUNCTION: iou
def iou(box1, box2):
"""Implement the intersection over union (IoU) between box1 and box2
Arguments:
box1 -- first box, list object with coordinates (x1, y1, x2, y2)
box2 -- second box, list object with coordinates (x1, y1, x2, y2)
"""
# Calculate the (y1, x1, y2, x2) coordinates of the intersection of box1 and box2. Calculate its Area.
### START CODE HERE ### (≈ 5 lines)
xi1 = max(box1[0],box2[0])
yi1 = max(box1[1],box2[1])
xi2 = min(box1[2],box2[2])
yi2 = min(box1[3],box2[3])
inter_area = max(0,xi2 - xi1)*max(0,yi2 - yi1) # 如果沒有交集的時候就取0，要不然兩個負數相乘也為正數
### END CODE HERE ###
# Calculate the Union area by using Formula: Union(A,B) = A + B - Inter(A,B)
### START CODE HERE ### (≈ 3 lines)
box1_area = (box1[2] - box1[0])*(box1[3] - box1[1])
box2_area = (box2[2] - box2[0])*(box2[3] - box2[1])
union_area = box1_area + box2_area - inter_area
### END CODE HERE ###
# compute the IoU
### START CODE HERE ### (≈ 1 line)
iou = inter_area/union_area
### END CODE HERE ###
return iou
box1 = (2, 1, 4, 3)
box2 = (1, 2, 3, 4)
print("iou = " + str(iou(box1, box2)))
iou = 0.14285714285714285
Expected Output:
iou =
0.14285714285714285You are now ready to implement non-max suppression. The key steps are:
- Select the box that has the highest score.
- Compute its overlap with all other boxes, and remove boxes that overlap it more than
iou_threshold. - Go back to step 1 and iterate until there’s no more boxes with a lower score than the current selected box.
This will remove all boxes that have a large overlap with the selected boxes. Only the “best” boxes remain.
Exercise: Implement yolo_non_max_suppression() using TensorFlow. TensorFlow has two built-in functions that are used to implement non-max suppression (so you don’t actually need to use your iou() implementation):
boxes_test = np.array([[1,2,3,4],[1,3,3,4],[1,3,4,4],[1,1,4,4],[1,1,3