[PYTHON] Kaggle Facial Keypoints Detection

Overview of the competition

In this competition, for each of the 7049 face image data, the x and y coordinates of the 15 parts that characterize the face are given as training data. (Example: x coordinate of the center of the left eye, y coordinate of the center of the right eye, etc.) It learns that data and gives the coordinates of the parts that characterize the face to the image of the test set.

I will start immediately.

Execution code

import pandas as pd
import numpy as np
import zipfile
import tensorflow as tf

Data given

Training data

Unzip and open the zip file.

Execution code

with zipfile.ZipFile('training.zip') as existing_zip:
    with existing_zip.open('training.csv') as traincsv:
        train=pd.read_csv(traincsv)
train

It is the data of 7049 × 31. The last column of the data is the image data, and the other columns are the x and y coordinates that indicate the part with the face. There also seems to be missing values. How many are there?

Execution code

train.isnull().sum()

output

left_eye_center_x              10
left_eye_center_y              10
right_eye_center_x             13
right_eye_center_y             13
left_eye_inner_corner_x      4778
left_eye_inner_corner_y      4778
left_eye_outer_corner_x      4782
left_eye_outer_corner_y      4782
right_eye_inner_corner_x     4781
right_eye_inner_corner_y     4781
right_eye_outer_corner_x     4781
right_eye_outer_corner_y     4781
left_eyebrow_inner_end_x     4779
left_eyebrow_inner_end_y     4779
left_eyebrow_outer_end_x     4824
left_eyebrow_outer_end_y     4824
right_eyebrow_inner_end_x    4779
right_eyebrow_inner_end_y    4779
right_eyebrow_outer_end_x    4813
right_eyebrow_outer_end_y    4813
nose_tip_x                      0
nose_tip_y                      0
mouth_left_corner_x          4780
mouth_left_corner_y          4780
mouth_right_corner_x         4779
mouth_right_corner_y         4779
mouth_center_top_lip_x       4774
mouth_center_top_lip_y       4774
mouth_center_bottom_lip_x      33
mouth_center_bottom_lip_y      33
Image                           0

There is a mess ...

test data

Also unzip the zip file and open it.

Execution code

with zipfile.ZipFile('test.zip') as existing_zip:
    with existing_zip.open('test.csv') as testcsv:
        test=pd.read_csv(testcsv)
test

This is 1783 image data.

File format to submit

Two other CSV files are given, but here the format of the file to be submitted is shown.

Execution code

df=pd.read_csv("IdLookupTable.csv")
sample=pd.read_csv("SampleSubmission.csv")

Execution code

df

IdLookupTable.csv shows the coordinates you want to present for each of the 1783 images. The tricky part of this competition is that the training data gives the coordinates of 30 facial parts, but you don't necessarily have to predict all the coordinates for the test data image.

Specifically, check the number of each coordinate included in the submitted file.

Execution code

df["FeatureName"].value_counts()

output

nose_tip_y                   1783
nose_tip_x                   1783
left_eye_center_x            1782
right_eye_center_y           1782
left_eye_center_y            1782
right_eye_center_x           1782
mouth_center_bottom_lip_x    1778
mouth_center_bottom_lip_y    1778
mouth_left_corner_x           590
mouth_center_top_lip_y        590
mouth_left_corner_y           590
mouth_center_top_lip_x        590
left_eye_outer_corner_x       589
left_eye_outer_corner_y       589
right_eye_inner_corner_y      589
right_eye_inner_corner_x      589
left_eye_inner_corner_y       588
right_eye_outer_corner_x      588
right_eye_outer_corner_y      588
left_eye_inner_corner_x       588
mouth_right_corner_x          587
mouth_right_corner_y          587
left_eyebrow_inner_end_x      585
right_eyebrow_inner_end_x     585
left_eyebrow_inner_end_y      585
right_eyebrow_inner_end_y     585
left_eyebrow_outer_end_x      574
left_eyebrow_outer_end_y      574
right_eyebrow_outer_end_x     572
right_eyebrow_outer_end_y     572

In this way, the required coordinates vary irregularly.

Next is a sample submission file.

Execution code

sample

The data to be submitted is simply the line number and the 27124 coordinate values.

Molding of training data

Needless to say, this is the main part of the competition. First, extract only the image data from train.csv and test.csv and make it into a numpy array. When converting the extracted image data into a numpy array, it is recommended to prepare an empty array and substitute it one by one.

Execution code

x_train=np.empty((7049,96,96,1))
x_test=np.empty((1783,96,96,1))

Execution code

for i in range(7049):
    train0=train["Image"][i].split(" ")     #Extract the i-th image array separated by spaces
    train1=[int(x) for x in train0]         #Convert each to int type and store in list
    train2=np.array(train1,dtype="float")   #Convert list to numpy array
    train3=train2.reshape(96,96,1)          #Mold the array into 96x96x1
    x_train[i]=train3                       #Assign as the i-th element of an empty numpy array

for i in range(1783):
    test0=test["Image"][i].split(" ")
    test1=[int(x) for x in test0]
    test2=np.array(test1,dtype="float")
    test3=test2.reshape(96,96,1)    
    x_test[i]=test3

Execution code

x_train =x_train / 255
x_test = x_test /255

Let y_train be the one with the Image column removed from train.csv.

Execution code

y_train=train.drop(['Image'],axis=1)

Handle the missing values you saw earlier. For the time being, apply a ffill that complements with the value in the previous line.

Execution code

y_train.fillna(method = 'ffill',inplace = True)

Check the missing value again.

Execution code

y_train.isnull().sum()

output

left_eye_center_x            0
left_eye_center_y            0
right_eye_center_x           0
right_eye_center_y           0
left_eye_inner_corner_x      0
left_eye_inner_corner_y      0
left_eye_outer_corner_x      0
left_eye_outer_corner_y      0
right_eye_inner_corner_x     0
right_eye_inner_corner_y     0
right_eye_outer_corner_x     0
right_eye_outer_corner_y     0
left_eyebrow_inner_end_x     0
left_eyebrow_inner_end_y     0
left_eyebrow_outer_end_x     0
left_eyebrow_outer_end_y     0
right_eyebrow_inner_end_x    0
right_eyebrow_inner_end_y    0
right_eyebrow_outer_end_x    0
right_eyebrow_outer_end_y    0
nose_tip_x                   0
nose_tip_y                   0
mouth_left_corner_x          0
mouth_left_corner_y          0
mouth_right_corner_x         0
mouth_right_corner_y         0
mouth_center_top_lip_x       0
mouth_center_top_lip_y       0
mouth_center_bottom_lip_x    0
mouth_center_bottom_lip_y    0
Image                        0

It was confirmed that there were no missing values.

Finally, standardize the data.

Execution code

for columns in train_y.columns:
    mean.append(train_y[columns].mean())
    std.append(train_y[columns].std())
    train_y[columns] = (train_y[columns] - train_y[columns].mean()) / train_y[columns].std()

Learning

Build the model.

Execution code

model=tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(6,(3,3), activation = 'relu', input_shape=(96,96,1)),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Conv2D(12,(3,3), activation = 'relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(512, activation = 'relu'),
    tf.keras.layers.Dense(30,activation='relu')
    ])
model.summary()

output

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_2 (Conv2D)            (None, 94, 94, 6)         60        
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 47, 47, 6)         0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 45, 45, 12)        660       
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 22, 22, 12)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 5808)              0         
_________________________________________________________________
dense_2 (Dense)              (None, 512)               2974208   
_________________________________________________________________
dense_3 (Dense)              (None, 30)                15390     
=================================================================
Total params: 2,990,318
Trainable params: 2,990,318
Non-trainable params: 0
_________________________________________________________________

Execution code

model.compile(optimizer='adam', loss='mean_squared_error', metrics=['accuracy'])

Let me learn.

Execution code

model.fit(x_train, y_train, epochs = 10)

output

Train on 7049 samples
Epoch 1/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 470.8903 - accuracy: 0.5540
Epoch 2/10
7049/7049 [==============================] - 19s 3ms/sample - loss: 283.7981 - accuracy: 0.6073
Epoch 3/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 152.0215 - accuracy: 0.6493
Epoch 4/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 151.0923 - accuracy: 0.6866
Epoch 5/10
7049/7049 [==============================] - 19s 3ms/sample - loss: 150.6215 - accuracy: 0.7188
Epoch 6/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 149.9657 - accuracy: 0.7289
Epoch 7/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 149.8715 - accuracy: 0.7371
Epoch 8/10
7049/7049 [==============================] - 19s 3ms/sample - loss: 149.7018 - accuracy: 0.7424
Epoch 9/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 149.4364 - accuracy: 0.7451
Epoch 10/10
7049/7049 [==============================] - 19s 3ms/sample - loss: 149.3164 - accuracy: 0.7510

Make predictions for the test model.

Execution code

pred = model.predict(x_test)

Creating a submission file

It's a little annoying from here, but you have to refer to the IdLookupTable.csv mentioned above and extract the coordinates required for each image. Not surprisingly, the pred consists of predictions of the x and y coordinates of 15 features for each of the 1783 images. (1783 × 30)

As an approach,

1. Get the features to extract for each image from IdLookupTable.csv
2. Next, encode the features with indexes from 0 to 29, respectively.
3. Extract the value corresponding to the combination of image ID and feature index from pred It is a flow.

Execution code

#1. IdLookupTable.Extract ImageId and feature name from csv.
lookid_list = list(df['FeatureName'])
imageID = list(df['ImageId']-1)
pre_list = list(pred)

Execution code

#2.Feature name 0~Encode to 29 indexes and store in the list.
feature = []
for f in looked_list:
    feature.append(lookid_list.index(f))

Execution code

#3.The one corresponding to the combination of the image ID and the feature name index is extracted from the prediction result.
#At that time, the value is obtained by performing the reverse conversion of standardization.
preded = []
for x,y in zip(imageID,feature):
    preded.append(pre_list[x][y] * std[y] + mean[y])

Finally, create a submission file.

Execution code

rowid = pd.Series(df['RowId'],name = 'RowId')
loc = pd.Series(preded,name = 'Location')

Execution code

submission = pd.concat([rowid,loc],axis = 1)

Summary

that's all. This will give you a score of around 3.8. If you apply Augmentation to the image or improve the model, the score will increase at all. I would like to post again as soon as I work on it. I would appreciate it if you could point out any mistakes. Thank you very much.