Regression Guided by Relative Ranking Using Convolutional Neural Network (R3CNN) for Facial Beauty Prediction

Abstract

Facial beauty prediction (FBP) aims to assess facial attractiveness with judgements based on human perception. Most of previous methods formulate FBP as a classification, regression or ranking problem of machine learning. However, humans not only represent facial attractiveness as a score, but also perceive the relative aesthetics of faces. Inspired by this observation, we formulate FBP as a specific regression problem guided by ranking information. Specifically, we propose a general CNN architecture, called R\n $^3$ \nCNN, to integrate the relative ranking of faces in terms of aesthetics to improve performance of facial beauty prediction. As R\n $^3$ \nCNN consists of both regression and ranking components, it is challenging to train and fine-tune it by existing techniques. To tackle this problem, we propose the following learning schemes for R\n $^3$ \nCNN: 1) hard pair sampling that generates challenging-to-predicted image pairs and pseudo ranking labels from true rating scores; 2) an assemble loss function that combines regression loss and pairwise ranking loss (PR-Loss); 3) a cascaded fine-tuning method that further improves prediction. Moreover, we build a benchmark dataset, called SCUT-FBP5500, containing 5,500 images of faces with diverse properties and labels. Experiments were performed on both the SCUT-FBP and the SCUT-FBP5500 benchmark datasets, where our method achieved state-of-the-art performance.