{"id":3259,"date":"2017-09-18T13:45:32","date_gmt":"2017-09-18T12:45:32","guid":{"rendered":"http:\/\/blogit.itu.dk\/ethos\/?p=3259"},"modified":"2017-09-18T13:49:30","modified_gmt":"2017-09-18T12:49:30","slug":"blog-post-by-dana","status":"publish","type":"post","link":"https:\/\/blogit.itu.dk\/ethoslab\/2017\/09\/18\/blog-post-by-dana\/","title":{"rendered":"Blog post by Dana Marie Yu"},"content":{"rendered":"
\n
\n
\n
\n

Your face is average. According to that guy.<\/strong><\/h2>\n

This is a blog post written by Dana Marie Yu. Dana was a Junior Researcher in ETHOS Lab in the spring semester 2017. She is on the MSc in Software Development programme at the ITU and during the fall of 2017 she is studying abroad in Japan.<\/em><\/p>\n

Your adolescent self can kindly dismiss the service of your mirror because the time spent examining your face – analyzing the size of your forehead or the shape of your nose – is over. Now there is face attribute recognition software to do it for you.<\/p>\n

In recent years, employment of deep learning techniques has significantly improved classification accuracy in widespread image recognition applications. Deep learning models are now also being applied to improve face attribute recognition. <\/p>\n

How does it work?<\/strong><\/p>\n

For supervised machine learning (supervised meaning the classification outputs are known), a labelled training data set is key. This data set has sample data points representing the classifications the model should recognize, and each data point is given a corresponding class label. This data is fed into a machine learning model that predicts a class for a given data point. It compares the predicted class label to the actual class label for the data point and recalculates internally to improve its accuracy score. The completed \u201ctrained\u201d model is then used to predict classifications for new unlabelled data points.<\/p>\n

The key to a classification model is in the training data set it used to learn from. In face attribute recognition, learning from sample input and class labels makes sense when predicting attributes like \u201cwears glasses\u201d and \u201cteeth visible\u201d since these can be universally defined. But what about the other attribute labels that include \u201cchin size\u201d, \u201cnose width\u201d, and \u201cface shape\u201d?<\/p>\n

One face image data set publicly available for training classification models, CelebA data set [1], is labelled with 40 attributes that include \u201cattractive\u201d, \u201cyoung\u201d, \u201cbig nose\u201d, and \u201cchubby\u201d [2].<\/p>\n

Building a quality and sizable training data set comes with established financial and temporal challenges. But where does basic data annotation in training data sets for face attribute recognition come from for many attributes that appear fairly subjective? <\/p>\n

One research team that recently developed a face attribute recognition model acknowledged the lack of labelled face attributes in existing face image data sets for training [3].<\/p>\n

Therefore, to build their own training data set, the team generated face attribute annotations by using an off-the-shelf face attribute predictor model [2] trained on the CelebA data set to label face attributes on each image in the CASIA-WebFace data set [4] taking the majority voted label for each attribute to set as the \u201cground-truth\u201d attribute label for each image [3]. <\/p>\n

To evaluate an accuracy score for their proposed network model, the team needed new face images and attribute annotations independent from the training data set. To generate these attribute annotations, they asked three annotators unrelated to their project to label the selected facial attributes on a randomly sampled subset of images from the FaceScrub data set [3, 5]. The majority voted labels among the three annotators became the \u201cground-truth\u201d face attribute labels for each image used to evaluate their model\u2019s attribute classification accuracy.<\/p>\n

One publicly available face image data set that includes labelled attributes, PubFig [6], was created by outsourcing the attribute annotation task to individuals via Amazon Mechanical Turk [7, 8], an online platform that matches online tasks that require human intelligence to online workers. A few manually labelled images annotated by the data set creators were submitted as examples, and the online workers were asked to select face images that exhibited a specified attribute. <\/p>\n

Each submitted labelling job was performed by three different workers, and only labels agreed upon by all three workers were used in the final data set. Via this process, \u201cground-truth\u201d face attribute annotations were collected for 65 attributes including \u201cAsian\u201d, \u201cIndian\u201d,  \u201cattractive woman\u201d, \u201cchubby\u201d, \u201ceyebrow shape\u201d, and \u201cmiddle-aged\u201d for the images in the PubFig data set [8]. <\/p>\n

Why is this concerning?<\/strong><\/p>\n

So far, the training data sets encountered have been annotated by taking the majority opinion of seemingly random individuals. In the examples above, up to three individuals gave their opinions for each attribute.<\/p>\n

In June 2017, a survey was conducted asking individuals to label the \u201ceyes distance\u201d face attribute in eight face images. The individuals labeled \u201ceyes distance\u201d using five categories that are currently being used by a commercial face recognition web service, Betaface API [9]: \u201cextra close\u201d, \u201cclose\u201d, \u201caverage\u201d, \u201cfar\u201d, and \u201cextra far\u201d. For each image, 200 human annotations were collected and compared to the face attribute label from Betaface API.<\/p>\n

In Images 1, 2, 3, 4, the majority of 200 labels matched the attribute label from Betaface API. However, Image 4 displays a bare majority of 1%.<\/p>\n

 <\/span><\/div>\n
\"\"<\/span><\/span><\/div>\n
Image 1<\/em><\/div>\n
 <\/span><\/span><\/div>\n
\"\"<\/div>\n
Image 2<\/em><\/div>\n
 <\/span><\/span><\/div>\n
\"\"<\/div>\n
Image 3<\/em><\/div>\n
 <\/span><\/span><\/div>\n
\"\"<\/div>\n
Image 4<\/em><\/div>\n
 <\/span><\/span><\/div>\n
 <\/span><\/span><\/div>\n
\n

In Images 5, 6, 7, 8, the majority label mismatched the attribute label from Betaface API.<\/p>\n<\/div>\n


\n\"\"
\n<\/span><\/span><\/div>\n
Image 5<\/em><\/div>\n
 <\/span><\/span><\/div>\n
\"\"<\/div>\n
Image 6<\/em><\/div>\n
 <\/span><\/span><\/div>\n
\"\"<\/div>\n
Image 7<\/em><\/div>\n
 <\/span><\/span><\/div>\n
\"\"<\/div>\n
Image 8<\/em><\/div>\n
 <\/span><\/span><\/div>\n
\n

Only one image resulted in over 75% majority, Image 5, and it mismatched the label from Betaface API. The survey results display a general spread of opinions across all \u201ceyes distance\u201d categories. <\/p>\n

The survey also showed a face image and its corresponding \u201ceyes distance\u201d label as classified by Betaface API for each of the five categories and asked individuals if they agreed or disagree with the software measurement. Only two out of the five images result in over 70% majority, and the rest are relatively split: <\/p>\n<\/div>\n

 <\/span><\/div>\n
\"\"<\/div>\n
Figure 1<\/em><\/div>\n
 <\/span><\/span><\/div>\n
\n

When asked how the “average” category was determined across attributes (and if there was any scientific research factored into this), the Betaface API CTO replied, “we have our own general average model where we fit faces to figure out relative sizes.\u201d<\/p>\n

Instead of looking at AI advancements in terms of \u201chumans vs computers\u201d, perhaps we should consider comparisons in terms of \u201chumans vs another set of humans who curate the training data set that prescribes a software system\u2019s predictive behavior.\u201d However, it is the perspectives of only certain humans that are being translated into how predictor models that are entering widespread use should classify face attributes given any input face. <\/p>\n

Automated face attribute recognition systems would certainly measure more consistently and save immensely on time. But whose definitions of how our faces should be measured are our face attributes being labelled by? How will this labelled data be used? For the time being, I might prefer to go back to my mirror. <\/p>\n<\/div>\n

\n

 <\/p>\n

Links and References<\/strong><\/h2>\n[1] CelebA data set http:\/\/mmlab.ie.cuhk.edu.hk\/projects\/CelebA.html<\/a><\/p>\n[2] \u201cMOON: A Mixed Objective Optimisation Network for the Recognition of Facial Attributes\u201d https:\/\/arxiv.org\/pdf\/1603.07027.pdf<\/a><\/p>\n[3] \u201cMulti-task Deep Neural Network for Joint Face Recognition and Facial Attribute Prediction\u201dhttp:\/\/www.yugangjiang.info\/publication\/icmr17-face.pdf<\/a><\/p>\n[4] CASIA-WebFace data set http:\/\/www.cbsr.ia.ac.cn\/english\/CASIA-WebFace-Database.html<\/a><\/p>\n[5] FaceScrub data set http:\/\/vintage.winklerbros.net\/facescrub.html<\/a><\/p>\n[6] PubFig data set http:\/\/www.cs.columbia.edu\/CAVE\/databases\/pubfig\/<\/a><\/p>\n[7] Amazon Mechanical Turk https:\/\/www.mturk.com\/mturk\/welcome<\/a><\/p>\n[8] \u201cAttributes and Simile Classifiers for Face Verification\u201d  http:\/\/www1.cs.columbia.edu\/CAVE\/publications\/pdfs\/Kumar_ICCV09.pdf<\/a><\/p>\n[9] Betaface API http:\/\/betafaceapi.com\/<\/a><\/p>\n

 <\/p>\n

 <\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Your face is average. According to that guy. This is a blog post written by Dana Marie Yu. Dana was a Junior Researcher in ETHOS Lab in the spring semester […]<\/p>\n","protected":false},"author":78,"featured_media":3272,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","ngg_post_thumbnail":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1,17,51],"tags":[],"class_list":["post-3259","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news","category-research","category-blog"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/blogit.itu.dk\/ethoslab\/wp-content\/uploads\/sites\/92\/2017\/09\/COLOURBOX28569446.jpg?fit=6319%2C2708&ssl=1","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/posts\/3259","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/users\/78"}],"replies":[{"embeddable":true,"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/comments?post=3259"}],"version-history":[{"count":5,"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/posts\/3259\/revisions"}],"predecessor-version":[{"id":3275,"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/posts\/3259\/revisions\/3275"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/media\/3272"}],"wp:attachment":[{"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/media?parent=3259"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/categories?post=3259"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogit.itu.dk\/ethoslab\/wp-json\/wp\/v2\/tags?post=3259"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}