Interesting undocumented Facebook API to identify friends in any personal photos

So last week I was casually browsing through Facebook feed and looking at some random photo posted by a friend. So hovering over the picture the Facebook displayed a rectangle over the face of my friend and showed "Do you want to tag X". This caught my attention. OK so Facebook does image recognition to identify not only faces of people but also identifies who it is. I wondered how FB does it so I spawned up my Chrome Dev Tools, to debug into how this whole thing works.

So it seems that FB has a bunch of div blocks which are the defining the face boxes with the suggestions for identity for each face box in data-recognizeduids. It also had another block of html with the names of the identified users.

OK so now the question is how is this data coming in the first place. It seems it is loaded when one clicks on a photo. So it must be loaded using some ajax call to the server later. Now to find out what call we switch to network tab which shows all the traffic between the browser and FB servers. So now it was all about finding the needle in haystack as there were hundreds of requests. So I started looking for the once with relevant names and struck gold when I found a request "/ajax/pagelet/generic.php/PhotoViewerInitPagelet". 

Then I searched the response for relevant content and found above block of html in the response. The params to the route looked encoded so after decoding them it seemed it sends bunch of params along with the id of the photo. The request looks something like this

https://www.facebook.com/ajax/pagelet/generic.php/PhotoViewerInitPagelet?dpr=2&ajaxpipe=1&ajaxpipe_token=<changed>&no_script_path=1&data={"fbid":"<changed>","set":"<changed>","type":"3","theater":null,"firstLoad":true,"ssid":<changed>,"av":"<changed>"}&__user=1&__a=1&__dyn=<changed>&__req=jsonp_2&__be=0&__pc=PHASED:DEFAULT&__rev=11111&__adt=2

Most of the params don't make much sense except for fbid which looks like the id of the photo. So I played around removing unnecessary params to find the ones that are actually needed for api to work and I found that it needed only the fb id and 2-3 other constant params. Chrome has a pretty neat feature where it allows you to copy a request as CURL request which has all headers and params so you could run it from the command line as is. Using that I replicated the request and found the same response which had the html data containing the information about the users. So I thought if we can use this API to find friends in any photo which might not be uploaded on FB. 

But for the API to work it needs the photo to be on FB. So I thought if we can temporarily upload a photo to FB without publishing it and use the API. Now to upload photos on FB I used the FB Graph API. I needed these photos not to be published and uploaded temporarily for using the API. The Graph Photo Upload API allows to upload the images with options "temporary"=> true and "published"=> false so that the images are uploaded temporarily and not show up in the feed. I created an app on FB Photo Friend Finder which uses publish_actions and user_photos permissions to allow access to the photo API. Once the photo is uploaded the API returns with the id of the photo which is then passed to the above PhotoViewerInitPagelet API which returns HTML data containing details of face suggestions. Now all I needed to do was parse the response to extract the data which can easily be done using HTML parser. I used Nokogiri in ruby for the same.

I put together a small Ruby script to do all of these. So you just run the script with path to image as input param and the script returns the number of people in the photo and identifies the friends. I found this API has a limitation it only suggests people in the photo who are in your FB friend list. I used Koala gem to access FB Graph API and JSON and Nokogiri libraries to parse the API response.
Here is the link to the script.
Script in action.

It was a lot of fun exploring and putting together the script.