Real-Time Facial Computing Web Service Construction and Mobile Device Implementation

Undergraduate #64
Discipline: Computer Sciences and Information Management
Subcategory: Computer Science & Information Systems

Christina Tsangouri - City College of New York
Co-Author(s): Wei Li, City College of New York, NY

Emotions are an incredibly important aspect of human communication. For visually impaired people, their inability to see the emotions of other people severely impairs their ability for social interaction. In order to help visually impaired people be able to recognize the emotions of the people around them, a system was designed which is able to inform users in real-time the emotions of the people around them. The emotions able to be recognized are anger, surprise, neutral, happy, disgust, fear, and sad. The system implemented, was designed as a framework consisting of a mobile application developed for Google’s Android operating system using Java and the Android Software Development Kit, a web service on the main server developed using Ruby on Rails, and an emotion-computing server.

The framework designed is based on a loop between a mobile device and a server. The mobile device captures real-time scenes through the built-in camera and detects faces continuously, which is implemented using OpenCV. Upon face detection, the face region is sent using HTTP protocol to the main server, which handles all the requests from the mobile device and assigns the image to the emotion-computing server for emotion recognition. The emotion-computing server is designed to compute the probability of each of the 7 emotions in the image based on a Convolutional Neural Network we trained. The main server then sends these probabilities to the mobile application and the most probable emotion is displayed to the user. This entire loop executes at about 2 frames per second. The application is designed with a visually impaired user in mind and includes features such as vibration upon face detection, and communicating the most probable emotion result through audio using music tunes in addition to displaying the result visually. The application developed, EmoComputing, is available on Android’s Google Play Market. Future work includes also implementing the framework on Apple’s iOS platform to aid with data collection for creating a more accurate emotion recognition model, an evaluation of the performance of the emotion recognition system in a real-time environment, and an evaluation of the user experience.

References: Li, Wei, Min Li, Zhong Su, and Zhigang Zhu. “A deep learning approach to facial expression recognition with candid images.” In Machine Vision Applications (MVA) 2015, 14th IAPR International Conference on, pp.279-282. IEEE, 2015.

Funder Acknowledgement(s): This work is supported by the National Science Foundation under Award #EFRI - 1137172, and the 2015 NSF EFRI-REM pilot program at the City College of New York.

Faculty Advisor: Zhigang Zhu, Tony Ro,