Testing Windows Machine Learning Preview through Emotion Recognition.

‍

The latest Windows 10 SDK Insider Preview -build 17110- comes with a new API known as Windows Machine Learning that enable developers to load pretrained ONNX models, bind to its input/output tensors and perform predictions effortlessly on any DirectX 12 compatible GPU. Deploying AI systems on Windows has never been easier.

Open Neural Network Exchange Format -ONNX- enables models to be trained in one framework and transferred to another for inference. It is just a file format on top of protobuf which carries the computational graph, as well as all the parameters required for making predictions.

Let’s see an example.

‍

Adding AI capabilities to an application

This post demonstrates the use of generic Windows.AI.MachineLearning (Preview) API to load an ONNX model, bind it to an input image and an output tensor, and perform the evaluation. To keep the focus on the new features, we left out all additional UWP stuff.

The full sample can be obtained from GitHub 👈. It uses the device’s camera to take a picture and feed the model.

‍

Prerequisites

To run this sample you’ll need the following:

• Windows 10 Insider Preview (Fast Ring) (build >= 17110)
• Windows 10 SDK Insider Preview (build 17110)
• Visual Studio 2017 Preview (15.6.1 Preview 1.0)

Notice that all tools are still on Preview, it’s not recommended to install them on productive systems.

‍

1. Create or Edit an existing UWP project

Open or create a new UWP project using Visual Studio 2017 Preview and make sure to select the target and minimum platform versions to: Windows 10 Insider Preview (10.0; Build 17110).

2. Load the ONNX pretrained model into the Visual Studio project

Now, we need a pretrained model for the specific task we’ll be adding in our UWP project.

There is a pretrained model for emotion recognition on the onnx models repository. Unfortunately, this model expects normalized input data. This will not work well with the Windows Machine Learning SDK since the API does not currently provide tools to do such kind of data pre-processing.

So, I trained a slightly modified version of the original Microsoft\FERPlus network that includes the normalization step into the model itself, thus creating a new model that directly accepts 64x64 gray scale images (i.e. without previous normalization).

Download the pretrained model from here, then copy it into the Assets folder and rename it as FERPlus.onnx if needed.

‍

3. Add a class for loading the model and perform the inference

Add a new class and name it EmotionRecognizer. Also, declare an auxiliary internal class named Results like the sample below, we will use it later for returning the predictions. Aditionally, declare the class variables: the model, the input image and the output tensor descriptors.

‍

2. Add the method responsible for loading the FERPlus.onnx model. This method also looks up the input and output variables that we’ll need latter for feeding the data into the model when performing the prediction.

‍

3. Finally, add the method that performs the inference/prediction. This example will return a Result object containing the likelihood of each emotion to be present on the picture’s face (the model assumes is receiving the image of a face).

‍

4. Test the recognizer

So far, we created our emotion recognizer. Now, to test it we first need to instantiate our class and invoke LoadModelAsync() once, then we need to invoke EvaluateAsync(..) with a VideoFrame as parameter every time we want to make a prediction. The VideoFrame is just a container for the image we want to evaluate.

Note that the model input receives a gray scale image of 64x64 pixels. You might want to create an image and feed it into the model. For best results, make sure the face dominates the image.

You can also use the example on GitHub that uses the device’s camera to take pictures to then feed the model. The sample also contains a few useful methods to Crop, Scale down and Convert to gray scale the images taken through the camera.

Nevertheless, just in case you want to test the EmotionRecognizer directly loading an image from a file… here’s an example by using the FileOpenPicker}

‍

Conclusions

Although this technology is still under active development and needs to grow and mature, it is a big step forward in order to have a better ecosystem for easly deploying Machine Learning models into windows applications. Also, it would be great to have this feature available for other kind of .NET applications like ASP.NET or NET Core.

At the moment, we are looking forward to the next Windows 10 major update 🤞.

Thanks to Nicolás Bello Camilletti

‍

Originally published by Emanuel Vecchio for SOUTHWORKS on Medium 14 March 2018

‍