Tutorial Series: using WinJS & WinRT to build a fun HTML5 Camera Application for Windows 8 (4/4)
In this fourth & last tutorial, we’re going to play with pixels manipulation. We will first use the EaselJS library and its filters part to apply some Sepia, Black & White and Blur effects on the images taken. Then, we will see how to optimize the performance by not blocking the UI Thread and by using the Web Workers to use more of the available cores of our CPUs. At last, we will see how to go even further with a WinRT C++ component using shaders to discuss directly with the many-cores of the GPU. I will do some small benchmarks to show you the benefit of paying such attention to performances. At last, as we start to have a lot of code, we’re going to start using the WinJS.Class helper to construct our nice objects to be used. This will be definitely the most advanced & technical article of the 4 tutorials. So be sure to be comfortable with the concepts already covered in the series:
1 – Accessing to the Camera stream and writing it on disk
2 – Working on the layout, adding video recording support and providing feedback to the user with CSS3 animations
3 – Using the FlipView to navigate through the live preview and the recorded files
4 – Manipulating the image to add some effects via the canvas tag with EaselJS or WebWorkers and via GPU Shaders with a WinRT component
You’ll find the Visual Studio solution matching the complete series at the end of this article.
To have again a better idea of what we’re going to built, here is a small video demonstrating what you’ll get at the end:
Download Video: MP4, WebM, HTML5 Video Player by VideoJS
Kudos: I’d like to thanks David Catuhe for the precious help he provided on parts of this article.
Step 1: Preparing the layout with the AppBar and Flyout controls
The idea here is to construct an application bar that will be only displayed when the user will navigate to the image template of the flip view. This bar will expose 3 buttons that will let the user launch more or less the same filters using the 3 different available components. When he will click on one of these buttons, the flyout will appear with the 3 available effects displayed as little preview images. It will look like that:
Ok, let’s start by the HTML part. Open default.html and insert the following HTML just after the flip view declaration:
<div id="filtersFlyout" data-win-control="WinJS.UI.Flyout" aria-label="{Filters controls flyout}">
<img id="effectBlur" width="128" height="96" src="" />
<img id="effectBW" width="128" height="96" src="" />
<img id="effectSepia" width="128" height="96" src="" />
</div>
<div id="appbar" data-win-control="WinJS.UI.AppBar" aria-label="Command Bar">
<button
data-win-control="WinJS.UI.AppBarCommand"
data-win-options="{id:'cmdDeFormFilters',label:'DeForm Filters',icon:'edit',type:'flyout'}">
</button>
<button
data-win-control="WinJS.UI.AppBarCommand"
data-win-options="{id:'cmdWorkersFilters',label:'Workers Filters',icon:'edit',type:'flyout'}">
</button>
<button
data-win-control="WinJS.UI.AppBarCommand"
data-win-options="{id:'cmdEaselJSFilters',label:'EaselJS Filters',icon:'edit',type:'flyout'}">
</button>
</div>
We’re just inserting some WinJS controls as usual. If you’re launching the application right now, you will have the application available (right click with your mouse or swipe from the bottom with touch). But the 3 buttons are currently doing nothing.
We now need to work on the JavaScript part. Jump into default.js and add this bunch of variables declarations:
// To know if we have already computed the thumbnails for the current
// image displayed or not
var thumbnailsGenerated = false;
var workInProgressElement;
var appbar;
var flyout;
// Pointers to the 3 images tags contained in the flyout control
var thumbnailBlur;
var thumbnailBW;
var thumbnailSepia;
// Current image displayed in the flipview if the image template is currently used
var currentImageToManipulate;
var currentItem;
// To mimic polymorphism
var componentToUse;
// The 3 types of filters processors: EaselJS, Web Workers and the WinRT C++ component from David Catuhe
var easelJSFilters;
var workersFilters;
var deFormFilters;
// Used to benchmark the time needed to apply an effect
var startDate;
We will use them during this tutorial and you will learn their role during each progressing step.
Ok, now, we need to add more logic into the init() function. Here is the code to add at the end of the function:
// Accessing to the AppBar WinJS Control and
// setting it as disabled by default
appbar = document.getElementById("appbar").winControl;
appbar.disabled = true;
// Everytime we're navigating into the flip view
flipView.addEventListener("pageselected", function () {
currentItem = screensList.getAt(flipView.currentPage);
// making the application bar available only if an image
// is displayed in the flipview
if (currentItem.type === "image") {
appbar.disabled = false;
}
else {
appbar.disabled = true;
}
// Resetting our boolean to false when the user
// navigate to another image to indicate we need
// to rebuild the thumbnails
thumbnailsGenerated = false;
});
flyout = document.getElementById("filtersFlyout");
// Getting the 3 images available in the flyout and adding the applyEffect handler
// on the click event
thumbnailBlur = flyout.querySelector("#effectBlur");
thumbnailBlur.addEventListener("click", function () {
applyEffect("Blur");
}, false);
thumbnailBW = flyout.querySelector("#effectBW");
thumbnailBW.addEventListener("click", function () {
applyEffect("BW");
}, false);
thumbnailSepia = flyout.querySelector("#effectSepia");
thumbnailSepia.addEventListener("click", function () {
applyEffect("Sepia");
}, false);
// We're using the very same flyout for each case but we then just need
// to indicate later on which component to use for the filter processing
document.getElementById("cmdEaselJSFilters").addEventListener("click", function () {
componentToUse = "EaselJS";
displayFlyout()
}, false);
document.getElementById("cmdWorkersFilters").addEventListener("click", function () {
componentToUse = "Workers";
displayFlyout();
}, false);
document.getElementById("cmdDeFormFilters").addEventListener("click", function () {
componentToUse = "DeForm";
displayFlyout();
}, false);
To avoid having the application bar being displayed on the camera live preview or when the user navigate to the video template, we’re disabling it by default. It’s only enabled when the type of the current item bind to the current flip view page is of type “image”. We’re then disabling or enabling it each time a navigation is being done and thus, each time the pageselected event is raised by the flip view. The rest of code should be straightforward and if not, just read the comments! 
Let’s build the draft body of the functions being called into the various click handlers:
function displayFlyout() {
// Generating the effects in thumbnails
// to let the user preview them before applying
// The thumbnailsGenerated boolean is handling our cache
if (!thumbnailsGenerated) {
currentImageToManipulate = new Image();
currentImageToManipulate.onload = function () {
// We need to generate thumbnails here
thumbnailsGenerated = true;
};
currentImageToManipulate.src = currentItem.url;
}
// displaying directly the flyout if the thumbnails have
// already been generated
flyout.winControl.show(document.getElementById("cmd" + componentToUse + "Filters"));
}
function applyEffect(type) {
// later implemented
}
If you’re now pressing F5, the application bar should only be displayable on some photo taken in the flip view and you will have this result:
As we don’t have implemented any filtering logic yet to generate the thumbnails.
Step 2: adding some filters effects with the EaselJS Library
As most of the JS libraries used on the web work inside HTML5 Windows 8 Modern Aps, if you need to build this kind of specific tasks, have first a look to the current available libraries on the web. On my side, I’m a huge fan of the EaselJS part of the CreateJS suite built by Grant Skinner and its team. If you don’t know this library yet, I’ve built a series of 6 tutorials on it starting it here: HTML5 Gaming: animating sprites in Canvas with EaselJS . This is also the library that have been used for the the Atari Arcade games ported to HTML5 with a cool IE10 touch experience.
For this tutorial, I have used this demo as a base: EaselJS Filters demo and more specifically this sample available on Github.
So, the first thing to do to be able to complete this second step of this tutorial is downloading the EaselJS library here. Create a new folder named “easeljs” under the “js” folder of your project and copy/paste the EaselJS JS files into it.
Add the proper references to some of these files into default.html :
<!-- EaselJS reference https://createjs.com -->
<script src="/js/easeljs/geom/Rectangle.js"></script>
<script src="/js/easeljs/geom/Matrix2D.js"></script>
<script src="/js/easeljs/utils/UID.js"></script>
<script src="/js/easeljs/display/DisplayObject.js"></script>
<script src="/js/easeljs/display/Bitmap.js"></script>
<script src="/js/easeljs/filters/Filter.js"></script>
<script src="/js/easeljs/filters/BoxBlurFilter.js"></script>
<script src="/js/easeljs/filters/ColorMatrixFilter.js"></script>
This is the minimum required to be able to use the filters. Now that we’ve got the libraries to work with, let’s build our own logic on top of it. Let’s organize also this logic in a proper way. For that, we’re going to use the WinJS.Class helper. If you want to know more about this, please read this article: Defining and deriving types with WinJS.Class. The helper helps us to construct object-oriented like code and maps that to the prototype nature of JavaScript for you. If you’re prefixing your variables and functions with an underscore, it means that it should be private. Otherwise, it will be exposed as public for the clients.
Add a new JS file named “easelJSFilters.js” under “js/easeljs”. Insert this code inside it:
(function () {
"use strict";
var Filters = WinJS.Class.define(
// The constructor needs an HTML Image Element
// and a function that will be callbacked at the
// end of the filter processing
function (image, callback) {
this._currentImageToManipulate = image;
this._callbackFunction = callback;
// Creating the full resolution Bitmap object and a 20% one to generate quickly
// some thumbnails if requested
this._bmpFull = new createjs.Bitmap(this._currentImageToManipulate);
this._bmpFull.cache(0, 0, this._currentImageToManipulate.width,
this._currentImageToManipulate.height);
this._bmpThumb = new createjs.Bitmap(this._currentImageToManipulate);
this._bmpThumb.cache(0, 0, this._currentImageToManipulate.width,
this._currentImageToManipulate.height, 0.2);
},
{
_applyFilter: function (bmp) {
// the filter is really applied after this line of code
// this is not an asynchronous call and it could take
// a lot of time to compute. It could then block the UI Thread
bmp.updateCache();
return bmp.getCacheDataURL();
},
_applyBlurEffect: function (bmp) {
var blurFilter = new createjs.BoxBlurFilter(10, 2, 2);
var margins = blurFilter.getBounds();
bmp.filters = [blurFilter];
return this._applyFilter(bmp);
},
_applyBWEffect: function (bmp) {
var greyScaleFilter = new createjs.ColorMatrixFilter([
0.33, 0.33, 0.33, 0, 0, // red
0.33, 0.33, 0.33, 0, 0, // green
0.33, 0.33, 0.33, 0, 0, // blue
0, 0, 0, 1, 0 // alpha
]);
bmp.filters = [greyScaleFilter];
return this._applyFilter(bmp);
},
_applySepiaEffect: function (bmp) {
var sepiaFilter = new createjs.ColorMatrixFilter([
0.393, 0.769, 0.189, 0, 0, // red
0.349, 0.686, 0.168, 0, 0, // green
0.272, 0.534, 0.131, 0, 0, // blue
0, 0, 0, 1, 0 // alpha
]);
bmp.filters = [sepiaFilter];
return this._applyFilter(bmp);
},
generateThumbnails: function (imagesArray) {
imagesArray[0].src = this._applyBlurEffect(this._bmpThumb);
imagesArray[1].src = this._applyBWEffect(this._bmpThumb);
imagesArray[2].src = this._applySepiaEffect(this._bmpThumb);
},
applyBlurEffect: function () {
var fullImageModifiedUrl = this._applyBlurEffect(this._bmpFull);
this._callbackFunction(fullImageModifiedUrl);
},
applyBWEffect: function () {
var fullImageModifiedUrl = this._applyBWEffect(this._bmpFull);
this._callbackFunction(fullImageModifiedUrl);
},
applySepiaEffect: function () {
var fullImageModifiedUrl = this._applySepiaEffect(this._bmpFull);
this._callbackFunction(fullImageModifiedUrl);
}
}
);
WinJS.Namespace.define("EaselJSFilters", { Filters: Filters });
} ());
This code exposes a public constructor taking an HTML image element and a callback function as parameters. It exposes also 4 public methods/functions: generateThumbnails, applyBlurEffect, applyBWEffect and applySepiaEffect. This code is just a refactored version of the EaselJS sample. The EaselJS library is using some working canvas to manipulate every pixel of our HTML image. If you look at its code, you’ll quickly find some document.createElement(“canvas”) for that. This code is also instantiating 2 Bitmap objects. 1 at the full resolution of the image taken with your WebCam. It will then depend of the default resolution output by your device. In my case, it’s 640x480 on my Intel UltraBook Touch machine but on some of my tablets, it’s 1280x720. To accelerate the thumbnails preview generation, I’m then building another Bitmap image with a 20% resolution to have less pixels to manipulate and thus, to be quicker.
To be able to use this code, insert this in the default.html:
<script src="/js/easeljs/easelJSFilters.js"></script>
Now, if you’re instantiating our fresh new object in your code, here is what the IntelliSense/autocompletion of VS2012 will propose you:
You will only see the publics functions described before. Ok, let’s now use this logic to first create our thumbnails. Insert this code into the body of the anonymous function used for currentImageToManipulate.onload event into displayFlyout() :
easelJSFilters = new EaselJSFilters.Filters(currentImageToManipulate, null);
// We're always using EaselJS to generate the preview thumbnails
var imagesArray = [thumbnailBlur, thumbnailBW, thumbnailSepia];
easelJSFilters.generateThumbnails(imagesArray);
Ok, we’re good to go to start playing with our little sample. Press F5 to launch the app, take a photo, navigate to this photo in the flipview and press the EaselJS filters button in the application bar. The 3 thumbnails should now be generated:
We now need to do something when the user is clicking on one of the thumbnails. For that, here is the code that you will need:
function applyEffect(type) {
var currentComponent;
flyout.winControl.hide();
startDate = new Date();
switch (componentToUse) {
case "EaselJS":
currentComponent = easelJSFilters;
break;
case "Workers":
currentComponent = workersFilters;
break;
case "DeForm":
currentComponent = deFormFilters;
break;
}
switch (type) {
case "Blur":
currentComponent.applyBlurEffect();
break;
case "BW":
currentComponent.applyBWEffect();
break;
case "Sepia":
currentComponent.applySepiaEffect();
break;
}
thumbnailsGenerated = false;
}
function updateFlipViewItem(url) {
var diff = new Date() - startDate;
console.log("Process done in " + diff);
//var messageDialog = new Windows.UI.Popups.MessageDialog("Process done in " + diff + " ms.");
//messageDialog.showAsync();
currentItem.url = url;
screensList.setAt(flipView.currentPage, currentItem);
}
And finally, update the call to the EaselJSFilters.Filters constructor to:
easelJSFilters = new EaselJSFilters.Filters(currentImageToManipulate, updateFlipViewItem);
If you’d like to start benchmarking the time needed to apply each effect, just uncomment the 2 lines of code commented in the updateFlipViewItem function.
Press F5 to play with the sample. You should now be able to apply blur, B&W and sepia effects to your photos. Congrats! 
Now I’d like to bring to your attention 2 important points:
- something really important about the differences between debugged JavaScript project from Visual Studio and the very same project tested in release mode directly from the start screen for instance
- provide visual feedbacks to the user
For the first point, it’s very easy to check it. Indeed, if I’m launching the sample in debug mode via Visual Studio 2012 and applying the blur effect, my CPU needs approx. 2800ms to apply the blur filter. Doing the same operation from the start screen or in release mode (CTRL+F5) only needs approx. 700ms!
The point is that the JavaScript code is compiled into native code when you’re pushing it into release mode like IE10 is more or less doing it with its Chakra engine. For debugging purposes, the JS code is not compiled in the debug mode. You can then see the benefit of compiling the JS code in Windows 8 projects! But the first conclusion also is that you should benchmark the performance of your application in release mode to have an accurate idea on how your app really behaves.
Let’s now talk about the second point. You will maybe have a bad experience on your machine/device with the current sample. Indeed, the time needed to process the image could take several seconds to accomplish. It then freezes the UI Thread and the user have no idea of what’s going on and he can’t interact anymore with the app. It would be a good idea to provide visual feedbacks to the user to let him know that the filtering is currently being computed and that he just needs to wait a bit.
For that, let’s add this piece of HTML at the end of default.html:
<div class="workInProgress hidden">
<div class="background"></div>
<div class="progressContainer">
<label class="progressRingText">
<progress class="win-ring withText"></progress>Processing</label>
</div>
</div>
It needs to be coupled with this CSS to be naturally inserted into default.css:
.workInProgress {
position: absolute;
top: 0px;
left: 0px;
width: 100%;
height: 100%;
display: -ms-grid;
-ms-grid-columns: 1fr;
-ms-grid-rows: 1fr;
opacity: 0.5;
z-index: 4;
}
.hidden {
display: none;
}
.workInProgress .progressContainer {
-ms-grid-column-align: center;
-ms-grid-row-align: center;
opacity: 1.0;
transform: scale(4.0);
}
.workInProgress .background {
-ms-grid-column-align: center;
-ms-grid-row-align: center;
width: 100%;
height: 100%;
background-color: #EE8100;
opacity: inherit;
}
The idea here is to add a full screen waiting screen with some slight opacity using the same color as our splash screen (#EE8100). The way to show or hide it is done by adding or removing the .hidden class via JS code.
To use it, add this code at the end of the init() function:
// Displayed while the filter is being applied to the image
workInProgressElement = document.getElementsByClassName("workInProgress")[0];
We now need to make it visible by removing the hidden class. For that, add this line of code at the beginning of the applyEffect function:
WinJS.Utilities.removeClass(workInProgressElement, "hidden");
And once the filter will be applied, we need to hide it again. For that, add this line of code at the end of the updateFlipViewItem function:
WinJS.Utilities.addClass(workInProgressElement, "hidden");
Launch the sample in F5 (debug mode) and try to add a filter. You will be maybe surprised to not see the progress element as expected. This is due to the mono-threaded nature of JavaScript. I have already covered this problem in this article: Introduction to the HTML5 Web Workers: the JavaScript multithreading approach . Have a look to the beautiful diagram of this article. This is exactly what is currently happening in our case. Even if the JS line of code that is removing the hidden class is before the code doing the filtering effect, the UI Thread reflects the change after. Indeed, the filtering effect of EaselJS is done on the main thread (the UI Thread). The window’s messages pump doesn’t have the time to process the UI changes before the filtering code is launched. Our waiting screen will then never be shown.
The first solution is then to bring a bit of oxygen to the UI Thread by using an old technic via a setTimeout(). We will see in the third part of this tutorial how to do something cleaner using workers. In the meantime, change the part of the applyEffect code with this one:
WinJS.Promise.timeout(100).then(function () {
startDate = new Date();
switch (componentToUse) {
case "EaselJS":
currentComponent = easelJSFilters;
break;
case "Workers":
currentComponent = workersFilters;
break;
case "DeForm":
currentComponent = deFormFilters;
break;
}
switch (type) {
case "Blur":
currentComponent.applyBlurEffect();
break;
case "BW":
currentComponent.applyBWEffect();
break;
case "Sepia":
currentComponent.applySepiaEffect();
break;
}
});
Providing 100ms to the UI Thread should be enough in most cases to let it displaying our waiting screen before launching the applyBlurEffect() code. Now, you should see this screen while the effect is being applied:
With this final result:
It starts to be really fun to use ! But there is room for performance improvement. Indeed, using only the UI Thread and thus only 1 of the available cores of the CPU is not very efficient. Let’s now see how to use more cores for more performance.
Step 3: using the JavaScript Web Workers to boost the performance
My Intel IvyBridge Core i7 CPU has currently 2 cores hyper-threaded and even the ARM tablets will have either 2 or 4 cores. We then often have 4 logical processors available on our machines. And you know what? Web workers are perfect candidates to help us maximizing our CPU’s usage.
For that, we need to clone the image and send it to several workers that will apply the effect of subparts of the image. Hopefully, this cloning mechanism is very fast under most modern browser. It’s done via a memcpy() operation in IE10 and thus in Windows 8 apps. I’ve shared my needs to David Catuhe who worked on that for me and he wrote an excellent article on this topic: Using Web Workers to improve performance of image manipulation. You should read it first before continue going through this tutorial if you’d like to understand everything.
Create a new folder named “workersFilters” under “ /js”. Add a new JS file named “tools.js” and copy/paste this code into it:
// This JS code will be processed inside a worker
// To achieve the black & white effect
var processBW = function (binaryData, l) {
for (var i = 0; i < l; i += 4) {
var r = binaryData[i];
var g = binaryData[i + 1];
var b = binaryData[i + 2];
var luminance = r * 0.21 + g * 0.71 + b * 0.07;
binaryData[i] = luminance;
binaryData[i + 1] = luminance;
binaryData[i + 2] = luminance;
}
};
// To achieve the sepia effect
var processSepia = function (binaryData, l) {
for (var i = 0; i < l; i += 4) {
var r = binaryData[i];
var g = binaryData[i + 1];
var b = binaryData[i + 2];
binaryData[i] = (r * 0.393) + (g * 0.769) + (b * 0.189);
binaryData[i + 1] = (r * 0.349) + (g * 0.686) + (b * 0.168);
binaryData[i + 2] = (r * 0.272) + (g * 0.534) + (b * 0.131);
}
};
function readProtected(binaryData, l, offset) {
if (offset < 0)
offset = 0;
if (offset >= l)
offset = l - 1;
return binaryData[offset];
}
// To achieve the blur effect
var processBlur = function (binaryData, l, depth) {
for (var i = 0; i < l; i += 4) {
var moyR = 0;
var moyG = 0;
var moyB = 0;
var count = 0;
for (var around = -depth; around <= depth; around++) {
var r = readProtected(binaryData, l, i + (around << 2));
var g = readProtected(binaryData, l, i + (around << 2) + 1);
var b = readProtected(binaryData, l, i + (around << 2) + 2);
moyR += r;
moyG += g;
moyB += b;
count++;
}
binaryData[i] = moyR / count;
binaryData[i + 1] = moyG / count;
binaryData[i + 2] = moyB / count;
}
};
Add another new JS file named “pictureProcessor.js” and insert this code into it:
importScripts("tools.js");
// Executed inside a web workers
// receiving the data canvas sent from the UI Thread
self.onmessage = function (e) {
var canvasData = e.data.data;
var binaryData = canvasData.data;
var l = e.data.length;
var index = e.data.index;
var type = e.data.type;
switch (type) {
case "BW":
processBW(binaryData, l);
break;
case "Blur":
processBlur(binaryData, l, 15);
break;
case "Sepia":
processSepia(binaryData, l);
break;
}
self.postMessage({ result: canvasData, index: index });
};
This 2 files are some slightly modified versions of David Catuhe’s code detailed in his article.
Ok, now that we have our library ready to be used, we need to build a new class mapping them to the same interface used before for EaselJS. Again, it’s just some refactoring job done on the original code built by David in his article.
Add a new JS file named “workersFilters.js” under “ /js/workersFilters” and copy/paste this code into it:
(function () {
"use strict";
var Filters = WinJS.Class.define(
// The constructor needs an HTML Image Element
// and a function that will be callbacked at the
// end of the filter processing and the number of
// workers we'd like to launch
function (image, callback, workersCount) {
this._currentImageToManipulate = image;
this._callbackFunction = callback;
this._workersCount = workersCount;
// Creating a working canvas to draw the image and
// access to every pixel of it
this._canvas = document.createElement("canvas");
this._canvas.width = this._currentImageToManipulate.width;
this._canvas.height = this._currentImageToManipulate.height;
this._tempContext = this._canvas.getContext("2d");
// Drawing the source image into the target canvas
this._tempContext.drawImage(this._currentImageToManipulate, 0, 0,
this._currentImageToManipulate.width,
this._currentImageToManipulate.height);
},
{
_applyFilter: function (type) {
var len = this._canvas.width * this._canvas.height * 4; // 4 = RGBA
// This is the length of array sent to the worker
var segmentLength = len / this._workersCount;
// Height of the picture chunck for every worker
this._blockSize = this._canvas.height / this._workersCount;
this._finished = 0;
// Launching every worker
for (var index = 0; index < this._workersCount; index++) {
var worker = new Worker("/js/workersFilters/pictureProcessor.js");
var that = this;
// we need a little closure for the post back message
worker.onmessage = function (e) {
that._onWorkEnded(e, that);
};
// Getting the picture
var canvasData = this._tempContext.getImageData(0, this._blockSize * index,
this._canvas.width, this._blockSize);
// Sending canvas data to the worker using a copy memory operation
worker.postMessage({ data: canvasData, index: index, length: segmentLength, type: type });
}
},
applyBlurEffect: function () {
this._applyFilter("Blur");
},
applyBWEffect: function () {
this._applyFilter("BW");
},
applySepiaEffect: function () {
this._applyFilter("Sepia");
},
// Function called when a job is finished
_onWorkEnded: function (e, that) {
// Data is retrieved using a memory clone operation
var canvasData = e.data.result;
var index = e.data.index;
// Copying back canvas data to canvas
that._tempContext.putImageData(canvasData, 0, that._blockSize * index);
that._finished++;
if (that._finished == that._workersCount) {
that._callbackFunction(that._canvas.toDataURL());
}
}
}
);
WinJS.Namespace.define("WorkersFilters", { Filters: Filters });
} ());
I’ve tried to comment the code to make it self-explicit. We’re now ready to use this new component. Reference this library in your default.html:
<script src="/js/workersFilters/workersFilters.js"></script>
The final part is now just to instantiate this component to be able to use it. The rest of the code is already able to call this new component. You just have to insert this line of code in the displayFlyout() function:
// the last parameter indicate the number of workers you'd like to use. 4 here.
workersFilters = new WorkersFilters.Filters(currentImageToManipulate, updateFlipViewItem, 4);
just after the instantiation of the EaselJS component.
Ok, we’re good to go logically. Launch the application in release mode (CTRL+F5 or from the start screen). Take several photos and compare the performance of the effects using the EaselJS library (which is currently mono-threaded) and the Web Workers versions. You should see some real performance boost.
For instance, on my machine (Intel Core i7-3667U), the EaselJS version needs 240ms to apply the black&white filter whereas the web workers version only needs 56ms to apply the very same effect. We will see a performance table at the end of this article summarizing the gap in several configurations.
At this stage, we’ve been able to boost the performance at least X2 and even higher for the color filters scenarios (X4 on quad-like CPUs). This is already awesome. But you know what? There is still room for improvements. Indeed, there are even more cores available in most of our devices. They are living in our GPUs. They are very specialized cores but there are largely outnumbering the classical CPU cores. And last but not least, there are very good at manipulating pixels. Well, let’s use them!
Step 4: using the GPU cores via shaders to have the best performance possible
GPUs have a lot of cores. For instance, the Intel Core i7 IvyBridge uses the HD4000 embedded GPU with 16 execution units. The ARM nVidia Tegra 3 that you’ll find in some of the Windows RT tablets uses 12 cores for the GPU: Tegra 3 Specifications . That’s very cool!
But GPUs have their own languages. 3D Games Developers know them very well. They are named pixel or vertex shaders. It looks like some low level C code being compiled into a form of assembler ready for our GPUs.
In Windows 8 Modern Apps, the only way to discuss directly with the GPU is to use the C++ language. If you’re like me a JavaScript or C# developer, you maybe don’t want to take time to learn C++ and the shaders model. The good news is that you don’t have to.
Thanks to the Windows 8 projection mechanism, JavaScript can do some direct calls to C++ components in the very same way it’s doing it to discuss with the WinRT APIs. This means that you just have to find “someone” who has the skills to built such a component and you will be able to use it directly from your HTML5/JavaScript project. If you’d like to know more about this architecture, please have a look to: Creating Windows Runtime Components
In my case, this skilled guy I was looking to is again David Catuhe. He has written a C++ WinRT component doing pixels manipulation named DeForm. He has written an article about it here: Creating a WinRT component using C++/CX: DeForm, a Direct2D effect toolkit and you can download the component directly from CodePlex here: https://deform.codeplex.com/
Ok, let’s now see how to use it in our current project.
Just download the compiled version of the component from CodePlex and unzip the archive somewhere on your hard drive.
In the “Solution Explorer” pane of Visual Studio, right-click on the “Reference” folder of your “ModernUIFunCamera” project and choose “Add Reference… ”:
Go into the “Browse” tab and click on the “Browse… ” button. Navigate into the folder where you’ve unzipped the DeForm component and add a reference to the component mapping to the CPU architecture you’re targeting (x86, x64 or ARM):
And that’s all ! We’re ready to use the component.
To be honest, the tricky part is now to find how to transfer the canvas’ data buffer to the C++ component. Indeed, the C++ component doesn’t know anything about the HTML5 canvas element. As we’ve spent a couple of hours on that with David, the best thing to do is to read its JS sample here: Source code of the JS client on CodePlex
But let me briefly explain you what’s going on. The very first thing to do is having access to the blob object of the canvas element by calling the msToBlob method. The MSDN documentation says that: “The Blob object represents immutable raw binary data, and allows access to ranges of bytes within the Blob object as a separate Blob. ”. That’s fine, that’s what I was looking for.
Next step is now to provide a stream (and more exactly an IRandomAccessStream) to the C++ component from this blob object. For that, simply call the msDetachStream method on the blob object. We’re then passing this structure to the C++ component which does its job on the stream living in the memory. The effect is then applied in memory at this stage.
Last step is then to retrieve the bytes living in memory by using the WinRT Windows.Storage.Streams.InMemoryRandomAccessStream type, building back a blob object from it with window.MSApp.createBlobFromRandomAccessStream and getting an URL access on the blob with URL.createObjectURL to change the source of the HTML image element.
Ok, now that you know how this works, let’s build the last component to use it in our project.
Create a new folder under “ /js” named “deForm” and create the JS file named “deForm.js” into it. Copy/paste this code:
(function () {
"use strict";
var Filters = WinJS.Class.define(
// The constructor needs an HTML Image Element
// and a function that will be callbacked at the
// end of the filter processing
function (image, callback) {
this._currentImageToManipulate = image;
this._callbackFunction = callback;
// We're using again a working canvas
// But this time it's in order to have access to the memory stream
// attached to the canvas blob element
var canvas = document.createElement("canvas");
canvas.width = this._currentImageToManipulate.width;
canvas.height = this._currentImageToManipulate.height;
var tempContext = canvas.getContext("2d");
// Drawing the source image into the target canvas
tempContext.drawImage(this._currentImageToManipulate, 0, 0,
this._currentImageToManipulate.width,
this._currentImageToManipulate.height);
var blob = canvas.msToBlob();
// The WinRT component will only understand streams and doesn't know
//about the HTML5 canvas element
this._stream = blob.msDetachStream();
},
{
_applyEffect: function (effects) {
var imageManipulator = new DeForm.ImageManipulator(this._stream);
for (var i = 0; i < effects.length; i++) {
imageManipulator.addEffectDescription(effects[i]);
}
// The filters are really applied after this line of code
imageManipulator.manipulate(false);
// Now we need to transform the stream manipulated by the C++ component into something
// useable in HTML5. First thing is to use WinRT to create an InMemoryRandonAccessStream
var inMemoryRandomAccessStream = new Windows.Storage.Streams.InMemoryRandomAccessStream();
// Then we're saving the stream sent back by the C++ component into it
imageManipulator.saveToStream(inMemoryRandomAccessStream, DeForm.CompressionMode.png);
// From this memory stream, we now need to have an object useable in HTML.
// The blob is perfect for that
var blob = window.MSApp.createBlobFromRandomAccessStream("image/png",
inMemoryRandomAccessStream);
// createObjectURL supports the blob type
var url = URL.createObjectURL(blob);
this._callbackFunction(url);
},
applyBlurEffect: function () {
var blurEffect = new DeForm.GaussianBlurEffectDescription(5.0, DeForm.GaussianBlurOptimization.Quality,
DeForm.GaussianBlurBorderMode.Hard);
this._applyEffect([blurEffect]);
},
applyBWEffect: function () {
var BWEffect = new DeForm.ColorMatrixEffectDescription();
BWEffect.colorMatrix = [0.2125, 0.2125, 0.2125, 0.0,
0.7154, 0.7154, 0.7154, 0.0,
0.0721, 0.0721, 0.0721, 0.0,
0.0, 0.0, 0.0, 1.0,
0.0, 0.0, 0.0, 0.0];
this._applyEffect([BWEffect]);
},
applySepiaEffect: function () {
var BWEffect = new DeForm.ColorMatrixEffectDescription();
BWEffect.colorMatrix = [0.2125, 0.2125, 0.2125, 0.0,
0.7154, 0.7154, 0.7154, 0.0,
0.0721, 0.0721, 0.0721, 0.0,
0.0, 0.0, 0.0, 1.0,
0.0, 0.0, 0.0, 0.0];
var SepiaEffect = new DeForm.ColorMatrixEffectDescription();
SepiaEffect.colorMatrix = [0.90, 0.0, 0.0, 0.0,
0.0, 0.70, 0.0, 0.0,
0.0, 0.0, 0.30, 0.0,
0.0, 0.0, 0.0, 1.0,
0.0, 0.0, 0.0, 0.0];
var effects = [BWEffect, SepiaEffect];
this._applyEffect(effects);
}
}
);
WinJS.Namespace.define("DeFormFilters", { Filters: Filters });
} ());
Again this code should be commented enough to help you definitely understanding the logic applied. Add a reference to this new JS file into “default.html”:
<script src="/js/deForm/deForm.js"></script>
And instantiate this component in the displayFlyout function like done before for the 2 other components:
deFormFilters = new DeFormFilters.Filters(currentImageToManipulate, updateFlipViewItem);
Before being able to run the project, you will maybe have to switch the current active solution platform from “Any CPU” to x86 or x64 depending on your machine. For that, right-click on the project, choose “Properties”. Click on the “Configuration Manager” button and switch away from “Any CPU”:
You should now be able to run the C++ DeForm effects on your image taken with your Camera, living inside the WinJS FlipView control!
Benchmarks
Ok, to bring you an idea of the performance interest of going from mono-threaded to web workers and finally to GPU shaders, here is a table summarizing some benchmarks I’ve done:
| Configuration | B/W with EaselJS | B/W with Workers | B/W with DeForm | Blur with EaselJS | Blur with Workers | Blur with DeForm |
| 4 CPUs - 640x480 | 241 ms | 62 ms | 58 ms | 718 ms | 331 ms | 58 ms |
| 4 CPUs - 3088x1737 | 4005 ms | 640 ms | 438 ms | 12842 ms | 5200 ms | 469 ms |
| 2 CPUs - 1280x720 | 5547 ms | 1013 ms | 806 ms | 16414 ms | 7340 ms | 711 ms |
We can then see some real benefits starting with the web workers even on 2 CPUs machines. For the GPU shaders, the difference become really visible when the processing request much more complexity & time. There are small differences on the black & white effect between workers and GPU Shaders but there is a huge difference for the blur effect.
As promised, you can download the final Visual Studio solution matching this complete series here: download the ModernUIFunCamera Tutorial 4 final solution
This is the end of this series around WinJS & WinRT. I hope you had fun reading those 4 tutorials and that it brought you some great ideas for some greats apps that will land into the Windows Store! 
David
Comments
Anonymous
March 21, 2013
Excellent post about JS vs. C++ performance in Windows 8 apps. This was very inspiring for my master thesis, thank you!Anonymous
November 26, 2013
Excellent tutorials, question: is there a way to make the camera full screen size??? like native camera app