Tutorial Series: using WinJS & WinRT to build a fun HTML5 Camera Application for Windows 8 (3/4)
In this third tutorial, we will use the WinJS FlipView control to navigate between our live preview UI and the various photos & videos taken during the life of the application. For that, we will need a specific template for each case: camera live preview, photo & video. To do that, you need to use a JS function as a template renderer that will return the appropriate piece of HTML for each case. We will also see that the FlipView is doing some virtualization by default which could lead to unexpected behavior if you’re not prepared for that.
As a reminder this tutorial is part of this series:
1 – Accessing to the Camera stream and writing it on disk
2 – Working on the layout, adding video recording support and providing feedback to the user with CSS3 animations
3 – Using the FlipView to navigate through the live preview and the recorded files
4 – Manipulating the image to add some effects via the canvas tag with EaselJS or WebWorkers and via GPU Shaders with a WinRT component
As usual, you’ll be able to download the Visual Studio solution matching the end of this third tutorial at the end of this article.
To have a better idea of what we’re going to built in this tutorial, here is a small video demonstrating of what you’ll get at the end:
Download Video: MP4, WebM, HTML5 Video Player by VideoJS
Step 1: preparing the HTML & CSS for the templates and the FlipView control
If you don’t know the FlipView control yet and how to use it, please have a look to the MSDN documentation:
Quickstart: Adding a FlipView (Windows Store apps using JavaScript and HTML) .
And to even better understand this article, please have a look the Windows SDK sample: FlipView control sample
In our case, we need 3 different templates for our FlipView control: a live preview, a photo and a video template.
As the first element of the list bind to the FlipView will be the live preview control, let’s start by this one. We just need to take the HTML built in the 2 previous tutorials and turn it into a WinJS Template. For that, we just have to copy/paste it into a new div marked with the data-win-control="WinJS.Binding.Template" attribute.
Here is then our first template:
<div id="applicationTemplate" data-win-control="WinJS.Binding.Template">
<div id="application">
<div id="control-bar">
<div id="input-switch">
<div class="top-switch">
<input id="still" type="radio" name="inputSwitch" value="still" class="switch camera"
checked="checked" />
</div>
<div class="bottom-switch">
<input id="vid" type="radio" name="inputSwitch" value="video" class="switch video" />
</div>
</div>
</div>
<div id="overlay">
<div id="capture-status" class="win-contentTitle">
Tap screen to capture media
</div>
</div>
<video id="live-preview">
</video>
</div>
</div>
Note: But to make a good template, it’s normally better to avoid using IDs the inner HTML as this block of HTML will be repeated by the templating engine. In our case, this template will be only instantiated once as we’ve got only 1 live preview control. So, we won’t run into issues in our very specific case.
Let’s now review the 2 templates that will be used (and repeated) for the photos and videos you’ll recorded:
<div id="imageTemplate" data-win-control="WinJS.Binding.Template">
<div class="media">
<img data-win-bind="src: url" src="" alt="" />
</div>
</div>
<div id="videoTemplate" data-win-control="WinJS.Binding.Template">
<div class="media">
<video data-win-bind="src: url" src="">
</video>
<button class="play-button">
</button>
</div>
</div>
The imageTemplate is more or less the same template we’ve used for the slide effect in the previous tutorial. The videoTemplate one is just using the video tag instead of the image one and add a layer on top of it with a play button. This play button is using the code xE102 of the Segoe UI Symbol font. Both templates use the binding engine of WinJS for their source properties.
Now that we’ve got our templates ready to use, let’s insert the FlipView control itself:
<div id="flip-view" data-win-control="WinJS.UI.FlipView">
</div>
Finally, insert this new piece of CSS to style our various containers:
/* FlipView CSS part */
.win-template
{
width: 100%;
height: 100%;
}
#flip-view
{
width: 100%;
height: 100%;
}
#flip-view *
{
outline: none;
}
#flip-view .media
{
display: -ms-grid;
-ms-grid-columns: 1fr;
-ms-grid-rows: 1fr;
background-color: #000;
width: 100%;
height: 100%;
}
#flip-view .media img
{
width: auto;
height: 100%;
-ms-grid-column-align: center;
-ms-grid-row-align: center;
background-color: #000;
}
#flip-view .media video
{
-ms-grid-column: 1;
-ms-grid-row: 1;
-ms-grid-column-align: center;
-ms-grid-row-align: center;
width: 100%;
height: 100%;
background-color: #000;
}
#flip-view .media .play-button
{
-ms-grid-column: 1;
-ms-grid-row: 1;
-ms-grid-column-align: center;
-ms-grid-row-align: center;
border-radius: 50%;
border: 2px solid #fff;
font-family: "Segoe UI Symbol";
font-size: 20pt;
line-height: 24pt;
color: #fff;
background-color: rgba(0, 0, 0, 0.7);
width: 48px;
height: 48px;
}
You should now be comfortable with this CSS. We’re using again CSS Grid to center our elements.
If you’re running the application now, you will simply have a black screen as we don’t have written the JavaScript code to handle our FlipView yet.
Step 2: inserting the live preview control as the first element bind to the FlipView
We need to instantiate a new binding list that will serve our FlipView control. This binding list will have 3 types of objects inside. Based on this type, our rendering function will choose the proper template to render and launch the appropriate code to initialize the various events handling and so on.
Replace the current init function by this one:
function init() {
screensList = new WinJS.Binding.List();
screensList.push({ "type": "application" });
flipView = document.getElementById("flip-view").winControl;
WinJS.UI.setOptions(flipView, {
itemDataSource: screensList.dataSource,
itemTemplate: WinJS.Utilities.markSupportedForProcessing(flipViewItemTemplateRendererFunction)
});
}
We’re pushing a dummy JS object with just the type property set to “application” into the binding list. This is just to help us knowing that associated to this object, we should render the live preview control. Then, we’re setting the FlipView’s options programmatically (data source and item template renderer) thanks to the setOptions functions. We now need the JS function acting as the item renderer:
function flipViewItemTemplateRendererFunction(itemPromise) {
return itemPromise.then(function (currentItem) {
var templateSelector = "#" + currentItem.data.type + "Template";
var controlTemplate = document.querySelector(templateSelector).winControl;
var template = controlTemplate.renderItem(itemPromise);
template.renderComplete = template.renderComplete.then(function (elem) {
switch (currentItem.data.type) {
case "application":
livePreview = elem.querySelector("#live-preview");
livePreview.addEventListener("click", capture);
console.log("starting the camera");
startCamera();
break;
}
});
return template.element;
})
}
This function will be called on each object inserted into the binding list. The first thing we’re doing is accessing to the type property of the current object bind for the currently processed item. We’re then selecting the associated template (applicationTemplate, imageTemplate or videoTemplate). We’re rendering and injecting them into the DOM by the renderItem function. Once this asynchronous operation is finished, we can associate some code to this new HTML generated. In the “application” case, we’re simply doing the same logic done in the previous tutorials: getting the video tag and rendering the live preview feed of the camera into it.
If you’re now pressing F5, you should visually obtain the exact result as the second tutorial. But we’re now living into a FlipView control! That’s a big difference. Aren’t you amazed? Ok, not yet I guess… But you’ll soon see the difference.
Step 3: inserting the photos & videos recorded into the FlipView
Like demonstrated in the video at the beginning at this article, if we slide from left to right, a new element will be displayed in the flipview. We then need to add items in the list bind to the control.
Let’s start by adding this little function:
function addItem(file, type, thumbnail) {
screensList.unshift({ "url": file, "type": type });
previewSlide(thumbnail);
}
It adds a new JS object to the list with the path to the URL of the photo or video just recorded, the value type (image or video) to help the renderer function choosing the right template and the URL to a thumbnail (for the video in fact). We add them at the beginning of the list with the unshift function. Use the push function to add the new element at the end of the list.
We now need to call this function instead of calling the previewSlide() directly into takePhoto() and stopRecording() . We also need to tune a little bit the stopRecording function to add a call to createObjectURL() to map the video stream recorded to an URL.
You will then have these 2 functions:
function stopRecording() {
mediaCapture.stopRecordAsync().then(function () {
document.getElementById("capture-status").innerText = "Tap screen to capture media";
console.log("Video saved on disk");
var videoUrl = URL.createObjectURL(recordedFile, { oneTimeOnly: true });
recordedFile.getThumbnailAsync(Windows.Storage.FileProperties.ThumbnailMode.singleItem).then(function (thumbnail) {
var thumbnailUrl = URL.createObjectURL(thumbnail, { oneTimeOnly: true });
addItem(videoUrl, "video", thumbnailUrl);
});
});
}
function takePhoto() {
pics.createFileAsync("photo.jpg", Storage.CreationCollisionOption.generateUniqueName).then(function (file) {
var photoProperties = Windows.Media.MediaProperties.ImageEncodingProperties.createJpeg();
mediaCapture.capturePhotoToStorageFileAsync(photoProperties, file).then(function () {
console.log("Image saved on disk.");
var thumbnailUrl = URL.createObjectURL(file, { oneTimeOnly: true });
addItem(thumbnailUrl, "image", thumbnailUrl);
});
});
}
Finally, we need to handle the “video” case in the flipViewItemTemplateRendererFunction() function. Add this case in the function then:
case "video":
addPlayControls(elem);
break;
And you’ll need the code of addPlayControls() of course:
function addPlayControls(elements) {
var button = elements.querySelector("button");
var video = elements.querySelector(".media video");
function _play() {
video.play();
WinJS.UI.Animation.fadeOut(button);
}
video.addEventListener("click", _play);
button.addEventListener("click", _play);
video.addEventListener("ended", function () {
WinJS.UI.Animation.fadeIn(button);
});
}
This function simply binds some code to the play button displayed in front of the video. You’ll notice that we’re using this time the WinJS.UI.Animation framework to handle the fade in/fade out effects. We could have used also some pure CSS3 transitions or animations code for that. But the WinJS framework provides some easy & cool effects to code & use. It generates some CSS3 on the fly for you.
Press F5 to play with the application. It starts to be really cool to play with! But maybe you’ll face some issues because we haven’t handled the virtualization of the WinJS list controls properly yet. Let’s talk about that in the next step.
Step 4: handling properly the virtualization of the WinJS controls
To reproduce the problem, launch the application, take for instance 2 photos, 1 video then 3 photos again. Navigate to the first element of the list sliding from left to right a couple of times and navigate back to the video element. You should normally see the play button but no more thumbnail image, just a black screen.
So, what happened? It happened that the WinJS list controls (ListView & FlipView) are very well written and are using a virtualization mechanism by default for performance and memory reasons. Usually, it then loads and renders the elements by bunch of 3. This means also that it finishes by unloading some element once the view window is greater than 3.
In our case case, we’ve created the URL bind to the template controls using the oneTimeOnly boolean set to true. This means that once the template control is unloaded by the virtualization mechanism, the memory area dedicated to this blob URL could be garbage collected.
There is then 2 ways to fix this problem:
1 – change to boolean to false to keep our blob URL in memory and avoid then garbage collecting. This is an easy fix but not a good practice for memory usage if your application does a lot of calls to createObjectURL.
2 – call the createObjectURL only when the template control is rendered in the flipViewItemTemplateRendererFunction() function.
The second option means that you will have to read the stream back from the disk to pass it the createObjectURL function every time the virtualization process will instantiate the template control. This is more memory efficient but generate more I/O. The first option is more I/O efficient but cost more memory. This is the usual balance we’re used to as developers my friends!
In our case, our sample application is not meant to generate thousand of videos and if we’re killing and re-launching it, we’re starting every time from scratch. We’re not reloading previous videos/photos taken. Using the first option should be fine for this tutorial.
In conclusion, change all calls to the createObjectURL to set the boolean to false. Re-launch the application with F5 and check that this boolean modification has solved our problems.
Ok, let’s now fix some other potential problems. While navigating into the flipview, the camera live preview video stream could be paused by the system. So when you’ll navigate back to the camera template, the video could appear as frozen. To fix that, in the startCamera() function, add this line of code:
livePreview.addEventListener("pause", handlePause);
And here is the handler to use:
function handlePause() {
console.log("pause invoked...");
livePreview.play();
}
It’s a simple trick that just force a re-play if the system tries to pause the live preview video stream of the camera.
A very last thing that could be good to do also is releasing some resources each time we’re re-starting the camera. For that, insert this function:
function releaseMediaCapture() {
if (livePreview) {
livePreview.src = null;
livePreview = null;
}
mediaCapture = null;
console.log("releasemediacapture");
}
And call it just before this line:
livePreview = elem.querySelector("#live-preview");
In the rendering function.
You can download the final Visual Studio solution matching the 3 complete tutorials here: download the ModernUIFunCamera Tutorial 3 solution
See you in the last tutorial that will be dedicated to manipulating the pixels of our elements via the canvas element with the EaselJS filters, then with some Web Workers and finally by using a WinRT C++ component discussing directly with the GPU via shaders. So continue the series here: Manipulating the image to add some effects via the canvas tag with EaselJS or WebWorkers and via GPU Shaders with a WinRT component
David
Comments
Anonymous
July 28, 2013
Only updates needed for it to work as 8.1 application (pre-beta) is swapping the deprecated MediaControl logic for muting with the systemMediaTransportControls. So the code would look something like this: var systemMediaTransportControls; then in app.onactivated(), replace MediaControl and event handler with the Windows.Media.SystemMediaTransportControl like this: // Windows.Media.MediaControl.addEventListener("soundlevelchanged", soundLevelChangedHandler); systemMediaTransportControls = Windows.Media.SystemMediaTransportControls.getForCurrentView(); systemMediaTransportControls.onpropertychanged = onPropertyChanged; and the handler to this: function onPropertyChanged() { if ( systemMediaTransportControls.soundLevel == systemMediaTransportControls.muted ) { console.log("Camera is being cut because of a switching action."); } else { startCamera(); console.log("Restarting the camera for live preview."); } } Also, like in the previous tutorial, you need to add the click handler to overlay instead of livePreview since at least with the pre-beta bits, the click event isn't getting pushed down. So this change needed: function flipViewItemTemplateRendererFunction(itemPromise) { ... // livePreview.addEventListener("click", capture); var overlay = elem.querySelector("#overlay"); overlay.addEventListener("click", capture); And that should do it. Jumping into the final lesson next. Great job, David! Love your teaching style.Anonymous
July 19, 2014
Hey , Nice tutorial. Could you give me some advice though. For a windows phone 8.1 app using winjs how do I play songs using system media transport controls? Please could you help me out?