HTML5 Gaming: benchmarking your sprites animations to target all devices & browsers
While meeting some game studios, I often have the same question coming over and over: if I’m writing/porting my game in HTML5, will it run well on the various targeted devices? Will it be playable or will the gameplay suffer too much? To answer to that question, I often use my own experience based on what I know and what worked well during my own tests. But I also had the feeling it wasn’t enough to provide some good advices. In the meantime, there were some obvious facts. For instance, we all know that mobile devices can’t animate as many sprites as desktop PC and preserve 60 fps. We know also that combining SVG and Canvas is a good idea to write games that scale across devices but this could also impact the performance. Moreover, even if GPU and hardware acceleration is available on mobile, their hardware architectures differ a lot from the PC and this impacts also a lot the performances. There is dozen of scenarios like that to address and to be aware of while writing HTML5 games for mobiles. But in which proportions?
With my friend David Catuhe, we then decided to measure these various scenarios and build a benchmark framework to have a better idea on what to pay attention for. It’s named the HTML5 Potatoes Gaming Bench framework. The concept is then to help you benchmarking your targeted platforms and to obtain indicators for your future games: number of simultaneous sprites supported, SVG & Canvas composition performance, usage of videos, etc. It’s a tool we’d like to provide you to help you benchmarking your own scenarios for your games.
David has published here an article explaining how we’ve built our benchmark framework: Benchmarking a HTML5 game: HTML5 Potatoes Gaming Bench . It took us time to validate it against tools like the ones that ship wit the Windows Performance Toolkit for instance. But we won’t dig into that part here. But this is an important article to read as we needed to rely on the stability of the framework for all the future benchmarks we will build on top of it.
This article will illustrate how to use this tool and will concentrate on the number of sprites you can use in your game to maintain a good frame rate on all platforms. We will also discover interesting details on how GPU are actually being used during these scenarios.
The questions I was asking to myself before benchmarking
During my various discussions with some games developers, we were often wondering ourselves a list of recurring questions:
- What the FPS of the benchmarked machine/browser with 200, 1000, 5000 sprites?
- What is the optimum number of sprites the benchmarked device can display at 30 fps and 60 fps?
- What is the impact of the resolution of the canvas used to draw the sprites? Can I use a canvas of 800x480 only or can I scale up to 1920x1080? If so, what’s the performance cost?
- What is the performance cost of using hardware scaling (i.e. setting a style.width & style.height higher than the canvas size itself)?
I was wondering also if the performance was due to GPU or CPU limited scenarios. The idea is to know if the CPU is waiting for the GPU to do its job or the opposite.
To answer all these questions, I’ve built some specific benchmarks to measure the scenario.
The core used by all benchmarks in this article
To animate my sprites, I’ve re-used some of my existing assets like these 2 articles:
- HTML5 Gaming: animating sprites in Canvas with EaselJS
- HTML5 Gaming: building the core objects & handling collisions with EaselJS
We will then use the famous EaselJS framework from the CreateJS suite as I know it runs everywhere and has already been optimized for performance.
Please read the main logic exposed in this embedded demo available on JSFiddle:
We’re going to re-use the very same logic inside our benchmark framework.
The various benchmarks are described like the simplified code below to be able to be integrated in the HTML5 Potatoes Gaming Bench framework:
(function () {
"use strict";
var canvas, context, stage, screen_width, screen_height, Monsters, contentManager;
var spritesNumber, displayBench, currentWorkBench, dynamicSpritesNumberLabel;
var shadowsEnabled = false;
var useRAF = false;
var init = function (workbench, spritesNumParam, shadowsEnabledParam, canvasSizeXParam,
canvasSizeYParam, then) {
// Initialization logic, creating the canvas, the EaselJS Stage, etc.
};
var buildFixNumberOfSprites = function () {
// Our Monsters collection
Monsters = new Array();
for (var xMonsters = 0; xMonsters < spritesNumber; xMonsters++) {
var newRandomMonster = new Monster(contentManager, shadowsEnabled, screen_width, screen_height);
Monsters.push(newRandomMonster);
stage.addChild(newRandomMonster);
}
createjs.Ticker.addEventListener("tick", tick);
// Best Framerate targeted (60 FPS)
createjs.Ticker.useRAF = useRAF;
createjs.Ticker.setFPS(60);
// Waiting 1s before doing taking first FPS due to some warm up needed on most machines
setTimeout(displayBench, 1000);
};
// tick function called-back by EaselJS
var tick = function () {
// looping inside the Monsters collection
for (var monster in Monsters) {
var m = Monsters[monster];
// Calling explicitly each tick method
// to launch the update logic of each monster
m.tick();
}
// update the stage:
stage.update();
}
var stopEaselJSBench = function () {
// cleaning things
}
var easelJSSpritesBench_200_NoShadow_640_480 = new POTATOES.GamingBench.Bench("Drawing 200 sprites
(640x480)", "https://blogs.msdn.com/davrous",
function (workbench) { // Init
init(workbench, 200, false, 640, 480, this.onInitCompleted);
}, function () { }, function (workbench) { // End
stopEaselJSBench();
});
POTATOES.GamingBench.registerBench(easelJSSpritesBench_200_NoShadow_640_480);
})();
You will need to build equivalent code to build your own benchmarks that will be monitored by the framework. The key point is to add your benchmark definition into the monitored collection via the registerBench function.
To validate my layout (but not the performance) on the various available browsers and devices, I’ve used Browser Stack. Thanks to Modern.IE, you can obtain a 3 months trial. It lets you test and validate your websites layouts on IE from 6 to 10, Firefox, Chrome, Opera, Android & iOS devices. A very useful tool you should have a look to.
Most of the below shared results were done on an Asus Ultrabook UX31A (Zenbook Touch) running Windows 8 Pro on a Core i5 integrating an HD4000 GPU from Intel. But I’ve done also some tests on the Nokia Lumia 920, on an iPad 2 under iOS 6.0, Surface RT and Xbox 360. I’m letting you testing the shared benchmarks on your own devices.
Average FPS with 200, 1000 & 5000 sprites
Let’s start by testing some arbitrary numbers of sprites to check how your device/browser will handle that.
Just below, inside an iframe, I’ve inserted a series of benchmarks that will run some sprints of 20 seconds to display in a 640x480 canvas: 200 sprites without shadow, 200 sprites with shadows enabled, 1000 sprites & 5000 sprites without shadow.
Just press the “launch prefixed sprites numbers series” button and it will run all of them.
You can also run this benchmark in full screen or in a separate window via this link: prefixed sprites numbers series benchmark.
At the end, you will have a summary of all benchmarks with a score. The score is simply the number of frames that your device was able to render in 20 seconds. The best logical score is then 1200. As you can render 60 frames per seconds * 20 seconds = 1200 frames. We’re using requestAnimationFrame when available. So it could sometimes occurs where the tick is calling us back just a bit above 16ms, that why you could have some cases where you will have 1201 like in the following screenshot:
If you’re clicking on the average FPS computed, it will display the graph of all instant measured FPS during the complete duration of the benchmark. We’re using the famous d3.js framework for that which generate some SVG.
Press the “Back” button to go back to the summary screen or simply touch the graph also.
Some results and their analysis
I’ve first run this benchmark on my Windows 8 machine with IE10, Chrome 26, Firefox 20 all with hardware acceleration enabled. The good news is that all the modern desktop browsers have some really good hardware acceleration layers in place. IE10 seems to have the best overall score (3030 on my machine) but can’t handle the shadows as well as Chrome (average 27 fps with 200 sprites vs 12 for IE10). Chrome has an overall score of 2870. It can’t maintain 60 fps with 1000 sprites on my screen whereas IE10 and Firefox don’t have any problem to achieve that. Firefox 20 has an overall score of 2913. The most difficult test for it is to handle the 200 sprites with shadows (average of 6 fps vs 12 and 27 for IE10 and Chrome). Except that, it has the exact same scores as IE10.
So, what should we conclude at this stage? The first thing is that you absolutely shouldn’t tell to your users: “please use this browser instead” as we’re talking here about building games for the web and all platforms. The users shouldn’t adapt their usages to your code, your code should adapt itself to the usages of your users on the web. This is a different story if you’re building a game only targeting the Windows Store Apps. In this case, you just have to benchmark IE10 to determine your assets limits.
But if we think about the web in general, we see that all browsers on can easily handle between 500 and 1000 sprites at 60 fps on my desktop machine. We absolutely shouldn’t use shadows if we want to keep a good frame rate. For an HTML5 desktop game targeting integrated Intel GPU like mine, you can use a bit less than 1000 animated sprites on screen without any issue. Please keep in mind that this test doesn’t take into account some collisions tests and/or the impact of a physics engine. But it already gives some interesting data.
On the mobile side, I’ve run the same series on my Nokia Lumia 920 (Windows Phone 8 is embedding IE10). It can maintain an average of 36 fps for 200 sprites without shadow. Activating shadows is a no-go as we’re falling at 1 fps.
So, we already have some interesting information to digest. The same rendering and JavaScript engine (IE10) is able to display 1000 sprites at 60 fps on a desktop PC using an integrated GPU but 200 sprites is already too much to handle for low-end hardware like the current ARM architectures. There is a 5X to 10X performance difference between a mobile and a desktop device. So the next step now is to find what is the optimum number of sprites to maintain 60 fps (for a good desktop experience) and also 30 fps (for a relatively good mobile experience).
Optimum number of sprites at 30 and 60 fps
To help you finding the best optimal lower number of sprites that will keep a 30 or 60 fps target, I’ve built the following benchmark series:
You can also run this benchmark in full screen or in a separate window via this link: launch 30 and 60 fps target series.
Some results and their analysis
This time the score provided at the end is the number of optimal sprites to maintain 30 or 60 fps. On IE10 on my Windows 8 Asus machine, the HD4000 is able to display 3750+ sprites at 30 fps and 1600 sprites at 60 fps. IE10 on my Nokia Lumia 920 running Windows Phone 8, I’m able to display 286 sprites at 30 fps and 75 at 60 fps. On Surface RT, IE10 is able to display 370+ sprites at 30 fps and 100+ sprites at 60 fps.
We’re then falling from 1600 sprites on desktop to 75 on mobile!
I’ve even gone further: I’ve benchmarked the version of IE available on the Xbox 360. Indeed, we can imagine building a HTML5 game running inside the Xbox 360 browser. I’ve done it with my game for instance. You can test it here: https://aka.ms/platformer . You can play to the game with the gamepad and we’ve got 60+ FPS! You can also play to the game on mobile/table touch devices thanks to Hand.JS (more details here: Creating an universal virtual touch joystick working for all Touch models thanks to Hand.JS).
Let benchmark it with the 30/60 fps target series:
The Xbox 360 can display 171 sprites to maintain 30 fps and 43 to maintain 60 fps.
So, using Internet Explorer as a base of comparison, Asus Zenbook > Surface RT > Nokia Lumia 920 > Xbox 360 to maintain 60 fps in HTML5 games with sprites animations.
I was also curious about changing another parameter to confirm some of my thoughts. I was convinced that the most importance piece of hardware to boost the performance in my HTML5 games was the GPU. Indeed, all modern browsers are now heavily using them to offload the job needed to be done for the layout engine. I needed then to have the same machine with 2 GPUs available. Everything else should remain the same (CPU, memory, hard drive, screen resolution, etc.). For that, I’ve been using a Sony Vaio Z13 that has an Intel Core i7 Sandy Bridge processor with an integrated HD3000 GPU plus a discrete nVidia GT330m GPU.
Using Intel HD3000, the Vaio is able to display 2900 sprites @30fps and 1100 sprites @60fps. Using the nVidia GT330m, the very same Vaio is able to display 5400 sprites @30fps and 2500 @60fps. We then have approximately a X2 performance boost by switching from the Intel GPU to the nVidia GPU. GPU really matters for HTML5 games even for 2D canvas games.
|
 |
Conclusion: you have 3 options here to build a cross-platforms game running fine everywhere. You’re taking the lowest value (43 sprites here) and you will never put more that this number of sprites on screen to be sure to have 60 fps everywhere on Xbox 360, Windows Phone 8, Surface RT and Windows 8 machine running HD4000 GPUs. Of course, you need to run the same benchmark on your other targeted platforms: Android tablets & phone, iOS tablets & phone, and so on to find the optimal magic number.
Taking this first option is the easiest one but it’s a pity that desktop machines are under exploited. A second option is then to build 2 versions of the game (like 2 versions of your websites): 1 for mobile and 1 for phone. You will just adjust the graphical complexity based on the performance of each platform. Or you can also take the optimal number on mobile to target 30 fps and you will be sure to run @60 fps on desktop. The idea is then to target 30 fps on mobile/tablets and 60 on desktop.
Lastly, the third and last option is to build a game engine that will itself dynamically adjust the complexity of the graphical engine based on the performance detected. Something that some game studios are frequently doing on PC for instance. It requires more work for sure and you need also to decide the kind of assets that will be displayed or not in your game without impacting the global gameplay.
But today, doing this kind of benchmark is really critical if you’re targeting the mobile markets. Use a “mobile first” approach, otherwise you will get into trouble to optimize the performance for these lower GPUs.
Impact of the canvas resolution
This time we’re going to always display the same number of sprites (500) but we’re going to vary the resolution of the canvas: 320x200, 640x480, 1024x768 and 1920x1080. I was convinced on my side that increasing the resolution would lower the average FPS for sure, even just to display some animated sprites. Well, let’s check that:
You can also run this benchmark in full screen or in a separate window via this link: launch various canvas resolutions series .
Some results and their analysis
Well, look at the result in IE10 on my machine:
On my machine, the various resolutions had 0 impact on the average framerate! This is also confirmed on my mobile WP8 devices and on Xbox 360. Only Chrome has a lower average FPS only in 320x200 for an unknown reason. This doesn’t seem logical. But it’s probably because of a specific optimization I’m not aware of. Firefox and all other rendering/JavaScript engines I’ve been testing on several devices provide the same result: resolutions has no impact on the global performance in my scenario.
Please note also that this doesn’t mean you shouldn’t take care of the resolution of your canvas for your global performance. This just means that for sprites animations, and with this specific benchmark, the resolution has no impact. The GPU seems to take the load without any problem even on mobile platforms. I was very surprised by these first results so I’ve done further tests and benchmarks.
To double check that GPUs weren’t saturated, I had the following idea. As I’ve got a laptop machine embedding 2 GPUs, a Sony Vaio Z13. It has a nVidia GT330m and an Intel HD3000 integrated GPU. The Intel HD3000 contains up to 12 scalar 128-bit execution units where the nVidia GT330m contains 48 128-bit execution units.
There are also tools that help to check the load of the GPU. For instance, GPU-Z can provide you the GPU Load in realtime. I’ve then run the last benchmark series first on the nVidia GPU with GPU-Z opened on the “Sensors” tab to verify the GPU load: 15% at 320x200, 30% at 640x480, 25% at 1024x768, 55% at 1920x1080 (don’t ask me why it’s lower in 1024 than in 640, I really don’t know why ;)).
Then, I’ve re-run the same benchmark on the HD3000: 30% at 320x200, 40% at 640x480, 62% at 1024x768 & 100% at 1920x1080. Logically, this time, the average FPS drop from 60 fps in 320x200 –> 1024x768 to 55 fps in 1080p.
Being below 60 fps is because we’re falling into a GPU limited scenario. You can’t do anything else than lowering the rendering resolution and/or display less sprites, etc. By the way, I’ve done some testing and using “hardware scaling” like described in this article: Unleash the power of HTML 5 Canvas for gaming has almost no impact on the global average FPS as this operation seems to be done with no effort by today’s GPU even on mobile. This is then a real good option to have in mind. If the frame rate is too low due to a too high GPU load, try to render your canvas in a lower resolution and stretch it full screen using this hardware scaling method.
There could be cases where the lost of frames will be due to CPU limited scenarios. To monitor that, you need profiler tools like the one embedded in most recent browsers. You will then see which parts of your code you should try to work on to regain some FPS. You need to avoid that the GPU is waiting too long for the CPU to do its job. Some cases could be optimized using the HTML5 Web Workers for instance.
But I’ve found also some very specific cases where dropping under 60 fps was due to more complex reasons. Here is one of them: GPU-Z shows a GPU load around 50% but the FPS is below 60 fps. Using the F12 tool included in IE10 to profile the code shows the following result:
It will only show that you’re limited by the drawImage function. drawImage is a native function handled by the browser. It’s probably mixing CPU usage and GPU usage. You can’t optimize this part at this level in JavaScript. So, you just need to deal with it!
Going even further
You can have a look to the set of tools available in the Windows Performance Toolkit and read the following methodology: Measuring Browser Performance with the Windows Performance Tools.
You’re also probably wondering why we have such slight FPS drops even with a relatively constant 60 average fps:
With David Catuhe, we had a precise idea why. But again it’s better to be able to verify it. I’ve then used the following methodology:
- Create a blank Visual Studio 2012 Windows Store App project. Indeed, as Windows 8 is using IE10 to run Windows Store App in HTML5, I had just to copy/paste the code of one of the benchmarks and I had a Windows Store app ready to be analyzed by Visual Studio. :) You can download this test project here if you’d like: HTML5PotatoesModernApp.zip
- I’ve setup a unique bench displaying 1000 sprites and gone into “Analyze” –> “JavaScript Analysis” –> “UI Responsiveness” –> “Launch Startup Project”:
After some processing, you will obtain such results:
We can see that this is the GC which is responsible for the drop of some frames. I remember reading interesting details about GC in this article: Are We Fast Yet? associated to this other benchmark: HTML5-Benchmark.
Conclusion
You know what I really like about this whole story? Even with a high level layer like HTML5, understanding the targeted architectures will always help you to optimize your code for your HTML5 games.
This is really what you should keep in mind to build your games that should scale across all HTML5 compatible devices. It’s now possible to have a unique code base to run everywhere. But layout & JavaScript compatibilities is just a small part of the story. You need to know which devices you’re going to target, understand their limitations & GPU characteristics and finally benchmark all of them to take the appropriate design decisions for your games.
I really hope that this article and our benchmark framework will help you in your future HTML5 games. We will soon work on similar articles focused on other topics important for HTML5 games.