EtchMark Under the Hood: Building a Website that Handles Touch, Mouse, and Pen – and Device Shakes
EtchMark is a new take on the classic Etch-A-Sketch drawing toy, showcasing IE11’s improved support for touch and emerging Web standards (including Pointer Events and Device Orientation). In this post, we’ll walk through several features that you can easily add to your own sites to build an experience that feels smooth and natural with touch, mouse, pen, and keyboard – and even responds to device shaking.
Structure of the Demo
EtchMark enables you to draw anything you want on the screen using touch, mouse, pen, or the arrow keys. The drawing surface is an HTML5 canvas element that we update any time a knob turns. In benchmark mode, we use the requestAnimationFrame API, which provides a smooth 60 frames per second animation loop and longer battery life. The drop shadows for the knobs are created using SVG filters. IE11’s hardware acceleration moves much of this work to the GPU, which leads to a blazing fast experience. Check out the video below to see these features in action, and then we’ll dive in and see how it’s built.
EtchMark uses HTML5 canvas, requestAnimationFrame, SVG filters, Pointer Events and Device Orientation APIs to create a new take on a classic toy
Touch, Mouse, Keyboard, and Pen Using Pointer Events
Pointer Events enable you to build experiences that work equally well with mouse, keyboard, pen, and touch – all by coding against a single API. Pointer Events are supported across the full range of Windows devices and are coming soon to other browsers as well. The Pointer Events specification is now a Candidate Recommendation at the W3C, and IE11 supports an un-prefixed version of the standard.
To get started, the first thing we need to do is wire up our pointer events in Knob.js. First we check for the standard un-prefixed version, and if that check fails we fall back to the prefixed version needed to enable IE10 support. In the example below, hitTarget is a simple div containing the knob image, sized to be a bit bigger so the user has space to land her finger easily:
if (navigator.pointerEnabled)
{
this.hitTarget.addEventListener("pointerdown", pointerDown.bind(this));
this.hitTarget.addEventListener("pointerup", pointerUp.bind(this));
this.hitTarget.addEventListener("pointercancel", pointerCancel.bind(this));
this.hitTarget.addEventListener("pointermove", pointerMove.bind(this));
}
else if (navigator.msPointerEnabled)
{
this.hitTarget.addEventListener("MSPointerDown", pointerDown.bind(this));
this.hitTarget.addEventListener("MSPointerUp", pointerUp.bind(this));
this.hitTarget.addEventListener("MSPointerCancel", pointerCancel.bind(this));
this.hitTarget.addEventListener("MSPointerMove", pointerMove.bind(this));
}
Similarly, we add the correct fallback for setPointerCapture to Element.prototype to ensure it also works on IE10:
Element.prototype.setPointerCapture = Element.prototype.setPointerCapture || Element.prototype.msSetPointerCapture;
Next let’s handle our pointerDown event. The first thing we do is call setPointerCapture on this.hitTarget. We want to capture the pointer so that all subsequent pointer events are handled by this element. It also ensures that other elements don’t fire events even if the pointer moves into their bounds. Without this, we’d run into problems when the user’s finger is on the edge of the image and the containing div: sometimes the image would get the pointer event and other times the div would. This would result in a jagged experience where the knob jumps around. Capturing the pointer is an easy way to solve this.
Pointer capture also works nicely when the user puts her finger down on the knob and then gradually moves off the hit target while continuing to rotate. Even if the finger isn’t lifted until it has moved several inches off the hit target, the rotation still feels smooth and natural.
The last thing to note about setPointerCapture is that we pass in the event’s pointerId property. This enables us to support multiple pointers, meaning the user can use a finger on each knob simultaneously without interfering with the other knob’s events. The support for multiple knobs means that when the user rotates both knobs at once, she gets a freeform drawing rather than only seeing vertical and horizontal lines.
We also want to set two flags on this, which points to our Knob object (the flags are per-knob):
- pointerEventInProgress - tells us whether or not the pointer is down
- firstContact - tells us if the user has just put their finger down
function pointerDown(evt)
{
this.hitTarget.setPointerCapture(evt.pointerId);
this.pointerEventInProgress = true;
this.firstContact = true;
}
Finally we want to reset the pointerEventInProgress flag when the user lifts up her finger (or mouse/pen):
function pointerUp(evt)
{
this.pointerEventInProgress = false;
}
function pointerCancel(evt)
{
this.pointerEventInProgress = false;
}
PointerCancel can occur in two different ways. The first is when the system has determined that a pointer is unlikely to continue producing events (for example, due to a hardware event). The event also fires if the pointerDown event has already occurred and then the pointer is used to manipulate the page viewport (for example by panning or zooming). For completeness it is always recommended to implement both pointerUp and pointerCancel.
With up, down, and cancel events wired up, we’re now ready to implement support for pointerMove. We use the firstContact flag so that we don’t over-rotate when the user first touches her finger down. After firstContact is cleared, we just calculate the movement deltas of the finger. We use trigonometry to turn our start and end co-ordinates into a rotation angle, which we then pass along to our drawing function:
function pointerMove(evt)
{
//centerX and centerY are the centers of the hit target (div containing the knob)
evt.x -= this.centerX;
evt.y -= this.centerY;
if (this.pointerEventInProgress)
{
//Trigonometry calculations to figure out rotation angle
var startXDiff = this.pointerEventInitialX - this.centerX;
var startYDiff = this.pointerEventInitialY - this.centerY;
var endXDiff = evt.x - this.centerX;
var endYDiff = evt.y - this.centerY;
var s1 = startYDiff / startXDiff;
var s2 = endYDiff / endXDiff;
var smoothnessFactor = 2;
var rotationAngle = -Math.atan((s1 - s2) / (1 + s1 * s2)) / smoothnessFactor;
if (!isNaN(rotationAngle) && rotationAngle !== 0 && !this.firstContact)
{
//it’s a real rotation value, so rotate the knob and draw to the screen
this.doRotate({ rotation: rotationAngle, nonGesture: true });
}
//current x and y values become initial x and y values for the next event
this.pointerEventInitialX = evt.x;
this.pointerEventInitialY = evt.y;
this.firstContact = false;
}
}
By implementing four simple event handlers, we have now created a touch experience that feels natural and sticks to your finger. It supports multiple pointers and enables the user to manipulate both knobs simultaneously to produce a freeform drawing. Best of all, because we used Pointer Events, the same code also works for mouse, pen, and keyboard.
Getting more fingers in the game: Adding Gesture Support
The Pointer Events code that we wrote above works great if the user rotates the knob using one finger, but what if she rotates it using two fingers? We had to use trigonometry to calculate the rotation angle, and calculating the correct angle with a second moving finger gets even more complex. Rather than trying to write this complex code ourselves, we take advantage of IE11’s MSGesture support.
if (window.MSGesture)
{
var gesture = new MSGesture();
gesture.target = this.hitTarget;
this.hitTarget.addEventListener("MSGestureChange", handleGesture.bind(this));
this.hitTarget.addEventListener("MSPointerDown", function (evt)
{
// adds the current mouse, pen, or touch contact for gesture recognition
gesture.addPointer(evt.pointerId);
});
}
With the events wired up, we can now handle gesture events:
function handleGesture(evt)
{
if (evt.rotation !== 0)
{
//evt.nonGesture is a flag we defined in the pointerMove method above.
//It will be true when we’re handling a pointer event, and false when
//we’re handling an MSGestureChange event
if (!evt.nonGesture)
{
//set to false if we got here via Gesture so flag is in correct state
this.pointerEventInProgress = false;
}
var angleInDegrees = evt.rotation * 180 / Math.PI;
//rotate the knob visually
this.rotate(angleInDegrees);
//draw based on how much we rotated
this.imageSketcher.draw(this.elementName, angleInDegrees);
}
}
As you can see, MSGesture gives us a simple rotation property which represents the angle in radians, so we don’t have to do all the math ourselves manually. This now gives us support for two-finger rotation that feels natural and sticks to your finger.
Device Motion: Adding a Little Shake
IE11 supports the W3C DeviceOrientation Event Specification, which enables us to access information about a device’s physical orientation and movement. When a device is being moved or rotated (or more accurately, accelerated), the devicemotion event is fired at the window and provides acceleration (both with and without the effects of gravitational acceleration on the device, expressed in meters/second2) in the x, y, and z axis. It also provides the rate of change in the alpha, beta, and gamma rotation angles in degrees/second.
In this case, we want to erase the screen anytime the user shakes the device. To do this, the first thing we do is wire up the devicemotion event (in this case we’re using jQuery):
$(window).on("devicemotion", detectShaking);
Next, we detect if the user has moved the device in any direction with an acceleration greater than our threshold value. Because we need to detect shaking, we have a counter to ensure that there are two such fast movements in a row. If we detect two fast movements, we erase the screen:
var nAccelerationsInARow = 0;
var detectShaking = function (evt)
{
var accl = evt.originalEvent.acceleration;
var threshold = 6;
if (accl.x > threshold || accl.y > threshold || accl.z > threshold)
{
nAccelerationsInARow++;
if (nAccelerationsInARow > 1)
{
eraseScreen();
nAccelerationsInARow = 0;
}
}
else
{
nAccelerationsInARow = 0;
}
}
For more information on device orientation and motion, please see this post on the IE Blog.
Orientation Lock
IE11 also introduces support for the Screen Orientation API and features like Orientation Lock. Since EtchMark is also a performance benchmark, we want to keep our canvas size the same across screen resolutions so that we’re doing the same amount of work on every device. This can make things really tight on smaller screens, particularly in portrait mode. To enable the best experience, we simply lock the orientation to landscape:
window.screen.setOrientationLock("landscape");
This way, no matter which way the user rotates the device, she always sees it in landscape mode. You can also use screen.unlockOrientation to remove the orientation lock.
Looking Ahead
Interoperable and standards-based techniques like Pointer Events and device orientation events enable new exciting possibilities for your web sites. IE11’s excellent touch support delivers a smooth stick-to-your-finger experience and interactivity. You can go even further with IE11 and MSGesture, making scenarios like calculating two-finger rotation angles as simple as accessing a property. Give these techniques a try on your own site, and we look forward to your feedback.
Jon Aneja
Program Manager, Internet Explorer
Comments
Anonymous
November 21, 2013
When will you stop pushing pointerevents and get real? No browser other than IE supports this.Anonymous
November 21, 2013
The comment has been removedAnonymous
November 21, 2013
Why don't you go complain to them instead of b*tching here? @ieblog Thanks for the informative post.Anonymous
November 22, 2013
Wohooo, new IE for new XBox! Remote debugging would be really nice, as "To develop and debug your website for Internet Explorer for Xbox, you can use Internet Explorer 10 on Windows 8." (msdn.microsoft.com/.../dn532261%28v=vs.85%29.aspx) is useless to debug IE10 for XBox One specific performance and user interaction issues. Also, a script that behaves nicely across all available IE9-11 on Windows 7/8/8.1 combinations runs away on IEXBO, so the provided means of debugging might be completely useless and misleading. If the timeline for bringing JIT to IEXBO is not overly top secret, I'd appreaciate If you'd share soon. Right now some computational tasks are about twice as fast in webkit, blink or gecko on previous-gen phones (half the number of CPU cores and signifficantly slower RAM). And we have to somehow explain that to our IEXBO users, as they otherwise might think our app is rubbish or even broken. Or lock them out. Please don't make planning the future too difficult for us developers. We've already enough worries, like that we have to support mouse, touch and pointer events, but Apple doesn't. It doesn't really matter that I like Pointer Events the best (especially the freedom of choice provided by pointer capture as opposed to Touch Events).Anonymous
November 22, 2013
Will the F12 developer tool Emulation feature ever get fixed? It's completely broken when I use Windows Phone mode. I get residual "graphics droppings" every time I change a resolution, it shows up as multiple layered scrollbars. When I resize the window a little, the emulation viewport suddenly expands to 100% of the window. The emulation tool in IE10 was much better.Anonymous
November 24, 2013
Please go on Channel 9 and make a presentation of IE11. I would like to know more about current Javascript 6 support.Anonymous
November 25, 2013
The comment has been removedAnonymous
November 28, 2013
Really wish you guys would post clean code to this blog and follow modern JavaScript coding standards! (no one uses C style coding conventions outside of C!) Here's your first sample cleaned up: var el = this.hitTarget; if(navigator.pointerEnabled){ el.addEventListener("pointerdown", pointerDown.bind(this)); el.addEventListener("pointerup", pointerUp.bind(this)); el.addEventListener("pointercancel", pointerCancel.bind(this)); el.addEventListener("pointermove", pointerMove.bind(this)); } else if(navigator.msPointerEnabled){ el.addEventListener("MSPointerDown", pointerDown.bind(this)); el.addEventListener("MSPointerUp", pointerUp.bind(this)); el.addEventListener("MSPointerCancel", pointerCancel.bind(this)); el.addEventListener("MSPointerMove", pointerMove.bind(this)); }Anonymous
November 30, 2013
Hi, please stop adding features that we'll need in next ages and focus missing ones for past ages. While upgrading IE to standard based you have accidently removed (weirdo) XSLT and XPath support. Currently there is no way to perform an xslt or executing xpath query. For example you have added to DOMParser which is great but how on earth have you forget XSLTProcessor ? I cannot use native dom object with Msxml2.XSLTemplate activex class. For xpath example see ; stackoverflow.com/.../xpath-in-internet-explorer-10-gone Our customers also is your customers and currently we are telling them to "do not use IE"Anonymous
November 30, 2013
@ed: Please post a link to modern JavaScript coding standards. Recommendations, but not standards, are easy to find from Google, jquery, ........Anonymous
November 30, 2013
The comment has been removedAnonymous
December 03, 2013
The comment has been removedAnonymous
December 04, 2013
IE 11 crashes on several GitHub pages and also the customized tracking protection list is broken. I have it set to manually block content and to show content that is at least on 3 pages... it shows the facebook tracker and a sourceforge tracker, and always keeps saying "not set, allow" even if i disallow and press ok and reopen it... it is back on "not set" again.... also it never refreshes and never shows new pages. why doesn't microsoft open a bug tracker / submission page like all other vendors? i have found lots of bugs in IE11. and even in IE10 that never got fixed.