Partager via

Making the web “just work” with any input: Mouse, Touch, and Pointer Events

  • Article

We recently announced support for the Touch Event API in Internet Explorer for Windows Phone 8.1 Update as a part of our commitment to make the Web “just work” for our users. This means IE now supports all three major pointing input APIs: Mouse Events, Touch Events, and Pointer Events. Mobile browsing experience gets better, but at the same time developers start wondering which API they should use.

To help developers understand when to use which API, we wanted to share our updated guidance, as well as our forward-looking thoughts on where we think the web platform should go next, considering Pointer Events is nearing support in Firefox while Google recently decided not to implement the specification.

Comparing and contrasting models

We want developers to use the input technology that is right for them. We provide interoperable Touch Events for the broadest reach across browsers today and leading-edge Pointer Events for simpler coding patterns, more capabilities, more device support, easier responsive design, and best hardware acceleration. Here’s a breakdown of qualities and capabilities for each model.

  Mouse Events Touch Events Pointer Events
Supports mouse   1  
Supports single-touch 2    
Supports multi-touch      
Supports pen, Kinect, and other devices 2    
Provides over/out/enter/leave events and hover      
Asynchronous panning/zooming initiation for HW acceleration      
W3C specification   3 3
Usable cross-browser on mobile devices     4
Usable cross-browser on desktop devices     4

1 In at least some Android browsers, touch events may fire for mouse input. However, this mouse input cannot be differentiated from actual touch input when using touch events.

2 All browsers fire some mouse events for single-touch input in order to preserve compatibility with mouse-based Web pages. However, it’s not currently possible to differentiate touch input from actual mouse input when using touch events. Similarly, Kinect gesture input on IE for Xbox will also fire some mouse events for basic compatibility.

3 Pointer Events are currently a Candidate Recommendation on track for Recommendation soon. Touch Events is a Recommendation. In 2012, the Web Events Working Group resolved to discontinue work on Touch Events Version 2 and close the WG in favor of standardizing Pointer Events.

4 Full support for pointer events is available in IE and on the way in Firefox. Partial support (the touch-action API) is available in Chrome, Firefox, and Opera and is being considered by Safari WebKit. Through polyfills, like Google Polymer Pointer Events and Hand.JS, the pointer event model can be used across browsers.

Check out this HTML5 Rocks article that explains some of these differences and issues in further detail.

Performance

Performance is critical for input, especially touch. The reality is you’re just sliding your fingers across a sheet of glass—but we want it to feel like you’re moving the Web page. This is why we invested 3 years of engineering to re-architect IE’s touch pipeline to be quad-threaded and GPU accelerated (starting in IE10 and improving in IE11), and we designed pointer events to leverage these types of architectures which have since also started to appear in other modern browsers.

Panning & zooming are typically the most critical of touch interactions, accounting for more than 60% of all touch gestures on IE and Chrome for Windows. Just about every native app, Web app, or site uses panning and/or zooming. It’s a gesture that’s very revealing when it comes to input performance. If panning is slow to start (jumping to your finger after a delay) or choppy (dropping frames), then you notice and remember it’s just glass.

Touch Events require running synchronous JavaScript before starting a pan or zoom, which can cause significant performance issues on the order of 100s of milliseconds or more. For this reason, switching to Pointer Events may dramatically improve your touch experience:

A demo of possible pan & zoom perf impact from Touch Events (both on an Android LG Nexus 5 and Windows Phone HTC 8x).
Pointer Events enable hardware accelerated, asynchronous touch actions, which often improves performance.

In this basic example test page (configured with a 1000ms delay), developed by Rick Byers on the Chrome team, with touch events in IE we can see the first frame of panning doesn’t render until after the touch contact has moved for 1020ms and travelled 4.8cm, which is clearly perceivable by the user. Simply replacing Touch Event handlers with Pointer Event handlers reduces this latency to just 31ms and 0.3cm—a greater than 32x improvement.

Although the delay in this test page is artificially introduced, it was authored to demonstrate a fundamental performance issue that's common on the web. In practice, we've seen pages better and worse than the delay exhibited by the 1000ms value. This variable unbounded dependency on the browser's primary thread leads to unpredictable user experiences and limits progress towards the optimal 1ms response time. Microsoft Research’s Advanced Sciences researchers have demonstrated the incredible user impact of this type of latency reduction. We’re still on the road to 1ms pan/zoom response time and see Pointer Events as a crucial step towards this.

Pointer Events also have additional functionality not present in Touch Events, such as the over/out/enter/leave transition events developers have asked for, and additional functionality comes with some additional cost. But we’ve invested in reducing those costs. The added hit-testing required for these features typically amounts to a mere ~0.15ms cost, and we’ve got ideas on how we might improve that further should we start to see evidence of this impacting developers’ ability to achieve 60FPS performance.

Compatibility

In our post announcing Touch Events support, we mentioned issues on the desktop Web that prevent us from compatibly bringing this API to those devices. For example, if Touch Events are enabled on desktop then the following common coding pattern breaks mouse support:

In this incorrect pattern, crucial mouse event handling is not wired up on any browser that supports Touch Events (regardless of whether the device actually has a touchscreen). This causes major functionality issues like menus, drop downs, share icons, and more not correctly functioning with mouse on sites like Web MD, Macys, Onet, Pages Juanes, Globo, Samsung, Taobao, Huffington Post, and more.

To make both the user touch experience and developers’ life easier, we want to do the following:

  • Add Touch Event support to IE on Windows Phone (done)
  • Work with site developers to fix the issues mentioned above in hopes that one day we can enable the APIs in a responsible way (work in progress: watch out for emails or tweets from our outreach team and webcompat.com).
  • Bring Touch Events to all of our platforms to make the support matrix less confusing for developers (work in progress)
  • Contribute to the implementation of Pointer Events on other platforms (work in progress on Firefox and Chrome)
  • Actively engage with site and framework developers to hear their feedback and desired direction for the web platform
  • Share our internal design docs, architectural diagrams, testing methodologies, and our code to other browsers (work in progress)

Many developers will find it easiest to use Pointer Events with a polyfill for other browsers, like Google Polymer Pointer Events or Hand.JS. But if you choose to use Touch Events, just be sure to also provide proper mouse support and never assume the existence of Touch Event APIs means the device has a touchscreen.

Going Forward

Input events remain one of the bigger interoperability pain-points across browsers and devices. Like most interop issues, this inevitably leads to both developer and user pain in the form of complex coding patterns and compatibility issues. We’re committed to making this better through interoperability work in our browser and standardization work at the W3C.

We want to build a platform that works for Web developers and our users. So give us your feedback—what works for you? What doesn’t? What would you like to see next? Sound off at @iedevchat and the Chrome and Firefox issue trackers.

Jacob Rossi
Program Manager

Comments

  • Anonymous
    September 05, 2014
    Microsoft want to inroduce us touch in everyones throats

  • Anonymous
    September 05, 2014
    The comment has been removed

  • Anonymous
    September 05, 2014
    Great work with Pointer Events. @Nick andd @Eduardo - Pointer Events are part of the standards, Touch Events are not. And PE is much better then TE.

  • Anonymous
    September 05, 2014
    @nick - feel free to use Touch Events in IE if that's your preference and it saves you time.  We recognize that model is what many sites use and some developers are OK with. That's why we implemented it and fully support it. Though in general, feedback we get is generally that web developers do think pointer events solve many problems (not to mention the capabilities outlined above not available in mouse/touch events).  Google's issue tracker is a good read to see some of this feedback: code.google.com/.../detail Additionally, it's worth reading jQuery and Dojo foundation's discussion on this issue. Their developers also have expressed that pointer events solve real problems for them: lists.w3.org/.../0071.html The test page isn't actually mine, it's Google's. It demonstrates a 2 different problematic scrolling scenarios: touch events and mousewheel handlers. Both of which actually do cause scrolling performance problems in sites that need solving. However, scrolling performance is perceived dramatically worse for touch than mouse because your finger is visibly on the page and you "feel" the lag.  Pointer events effectively eliminates this lag, bringing web apps closer to the performance you expect in native apps.  

  • Anonymous
    September 05, 2014
    The comment has been removed

  • Anonymous
    September 05, 2014
    The comment has been removed

  • Anonymous
    September 05, 2014
    Overall this is a good post Jacob, thank you in particular for outlining your vision for the future.  As always I think we have lots to collaborate together on in making the web better. That said, I have to point out two specific details in this post which seem to me to be disingenuous.  First, I know you're well aware that we did NOT "recently change our minds" about pointer events.  We've been clear since nearly the beginning that we weren't sure if the benefits justified the costs.  In Feb 2013 I wrote that "we're actively debating whether the benefits justify the complexity ..." in the bug that you site (http://crbug.com/162757#c11) and we repeated that sentiment at Google I/O and in your intent-to-implement thread on blink-dev (groups.google.com/.../forum).  It's true that I was more optimistic in 2013 that we'd eventually ship it, but at no point did we ever say we'd made a decision to do so. Secondly, your charts and specific numbers for the scroll start delay demo imply that there is some significance to the specific difference in magnitude.  I'm sure you're well aware that touchevents take 1000ms in this case because I wrote a 1000ms busy loop into the test!  Why not write a demo with a 100,000ms busy loop so that pointer events are 3200x better instead of just 32x better?  I notice you didn't use my test page which provides UI to set the delay value (www.rbyers.net/janky-touch-scroll.html) - was that because that version makes it clear the specific delay is completely synthetic and under the control of the page? I'm all for having legitimate technical debate about what's best for the web.  But if you stoop to misleading your readers to attempt to make your argument appear stronger it only works against our efforts to improve the web together.  Of course we've had enough drinks together that I would never hold a grudge against you personally :-)

  • Anonymous
    September 05, 2014
    By the way, you can easily highlight the scroll start problem with touch events using actual data instead of this synthetic test case where the numbers are meaningless.  Just check out our perf tests tracking our worst case scenario (scrolling during page load): chromeperf.appspot.com/report.  Here you can see, for example, that scrolling a desktop Google+ page during page load on a Nexus 4 takes ~300ms for the scroll to start!  We think this is terrible, and so are actively working on ways to make it better (eg. http://crbug.com/173543).

  • Anonymous
    September 05, 2014
    Link of "browsers" from "other modern browsers" is dead.

  • Anonymous
    September 05, 2014
    The comment has been removed

  • Anonymous
    September 05, 2014
    I am not sure how the test page by Rick is supposed to prove anything regarding the touch versus pointer event debate. The use case of the test page seems to (or should) be - Upon touch (or pointer) start, the developer needs to do some calculation before deciding whether or not to let the browser follow its default behavior (scroll, pan). Unless I am mistaken, pointer events simply do not let you do that. You cannot decide, based on the (geometric) point at which the event was fired, what to do. Touch events do. So while touch events give you more power, if handled incorrectly, it may have performance issues. Depending on the case, of course, performance issues can be easier or harder to overcome than issues of any missing functionality. If the developer incorrectly runs a long running code on touch start and the code is not supposed to affect the default behavior, then the code is wrong, not the API. @Yannick - Touch Events are a W3C recommendation (which means "a standard" to me). Pointer Events are a Candidate Recommendation (which means "a candidate for a standard, but not yet a standard" to me). So, Touch Events are definitely a standard. Pointer Events - not yet. But that does not matter at all, really.

  • Anonymous
    September 06, 2014
    I wish if I could help someone making this comment system like GitHub commend system (supporting Markdown with backtick) and email notifications etc. so to make it more appealing for community members.. Hopefully some day the person in-charge for MSDN blogs will understand why developers feel comfortable interacting on GitHub issue tracker than any other place.. And so will CodePlex team (Ask anyone at http://github.com/Microsoft .. if they would ever prefer Codeplex over GitHub ever again?) and so will Connect team and since so forth.. Quality feedback system requires creativity and one of the state-of-the-art way is http://conceptsharing.com/. Throw GitHub issue tracker features to the mix and get the best feedback system ever!

  • Anonymous
    September 06, 2014
    @RickByers. Sorry you felt parts were disingenuous. I love your variable demo page for latency--didn't know that existed. Thanks! It's much better because it demonstrates the root issue: the impact is variable and unbounded. We've seen the impact be both better and worse than the 1000ms example.  I think there's some confusion in what factors determine the actual value of the impact. It is by no means just the amount of script executed in the event handlers. On many phones, there can be hundreds of millisecond (or even seconds) delays before the event loop gets to even dispatch a touch event, not to mention the script that might run thereafter. This busy time is often hard to predict or improve in complex apps without rearchitecture of the sites code. Even 300ms is incredibly noticeable and breaks the physical illusion of touch. I used the test page you linked from your blog (1s version).  I've updated our post to now use the better variable one. Probably a good idea to update yours too. ;) I also removed the charts to prevent the risk that they are taken out of the context where I explain the variable nature here. Regarding "changing their mind", this was a reference to pointer events being in Blink's 2014 goals at the beginning of the year (groups.google.com/.../forum) and Google actively investigating through Polymer. But now that's now longer the case and Google is no longer investing in the Polymer implementation of Pointer Events (or a Chrome implementation). That constitutes a change in my opinion. But this post isn't about the historical record here (its about our updated developer guidance and education given the latest state of the world), so I've also updated this to just state Google decided not to implement. If you think there's any other clarifications, let me know.

  • Anonymous
    September 06, 2014
    Speaking of performance comparisons, have you considered testing:

  • Anonymous
    September 08, 2014
    @Brenno - thanks for letting us know. The link has been fixed now.

  • Anonymous
    September 08, 2014
    That's a very professional response. I appreciated reading your civility.

  • Anonymous
    September 08, 2014
    @Jacob Rossi. That's a very professional response. I appreciated reading your civility.

  • Anonymous
    September 08, 2014
    The comment has been removed

  • Anonymous
    September 08, 2014
    @RickByers @Jacob Rossi, Thank you for having these conversation publically. This will only improve the evolving Web ecosystem! Google and Microsoft along with other W3C members can work together to make things better by making a positive influence on each other; as opposed to make it a matter of ego "Mine is better than yours". IE is one of the oldest browser and therefore we expect the team to share knowledge and experience with the open web community as well as admit to the mistakes and fix them quickly. So we, the developers, just write a standard code trusting your expertise to run it efficiently on client machine and your commitment to standard.. Same goes for Chrome not for the web only but for sake of Node.js as well! We aren't living in utopian times, but we can definitely mitigate the friction by sharing thoughts! We love web and we respect both browsers and the people behind it.. <3<3

  • Anonymous
    September 08, 2014
    The comment has been removed

  • Anonymous
    September 09, 2014
    The comment has been removed

  • Anonymous
    September 11, 2014
    thanks, but I wish if I could help someone making this comment system like GitHub commend system (supporting Markdown with backtick) and email notifications etc. so to make it more appealing for community members.. Hopefully some day the person in-charge for MSDN blogs will understand why developers feel comfortable interacting on GitHub issue tracker than any other place.. http://techleaks.us/

  • Anonymous
    September 12, 2014
    The comment has been removed

  • Anonymous
    September 13, 2014
    @Rick - if Pointer Events was called out as a "priority" for Blink for 2014, why was no work done on it? Or if there was already concern about it, why was it called out as a priority? That's the position that makes it look like Google taking a step back. What does "priority" mean when it's used like that?

  • Anonymous
    September 13, 2014
    @pmbAustin - Opinions differ and the reality differs. Google does think it is a good model (Apple does not), but implementing it right now will have performance costs for no major benefit (not a lot of websites use them, I believe, at the moment, because Windows Phone is not popular). Once Firefox ships this successfully (without major performance issues), Google might change its mind. Remember that the decision is not set in stone. "Like every other browser" - only Internet Explorer has shipped pointer events. Firefox are on their way, but they are not there yet. So, one browser.

  • Anonymous
    September 14, 2014
    I'm glad to see this open discussion on the topic but unfortunately it is taking place on the IE Blog. Since the IE Blog comment system is known to be the worst comment system ever designed and it will routinely eat posts I have no intention of wasting any time writing involved replies here. Replace the existing comment system and maybe we will come back.