Ping Pong Quest Follow Up
(Published on behalf of Adam Wilson)
Well that was exciting. In a little over 5 days we managed to get over 2000 insiders running the Ping Pong App and generating over 26 000 datapoints per day. The team is still pouring over the data, but we have been able to track down some interesting information about how notifications are delivered to your machines. I’d like to personally thank every single person who took the time to install the app on their device, your dedication to making Windows great inspires us and the data you provided will go a long way to making notifications more reliable.
Feedback
Before we go into the results there were a few pieces of feedback to be addressed.
- I’m sorry for the confusion from our non-English speaking users, there was a miscommunication and internationalized versions of the app were not available in the store. There is a bright side, everyone who ran the app generated useful data. So even if you ended up with an app you couldn’t read - you still generated useful data to improve Windows for your home region.
- The store download issues are being addressed by the store team. There were a few systemic failures over the weekend that Ping Pong got caught up in. Rest assured the store team is tracking them down (and I’m tracking them tracking them down J)
It seems that engineers aren’t allowed to design interfaces for a good reason, which meant the app was a little more complicated than it could have been. Here is a quick walk through of what the app is showing you:
Chat
This pane lists the notifications that have been sent to your device or are pending. The colors are:
Message Background |
Notification Status |
Gray |
Notification pending |
Green |
Notification received |
Yellow |
Notification delayed, but not lost |
Red |
Notification lost |
There is also a bit of information about the notifications that you can see in the message box if you are interested in seeing when the notification was received.
Push
This column lists the number of notifications received and dropped as well as configuration options. We are going for 100% on the number received, but as you can see that doesn’t always happen (and in the screenshot you can see that my test machine lost a few)
Interval Type: Allows you set how often notifications are going to be delivered to your device. We’d prefer for you to leave it set as the default value of server. This lets our server schedule follow up notifications based on how often they are being lost on the way to your device, but there are other options if you’d like more control.
Diagnostics
This panel has some information that we thought the developers in the audience would like to see, or those who are more interested in the internals of notifications on Windows.
ClientID: The ID that the notification service uses to track your device. This is how the Microsoft servers know to send your notifications only to your phone.
Channel: This is the address of the Ping Pong app on your device. Every app who sends notifications gets one of these URIs for each device it is installed on. This URI is how an app like Facebook or WhatsApp is able to communicate that they have a notification to be sent to a device.
Request New WNS Channel: Apps must refresh the channel or it will expire and stop allowing notifications to flow to your device. Ping Pong automatically refreshes the channel on a regular cadence, but this button allows you to force an update if the boxes above are blank.
Clear Chat History: Clears the record of past notifications from the left pane.
Force Watson (Phone Only) : This is an experimental button that snuck out of the lab. We are looking at enabling Ping Pong to trigger the error reporting dialog which sends crash information to Microsoft, but currently it doesn’t work. We’ll let you know if there is something that you can do with it in a later quest.
As a side note, the engineering teams really do look at the reports that users send back. I didn’t believe the reports from my machine were doing anything until I started working here and saw how they get treated. Even on a small feature team, we spend time every week sorting through the reports and finding issues to investigate and fix.
Show Notifications: Lets you choose if you want toasts when a notification is received. Personally I keep if off unless it is a test machine. If the box is unchecked, the app will check for lost notifications silently in the background without showing any toast messages.
Results
This was a great chance for the team to take a look at the flow of notifications from a number of different regions and device configurations that are usually hard to get debug data from. There are a couple interesting things that jump out from glancing at the data:
Caching
You folks hammered the caching code paths on the server. In preliminary tests with Microsoft employees <5% of notifications had to be cached in the server for later delivery because the device wasn’t connected at the time. With insiders we were seeing as many as 20% of notifications being cached, as shown by the larger yellow bars in the graph below:
That is great for us to see, as it lets us make sure a lot of tricky scenarios surrounding devices connecting and disconnecting are working.
Dedication
The quest went live mid-way through the day on July 31st (Pacific Time), and the response was unbelievable. We had expected to leave the quest up for a few weeks before hitting the user cap, but we managed to surpass the target in a single weekend.
What is most amazing to me is that the quest came down on August 3rd, but as of August 6th insiders are still running the app and generating feedback. This data is hugely valuable to track the flow over time and we are going to continue to analyze every single report that comes in.
Errors
This is still under investigation by the team, but we got a couple of new error states that need to be tracked down. Our best development and quality teams (in my humble opinion) are currently investigating the issues and will be pushing fixes out as quickly as possible.
One last thing!
I want to once again thank all the insiders for installing the app and taking the time to help us out. Hopefully you are enjoying Windows 10 and all the new features that it brings.
The notifications team has working hard to improve notifications on Windows and hopefully you’re going to love the results.