How Do You Debug A Movie?

How do you debug a movie? Before I get too old and forget, here is the story of one bug that I had to find when playing King Kong in the Xbox HD DVD player. It concludes with yet another reason I am so glad we’re not doing a Blu-ray player.

The HD DVD team originally wrote the player software for what became the Toshiba A1 series HD DVD players. Those players were basically x86-based laptop boards with an optical drive and video decoding hardware added, running Linux. Toshiba wrote the audio/video pipeline and we wrote the rest of the player. After that initial Toshiba release we really got going on a version for the Xbox 360, which shared much of the codebase but had some notable differences, plus we got to write our own AV pipeline. Progress was rapid (thanks in no small part to the ninjas that the Xbox team had lent us to get things going) [and yes Ninja was their official job title] and soon the Test team were running real movies from shiny discs through the player.

I ended up with a particular bug in King Kong which involved picture-in-picture (PIP). I got it because I owned PIP and its positioning logic, and the bug was that at a particular point in the movie (12:08) when PIP had been enabled, the PIP window was supposed to disappear. Well the video disappeared, but a black PIP-sized rectangle was left behind for about 12 seconds. This was a Universal title and used one of the earliest versions of U-Control to cause certain things to appear and disappear from the display at certain times. There were other times when this same weird rectangle would show up, but this was the first one.

The first thing we did with a problem like this was to look at the source. HDi used ECMAScript (aka Javascript) and the source code was right there on the disc. The .js files were usually packed into an .aca file (think ZIP file without the compression) and sometimes it might be AACS encrypted, but as a player manufacturer we legally had "the keys to the lock" so that was not a barrier to us reading the source. (The .aca file was also multiplexed into the audio/video stream so that usually the player does not have to pause playback to load a script, or a bitmap, which was important as seek times on optical discs are generally terrible).

The U-Control code got a lot more sophisticated over time, but the basic premise was that there was a list of timecodes and a list of actions to perform when those timecodes were hit. These actions included things like fading PIP audio up and down, showing and hiding PIP video, displaying overlay graphics and the like. The first place I look was the most obvious: add some logging to the PIP alpha code and see what and when it was being used. If PIP video was present in the video stream it was always run through the secondary video decoder, but you couldn’t see it as it started with an alpha value of Transparent. In order to see it, the js code would change the alpha value to Opaque (or in the case of U-Control it faded up the alpha from one to the other over a few frames). The logging revealed that in the failing case at 12:08, the alpha API was being called with 255 (opaque) instead of zero (transparent). As there was no actual secondary video stream at that time, we rendered an empty stream (ie a black rectangle).

At some point we learnt that King Kong was going to be bundled with the HD DVD drive itself, so we had to make very sure every aspect of the title worked perfectly on our player. Pressure: I can handle it.

I next spent a while poring over the U-Control logic: this was the first such title I ever had to debug, but it was made a bit easier because there was a bunch of debugging in the .js file itself, put there by the original authors. All I had to do was change a debug variable to true, and a bunch of debug spew was printed as the title ran. In order to "edit" js files on a read-only disc packed into an .aca file we had debug code in the player that allowed us to substitute "our" version of any .js file (read from the Xbox hard-disc) instead of the one from the optical disc. That way we could play with the title code to add debugging or change logic, and see the effect on the movie immediately.

The logging showed that there was a problem parsing the timecode: The JScript function parseInt("08") was returning zero (it was parsing the seconds part of the timecode), and clearly the code was expecting it to return 8. Further investigation revealed that our JScript engine (cloned from the IE one) treated zero-prefixed numbers as octal, and as “08” is not a valid octal number it returned zero. Toshiba’s ECMAScript engine (which they wrote from scratch) didn’t think it was octal, so returned 8. (In the early days all HD DVD title development was done on Toshiba A1 Emulators: special players with all kinds of extra, like hard discs and debug tools. Only later did authors start using Xboxes to test, and very much later to develop against). This confusion over parseInt and octal was not new to web page designers it turns out.

So I changed the parseInt function code in the script engine to default to decimal, and now King Kong worked fine. The player was released and shipped along with a copy of King Kong and everyone was happy. Well, for a while. At some point the Test team ran an ECMASCript compliance test on the player, and discovered that my "fix" to parseInt actually caused a test to fail, so the code was revisited and a better, more compliant fix was made, that kept King Kong working and also conformed more exactly to the ECMAscript spec (and hence to the HD DVD "gold standard", the Toshiba A1).

This particular bug was a player bug, as were many. However there were other categories of bugs, some caused by the title authors themselves. Often we would identify them at "check-disc" time (ie a pre-release copy of the disc some of the studios would send us) and we could tell them what to fix in their code before the disc was mastered. However sometimes we would be too late, or it would be from a studio that didn’t send us check-discs, so we would have to add specific modifications to the player that only kicked in when a particular title was played, and then push a player update out over Xbox Live.

Over time our methods for debugging title problems got more sophisticated, and much of what we learnt ended up in the HD DVD Emulator that I created. That way title authors could themselves debug issues with their code and content without having to burn and rush us a check-disc. Of course if there was a suspicion of player bug then we’d have to get involved in the investigation.

I am so glad we didn’t do a Blu-ray player. The poor folks who have to work out why BD-J titles don’t work correctly don’t have the luxury of seeing the original author’s source code, with comments and debug info. All they have is compiled Java bytecode: ug.

Comments

• Anonymous
  March 06, 2009
  The comment has been removed

• Anonymous
  March 08, 2009
  The comment has been removed

• Anonymous
  March 14, 2009
  The comment has been removed

• Anonymous
  March 25, 2009
  The comment has been removed

• Anonymous
  March 27, 2009
  Great story Andy. I was wondering what it might take to turn some of our A1's into mini pc's. Maybe I'll get to that some day after the days of BD-J are over. :-) Cheers!

• Anonymous
  March 27, 2009
  Thanks D B. As A1s were the development platform, you could add a hard disc to try and turn it into a PC. However the video would be a challenge: there was nothing like a PC video card on there. It had a serial port so you could TTY to it: that's how we would develop the Linux stuff in the early days, we would actually edit the source on a PC (via an NFS share) then compile, link & run the result on the development A1s. Once AACS came up we lost that ability and only Toshiba could get new bits onto the devices.