Compartilhar via

Sacrifice me!

  • Artigo

I'm on BVT [Build Verification Test] duty this week -- lucky me (not!).  Each week we Sacrifice One Person (capitalized because that's a Management Pattern, don't you know; see https://Alistair.Cockburn.us for more info) to BVT duty.  On a good week, said duty is simple:  check the results mail from the overnight BVT run, see that all tests passed, pass on the good news via that day's BVT mail to the team.

Good weeks do occur but they are somewhat rare.  More often than not, one or more of the following occurs:

  • The automation system is hosed.
  • Enough machines fail to reimage that the automated BVT process hangs.
  • A test fails.

If the automation system is completely broken, BVT Person has the pleasure of executing the BVTs manually.  This is kind of fun the first day because you do things you might not normally do, but it gets old fast.  If the automation system is broken in the right kind of way, we can manually prepare the machine and then kick off an automated run.  Similarly, if machines had problems reimaging, we manually prepare them and then kick off an automated run.

The first step towards diagnosing a test failure is to inspect the test case's run's log.  This is often sufficient information to understand what went wrong, but it's usually a good idea to rerun the test and watch what it does, just to be sure that what you think happened is what actually happened.  I'm always happy when rerunning the test makes the problem sufficiently clear that I can write a bug.  When it's not, I have to manually execute the steps in the test.  If the test passes when executed this way, I write a bug against the script and assign it to the script's owner.  (Yes, it may actually be a bug in my app, but it's the test owner's responsibility to figure out what exactly is going on.)  If the test fails when run manually, I write a bug against the feature and assign it to the feature's developer.

This week just about everything that could go wrong has:  Our automation system isn't playing well with the current Longhorn installs, so I have to manually set up a machine and kick off the BVT automation.  A few tests are failing due to bugs in my app.  A couple other tests are failing due to bugs in the tests themselves.  On top of everything else, we're trying to take a new Longhorn build.

Integration is perhaps the worst part of BVT duty.  We're taking a big bet on Longhorn and Avalon.  Those teams are continuously adding features we want to take advantage of (not to mention fixing bugs that are blocking us), so we want to take as many new builds from them as possible.  Moving to a new Longhorn build is rather involved:

  1. A developer and a tester each install the build.
  2. The developer gets our app compiling on the new build, and the tester does the same for our test infrastructure and test cases.  Longhorn and Avalon are both being actively developed, which means their APIs are a moving target, which means code edits are often required before our app will compile.
  3. The tester runs all of our BVTs and Exit Criteria tests (user scenarios and other functionality we have decided must work in order to exit a milestone) on the new build.
  4. Any tester with a failing test analyzes the failure.
  5. Anybody (Dev or Test or PM) who sees that their feature is sufficiently broken vetoes taking the build.
  6. The developer moves our build machine to the new Longhorn build.  The developer and the tester check in any code changes necessitated by the new build.
  7. Each team member moves to the new Longhorn build.  This takes an hour or so between installing the new Longhorn, resetting any custom settings, and installing whatever apps each person uses.

Between BVTs and testing the new Longhorn build, then, fifteen hours or so of my time this week will disappear into the ether.  It's a major pain, but that's what Sacrifice One Person is all about:  piling the pain on onto a single person so that the rest of the team can continue to make progress.  Luckily, my BVT tour of duty only comes 'round every ten weeks, which gives my tender hide plenty of time to heal and re-harden before the next tour!  <g/>

*** Comments, questions, feedback?   Or just want a fun job on a great team? </g>  Send two coding samples and an explanation of why you chose them, and of course your resume, to me at michhu at microsoft dot com. I need a tester, and my team needs a data binding developer, program managers, and a product manager. Great coding skills required for all positions.