Share via

Testing... 1... 2... Is this thing on? (Posted by Avi)

  • Article

I wanted to kick off this new blog with a post describing the team, what we do, who we are, etc.

Internally, we're known as the "DDIT Web Team", as in "Developer Division IT Web Team". Developer Division IT is the group in DevDiv that encompasses the Build Lab, including all the cruft that's required to keep it up and running - build ops, build tools development, hardware maintenance, build testing, customer management, and the web team.

The Web Team consists of 4 full time employees (Aaron, Arturo, Paul and myself [Avi]) and 3 contract employees (Hatim, Viktor and Joe [joining us on Monday!]). All but me are developers - I'm what they often refer to as "overhead", or "comic relief" if they're being nice :)

We started out much smaller a few years ago. The lab was running maybe 2-3 builds per day, and was using an old clunky web page to report their status. Perhaps 20% of the time, the web page was wrong - causing the customers to ignore it completely and just call the lab for status. This was our first lesson: If a status site is ever wrong, it may as well be always wrong. If people don't trust it, they'll ignore it and get around it. So, our first order of business was rewriting the status site. We wrote it from scratch, designing it to be solid and to cater to the various type of customers we had… We've not looked back since.

In the past two years, the build lab has grown immensely! They're now doing an average of over 70 builds per day, generating over 1.5 TB (yes, that's a "T") of data each day. With that kind of size come problems that just don't exist at a smaller scale, and the web team has needed to grow in order to keep things running smoothly (although we haven't scaled to nearly the same degree ;)).

We've reached a point where the build lab cannot function without the web tools. Some examples of tools we write/maintain for the lab:

  • Build status tracking.
  • Build break tracking/assigning/fixing.
  • Statistics on build breaks, what causes them, who causes them, etc.
  • Space management (how do you keep track of hundreds of builds, over 100TB of space, who needs each build, till when, why, etc?)
  • Customer build request page.
  • Kicking off builds and managing their resource requirements (we have over 1000 servers; which builds will kick off on which servers, when? etc).

Very recently, a re-org has placed the DDIT group within the DevDiv Engineering Excellence org, which puts us in a position to write tools for the whole division, reducing the amount of duplication that may otherwise exist. Examples of tools we own that are used by the whole division (and sometimes by other divisions in the company):

  • Custom statistics tracking (track each team's stats for bugs, test results, etc).
  • Documentation management (online docs/FAQs).
  • Server/asset management and monitoring.
  • Product performance reporting.

There are plenty more examples, but you get the idea. We're in a position where the few of us are supporting over 80 applications, some in use by thousands of people, some critical to the whole division. It's exciting :)

This blog will contain entries from the devs about issues related to ASP.NET, C# and SQL; tips, tricks, the usual. They're doing some fantastic stuff, so there's a lot to learn from them. I'll also try to inject posts about running a web team and supporting a large volume of applications.

Stay tuned; the devs are itching to post stuff. We'd love to get your feedback!

Avi

Comments

  • Anonymous
    May 17, 2005
    This type of thing is critical to managing a project successfully. Whilst waiting for VSTS to emerge from it's embryonic cocoon, I have developed a lot of reports in Reporting Services that combine build and issue statistics. It would be great to see some screenshots and/or samples of the kind of work you are doing.

    Bruce

  • Anonymous
    May 17, 2005
    Bruce, I could give you screenshots, but I think the lack of context would render them kinda useless. But I'll try to describe one of the stats that we track...

    Whenever a build breaks, we have an issue tracking system which allows us to assign the break to people until we get some kind of resolution. When that happens, the issue's cause is classified, along with the person who caused it (or failing that, the person in the best position to have prevented it).

    Given that information, we can now use active directory to roll up from person, to team, to org, etc.

    We also know when the issue was opened, how long it stayed open, and why it happened in the first place.

    At that point, we can bust out any number of interesting stats. For example:

    * What are the most frequent causes of breaks?
    * Which teams are the more frequent "breakers"?
    * Similar to above: But instead of using frequency, use the total (or average) time cost of each issue (this is probably more interesting).
    * How do these values change over time?
    * How do these values change in relation to our milestones (we always see the build breaks dwindle away as we get near shipping)
    etc...

    That's one facet of our tracking - dealing with build breaks. We also have tracking for other stuff:

    * Code breaks (how many bugs, which team, trends, predictions, etc)
    * Build sizes (is it growing? [always does...])
    * Build times (taking longer?)

    I'm expecting that VSTS will be great for this sort of stuff. And the larger your organization is, the more valuable the information.

    I have a mere 6 devs at the moment, and seeing the graph of incoming work items versus the rate at which we're resolving them is crucial (not to mention horrifying ;)).


    -- Avi

  • Anonymous
    May 17, 2005
    Hi this is excellent article

  • Anonymous
    May 17, 2005
    Hi this is excellent article posted by them

  • Anonymous
    September 01, 2005
    Your blog is very interesint

  • Anonymous
    February 21, 2006
    Phoenix hotels

  • Anonymous
    February 21, 2006
    Vienna hotels

  • Anonymous
    March 21, 2006
    I found your page from google but i like it so much

  • Anonymous
    June 03, 2006
    i like your website very much but please do get us more information about it

  • Anonymous
    November 19, 2006
    I wanted to kick off this new blog with a post describing the team, what we do, who we are, etc. I do not agree. Go to http://www.aplushotels.info/judaic_Italy/unbusinesslike_Lombardia/admissibility_Milan_1.html

  • Anonymous
    December 14, 2006
    I wanted to kick off this new blog with a post describing the team, what we do, who we are, etc. I do not agree. Go to http://www.lifehotels.info/crime_Austria/trike_Eastern%20Austria/unhook_Vienna_1.html

  • Anonymous
    December 20, 2006
    I wanted to kick off this new blog with a post describing the team, what we do, who we are, etc. I do not agree. Go to http://www.lowcostlodging.info/pimento_Spain/vernation_Comunidad%20Valenciana/savvy_Alicante_1.html

  • Anonymous
    August 14, 2007
    I wanted to kick off this new blog with a post describing the team, what we do, who we are, etc. I do not agree. Go to http://apartments.waw.pl/