My philsophical musings about building setup for software.
Mike Gunderloy, who has written a couple articles about the WiX toolset, posted a comment on a previous blog entry. In the comment, he suggests that splitting a setup project into fragments only moves the problem but doesn't solve it. Thus, he argued that adding a tool to generate the fragments has value. I still didn't agree. Then I realized that I disagreed because I have some philosophies about how to build software and setup for software that I've never posted here. So, I thought I'd share those philosophies today. Please note that these are guidelines that I use when discussing setup build processes with other people not hard rules.
The developer that wrote the feature knows best what needs to be authored into setup.
This part of my philosophy is based on the fact that the person that knows the most about a particular resource is the developer that wrote the resource. Seriously, if the developer who wrote the code doesn't know where his or her files need to be installed or doesn't know what registry keys are necessary to make the feature work (or whatever other resources are necessary) who does?
Yes, there have been (too many) cases in my career where the developer who wrote the code said, "Uhh, I don't know what my code depends on." However, in those cases it was pretty easy to look at him or her (or his or her manager) and ask, "Well, uhh, shouldn't you?" In every case, they went off and figured out what was necessary to get their software "in the box".
Fundamentally, if developers don't know what their dependencies are there is very little chance their project will have a solid security, performance, or deployment story. Area experts may be necessary to help developers work through complicated issues but, in general, developers must be aware of what their software is doing.
Setup authoring is a part of the development process.
Every team I have interacted with (the grand majority of Microsoft and some smaller companies), the developers on the team are expected to add their source files to the makefile before checking in new files to the project. Yet, many of those teams have people who are solely responsible for adding new files to the setup project. In many cases, one set of developers write registry keys into the code for SelfReg (this is evil, and I'll explain why one day) and the setup developers have to reverse engineer the registry keys out of the built executables to author setup. Why didn't the original developer just author the registry key into setup in the first place?
There was a point in time where it was arguably difficult to distribute the authoring to all developers when you had to buy custom tools that didn't fit well into the development process. However, today there are a few alternatives out there (like the WiX toolset) that allow setup to be treated like source code. Now there are no excuses I've heard that hold water why setup authoring for the majority of the resources in setup cannot be distributed across the developers in the organization. Doing that distribution leaves only the "look and feel" and integration between the individual pieces of setup to the core setup developers. All other excuses have always been just whinging (as Peter might say).
This part of my philosophy and the one above are the reasons I disagree when Mike says that "breaking up the package into multiple source fragments pushes the problem back one level, but doesn't necessarily solve it." Breaking up your setup in to multiple text files enables developers to maintain setup authoring the same way they maintain all the other code that is part of a project. And that brings me to my final point.
Text files should be the only inputs into the build process.
Over the years, developers have created fairly significant processes for tracking, merging, and reverting changes to text files. For example, Concurrent Version System (better known as CVS) is used heavily by SourceForge.net can show you the individual lines changed in a text file over time. For a demonstration, take a look at Compiler.cs from v1.6 to v1.7 in CVS at SourceForge.net for WiX. It is not generally possible to visualize the differences between two revisions of a binary file.
Every input into the build process that is manipulated by a human will eventually hose the process at least once. Being able to text search (using even the simplest tools like grep) for the exact change that caused the problem significantly improves build throughput. Binary files usually hide the information by requiring custom tools to be opened and queried to find the break.
Of course, once you find the break it is really nice to be able to fix the problem by launching your favorite text editor and tweaking the line(s) of code with the fault. Requiring a tool to be installed on the build machines increases your impedance to fixing the issue.
Ultimately, keeping binary files out of your build process simplifies your life. That fact is why I disagree with Mike when he suggests it makes sense to "put one more tool into the chain, something with a friendly interface that could spit out the source fragments as needed? I could see doing that with Access/Excel, among other things."
Note I do think it is reasonable for developers to use tools that help generate text files. Those text files then can get checked into source control and built as part of the standard process. However, the tools need to generate human friendly text files. Text files that can only be modified with a custom tool are only half a step better than raw binary files.
This is the philosophy that I developed during my tenure in Office and have promoted across the company for a couple years now. Many teams have had an incredible amount of success with their setup processes when following these guidelines. There have been many cases where I was asked by other teams to talk about the philosophy. At the end of my talk I always left them with, "If you have someone resisting change, have them contact me. I'll be happy to talk to him or her to discuss how successful other teams have been with this process and see if there isn't some way to address their concerns." I'd be happy to do the same for you.
PS: This philosophy has worked out extremely well for me, but if you have a unique situation that you think wouldn't work using these guidelines, I'd be very interested to hear about them. I'm always looking for alternative views (as long as people aren't just whinging <smile/>).
Comments
- Anonymous
May 16, 2004
"Note I do think it is reasonable for developers to use tools that help generate text files." We're not disagreeing - at least not about that, or about the need for having text files be the only input into the process.
But isn't your registry key example the perfect example of why it makes sense to provide targeted setup-authoring tools to every developer? The developer should absolutely know "my piece is X, and it depends on Y and Z, and it needs these registry keys." But I think the team's toolmaker (or whoever) can profitably provide a tool that translates that knowledge into the right XML input for WiX (or whatever other process the team is using), thus avoiding any chance that a developer who isn't up on setup will write SelfReg code.
So, to sum up where I'm coming from: setup should be a distributed activity, but it's not realistic to expect every developer to know the ins and outs of the setup rules from the start, and that's where some tool can come in - to take the knowledge that the developer has about their own bits and translate it into a proper chunk of the setup source. - Anonymous
May 16, 2004
The comment has been removed - Anonymous
May 16, 2004
Rob, what an amazing post. I wish I could force the people in my company to read some of your blog entries, because they are so much more than philosophical musings!
I have a presentation to management about the ins and outs of MSI soon, and I think I will include these principles in there (along with "What is setup" and the two axioms :-)). We are busy (or plan to) inform management about setup in general and MSI in particular, as I desperately want to get away from the current "ask the setup developer - he/she can make it work" scenario.
The more I work with MSI, the more I realize how closely the install should be integrated with the rest of product development. The problem is getting management to realize this too. It is so convenient for them to just "hand it off" to the setup developer. Only problem is, when using MSI, you can't just "make it work" - you have to know the limitations and advantages and rules!
And WiX is the perfect tool to do this with (the more I work with it, the more I like it), if there can be some way to simplify file handling etc. for developers. Luckily, as you pointed out in an earlier blog entry, some tools are being developed (and I'm playing around with this myself). - Anonymous
May 16, 2004
The comment has been removed - Anonymous
May 17, 2004
Mike,
As noted in my blog entry, I’m completely comfortable with having development environments that help build WiX files to ease a developers’ life. I use VisualStudio today to do all my WiX authoring because it will do "intellisense" on the WiX schema. That saves me a lot of time. For example, with VisualStudio, writing a registry key is trivial because I type "<Reg" + space, then "I" + "=" then "K" + "=" + select from dropdown list then "P" + "=" then "N" + "=" then "V" + "=" and I end up with something like:
<Registry Id="" Key="HKCR" Path="" Name="" Value=""/>
Hopefully, any Windows developer could figure out what to do with that. Granted, I am expecting developers to spend a little bit of time understanding the basics of setup just like I am expecting developers to know a little bit about programming secure code and “not slow” code. If tools can help that learning curve and make life better, that’s great.
Also, I’m not arguing that every developer should need to know the installation in depth. Every team will need an area expert that they can leverage to take care of the more complicated things like integrating all the smaller pieces into the larger whole (or "wholes" if there are multiple SKUs). The interesting thing is that even on large projects your core setup team can be very small if you distribute authoring across all the developers. - Anonymous
May 17, 2004
The comment has been removed - Anonymous
May 17, 2004
David,
My point about developers understanding dependencies should take care of your build machine issues. If a developer creates a dependency on a tool that is not in the build why is it acceptable for the developer to check in his or her code? Seriously, if developers don’t understand what is in the build lab (and I agree the build lab should be as minimal as possible) how do you prevent them from checking in stuff that breaks the lab all the time? Having developers author setup (or not) wouldn’t affect this.
All the other issues you raise are things that I would expect the core setup team to manage. Understanding and debugging those cases is not something every developer needs to do. Area experts (that know everything about conditionals in MSI, upgrades/patching, and CustomActions and such) will always be needed. However, I don’t believe you spend your area experts’ time doing work that can (and should) be distributed across all the developers.
I think your last point argues for code reviews by the area experts. I think code reviews is a great idea for setup, security, and performance. However, that does not mean that the setup authoring should not be distributed across your development organization. In fact, over time, I think you will find that a number of developers will become setup area experts on their own. - Anonymous
May 17, 2004
The comment has been removed - Anonymous
May 17, 2004
James: Actually, I do have something relevant (mostly... I'm probably nitpicking) to add. You mention SQL server as a particularly bad offender of the "setup should be simple" mantra, and you say that an average user gets confused. But why, exactly, is an average user installing SQL server? It seems like that would be an automatic contradiction. If it's a developer installing it to write code against, they're really going to have to understand what they've installed, or they won't know the requirements of their code.
It's very similar with Oracle (I used to run Solaris boxes and networks at school). Yes, it's a giant pain to get Oracle running, and you need (or needed at the time) to write some scripts to get things working. How do they expect an average user to get Oracle installed? Well they don't, of course.
Now I personally believe that SQL Server is easier to install than Oracle, but that it could be made easier. But I still don't understand why your average user would want or need to do so. - Anonymous
May 17, 2004
I am the lead developer at an ISV, which makes software used by small businesses which do not have a particularly technical orientation. Our product consists (among other things) of a client application, a middleware layer, and a sql server. Our customers need to install this without our supervision. The middleware layer uses SQLDMO to discover sql servers, so we have to include that also.
Ideally, I should be able to use some sort of MSDE redistributable installer to make a new sql server instance on the machine, prompting the user for their sa password as part of the install. Then I could create my sample database as part of my app's install.
This is really, really hard to do in the current state of affairs. We ship a separate MSDE installer, and hope that our users read the readme file.
In a perfect world, MSDE would be installed with the OS, along with MSMQ, but convincing the powers that be that that should be on every desktop is beyond my ability. - Anonymous
May 18, 2004
The comment has been removed - Anonymous
May 18, 2004
Amen, James. I've been there right with you since 1998.
The only thing I ask is that you think twice about creating your own installation engine. Multiple installation engines has the capacity for truly hurting our customers. There are so many interoperation problems with multiple installation engines that we are better off getting one good engine in the platform than fragmenting the world. I lived in a world with several installation engines and customers were getting completely hosed. I don't want to go back there.
Your well-informed and completely appropriate "rant" here provides yet another example (that I have been throwing around for years) of things that need to get fixed.
Just know that you're not alone. <smile/> - Anonymous
May 18, 2004
James: I'll second the amen. Also, I clearly didn't understand the scenario you were saying was broken. What you described is definitely not the way that should work.
On my first read through your original post, I thought you were writing about a stock SQL Server install, rather than merged into your own. Apologies for the misunderstanding. - Anonymous
May 18, 2004
BTW, I'm in the Windows Installer 3.0 Beta... none of this has been addressed. (since the 3.0 specs have been publically released I think I'm find to disclose this?) - Anonymous
May 18, 2004
James,
Yes, the What's New page for MSI 3.0 can be found here: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/msi/setup/what_s_new_in_windows_installer_version_3_0.asp
And you are correct, none of the issues listed in this thread have been addressed in the next version of the Windows Installer. - Anonymous
May 19, 2004
whew! :)
Now if only they would fix all of this stuff... We don't need a new version just for easier updates. - Anonymous
May 19, 2004
The comment has been removed - Anonymous
June 25, 2004
Imagine a blog entry where I ramble along after having my wisdom teeth pulled and come up with something of a vision statement. - Anonymous
July 11, 2004
Imagine a blog entry where I am reminded how important rollback is in setup and take a few minutes remind everyone. - Anonymous
August 28, 2004
The comment has been removed - Anonymous
April 19, 2005
Experiance suggests there are windows of opportunity for breaking the cycle of the centralized setup... - Anonymous
September 12, 2005
Imagine a blog entry where I talk about my first day at PDC. - Anonymous
September 13, 2005
Imagine a blog entry where I talk about my second day at PDC 2005. - Anonymous
January 31, 2006
The comment has been removed - Anonymous
January 31, 2006
Imagine a blog entry where I try to answer Kurt's question about how large a development team should be to use distributed setup development.