Create a Setup Project for a 64-bit Windows Service in Visual Studio 2010
I was having a lot of trouble with this, so I'm going to share it. Visual Studio 2010 setup projects still do not fix the way you create a setup project. The problem is that Visual Studio doesn't add the x64 version of InstallUtilLib.dll to the MSI and so you will receive a BadFormatException error message and the installer will crash and burn.
The workaround is explained here: