Error: VMM 2008 R2 Service Account Changes Not Supported without Uninstall
We recently did a slight shakeup of all our service accounts as our engineering lab that we use to test our code against uses all of the System Center products. This is by design since our team is in the division that focuses on Microsoft’s Enterprise Management tools. We are 95 percent virtualized in our engineering lab and we’ve been using System Center Virtual Machine Manager (VMM) 2008 since mid-2007 when it was in beta. This setup has pretty much utilized the same service account since the beginning so this was a “new” experience trying to move to a new service account.
Why not use Network Service? As described in this blog post, it is recommended to utilize a domain service account in various situations. We primarily met option #2 (trusted domain for hosts) . I’ve found it also to be extremely useful when you are using VMM with Operations Manager 2007 (SCOM) because the Performance & Resource Optimization (PRO) integration requires that each application’s service account is an administrator on the other’s box.
The purpose of my post is to share what I learned from Michael Michael when I had debugged as much as I possibly could to figure out why when running VMM under the new service account, I was unable to access VMM using the VMM Administration console.
Symptoms: VMM Service starts but VMM Administration Console Returns Error
The first symptom that I received immediately after changing the service account was the following -
This was frustrating as I already knew for a fact that I had added the new service account to the SCVMM Administrators role.
The ironic thing is during the debugging of this using the technique outlined here, the error returned was the VMM database was corrupt. This was odd to me because I could easily shutdown the VMM Service and replace the service account back to the previous account and restart and VMM would work fine.
Error from DebugView:
00000517 8.94344139 [3696] at Microsoft.VirtualManager.Utils.LockableObject.Dispose()
00000518 8.94344139 [3696] at Microsoft.VirtualManager.Engine.ImageLibrary.VMBase.Dispose()
00000519 8.94344139 [3696] at Microsoft.VirtualManager.Engine.BitBos.VMRefresherBase.UpdateHostandVMs(VMRefresherType refresherType, Guid vmObjectId, VM tempVm)
00000520 8.94344139 [3696] at Microsoft.VirtualManager.Engine.BitBos.VmFullRefresher.RefreshData(HostReference hostRef)
00000521 8.94344139 [3696] *** Carmine error was: DatabaseInconsistent (2603); 50001
00000522 8.94447803 [3696] 0E70.1264::08/11-02:27:40.221#04:RefreshDriver.cs(207): Terminating error - Refresher failed DatabaseInconsistent (2603); 50001
00000523 8.94479942 [3696] 0E70.1264::08/11-02:27:40.221#04:RefreshDriver.cs(207): Microsoft.VirtualManager.DB.DBCorruptionException: Unable to connect to the VMM database because the database is in an inconsistent state.
00000524 8.94479942 [3696] Contact an experienced SQL administrator whenever this error occurs. In some cases, it may be necessary to restore the VMM database. If the problem persists, contact Microsoft Help and Support.
VMM Security: Service Account Membership Review
If you’ve spent time learning VMM, you will probably start to accumulate a listing in your mind of all the various places the service account has and/or needs permissions. This was the case and so when I set out to replace the service account I took that non-written down list and started verifying using different methods, either manual or listing group memberships via script. To make my life easier, and maybe help others in the future, I thought I would list here where the VMM Service Account actually has permissions and what those permissions are -
Permission Where? VMM ACLs on Hyper-V physical hosts using Local Group On each physical host, there is a local group named Virtual Machine Servers VMM Service Account needs DBO access to the Virtual Manager DB in SQL On the SQL Server, the service account needs a login to the server and DBO/Admin rights to the Virtual Manager DB VMM Service Account needs VMM console permissions via The Administration | Roles | Administrators group in SCVMM VMM Service Account needs to be an administrator on each Hyper-V host it is to manage On each physical host, add service account to the Local Administrators group VMM Service Account User & System Policies In local security policy, under user rights assignments, ensure that new service account is located in Logon as a Service
After taking the time to ensure that all permissions, security roles, and policies were set for the new service account, it was time to flip the switch and restart VMM.
Solution: VMM doesn’t support changing Service Accounts like many Windows services. Re-install.
At first reading, you will probably fall out of your chair when you read “Re-install” but it isn’t nearly as bad as you might first believe. I found it a bit odd that this isn’t a supported solution to switch service accounts because enterprises often move or change domain structures, ownership, etc. and one of the first things many do is change service accounts to ones they own.
However, VMM has this option when uninstalling called “Retain Data” that basically instructs VMM setup to not modify or remove the VMM database (e.g. Virtual Manager DB). This retention capability is how you effectively implement the solution to changing your service account for VMM if you so desire.
How to change service account for VMM:
NOTE: I recommend that you backup the Virtual Machine database. For instructions on doing this, see the following TechNet article.
- Click Start, Programs, Programs & Features (assuming Windows 2008 or greater OS here)
- Local Virtual Machine Manager 2008 R2 Server and click Uninstall
- When prompted, select the option to Retain Data, and then click through to remove VMM 2008 R2
- Re-install VMM using the new service account when prompted what service account to use
Each physical host requires “Update Agent”
After the installation is complete you can now open the Virtual Machine Manager console and not get the error shown above. The one thing you will get is a warning sign next to each Host Group and\or physical server. To correct this, do the following:
- Right-click on each host in VMM Administrator
- Select Update Agent
- Provide credentials to connect to the host such as your Administrator credentials
You will now be successful at your goal – you’ve now changed the VMM Service Account.
Summary
Although this might be a sparse activity for many (changing Service Accounts), I was so shocked personally by the procedure (uninstall) that I simply couldn’t resist writing up this experience. I’m so accustomed to being able to easily change the credentials which a Windows service runs under that when I hear that an uninstall is required to do such a thing I was shocked!
Thus, if you are ever finding yourself in this situation please don’t spend anytime searching on how to do this – just uninstall and start over.
Enjoy!
Thanks,
-Chris