UR3 for SCOM 2012 R2 – Step by Step
KB Article for OpsMgr: https://support.microsoft.com/kb/2965445
KB Article for all System Center components: https://support.microsoft.com/kb/2965090
Download catalog site: https://catalog.update.microsoft.com/v7/site/Search.aspx?q=2965445
Key fixes:
- Reliability fix: A deadlock condition occurs when a database is connected after an outage. You may experience this issue may when one or more HealthServices services in the environment are listed as Unavailable after a database goes offline and then comes back online. Management servers cannot reconnect to SQL after a SQL outage because of thread exhaustion.
- The Desktop console crashes after exception TargetInvocationException occurs when the TilesContainer is updated. You may experience this issue after you leave the console open on a Dashboard view for a long time.
- The Password expiration monitor is fixed for logged events. To make troubleshooting easier, this fix adds more detail to Event IDs 7019 and 7020 when they occur.
- The Health service bounces because of high memory usage in the instance MonitoringHost: leak MOMModules!CMOMClusterResource::InitializeInstance. This issue may be seen as high memory usage if you examine monitoringhost.exe in Performance Monitor. Or, the Health service may restart every couple of days , depending on the load on the server.
- The Health service crashes in Windows HTTP Services (WinHTTP) if the RunAs account is not read correctly.
- Windows PowerShell stops working with System.Management.Automation.PSSnapInReader.ReadEnginePSSnapIns. You may see this issue as Event ID 22400 together with a description of "Failed to run the Powershell script."
- The PropertyValue column in the contextual details widget is unreadable in smaller widget sizes because the PropertyName column uses too much space.
- The update threshold for monitor "Health Service Handle Count Threshold" is reset to 30,000. You can see this issue in the environment, and the Health Service Handle Count Threshold monitor is listed in the critical state.
- An acknowledgement (ACK) is delayed by write collisions in MS queue when lots of data is sent from 1,000 agents.
- The execution of the Export-SCOMEffectiveMonitoringConfiguration cmdlet fails with the error "Subquery returned more than 1 value.”
- The MOMScriptAPI.ReturnItems method can be slow because a process race condition may occur when many items are returned, and the method may take two seconds between items. Scripts may run slowly in the System Center Operations Manager environment.
- When you are in the console and click Authoring, click Management Pack, click Objects, and then click Attributes to perform a Find operation, the Find operations seems unexpectedly slow. Additionally, the Momcache.mdb file grows very large.
- A delta synchronization times out on SQL operations with Event ID 29181.
- Operations Manager grooms out the alert history before an alert is closed.
- The time-zone settings are not added to a subscription when non-English display languages are set. Additionally, time stamps on alert notifications are inaccurate for the time zone.
- Web Browser widget requires the protocol (http or https) to be included in the URL.
- You cannot access MonitoringHost's TemporaryStoragePath within the PowerShell Module.
- The TopNEntitiesByPerfGet stored procedure may cause an Operations Manager dashboard performance issue. This issue may occur when a dashboard is run together with multiple widgets. Additionally, you may receive the following error message after a time-out occurs:
[Error] :DataProviderCommandMethod.Invoke{dataprovidercommandmethod_cs370}( 000000000371AA78 )
An unknown exception was caught during invocation and will be re-wrapped in a DataAccessException. System.TimeoutException: The operation has timed out. at Microsoft.EnterpriseManagement.Monitoring.DataProviders.RetryCommandExecutionStrategy.Invoke(IDataProviderCommandMethodInvoker invoker) at Microsoft.EnterpriseManagement.Presentation.DataAccess.DataProviderCommandMethod.Invoke(CoreDataGateway gateWay, DataCommand command)
Xplat updates:
Slow results are returned when you run the Get-SCXAgent cmdlet or view UNIX/Linux computers in the administration pane for lots of managed UNIX/Linux computers.
Note To apply this hotfix, you must have version 7.5.1025.0 or later of the UNIX/Linux Process Monitoring, UNIX/Linux Log File Monitoring, and UNIX/Linux Shell Command Template management pack bundles.Accessing the UNIX/Linux computers view in the administration pane can sometimes trigger the following exception message:
Microsoft.SystemCenter.CrossPlatform.ClientLibrary.Common.SDKAbstraction.ManagedObjectNotFoundException
Lets get started.
From reading the KB article – the order of operations is:
- Install the update rollup package on the following server infrastructure:
- Management servers
- Gateway servers
- Web console server role computers
- Operations console role computers
- Apply SQL scripts.
- Manually import the management packs.
- Update Agents
Now, we need to add another step – if we are using Xplat monitoring – need to update the Linux/Unix MP’s and agents.
5. Update Unix/Linux MP’s and Agents.
1. Management Servers
Since there is no RMS anymore, it doesn’t matter which management server I start with. There is no need to begin with whomever holds the RMSe role. I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.
I can apply this update manually via the MSP files, or I can use Windows Update. I have 3 management servers, so I will demonstrate both. I will do the first management server manually. This management server holds 3 roles, and each must be patched: Management Server, Web Console, and Console.
The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location:
Then extract the contents:
Once I have the MSP files, I am ready to start applying the update to each server by role.
***Note: You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator (SA) role to the database instances that host your OpsMgr databases.
My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:
This launches a quick UI which applies the update. It will bounce the SCOM services as well. The update does not provide any feedback that it had success or failure. You can check the application log for the MsiInstaller events for that:
Log Name: Application
Source: MsiInstaller
Date: 8/6/2014 3:00:46 PM
Event ID: 1022
Task Category: None
Level: Information
Keywords: Classic
User: OPSMGR\kevinhol
Computer: SCOM01.opsmgr.net
Description:
Product: System Center Operations Manager 2012 Server - Update 'System Center 2012 R2 Operations Manager UR3 Update Patch' installed successfully.
You can also spot check a couple DLL files for the file version attribute.
Next up – run the Web Console update:
This runs much faster. A quick file spot check:
Lastly – install the console update (make sure your console is closed):
A quick file spot check:
Secondary Management Servers:
I now move on to my secondary management servers, applying the server update, then the console update.
On this next management server, I will use the example of Windows Update as opposed to manually installing the MSP files. I check online, and make sure that I have configured Windows Update to give me updates for additional products:
This shows me two applicable updates for this server:
I apply these updates (along with some additional Windows Server Updates I was missing, and reboot each management server, until all management servers are updated.
Updating Gateways:
I can use Windows Update or manual installation.
The update launches a UI and quickly finishes.
Then I will spot check the DLL’s:
I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:
2. Apply the SQL Scripts
In the path on your management servers, where you installed/extracted the update, there are two SQL script files:
%SystemDrive%\Program Files\System Center 2012\Operations Manager\Server\SQL Script for Update Rollups
First – let’s run the script to update the OperationsManager database. Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file. Make sure it is pointing to your OperationsManager database, then execute the script.
Click the “Execute” button in SQL mgmt. studio. The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.
You will see the following (or similar) output:
or
IF YOU GET AN ERROR – STOP! Do not continue. Try re-running the script several times until it completes without errors. In a large environment, you might have to run this several times, or even potentially shut down the services on your management servers, to break their connection to the databases, to get a successful run.
Technical tidbit: This script has been updated in UR3. Even if you previously ran this script in UR1 or UR2, you must run this again.
Next, we have a script in UR3 to run against the warehouse DB. Do not skip this step under any circumstances. From:
%SystemDrive%\Program Files\System Center 2012\Operations Manager\Server\SQL Script for Update Rollups
Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file UR_Datawarehouse.sql. Make sure it is pointing to your OperationsManagerDW database, then execute the script.
If you see a warning about line endings, choose Yes to continue.
Click the “Execute” button in SQL mgmt. studio. The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.
You will see the following (or similar) output:
3. Manually import the management packs?
We have 6 updated MP’s to import (MAYBE!).
The TFS MP bundles are only used for specific scenarios, such as DevOps scenarios where you have integrated APM with TFS, etc. If you are not currently using these MP’s, there is no need to import or update them. I’d skip this MP import unless you already have these MP’s present in your environment.
The Advisor MP’s are only needed if you are using System Center Advisor services.
However, the Image and Visualization libraries deal with Dashboard updates, and these need to be updated.
I import all of these without issue.
4. Update Agents
Agents should be placed into pending actions by this update (mine worked great) for any agent that was not manually installed (remotely manageable = yes):
If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending
You can approve these – which will result in a success message once complete:
Soon you should start to see PatchList getting filled in from the Agents By Version view under Operations Manager monitoring folder in the console:
5. Update Unix/Linux MPs and Agents
Next up – I download and extract the updated Linux MP’s for SCOM 2012 SP1 UR3
https://www.microsoft.com/en-us/download/details.aspx?id=29696
7.5.1025.0 is current at this time for SCOM 2012 R2 UR2.
****Note – take GREAT care when downloading – that you select the correct download for R2. You must scroll down in the list and select the MSI for 2012 R2:
Download the MSI and run it. It will extract the MP’s to C:\Program Files (x86)\System Center Management Packs\System Center 2012 R2 Management Packs for Unix and Linux\
Update any MP’s you are already using. These are mine for RHEL, SUSE, and the universal Linux libraries:
You will likely observe VERY high CPU utilization of your management servers and database server during and immediately following these MP imports. Give it plenty of time to complete the process of the import and MPB deployments.
Next up – you would upgrade your agents on the Unix/Linux monitored agents. You can now do this straight from the console:
You can input credentials or use existing RunAs accounts if those have enough rights to perform this action.
6. Update the remaining deployed consoles
This is an important step. I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc. These should all get the UR3 update.
Review:
Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.
Known issues:
See the existing list of known issues documented in the KB article.
1. Many people are reporting that the SQL script is failing to complete when executed. You should attempt to run this multiple times until it completes without error. You might need to stop the Exchange correlation engine, stop the services on the management servers, or bounce the SQL server services in order to get a successful completion in a busy management group. The errors reported appear as below:
------------------------------------------------------
(1 row(s) affected)
(1 row(s) affected)
Msg 1205, Level 13, State 56, Line 1
Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Msg 3727, Level 16, State 0, Line 1
Could not drop constraint. See previous errors.
--------------------------------------------------------
Comments
- Anonymous
January 01, 2003
Xahmed: As far as I know, these updates are cumulative, so you don't need to deploy UR1 and UR2, you can just deploy UR3.
Kevin: Thanks for this comprehensive guide on updating. Love the part about checking DLL versions to see if the update actually was applied. - Anonymous
January 01, 2003
Thank You. - Anonymous
January 01, 2003
Thank you - Anonymous
January 01, 2003
The comment has been removed - Anonymous
January 01, 2003
@steven - No. Update Rollups are cumulative. - Anonymous
January 01, 2003
@Steve -
The KB article is in error. I will notify those who own that to get it updated. Those were issues in UR2, in my testing they do not exist as issues in UR3. Thanks for pointing that out. - Anonymous
January 01, 2003
The comment has been removed - Anonymous
August 07, 2014
The Known Issues section of the KB article indicates that the agents need to be updated via the Repair Install process, not by simply approving them from the Pending Management view. Is this not correct? Your article indicates that they can be updated normally from the Pending Management view. - Anonymous
August 07, 2014
Thank you Kevin for a detailed step-by-step! Really appreciate it. - Anonymous
August 11, 2014
This is updated as of 8-11-2014 In general - you should evaluate all hotfixes available, and only apply - Anonymous
August 11, 2014
Good to see you back! However, this webpage appears to render images with the word "image" and not a real pic in ff and in IE I get the red X. - Anonymous
August 14, 2014
We did the update here and even in our tiny environ (87 agents, 2 MS, 2 GW), we had to stop the SCOM services on our MSs to do the SQL scripts. Otherwise, it went flawlessly, we had a couple of agents that didn't update, but that had nothing to do with SCOM (full C: drives, anyone?) - Anonymous
August 15, 2014
Glad you are back. Thanks for the details! - Anonymous
August 20, 2014
We update our agents manually (via SCCM). So far I have installed the URS agent update manually to about 10 servers. Every server wanted to reboot. I thought Microsoft committed to no reboots for agent updates. We can suppress the reboot in SCCM but will this cause any issues with the agent? - Anonymous
August 25, 2014
Hi Kevin,
A few days ago I've started receiving alerts in SCOM regarding "Failed to store data in the Data Warehouse. The operation will be retried.
Exception 'InvalidOperationException': The given value of type String from the data source cannot be converted to type nvarchar of the specified target column".
I've searched through the net and didn't find any proper solution regarding this issue.
In addition since I get these errors I'm experiencing problems with my RMS. Every time and now it turns to gray along with the Gateway server and only after I restart the SCOM services it returns to green.
1. Do you think that these issues are related ?
2. Is this fix mentioned in the post here can fix this issue ?
Thx in advance.
Dudu Sakharovich - Anonymous
August 26, 2014
The comment has been removed - Anonymous
August 26, 2014
Hi actually i'm planning to install new SCOM 2012 R2 so shall i install only UR3 or start from UR1 until UR3.
Thanks - Anonymous
September 10, 2014
cool and useful docu, thanks a lot! - Anonymous
September 11, 2014
Here is a simple work around.
Known issues: Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Stop all management server services (Monitoring, Data Access, Management system) on all the Management servers and rerun the SQL update.
Results Command completed Successfully. - Anonymous
September 16, 2014
Hi all,
during installation of our agents, I noticed an error event about MsiInstaller:
Product: Microsoft Monitoring Agent -- Error 1923.Service '@C:Windowssystem32AdtAgent.exe,-500' (AdtAgent) could not be installed. Verify that you have sufficient privileges to install system services.
My account is Domain Admin and machine administrator, also the UR3 update is correctly installed:
Product: Microsoft Monitoring Agent - Update 'System Center 2012 R2 Operations Manager UR3 Update Patch' installed successfully.
Can I safely ignore this event?
Thanks,
David - Anonymous
September 23, 2014
The comment has been removed - Anonymous
September 30, 2014
you are simply the best in SCOM - Anonymous
October 13, 2014
Hi, i want to ask that i have two Management server now after upgrading my servers with above RU if later i add another management server or OpsMgr server role in current hierarchy then what would be case, mean i have to update that new server and SQL datbases? - Anonymous
October 14, 2014
Kevin good afternoon I already manually updated a fresh SCOM 2012 R2 install to the UR3 as it is pretty well explained in this post, but after that I noticed that the new widgets view does not show the new one, I am pretty sure that I followed this guide step by step, and the SQL scripts run OK. I appreciate any coment. Thanks in advance. - Anonymous
October 15, 2014
Hi Kevin,
We have updated from SCOM 2012 SP1 to R2 CUR3 and after the upgrade we tried to update the Unix SCOM agent and found that when we clicked update agent it caused the SCOM console to hang. Have tried SP1 and R2 consoles and both have same issue.
Anyone experience this? - Anonymous
October 16, 2014
Hi Kevin,
Worked it out, turns out that the SQL broker was not enabled, once enabled this looks to have fixed the issue. - Anonymous
October 17, 2014
Why did you extract KB2965445-i386-Agent and KB2965445-AMD64-Agent ?
You didn't use or run these files anywhere in this guide...
Please explain what needs to be done with these files - Anonymous
October 20, 2014
do i need to update the previous update UR1 & UR2 before the UR3? - Anonymous
November 11, 2014
Has anyone seen an agent in the "Agents by Version" view not show the updated patch level even though the Application log on the agent machine shows it was updated properly? Out of my ~10 test agents one is having an issue. I've cleared the cache, restarted the agent, checked logs, everything looks good except it's not reporting properly in the OM console. - Anonymous
December 18, 2014
Thanks Kevin. - Anonymous
February 02, 2015
The comment has been removed - Anonymous
February 10, 2015
The comment has been removed - Anonymous
February 10, 2015
Let me clarify, I get these errors when I'm trying to run the Data Warehouse sql command