UR6 for SCOM 2012 R2 – Step by Step
KB Article for OpsMgr: https://support.microsoft.com/kb/3051169
KB Article for all System Center components: https://support.microsoft.com/en-us/kb/3051171
Download catalog site: https://catalog.update.microsoft.com/v7/site/Search.aspx?q=3051169
Key fixes:
-
- The "Remove-DisabledClassInstance" Windows PowerShell command times out without completing
The cmdlet takes lots of time to complete in large SCOM installations and sometimes times out without completing its work. This is fixed by optimizing the query that is used for deletion. - Duplicate closed alerts
The Find option in a message view sometimes gives results of the current search together with the results from previous searches. This occurs because of an update issue in the grid and is fixed so that values are displayed that correspond only to the current search. - Topology widget objects lose location when they are opened in a console that has a different locale and decimal format
Topology widgets that are created in one locale but are viewed from another locale display an incorrect location because of the widget's inability to interpret locale-specific formatting. This issue is fixed, and users can now create and view topology widgets from different locales. - WebConsole Details widget does not display anything
When you click a message in the Alert widget, the details of the message are not displayed in the details widget on WebConsole. This issue is fixed by fixing the XAML page of the details widget. The XAML page was preventing the data from being displayed on the page. - Top 10 Performance widgets (WebConsole) are sometimes empty
The Top 10 Performance widgets (WebConsole) are sometimes empty because of a performance issue. This issue was fixed in Update Rollup 3 through an MPB file that requires a manual import. Because many users missed importing the MPB, a fix is included in this update rollup as an SQL query update. - Problem with decoding SCOM trace log files
Sometimes Operations Manager trace logs do not decode but generate the error "Unknown(X ): GUID=XXX (No Format Information found)." This occurs because the latest TMF files that contain formatting information for the related traces are missing. All TMF files are updated to make sure that traces are formatted correctly.
- The "Remove-DisabledClassInstance" Windows PowerShell command times out without completing
Xplat updates:
-
JEE: WebLogic 12.1.3 servers on Linux or Solaris are not discovered
WebLogic 12cR1 Rel3 Java Application Servers that are installed on Linux and Solaris are not discovered.
Note The UNIX or Linux agent must be updated after you install the Management Pack update to resolve this issue.
Relevant log messages:
/var/opt/microsoft/scx/scx.log2015-02-04T15:28:18,797Z
Trace [scx.core.common.pal.system.appserver.appserverenumeration:379:15739:139715041470400] Weblogic process does not contain the 'platform.home' or 'weblogic.system.BootIdentityFile' commandline argument.
2015-02-04T15:28:18,797Z
Trace [scx.core.common.pal.system.appserver.appserverenumeration:462:15739:139715041470400] Merging previously known instances with current running processes
-
In rare cases, lots of omiagent processes may be observed on a UNIX or Linux computer
If a single instance of data that is returned by an OMI provider is larger than 64 kilobytes (KB), the omiagent process becomes unresponsive, and additional omiagent processes are created. This issue can occur on a managed UNIX- or Linux-based computer if a rule or monitor returns an instance of data larger than 64 KB. This is most likely to be seen when a custom shell command or script rules return lots of data in StdOut.
Note After you upgrade the Operations Manager UNIX/Linux agent to the version that is contained in this Management Pack update, an instance that exceeds 64 KB will be truncated and will generate the following message in the scx.log file. You can use the log message to identify and change the workflow that is returning too much data.ExecuteCommand: Exceeded maximum output size for provider (64k), output truncated. Monitoring will not be reliable! Command executed: "
The SSLv3 protocol cannot be disabled in OpenSSL as used by the UNIX and Linux agents
With this update, you can now disable the SSLv3 protocol for the Operations Manager UNIX and Linux agents. You can do this by adding a property to omiserver.conf (/etc/opt/microsoft/scx/conf). If SSLv3 is disabled, the UNIX and Linux agent will reject connections that cannot be negotiated by using TLS encryption. If SSLv3 is enabled (this is the default configuration), SSLv3 will be used only if TLS encryption cannot be negotiated.
omiserver.conf options for configuration SSLv3NoSSLv3=true|false
Enable/Disables use of SSLv3
Lets get started.
From reading the KB article – the order of operations is:
- Install the update rollup package on the following server infrastructure:
- Management servers
- Gateway servers
- Web console server role computers
- Operations console role computers
- Apply SQL scripts.
- Manually import the management packs.
- Update Agents
Now, we need to add another step – if we are using Xplat monitoring – need to update the Linux/Unix MP’s and agents.
5. Update Unix/Linux MP’s and Agents.
1. Management Servers
Since there is no RMS anymore, it doesn’t matter which management server I start with. There is no need to begin with whomever holds the RMSe role. I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.
I can apply this update manually via the MSP files, or I can use Windows Update. I have 3 management servers, so I will demonstrate both. I will do the first management server manually. This management server holds 3 roles, and each must be patched: Management Server, Web Console, and Console.
The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location:
Then extract the contents:
Once I have the MSP files, I am ready to start applying the update to each server by role.
***Note: You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator (SA) role to the database instances that host your OpsMgr databases.
My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:
This launches a quick UI which applies the update. It will bounce the SCOM services as well. The update does not provide any feedback that it had success or failure. You can check the application log for the MsiInstaller events for that:
Log Name: Application
Source: MsiInstaller
Date: 5/11/2015 12:39:05 PM
Event ID: 1036
Task Category: None
Level: Information
Keywords: Classic
User: OPSMGR\kevinhol
Computer: SCOM01.opsmgr.net
Description:
Windows Installer installed an update. Product Name: System Center Operations Manager 2012 Server. Product Version: 7.1.10226.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2012 R2 Operations Manager UR6 Update Patch. Installation success or error status: 0.
You can also spot check a couple DLL files for the file version attribute.
Next up – run the Web Console update:
This runs much faster. A quick file spot check:
Lastly – install the console update (make sure your console is closed):
A quick file spot check:
Secondary Management Servers:
I now move on to my secondary management servers, applying the server update, then the console update.
On this next management server, I will use the example of Windows Update as opposed to manually installing the MSP files. I check online, and make sure that I have configured Windows Update to give me updates for additional products:
This shows me three applicable updates for this server:
I apply these updates (along with some additional Windows Server Updates I was missing, and reboot each management server, until all management servers are updated.
Updating Gateways:
I can use Windows Update or manual installation.
The update launches a UI and quickly finishes.
Then I will spot check the DLL’s:
I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:
2. Apply the SQL Scripts
In the path on your management servers, where you installed/extracted the update, there are two SQL script files:
%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups
(note – your path may vary slightly depending on if you have an upgraded environment of clean install)
First – let’s run the script to update the OperationsManager database. Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file. Make sure it is pointing to your OperationsManager database, then execute the script.
You should run this script with each UR, even if you ran this on a previous UR. The script body can change so as a best practice always re-run this.
Click the “Execute” button in SQL mgmt. studio. The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.
You will see the following (or similar) output:
or
IF YOU GET AN ERROR – STOP! Do not continue. Try re-running the script several times until it completes without errors. In a large environment, you might have to run this several times, or even potentially shut down the services on your management servers, to break their connection to the databases, to get a successful run.
Technical tidbit: Even if you previously ran this script in UR1, UR2, UR3, UR4, or UR5, you should run this again for UR6, as the script body can change with updated UR’s.
Next, we have a script to run against the warehouse DB. Do not skip this step under any circumstances. From:
%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups
(note – your path may vary slightly depending on if you have an upgraded environment of clean install)
Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file UR_Datawarehouse.sql. Make sure it is pointing to your OperationsManagerDW database, then execute the script.
If you see a warning about line endings, choose Yes to continue.
Click the “Execute” button in SQL mgmt. studio. The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.
You will see the following (or similar) output:
3. Manually import the management packs?
There are 25 management packs in this update!
The path for these is on your management server, after you have installed the “Server” update:
\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Management Packs for Update Rollups
However, the majority of them are Advisor, and language specific. Only import the ones you need, and that are correct for your language. I will remove all the Advisor MP’s for other languages, and I am left with the following:
The TFS MP bundles are only used for specific scenarios, such as DevOps scenarios where you have integrated APM with TFS, etc. If you are not currently using these MP’s, there is no need to import or update them. I’d skip this MP import unless you already have these MP’s present in your environment.
The Advisor MP’s are only needed if you are using System Center Operational Insights (Previously known as Advisor) services.
However, the Image and Visualization libraries deal with Dashboard updates, and these always need to be updated.
I import all of these shown without issue.
4. Update Agents
Agents should be placed into pending actions by this update (mine worked great) for any agent that was not manually installed (remotely manageable = yes): One the Management servers where I used Windows Update to patch them, their agents did not show up in this list. Only agents where I manually patched their management server showed up in this list. FYI.
If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending.
In this case – my agents that were reporting to a management server that was updated using Windows Update – did NOT place agents into pending. Only the agents reporting to the management server for which I manually executed the patch worked.
You can approve these – which will result in a success message once complete:
Soon you should start to see PatchList getting filled in from the Agents By Version view under Operations Manager monitoring folder in the console:
5. Update Unix/Linux MPs and Agents
Next up – I download and extract the updated Linux MP’s for SCOM 2012
https://www.microsoft.com/en-us/download/details.aspx?id=29696
7.5.1043.0 is current at this time for SCOM 2012 R2 UR6.
****Note – take GREAT care when downloading – that you select the correct download for R2. You must scroll down in the list and select the MSI for 2012 R2:
Download the MSI and run it. It will extract the MP’s to C:\Program Files (x86)\System Center Management Packs\System Center 2012 R2 Management Packs for Unix and Linux\
Update any MP’s you are already using. These are mine for RHEL, SUSE, and the Universal Linux libraries.
You will likely observe VERY high CPU utilization of your management servers and database server during and immediately following these MP imports. Give it plenty of time to complete the process of the import and MPB deployments.
Next up – you would upgrade your agents on the Unix/Linux monitored agents. You can now do this straight from the console:
You can input credentials or use existing RunAs accounts if those have enough rights to perform this action.
Mine FAILED, with an SSH exception about copying the new agent. It turns out my files were not updated on the management server – see pic:
I had to restart the Healthservice on the management server, and within a few minutes all the new files were there.
Finally:
6. Update the remaining deployed consoles
This is an important step. I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc. These should all get the matching update version.
Review:
Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.
Known issues:
See the existing list of known issues documented in the KB article.
1. Many people are reporting that the SQL script is failing to complete when executed. You should attempt to run this multiple times until it completes without error. You might need to stop the Exchange correlation engine, stop all the SCOM services on the management servers, and/or bounce the SQL server services in order to get a successful completion in a busy management group. The errors reported appear as below:
------------------------------------------------------
(1 row(s) affected)
(1 row(s) affected)
Msg 1205, Level 13, State 56, Line 1
Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Msg 3727, Level 16, State 0, Line 1
Could not drop constraint. See previous errors.
--------------------------------------------------------
Comments
Anonymous
January 01, 2003
@Rick - generally the patchlist is pretty reliable. Sometimes agents need a reboot before they submit the discovery data in - it takes some time to get them all.
If you log in and want to check - spot check a DLL that gets updated, same as I do my management server examples above.Anonymous
January 01, 2003
@Guru -
Update Rollups are a Rollup - they are cumulative. There is no requirement for any other previous hotfixes.Anonymous
January 01, 2003
@ Imtiaz -
I'd verify you updated the visualization management packs included with the UR6, and then see if you can repro by installing another web console server somewhere else (on a management server preferred), and then if those fail open a support case.Anonymous
January 01, 2003
@Francisco -
Just make sure you are looking at the exact files I reference. For instance - the web console only updates a single file in that directory. Also make sure you are using the correct localized update if non-English.Anonymous
January 01, 2003
@Francisco - No, Update Rollups are cumulative.Anonymous
January 01, 2003
The comment has been removedAnonymous
January 01, 2003
@ jv1031 -
There are no known issues like that with UR6, so it sounds very much like something environmental in your deployment.Anonymous
January 01, 2003
@Visitor -
No. My article doesn't "miss" this step. I choose not to apply it. The KB article calls out all kinds of scenarios, registry keys, etc..... that should NOT be changed from the default UNLESS you are specifically affected by that issue. My article calls out the specific steps that everyone should apply. Then, IF you are having a specific problem with the web console as called out in that scenario (which I have never seen) then by all means people should apply those changes.Anonymous
May 11, 2015
This is updated as of 5-11-2015 In general - you should evaluate all hotfixes available, and only applyAnonymous
May 12, 2015
SCOM Console update - " These should all get the UR5 update." I guess you meant all should get the UR6 update.. :)Anonymous
May 13, 2015
I have about 50 agents in my lab SCOM environment. I approved them for update and the update succeeded for most. Nonetheless, only a few of those that were successfully updated now show up in the Agent by Version with the Patch List correctly reflecting that they're UR6. I don't understand why a few show up correctly, but many do not. I also noticed that if I log on to a device that has been successfully updated, there's nothing easily found that indicates that the agent update worked - e.g. the Microsoft Monitoing Agent applet in the Control Panel, Properties Panel shows the version as 7.1.10184.0. It just seems like it should be easier to tell whether an agent updated was successfulAnonymous
May 14, 2015
Thanks for the feedback, Kevin. When I checked several of the agent files, they seem to be the same dates/sizes as the files on a server that is reporting as UR6 in the Patch List. I did try some more things -- remove agent from the server and then put on base agent followed by the UR6 update. I rebooted the agent machine. I rebooted the SCOM server. No joy. I find the Patch List useful, so I'd really like to know what I can do to get them all reporting correctly. Time to open a case with MS Support?Anonymous
May 18, 2015
after installing this UR no alerts appear on the console from any server, waited a couple of days and nothing, rebooted server, it is connect to microsoft operational insights. i have done this twice. i restored server from backup and both times the issue remains.
no previous urs installed.Anonymous
May 19, 2015
Hi Kevin
Thank you for writing this. I've updated my SCOM environment according to your blog.
The issue I had with UR5 was "WebConsole Details widget does not display anything"
Now everything is updated to UR6 successfully. Even the SQL scripts.
But... the main issue remains :-(
When creating summary dashboard with Top N widget in the console, the data is displayed correctly.
The WebConsole shows the only the column layout with names but no data.
Suggestions will be very appreciated!Anonymous
May 21, 2015
Hi Kevin, thanks for your amazing work! i have a question, should I have installed UR5 before this one?Anonymous
May 26, 2015
In the Microsoft KB-Article for UR 6 there is a section where a modification of the web.config is described.
This section misses on your Step-by-Step-Guide :)Anonymous
May 26, 2015
is there a good troubleshooting guide for SCOM?Anonymous
May 26, 2015
Hey Kevin, I have 2012 R2 UR4 right now, can I directly update to UR6 without applying UR5?Anonymous
May 26, 2015
Thanks Kevin for your reply, I am following your guide but I don't see my dlls getting update, just the server part but not web console and console, they remain in v 7.1.10226.0 (and should be in 7.1.10226.1064).
I have closed the consoles, rebooted the servers and reinstalled the updates but they remain in v .0, is this ok or I'm doing something wrong?
Thanks in advice for your help.Anonymous
May 27, 2015
I am unable to download UR6 from https://support.microsoft.com/en-us/kb/3051169, Site seems to be down!!!Anonymous
May 28, 2015
Hi Kevin.
In my test environment, Advisor had changed it name to OpsInsights, but after upgrading production, it still says advisor. Do you know if i am missing a key piece here? All Advisor MP's seems to be the same version in test and productionAnonymous
May 28, 2015
The comment has been removedAnonymous
May 29, 2015
The comment has been removedAnonymous
June 01, 2015
The comment has been removedAnonymous
June 01, 2015
Kevin – Thank you very much, that did it, all my agents are now upgraded. I Appreciated your help.Anonymous
June 02, 2015
after the update the console no longer displays new alerts, I have cleared cache, restarted. no go.Anonymous
June 03, 2015
Hi, after i upgraded to UR6 i'm receiving alot errors mostly from all our domain controllers, i.E:
- ADWS Service Status : The script AD_ADWS_Service.vbs failed retrieve the service state. The error was:
The error returned was: 'Object required' (0x1A8)
- AD CPU Overload : The script AD LSASS CPU failed to retrieve the raw performance data for the lsass process via WMI.
The error was: The error returned was: 'Object required' (0x1A8)
- AD Replication Monitoring : encountered a runtime error.
Failed to convert to UTC time.
The error returned was: '' (0x80041013)
And some more, anyone saw similar errors on the domain controllers and have any idea to fix them?
Regards,
OliverAnonymous
June 03, 2015
@Oliver - that sounds more like a memory leak on the OS for the domain controller when we see those specific errors. When was their last reboot and have you patched them fully?Anonymous
June 03, 2015
@Kevin: you could be right, i guess we have serious issues on our domain controller,. just wanted to check the up time ;-): C:Windowssystem32>systeminfo | find "Up Time"
Loading Operating System Information ...ERROR: No more threads can be created in
the system.
I'll reboot some DC's tonight and check if its going better.Anonymous
June 05, 2015
@Francisco @Kevin
Just confirming, from within the Operations Console, going to Help > About, the console version will still show 7.1.10226.0 after UR6 upgrade, not 7.1.10226.1064Anonymous
June 11, 2015
"@Francisco @Kevin
Just confirming, from within the Operations Console, going to Help > About, the console version will still show 7.1.10226.0 after UR6 upgrade, not 7.1.10226.1064"
I have the same issue.
Je recontre le même pb. L'upgrade semble être fait, mais le build number n'a pas changé.Anonymous
June 16, 2015
Hi Kevin - just wanted to say great job on the guide, this is the second one I've used and so far, I've not encountered any issues. Keep up the good work ! DaveAnonymous
June 23, 2015
Im deploying a fresh SCOM R2 install in my org, Installed the media, followed the guide to update to UR6, but the Version number hasn't changed, any ideas?Anonymous
June 23, 2015
Great step by step tutorial, thanks. I've got to do this today for a clientAnonymous
June 23, 2015
Thanks great tutorial, i've just finished UR5 last month and client today asked for me to update to UR6. This should helpAnonymous
June 23, 2015
Thanks great tutorial, i've just finished UR5 last month and client today asked for me to update to UR6. This should helpAnonymous
June 23, 2015
Thanks great tutorial, i've just finished UR5 last month and client today asked for me to update to UR6. This should helpAnonymous
June 23, 2015
Hi Kevin,
A little issue I encountered while updating agents and how I used ORCH 2012 R2 to resolve it. Runbook included in post
http://mundoscom.blogspot.ca/2015/06/health-service-failure-runbook.htmlAnonymous
July 01, 2015
I performed this update today - all of the installations were successful, All of my agents went into pending but when I attempt to approve a pending agent the console freezes. I don't get any error messages, or a "not responding" tag. but after 2 hours I had to manually end the task.
Any thoughts?Anonymous
July 09, 2015
Hey Imtiaz Ahmad, Make sure that your web server is also a management server. I've had problems where powershell doesn't run on any web console, if the web server is also not a management serverAnonymous
July 09, 2015
Thanks Kevin, you are a gem!Anonymous
July 14, 2015
Product REGISTRATION failed. Hi Kevin, just installed SCOM 2012 R2 on my production environment. This is clean installation, not an upgrade. I'm assuming the problem I have is because I've tried to register SCOM after update to UR6 - am I right? Do I have to scrap it and install it from scratch?
I’ve seen UR6 uninstall options, but what with the DB update script? Can this action be reversed as well?
I have also problem with reports-charts are missing. “A generic error occurred in GDI+” I’ve seen some solutions for SQL 2008, but my is DB sits on SQL 2012 SP1. I came across a note about kb2823372, but I can’t install it on my 2012 R2 server.
Thank you in advanced.Anonymous
July 14, 2015
@Ralph -
I have not seen where applying UR6 will not allow you to register your product. I seriously doubt I could repro this. Can you?
You will have to be more specific on the reports/charts issue. Sounds like a .NET issue on the machine you are running the console.Anonymous
July 14, 2015
Product REGISTRATION failed. Hi Kevin, firstly I'd like to thank you for prompt reply. Registration problem has been sorted out. My bad... I didn't run it as an admin.
I'm happy to say you were right again - .NET issue. Reports were generated as they should just without charts. There was a red cross or a message “A generic error occurred in GDI+” instead proper graphs. To double-check quickly I installed the console on one of our DCs and it's running like a charm :)
BTW. I'm a big fan of your blog for years! Thank you so much for your all work and efforts to help SCOM admins. Best regards!Anonymous
July 23, 2015
Great write up Kevin. Thanks. Something odd I've noticed with UR updates in 2012R2 is that the old agent .msp file is not removed from the AgentManagement directory. I'm pretty sure in 2007 R2 updates the old agent update files were deleted leaving only the current file. This isn't a big deal of course, but we have some automation that applies KB*.msp during agent installation from that directory. We just rename / delete the old files.Anonymous
July 23, 2015
@Doug -
You are correct. Originally we didn't clean them up, then we did, now we regressed and aren't cleaning them up again. I pointed this out to the PG but they didn't feel it was a big deal I suppose. I prefer it cleaned up because it is, well, cleaner. Supposedly, the MSI examines the update files and ignores the old ones. From looking at the install logs, that appears to be the case. There is no harm in manually deleting the old unused ones from the folder, but it doesn't hurt to leave them there, either.Anonymous
July 23, 2015
Hi Kelvin,
Is any new Dashboard Widgets comes with UR6 ?Anonymous
July 28, 2015
Cannot install UR6 on either of two management servers. Getting error 1603 on MSI in event viewer and capturing a log only shows errors related to APM performance counters? I'm definitely a local admin and scom admin and ran from elevated command prompt on server 2012 R2 Hyper-V VM.
I have a ZIP of the full log file but it's too bloated to post. As I said, I tried this on two different Mgmt servers.
Thanks.Anonymous
July 29, 2015
Resolved UR6 install on the main Mgmt server by installing all updates. UR6 then installed successfully with no more loadperf errors. The other DB/Mgmt server is having the same issue even after all updated. MSI Installer cranks up and then shortly multiple 3011 and 3012 errors from Loadperf after which the UR6 fails.
Any thoughts anyone?Anonymous
July 29, 2015
lodctr /R resolved that issue on the other server. Also running SCEP which I temporarily disabled.Anonymous
August 07, 2015
I have updated our Lab from SCOM 2012 SP1 to R2 and I have no issues during the upgrade. When I was installing the UR6 patch for the console I am encountered the error of the "The upgrade patch cannot be installed by the Windows Installer service because the program to be upgraded may be missing. or the upgrade patch may update different version of the program. Verify that the program to be upgraded exists on your computer and that you have the correct patch.
Current version of console: 7.1.10226.0
MS version: 7.1.10226.1064
Can you please help? I should be able to upgrade directly to UR6 from SCOM 2012 R2 right?
Thank you.Anonymous
September 01, 2015
Great article! Clear instructions and detailed explainations. Thank you!Anonymous
December 31, 2016
HiThanks for your helpful article.I have a problem with scom web console that alert detail view is empty.i have installed UR2, UR3, UR11 and then UR6 but my problem not fixed yet.what shall i do?- Anonymous
December 31, 2016
Wait, you installed UR11, then went and installed ur6?
- Anonymous