System Center Operations Manager (SCOM) Management Group Performance Optimizations
Introduction
The performance of a System Center Operations Manager (SCOM) management group is often a discussion topic for many organizations. In most cases users feel that the performance was either not optimal since the initial installation or has degraded over time. Some of the most common pain points are slow or unresponsive Operations Console and failing workflows or data inserts. The same challenges were often described in various threads on the Operations Manager Microsoft Technet forums.
This article is actually a collection of SCOM optimization and tuning tips. The information is presented in form of a list and each recommendation in it is backed up, using either a quotation from a Microsoft Docs or Blog article or the statement of a renown subject matter expert. The statements below are considered generic and apply for any of the recent SCOM versions, like 2012 R2, 2016 and 2019.
The Challenge
From:
Operations Manager SQL Server design considerations (Optimizing SQL Server)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#optimizing-sql-server
“In general, previous deployment experience with customers shows that performance issues are typically not caused by high resource utilization (that is, processor or memory) with SQL Server itself; rather it is directly related to the configuration of the storage subsystem. Performance bottlenecks are commonly attributed to not following recommended configuration guidance with the storage provisioned for the SQL Server database instance.”
** **
The essence of this statements is that the SCOM backend and particularly the storage subsystem is of major importance for the performance of your Operations Manager Management Group.
List of recommendations
**1. ** Sizing Helper Tool
Use the SCOM Sizing Helper Tool to estimate the sizes of the SCOM databases (both Operational and Data Warehouse)
From:
Operations Manager SQL Server design considerations (Initial database sizing)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#initial-database-sizing
“The initial database size, as suggested by the Sizing Helper, should be allocated to its predicted size, to reduce fragmentation and corresponding overhead, which can be specified at setup time for the Operational and Data Warehouse databases. If during setup not enough storage space is available, the databases can be expanded later by using SQL Management Studio and then re-indexed thereafter to defragment and optimize accordingly. This recommendation applies also to the ACS database"
**Sizing Helper Tool:
**Sizing SCOM 2012R2 and 2016
https://blogs.technet.microsoft.com/kevinjustin/2017/01/06/sizing-scom-2012r2-and-2016/
** **
The correct sizing should be done according to the features, used (or planned to be used) within the Management Group. The tool can be used, of course also for sizing SCOM 2019 Management Groups.
**2. ** Operational Database
** **
2.1. Pre-sizing the Operational Database
Pre-size the SCOM Operational Database.
Kevin Holman, from:
What SQL maintenance should I perform on my SCOM 2016 databases?
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
“Ensure you pre-size your databases and logs so they are not always auto-growing, have plenty of free space as required to be supported.”
2.2. Operational Database free space
Ensure the Operational Database has enough free space.
Kevin Holman, from:
What SQL maintenance should I perform on my SCOM 2016 databases?
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
“The Operations Database needs 50% free space at all times. This is for growth, and for re-index operations to be successful. This is a general supportability recommendation, but the OpsDB will alert when this falls below 40%.”
2.3. Operational Database Transaction log Size
A good practice when sizing the Operational Database Transaction Log is to configure it to be 20% - 50% of DB file size.
Kevin Holman, from:
What SQL maintenance should I perform on my SCOM 2016 databases?
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
“For transaction log sizing - we don't have any hard rules. A good rule of thumb for the OpsDB is ~20% to 50% of the database size....”
**Jonathan Almquist (SCOM Expert and former Microsoft Premier Field Engineer), **from:
OpsMgr, OpsMgrDW vs transaction log ratio
https://social.technet.microsoft.com/Forums/systemcenter/en-US/a05a358d-0017-4075-ae06-aa32fbf1a07a/opsmgr-opsmgrdw-vs-transaction-log-ratio?forum=operationsmanagerdeployment
“I like to set the log file 50% of the data file. This offers plenty of space for the operational database to perform its regular grooming, and also gives you an insurance policy in cases where an alert storm happens. Alert storms are notorious for stopping the groom procedures, which will cause the database to continue growing until there is zero space and SCOM is completely down - you want to stay away from this scenario at any cost, and sizing log file to 50% of data file is a safe setting without over-allocating.”
2.4. Ensure Autogrow is disabled
There are a couple of important reasons NOT to enable autogrow on the Operational Database:
· Performance
When large amount of data is inserted into the database (alert storms for example) and it needs to grow in size, the backend could experience a major performance degradation, because all transactions are temporary suspended during the growth.
· Rapid Database size growth
If for some reason (poor workflow design - rules, monitors, discoveries, etc.) large amount of data (performance data, alerts) is inserted into the database and autogrow is enabled, the database could expand at very high pace and its size could get enormous and overtake your SQL storage subsystem. The actual problem with this is that the database cannot be shrunk in such cases, because shrinking it can cause serious fragmentation problems, which are very hard to resolve and the only way out of it would be moving the data to a new database.
Sameer Mhaisekar, a SCOM community contributor has written a very useful article on this particular topic (autogrow). Please check it out here:
Should You Enable Autogrowth on SCOM Operations Database?
https://social.technet.microsoft.com/wiki/contents/articles/52398.should-you-enable-autogrowth-on-scom-operations-database.aspx
2.5. Operational Databse Recovery Model
Choosing the proper setting when it comes to the recovery model of the SCOM databases is also an important point.
Kevin Holman, from:
What SQL maintenance should I perform on my SCOM 2016 databases?
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
“We default to “simple” for all our DB’s. This should be left alone…. unless you have *very* specific reasons to change this. Some SQL teams automatically assume all databases should be set to “full” recovery model. This requires that they back up the transaction logs on a very regular basis, but give the added advantage of restoring up to the time of the last t-log backup.”
What could be the reason for settings the TempDB recovery model to “full”? The answer is found again here:
From:
Operations Manager SQL Server design considerations (Always On and recovery model)
/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#always-on-and-recovery-model
“Although not strictly an optimization, an important consideration regarding Always On Availability Group is the fact that, by design, this feature requires the databases to be set in the “Full” recovery model. Meaning, the transaction logs are never discarded until either a full backup is done, or only the transaction log. For this reason, a backup strategy is not an optional but a required part of the AlwaysOn design for Operations Manager databases. Otherwise, with time, disks containing transaction logs will fill up.”
**3. ** Data Warehouse
** **
3.1. Sizing the Data Warehouse
Although the Data Warehouse is not that critical as the Operational Database, it is good practice to pre-size it, or at least size it in way that prevents it from growing since day one.
Kevin Holman, from:
What SQL maintenance should I perform on my SCOM 2016 databases?
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
“Ensure you pre-size your databases and logs so they are not always auto-growing, have plenty of free space as required to be supported.”
3.2. Data Warehouse free space
Ensure the Data Warehouse has enough free space.
Kevin Holman, from:
What SQL maintenance should I perform on my SCOM 2016 databases?
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
”For the Data warehouse.... we do not require the same 50% free space. This would be a tremendous requirement if we had a multiple-terabyte database!”
3.3. Data Warehouse Transaction log size
Here are some useful recommendations about sizing the Data Warehouse Transaction logs:
Kevin Holman, from:
What SQL maintenance should I perform on my SCOM 2016 databases?
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
“For the Data warehouse, it depends on how large the warehouse is – but you will probably find steady state to require somewhere around 10% of the warehouse size or less. When we are doing any additional grooming of an alert/event/perf storm…. or changing grooming from 400 days to 300 days – this will require a LOT more transaction log space – so keep that in mind as your databases grow.”
**Jonathan Almquist (SCOM Expert and former Microsoft Premier Field Engineer), **from:
OpsMgr, OpsMgrDW vs transaction log ratio
https://social.technet.microsoft.com/Forums/systemcenter/en-US/a05a358d-0017-4075-ae06-aa32fbf1a07a/opsmgr-opsmgrdw-vs-transaction-log-ratio?forum=operationsmanagerdeployment
“If you have sized your data warehouse and want to set initial size to what is needed 400 days from today, I'd say having a comparable log size as the operational database should be just fine - this could be as little as 10GB, but usually would not need any more than 20-30GB.”
3.4. Data Warehouse Recovery Model
The recommendations about the Operational Database Recovery Model (2.5) also apply to the Data Warehouse DB.
**4. ** Operational Database TempDB
The TempDB of the Operational Database is one of the most important and at the same time underrated components, when it comes to performance of SCOM. That is why it gets its own section here.
From:
Operations Manager SQL Server design considerations (Optimizing SQL Server)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#optimizing-sql-server
"Performance bottlenecks are commonly attributed to not following recommended configuration guidance with the storage provisioned for the SQL Server database instance. Such examples are:
· Configuration of TempDB is incorrect with respect to placement, sizing, etc.”
4.1. Initial Size
It is always a good idea to pre-size your SCOM Operational Database TempDB, but still configure it to autogrow. You can start with 20-40% of the DB file size.
4.2. TempDB Sizing
Sizing of the TempDB is a crucial part of the backend configuration. Ensure the TempDB files are pre-sized to accommodate the needs of Operations Manager.
From:
TempDB Database (Optimizing TempDB performance in SQL Server)
https://docs.microsoft.com/en-us/sql/relational-databases/databases/TempDB-database?view=sql-server-ver15#optimizing-TempDB-performance-in-sql-server
“Preallocate space for all TempDB files by setting the file size to a value large enough to accommodate the typical workload in the environment. This prevents TempDB from expanding too frequently, which can affect performance. The TempDB database can be set to autogrow, but this should be used to increase disk space for unplanned exceptions.”
From:
Operations Manager SQL Server design considerations (Optimizing SQL Server)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#optimizing-sql-server
“The size and physical placement of the TempDB database can affect the performance of Operations Manager. For example, if the size that is defined for TempDB is too small, part of the system-processing load may be taken up with autogrowing TempDB to the size required to support the workload every time you restart the instance of SQL Server.”
Ensure each TempDB data file has the same size.
From:
TempDB Database (Optimizing TempDB performance in SQL Server)
https://docs.microsoft.com/en-us/sql/relational-databases/databases/TempDB-database?view=sql-server-ver15#optimizing-TempDB-performance-in-sql-server
“Data files should be of equal size within each filegroup, as SQL Server uses a proportional-fill algorithm that favors allocations in files with more free space. Dividing TempDB into multiple data files of equal size provides a high degree of parallel efficiency in operations that use TempDB.
4.3. TempDB Recovery Model
Ensure that the TempDB recovery model is set in accordance to your SQL configuration.
From:
Operations Manager SQL Server design considerations (Optimizing SQL Server)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#optimizing-sql-server
“To achieve optimal TempDB performance, we recommend the following configuration for TempDB in a production environment:
· Set the recovery model of TempDB to SIMPLE. This model automatically reclaims log space to keep space requirements small.”
4.4. Number of TempDB files
This is an important consideration if an older SQL version (SQL Server 2012, SQL Server 2014) is used for the backend. Starting with SQL Server 2016 the installation wizard automatically configures the recommended number of TempDB files.
From:
Operations Manager SQL Server design considerations (Optimizing SQL Server, section “Optimize TempDB”)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#optimizing-sql-server
**“**Create as many files as needed to maximize disk bandwidth. Using multiple files reduces TempDB storage contention and yields significantly better scalability. However, do not create too many files because this can reduce performance and increase management overhead. As a general guideline, create one data file for each logical processor on the server (accounting for any affinity mask settings) and then adjust the number of files up or down as necessary. As a general rule, if the number of logical processors is less than or equal to 8, use the same number of data files as logical processors. If the number of logical processors is greater than 8, use 8 data files and then if contention continues, increase the number of data files by multiples of 4 (up to the number of logical processors) until the contention is reduced to acceptable levels or make changes to the workload/code. If the contention is not reduced, you may have to increase the number of data files more.”
4.5. TempDB disk subsystem
One of the most important factors, influencing the performance of the Operations Manager Management Group is the TemDB disk subsystem. If there is an option, SSD disks should be used for it.
From:
TempDB Database (Optimizing TempDB performance in SQL Server)
https://docs.microsoft.com/en-us/sql/relational-databases/databases/TempDB-database?view=sql-server-ver15#optimizing-TempDB-performance-in-sql-server
“Put the TempDB database on a fast I/O subsystem. Use disk striping if there are many directly attached disks. Individual or groups of TempDB data files do not necessarily need to be on different disks or spindles unless you are also encountering I/O bottlenecks.
Put the TempDB database on disks that differ from those that are used by user databases.”
**5. ** Other important recommendations, concerning the SCOM databases
** **
5.1.1. Operations Manager databases in virtual environments
If SQL server runs on virtual machines, consider placing the SCOM databases on a direct attached storage (DAS).
From:
Operations Manager SQL Server design considerations (Optimizing SQL Server, section “Optimize TempDB”)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#optimizing-sql-server
“In virtual environments, for performance reasons, it is recommended that you store the operational database and data warehouse database on a direct attached storage, and not on a virtual disk. Always use the Operations Manager Sizing Helper to estimate required IOPS, and stress test your data disks to verify. You can leverage the SQLIO tool for this task. See also Operations Manager virtualization support for additional guidance on virtualized Operations Manager environment.”
5.1.2. Separation of data files and log files
Separate data files from log files on different Volumes/LUNS/Arrays, because they have different write I/O patterns.
From:
Operations Manager SQL Server design considerations (Optimizing SQL Server)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#optimizing-sql-server
"Performance bottlenecks are commonly attributed to not following recommended configuration guidance with the storage provisioned for the SQL Server database instance. Such examples are:
· Hosting transaction logs and database files on the same volume. These two workloads have different IO and latency characteristics.”
5.1.3. NTFS allocation size
Configure the proper NTFS allocation size on the partitions, hosting the data files.
From:
Operations Manager SQL Server design considerations (Optimizing SQL Server, section “NTFS allocation unit size”)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#ntfs-allocation-unit-size
“When formatting the partition that will be used for SQL Server data files, it is recommended that you use a 64-KB allocation unit size (that is, 65,536 bytes) for data, logs, and tempdb. Be aware however, that using allocation unit sizes greater than 4 KB results in the inability to use NTFS compression on the volume. While SQL Server does support read-only data on compressed volumes, it is not recommended.”
5.1.4. XP_cmdshell
Disable XpCommandShell (xp_cmdshell)
Kevin Justin (Microsoft Dedicated Field Engineer), from:
Optimize SQL for SCOM
https://blogs.technet.microsoft.com/kevinjustin/2017/05/24/optimize-sql-for-scom/
“Disable XpCommandShell”
A short description of the feature and a guide on how to disable it can be found here:
xp_cmdshell (Transact-SQL)
/en-us/sql/relational-databases/system-stored-procedures/xp-cmdshell-transact-sql?view=sql-server-2016
5.1.5. SQL Max degree of parallelism (SQL MaxDop)
This one has direct impact on how your Operations console is performing. Ensure that SQL MaxDop is properly configured.
From:
Operations Manager SQL Server design considerations (Optimizing SQL Server)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#optimizing-sql-server
"Performance bottlenecks are commonly attributed to not following recommended configuration guidance with the storage provisioned for the SQL Server database instance. Such examples is overlooking the basic SQL Server configuration such as using AUTOGROW for database and transaction log files,MAXDOP setting for query parallelism, creating multiple TempDB data files per CPU core, etc..”
Stick to the following recommendations when configuring SQL MaxDop.
From:
Operations Manager SQL Server design considerations (Optimizing SQL Server, section “Max degree of parallelism”)
https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#max-degree-of-parallelism
- For servers that use more than eight processors, use the following configuration: MAXDOP=8
- For servers that use eight or fewer processors, use the following configuration: MAXDOP=0 to N Note In this configuration, N represents the number of processors.
- For servers that have NUMA configured, MAXDOP should not exceed the number of CPUs that are assigned to each NUMA node.
- For servers that have hyperthreading enabled, the MAXDOP value should not exceed the number of physical processors.
- For servers that have NUMA configured and hyperthreading enabled, the MAXDOP value should not exceed number of physical processors per NUMA node.
Here is also some additional information about Max Degree Of parallelism Server Configuration Option:
Configure the max degree of parallelism Server Configuration Option
/en-us/sql/database-engine/configure-windows/configure-the-max-degree-of-parallelism-server-configuration-option?view=sql-server-ver15
5.1.6. Cost Threshold for Parallelism
Its important to first understand what Cost Threshold for Parallelism actually does.
From:
Configure the cost threshold for parallelism Server Configuration Option
/en-us/sql/database-engine/configure-windows/configure-the-cost-threshold-for-parallelism-server-configuration-option?view=sql-server-ver15
“The cost threshold for parallelism option specifies the threshold at which SQL Server creates and runs parallel plans for queries. SQL Server creates and runs a parallel plan for a query only when the estimated cost to run a serial plan for the same query is higher than the value set in cost threshold for parallelism. The cost refers to an estimated cost required to run the serial plan on a specific hardware configuration, and is not a unit of time. The cost threshold for parallelism option can be set to any value from 0 through 32767. The default value is 5.”
Before configuring this option, please consider the following important recommendation by Microsoft:
“This option is an advanced option and should be changed only by an experienced database administrator or certified SQL Server professional.”
Although Microsoft claims in the same article that:
“While the default value of 5 is adequate for most systems, a different value may be appropriate. Perform application testing with higher and lower values if needed to optimize application performance.”
Many SQL professionals claim that the default setting was actually good years ago, but since things have changed nowadays, a much higher value is now considered a best practice. Most of them confirm that 50 is a good default value for Cost Threshold for Parallelism.
There are a lot of quality posts on the topic, where you can find more details.
5.1.7. “Optimize for ad hoc workloads”
Enable “Optimize for ad hoc workloads” on instance level. This is a setting, which is often neglected, but plays an important role and should be addressed when configuring the databases. What does it do?
From:
optimize for ad hoc workloads Server Configuration Option
/en-us/sql/database-engine/configure-windows/optimize-for-ad-hoc-workloads-server-configuration-option?view=sql-server-ver15
“The optimize for ad hoc workloads option is used to improve the efficiency of the plan cache for workloads that contain many single use ad hoc batches. When this option is set to 1, the Database Engine stores a small compiled plan stub in the plan cache when a batch is compiled for the first time, instead of the full compiled plan. This helps to relieve memory pressure by not allowing the plan cache to become filled with compiled plans that are not reused.”
SCOM is an Online transaction processing (OLTP) workload and as such executes the same queries multiple times over and over so it would benefit greatly from this particular setting.
5.1.8. Separating DB Instances
It is recommended to have individual SQL instances for each of the databases – Operational and Data Warehouse. A separate instance for Reporting is also recommended.
Cameron Fuller **(SCOM Expert and Microsoft MVP), **from:
SQL Instances and System Center 2012 (#SYSCTR #SQL #SCOM #SCCM #SCSM)
http://www.systemcentercentral.com/sql-instances-and-system-center-2012-sysctr-sql-scom-sccm-scsm/
- Splitting to multiple instances creates a different Tempdb for each instance
- Splitting to multiple instances allows you to configure memory for each instance.
- Splitting to multiple instances provides the ability to have different SA accounts which can be configured for improved security.
- Splitting to multiple instances provides different Server and Agent Services (databases on each instance can be managed independently).
- Splitting to multiple instances may be required to support different collations.
- Splitting to multiple instances may be required to support different SQL version, SP, and maybe even CU support levels.
- Splitting to multiple instances provides a division/separation for maintenance. IE: Taking down one instance does not impact multiple system center components.
From:
SQL Server Design Considerations (Capacity and storage considerations, section “Operations Manager data warehouse database”)
/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#operations-manager-data-warehouse-database
“Consider placing the Reporting data warehouse on a separate server from the Operations Manager database. Although smaller-scale deployments can often consolidate the Operations Manager database and Reporting data warehouse on the same server, it is advantageous to separate them as you scale up the number of agents and the volume of incoming operational data. When the Reporting data warehouse and Reporting Server are on a separate server from the Operations Manager database, you experience better reporting performance.”
5.1.9. SQL Memory settings
SQL Server offers the possibility to configure the minimum and maximum values and reserve memory, which is then used by each instance. It is recommended to specify at least 4GB of RAM as minimum amount.
From:
SQL Server Design Considerations (Optimizing SQL Server, section “Reserve memory”)
/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#reserve-memory
“SQL Server allows you to configure the minimum and maximum amount of memory that should be reserved and used by its process. Specify at least 4 GB of RAM as minimum amount. This should be done for every SQL node hosting one of the Operations Manager databases (Operational, Data warehouse, ACS).”
And the recommended approach is (from the same article):
“First start by reserving 1 GB of RAM for the OS, 1 GB for each 4 GB of RAM installed from 4–16 GB, and then 1 GB for every 8 GB RAM installed above 16 GB RAM. Then monitor the Memory\Available MBytes performance counter in Windows to determine if you can increase the memory available to SQL Server above the starting value.”
**6. ** Troubleshooting DB Performance Problems
Although this is not a optimization recommendation like the rest in this list, it is important to shed some light on some troubleshooting techniques in case of data insertion problems.
How to troubleshoot Operations Manager DB performance problems
https://techcommunity.microsoft.com/t5/system-center-blog/how-to-troubleshoot-operations-manager-db-performance-problems/ba-p/350927
**7. ** Database Maintenance
Proper database maintenance is equally important for the performance as its correct configuration. There is one article (a MUST read) that covers this in great details, one has already been referenced numerous times in this post:
**Kevin Holman:
**What SQL maintenance should I perform on my SCOM 2016 databases?
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases
** **
Please ensure that the following SQL maintenance tasks are configured at minimum:
7.1. Index Rebuild
As per Kevin Holman’s recommendation, when SCOM 2016 is used with SQL 2016 a indexing rebuild job should be considered. This is to avoid the issue, described in the following Blog Post:
Possible increased unused disk space when running SCOM 2016 on SQL2016
/en-us/archive/blogs/germanageability/possible-increased-unused-disk-space-when-running-scom-2016-on-sql2016
“To resolve this, and likely improve performance of SCOM – I recommend that each SCOM customer set up SQL agent jobs, that handles Index maintenance for the entire OpsDB, once a day. I’d say given the other schedules, a start time between 3:00 AM and 6:00 AM would likely be a good time for this maintenance. That lets the built in maintenance run, and wont conflict with too much. You should try and avoid having anything running at 4:00 AM because of the Alert Auto Resolution. We don’t want any blocking going on for that activity.”
7.2. CheckDB
It is generally recommended that each of the SCOM Databases is regularly checked with checkDB.
Kevin Holman, from:
What SQL maintenance should I perform on my SCOM 2016 databases?
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases
“For the above reasons, I would be careful with any maintenance jobs on the Data Warehouse DB, beyond a CHECKDB and a good backup schedule.”
**8. ** Power management settings on the SQL Servers
There are number of Blog posts and articles, which cover the effects of the “Power Plan” on SQL Server. In order to avoid Performance issues on the SQL Servers (like the one described in “Slow Performance on Windows Server when using the “Balanced” Power Plan”), ensure that the Power Plan on your SQL Servers is set to “High Performance”
**9. ** Optimizing the SCOM Management Servers
Almost all of the recommendations listed so far were concerning the SCOM databases and their configuration respectively maintenance. There are plenty of configuration settings though that can optimize the performance of SCOM also on application level. Let us start with some registry tweaks, presented by Kevin Holman. In the article you will find a short explanation of each setting and its impact on the SCOM environment.
**Kevin Holman, **from:
Recommended registry tweaks for SCOM 2016 management servers
https://kevinholman.com/2017/03/08/recommended-registry-tweaks-for-scom-2016-management-servers/
reg add "HKLM\SYSTEM\CurrentControlSet\services\HealthService\Parameters" /v "State Queue Items" /t REG_DWORD /d 20480 /f
reg add "HKLM\SYSTEM\CurrentControlSet\services\HealthService\Parameters" /v "Persistence Checkpoint Depth Maximum" /t REG_DWORD /d 104857600 /f
reg add "HKLM\SOFTWARE\Microsoft\System Center\2010\Common\DAL" /v "DALInitiateClearPool" /t REG_DWORD /d 1 /f
reg add "HKLM\SOFTWARE\Microsoft\System Center\2010\Common\DAL" /v "DALInitiateClearPoolSeconds" /t REG_DWORD /d 60 /f
reg add "HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0" /v "GroupCalcPollingIntervalMilliseconds" /t REG_DWORD /d 900000 /f
reg add "HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Data Warehouse" /v "Command Timeout Seconds" /t REG_DWORD /d 1800 /f
reg add "HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Data Warehouse" /v "Deployment Command Timeout Seconds" /t REG_DWORD /d 86400 /f
**10. ** Optimize Memory Thresholds for the Health Service
Health Service Memory Thresholds is an important topic, because it directly impacts the performance of the SCOM agents. Many have experienced this behaviour and blogged about it. Here an example:
SCOM 2016 – Agent (Health Service) high CPU utilization and service restart
https://www.pohn.ch/scom-2016-agent-health-service-high-cpu-utilization-and-service-restart/
And if you think that this does not concern you, please remember what Kevin Holman states in his blog post.
**Kevin Holman, **from:
Stop Healthservice restarts in SCOM 2016
https://kevinholman.com/2017/05/29/stop-healthservice-restarts-in-scom-2016/
“This is probably the single biggest issue I find in 100% of customer environments.
YOU ARE IMPACTED. Trust me.”
Although this issue can remain unnoticed in many cases, it is essential to adjust the default Health Service Memory Thresholds as suggested in the referenced article. Kevin Holman has even created a management pack with all the recommended overrides, which you can simple download and import in your environment. It can be found on the Microsoft Technet Gallery:
SCOM Agent Threshold Override Management Pack
https://gallery.technet.microsoft.com/SCOM-Agent-Threshold-b96c4d6a
**11. ** Workflow optimizations for large environments
There are also other types of performance issues – those caused by workflows, defined in certain management packs. Please make sure you read carefully the post below and take appropriate actions if necessary.
Kevin Holman:
QuickTip: Disabling workflows to optimize for large environments
https://kevinholman.com/2017/08/15/quicktip-disabling-workflows-to-optimize-for-large-environments/
The article contains two more references, which are equally important and should also be considered, especially if the features mentioned are not used in your management group:
- Removing the APM management Packs (if the feature is not used)
** **
**Kevin Holman, **from:
Reinstalling your SCOM agents with the NOAPM switch
https://kevinholman.com/2017/08/05/reinstalling-your-scom-agents-with-the-noapm-switch/
“On another note – if you have no plans to use the APM feature in SCOM – you should consider removing those MP’s which get imported by default. They discover by default a LOT of instances of sites, services, and instances of classes where APM components are installed on the agents.”
- Removing the OMS and Advisor management packs (if the feature is not used)
** **
**Kevin Holman:
**How to remove OMS and Advisor management packs
https://kevinholman.com/2016/03/26/how-to-remove-oms-and-advisor-management-packs/
Here is also an old, but “gold” guide on tuning event collection in your SCOM environment. Kevin Holman describes how you can approach a possible over-collection of events in your environment.
**Kevin Holman:
**Tuning tip – turning off some over-collection of events
https://kevinholman.com/2009/11/25/tuning-tip-turning-off-some-over-collection-of-events/
**12. ** Updating your SCOM Management Group
Always Install the latest Update Rollup for your SCOM Version. Update Rollups are cumulative, meaning that you need to install only the latest one and almost always include important fixes or critical improvements. The description of the term “Update Rollup” says it all.
From:
Description of the standard terminology that is used to describe Microsoft software updates
https://support.microsoft.com/en-us/help/824684/description-of-the-standard-terminology-that-is-used-to-describe-micro
“Definition: A tested, cumulative set of hotfixes, security updates, critical updates, and updates that are packaged together for easy deployment.”
Conclusion
Having a satisfying SCOM management group performance is not hard thing to achieve if you follow some basic principles. A key factor is planning your deployment in detail before even starting with the actual implementation, so that you can address all those important topics in advance and ensure each of them gets the attention needed.
Make sure all your suggestions are technically justified, so that the teams or persons, involved in your project understand why all those configurations really matter and what the benefit would be.
Last, but not least, make sure you test the effects of all the settings in test environment first. Some of them can even introduce the opposite result and cause troubles, if not set according to the needs of the environment.
References
SCOM Sizing Helper. The most misunderstood Sizing Tool ever.
http://www.kuskaya.info/2019/05/12/how-to-properly-size-system-center-operations-manager-distributed-environment-with-microsoft-scom-sizing-helper-tool/
What SQL maintenance should I perform on my SCOM 2016 databases?
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
OpsMgr, OpsMgrDW vs transaction log ratio
https://social.technet.microsoft.com/Forums/systemcenter/en-US/a05a358d-0017-4075-ae06-aa32fbf1a07a/opsmgr-opsmgrdw-vs-transaction-log-ratio?forum=operationsmanagerdeployment
xp_cmdshell (Transact-SQL)
/en-us/sql/relational-databases/system-stored-procedures/xp-cmdshell-transact-sql?view=sql-server-2016
Configure the max degree of parallelism Server Configuration Option
/en-us/sql/database-engine/configure-windows/configure-the-max-degree-of-parallelism-server-configuration-option?view=sql-server-ver15
MAXDOP of Confusion (Dear SQL DBA Episode 8)
https://littlekendra.com/2016/07/14/max-degree-of-parallelism-cost-threshold-for-parallelism/
SQL Server 2016, Minimal logging and Impact of the Batchsize in bulk load operations
https://blogs.msdn.microsoft.com/sql_server_team/sql-server-2016-minimal-logging-and-impact-of-the-batchsize-in-bulk-load-operations/
How to troubleshoot Operations Manager DB performance problems
https://techcommunity.microsoft.com/t5/system-center-blog/how-to-troubleshoot-operations-manager-db-performance-problems/ba-p/350927
Description of the standard terminology that is used to describe Microsoft software updates
https://support.microsoft.com/en-us/help/824684/description-of-the-standard-terminology-that-is-used-to-describe-micro
How to remove OMS and Advisor management packs
https://kevinholman.com/2016/03/26/how-to-remove-oms-and-advisor-management-packs/
Tuning tip – turning off some over-collection of events
https://kevinholman.com/2009/11/25/tuning-tip-turning-off-some-over-collection-of-events/
Reinstalling your SCOM agents with the NOAPM switch
https://kevinholman.com/2017/08/05/reinstalling-your-scom-agents-with-the-noapm-switch/
QuickTip: Disabling workflows to optimize for large environments
https://kevinholman.com/2017/08/15/quicktip-disabling-workflows-to-optimize-for-large-environments/
SCOM Agent Threshold Override Management Pack
https://gallery.technet.microsoft.com/SCOM-Agent-Threshold-b96c4d6a
SCOM 2016 – Agent (Health Service) high CPU utilization and service restart
https://www.pohn.ch/scom-2016-agent-health-service-high-cpu-utilization-and-service-restart/
Recommended registry tweaks for SCOM 2016 management servers
https://kevinholman.com/2017/03/08/recommended-registry-tweaks-for-scom-2016-management-servers/
Slow Performance on Windows Server when using the “Balanced” Power Plan
https://support.microsoft.com/en-au/help/2207548/slow-performance-on-windows-server-when-using-the-balanced-power-plan
Possible increased unused disk space when running SCOM 2016 on SQL2016
/en-us/archive/blogs/germanageability/possible-increased-unused-disk-space-when-running-scom-2016-on-sql2016
SQL Instances and System Center 2012 (#SYSCTR #SQL #SCOM #SCCM #SCSM)
http://www.systemcentercentral.com/sql-instances-and-system-center-2012-sysctr-sql-scom-sccm-scsm/
optimize for ad hoc workloads Server Configuration Option£
/en-us/sql/database-engine/configure-windows/optimize-for-ad-hoc-workloads-server-configuration-option?view=sql-server-ver15
Configure the cost threshold for parallelism Server Configuration Option
/en-us/sql/database-engine/configure-windows/configure-the-cost-threshold-for-parallelism-server-configuration-option?view=sql-server-ver15
Optimize SQL for SCOM
https://blogs.technet.microsoft.com/kevinjustin/2017/05/24/optimize-sql-for-scom/
TempDB Database (Optimizing TempDB performance in SQL Server)
/en-us/sql/relational-databases/databases/TempDB-database?view=sql-server-ver15#optimizing-TempDB-performance-in-sql-server
See also
Should You Enable Autogrowth on SCOM Operations Database?
https://social.technet.microsoft.com/wiki/contents/articles/52398.should-you-enable-autogrowth-on-scom-operations-database.aspx
System Center 2012 R2 Operations Manager: Survival Guide
https://social.technet.microsoft.com/wiki/contents/articles/20796.system-center-2012-r2-operations-manager-survival-guide.aspx