Monitoring Hybrid Cloud – Step-By-Step

It is always important to understand how our server workloads are doing.  Getting visibility into our systems allows us to proactively deal with future issues.  As an example, monitoring can let us know that we are running out of disk space on one of our volumes so we can order drives and get them install before a catastrophic event happens.  Monitoring can also alert us of future outages due to hardware failure.  I have been able to save downtime by catching through monitoring that a disk drive or disk controller is having problems.  Without monitoring we would not find out about these things until they fail completely and the service is brought down hard.  Though virtualization technology has changed our world for the better, it brought with it some unique challenges.  The biggest of which is probably that we have to use tools to understand our infrastructure.  No longer could we go “see” our “touch” our servers.  This problem is compounded even more as we move to the cloud.  Not only are these workloads not directly visible, they are not even in our data centers.  In the technology space, if we want to continue to grow we have to continue to learn and continue to solve problems.  When technology professionals face a problem, it is their nature to come up with innovative ways to make the problems go away.   Monitoring tools give us the capability of not only seeing all of our on-premises machines but also all of our virtualized machines and yes, even all of our cloud infrastructure.  We can not only see how they are doing and anticipate failure but we can audit (track who changed what) and even evaluate performance. By truly seeing our infrastructure, we can proactively make sure it stays running (up to standard SLAs) and we keep our jobs and prosper.

Different people will have different needs in terms of monitoring.  Datacenter administrators will want to see everything everywhere.  Application owners will want to see the infrastructure running their applications. Developers will want to see their test machines and be able to see how their application impacts performance.  This information can give them insight into scalability and optimization needs within their code.  Monitoring can go much deeper than just showing availability.  It can even show them where in the application the time lags may be occurring.  Monitoring for performance as an example can peer inside of an application, (yes, even behind a login screen) and see how long it takes to get to a particular page and load it, how long it takes to run a report, how long it takes to run a query, what part of a query is taking time, etc.  Monitoring infrastructure in the Hybrid Cloud Age has truly grown up and become a vital component to save businesses money, to save employees (Administrators, Developers, Help Desk, Application Owners, others) much time and aggravation.   However, in my personal opinion the best “feature” of monitoring is that it prevents people from being able to point the finger somewhere else.  With effective system monitoring, the root cause of current and future issues are clear.  DBAs, developers and administrators no longer need to duke it out to determine who needs to fix the issues.

In a Hybrid cloud environment there is no doubt there is one tool that is the clear and undisputed leader.  That tool is Microsoft Operations Manager which is an integral part of System Center.

There are other (free) tools within Windows and Azure that can be used to manage and monitor your workloads (Resource Monitor, Azure Portal Monitor, more) but only System Center Operations Manager can do it all for your internal infrastructure (physical and virtual) as well as your cloud infrastructure.   You can read about the system requirements for Operations Manager at https://technet.microsoft.com/en-us/library/jj656654.aspx.  For the prior version of Operations Manager I did a number of introductory and how to posts.  You can search SCOM on ITProGuru.com to find them.  In the very near future, I am planning on writing some very detailed step-by-step guides for getting up and running on the latest version System Center Operations Manager.  If you just want to setup a lab environment you might want to checkout the Powershell Deployment Toolkit. Complete information on how to install, deploy and monitor an entire hybrid cloud infrastructure would take many articles (which I hope to do in the coming months).  For this post, I want to just focus on leveraging the Azure Management Pack in Operations Manager to monitor the Azure Cloud.

 

After getting System Center Operations Manager up and running you need to download the Azure Management Pack.  See the management pack guide (inside the download file) for more detailed information than provided in this post.

Prerequisites The following requirements must be met to run this Management Pack: (the Management Pack will not Check or Configure for you) Supported Operating System – Windows Server 2008 R2 SP1, Windows Server 2008 Service Pack 2 – You must have an Operations Manager 2012 SP1 or later environment (this Management Pack will not import on Operations Manager 2007 R2 or Operations Manager 2012 RTM) Internet Connection · At least one machine in your server pool must have internet connectivity in order to be able to communicate with Windows Azure Windows Azure Diagnostics must be enabled. For more information about Windows Azure Diagnostics, see Collect Logging Data by Using Windows Azure Diagnostics (https://go.microsoft.com/fwlink/?LinkId=186765). Forward Diagnostic Data Windows Azure Diagnostics must be configured to forward diagnostic data to a Windows Azure storage. For more information about configuring Windows Azure Diagnostics, see Transferring Diagnostic Data to Windows Azure Storage (https://go.microsoft.com/fwlink/?LinkId=191347). The connection string for Windows Azure Diagnostics must be specified in the service configuration file (.cscfg) and should not be hardcoded in the application. By default, the Management Pack for Windows Azure Applications references the DiagnosticsConnectionString setting name in the service configuration file. If a different name is used in the service configuration file, you must override the Diagnostics Storage Account Setting Name parameter for Windows Azure Hosted Service Total Discovery.

Installing Management Packs

After downloading the file and making sure your environment has met the prerequisites you can install the Management pack. Run the “System Center Management Pack for Windows Azure.msi” file to save the management packs to your SCOM server.   Then you have to import them. (Administration – right-click Management pack).  There are two different management packs to import:

• Microsoft.SystemCenter.WindowsAzure.mpb
• Microsoft.SystemCenter.WindowsAzure.SLA.mpb

By default, this Management Pack does not provide any functionality on import. In order to monitor Azure resources, you must go through the following steps

Discovering Azure Resources

After importing the Management Pack into your environment, you will need to provide information about the subscriptions that have resources that you would like to monitor. To do this, navigate to the Administration pane in the Operations Manager Console, and then navigate to the “Windows Azure” node and enter the subscription id, certificate and certificate password.

1. Log on to the computer on which the Operations console is installed using an account that is a member of the Operations Manager Administrators role.

2. In the Operations console, click Administration.

3. At the bottom of the navigation pane, click Windows Azure.

4. Under Required Configuration Tasks, click Add subscription.

5. On the Subscription Configuration page, enter a subscription id, provide a path to the certificate file (See https://ITProGuru.com/GuruCert; download cert from Azure – Settings – Management Certificates), and provide the certificate password.

You can get your subscription id from the Azure Portal under Settings – Subscriptions – Subscription ID

 

Then click Next.

Note that the wizard tries to communicate with Windows Azure at this stage to validate the information you provided. The wizard will not let you finish until the provided credentials have been validated.

6. On the Server Pool page, as shown in the following illustration, select a resource pool to communicate with Windows Azure. The resource pool must have Internet access. If you use a proxy server in your environment for internet access, select Use a proxy server to connect and enter the proxy address. If no proxy server is selected, your IE proxy settings will be used. Click Add Subscription.

7. On the Summary page, confirm the settings, and then click Add subscription.

This will discover all Azure resources in this subscription, but will not monitor them by default. The discovery includes Cloud Services, Storage, and Virtual Machines.  Before any discovery happens you will need to either wait for the system polled refresh or run a simple PowerShell command to cycle the health service using the command Restart-Service HealthService

Note that discovery may take 5-10 minutes to complete but depending on your configuration, it could take even longer.

1. In the Monitoring workspace, in the Windows Azure folder and the Azure Resource Inventory folder, click Discovered Deployment and verify that the deployments that are part of your subscription are listed in the results pane. Do similar checks for the Discovered Cloud Services, Discovered Virtual Machines, Discovered Storage, Discovered Role Instance, Discovered Roles, Discovered SQL Azure Instances, and Discovered Subscriptions views.

If the application that you added is not displayed correctly when you verify the configuration, check the Operations Manager event log for related events such as the password being incorrect.

Choosing the resources to monitor

After you have completed the Administration wizard, you will need to specify which discovered Azure Resources you want to monitor. Do this by navigating to the Authoring Pane of the Operations Manager Console and selecting the Windows Azure Monitoring template.

1. Log on to the computer on which the Operations console is installed using an account that is a member of the Operations Manager Administrators role.

2. In the Operations console, click Authoring.

3. At the bottom of the navigation pane, click Add monitoring wizard.

4. In the Add Monitoring Wizard, select the Windows Azure Monitoring monitoring type, and then click Next.

5. On the General page, enter a name and description. Select an existing or new management pack in which to save the discovery, and then click Next.

Note: By default, when you create a management pack object, disable a rule or monitor, or create an override, Operations Manager saves the setting to the Default Management Pack. As a best practice, you should create a separate management pack for each sealed management pack that you want to customize, instead of saving your customized settings to the Default Management Pack. For more information, see Customizing Management Packs (https://go.microsoft.com/fwlink/?LinkId=140601).

6. On the Subscription page, select the subscription for which you want to specify resources to monitor. Note that this list will only contain subscriptions you have specified in the wizard in the administration section. Then click Next.

7. On the Cloud Services page, click Add and search for Cloud Services and Deployments you would like to monitor. Add them for monitoring by selecting them and choosing Add

8. On the Virtual Machines page, click Add and Search for Virtual Machines you would like to monitor. Add them for monitoring by selecting them and choosing Add.

9. On the Storage page, click Add and search for Storage you would like to monitor. Add them for monitoring by selecting them and choosing Add

Verify:

1. In the Monitoring workspace, in the Windows Azure folder and the Monitored Azure Resources folder, click Cloud Service State and verify that the Cloud Services you selected to be monitored are listed in the results pane. Do similar checks for the Role Instance State, Role State, Subscription State, Virtual Machine State and Storage State views.

2. Note that it may take a few minutes before the Azure resources you have selected for monitoring appear in the correct state views. It may also take a few minutes before they become “monitored”.

 

Viewing Information in the Operations Console

The Management Pack for Windows Azure Fabric defines the following views which are displayed in the Monitoring workspace under the Windows Azure folder:

Azure Resource Inventory Views – shows all discovered resources for the subscriptions you have specified (you specify subscriptions where you want to discover resources by running the wizard in the administration space)

·        Monitored Azure Resources Views – shows the state of all Azure resources you have chosen to monitor (you specify resources to monitor by running the wizard in the authoring space)

·        Active Alerts View – shows all active alerts for Windows Azure monitored resources

·        All Alerts Last 24 hours – shows all alerts generated in the last 24 hours for Windows Azure monitored resources

·        Service State – shows state of applications defined using the new Distributed Application template. Note that this view is empty until at least one such distributed application is defined

·        Topology Dashboard – shows how your services are “connected”

·        Performance views – shows the following for all role instances

·      ASP.NET performance

·      Disk Capacity

·      Memory Utilization (Physical)

·      Network Adapter Utilization

·      Processor Performance

·      Role Instance Performance (All counters

·        Storage Account Size

Grooming Data from Windows Azure Storage Services

Windows Azure Diagnostics writes performance and event information to Azure Storage, but does not delete it. This means that the tables in the Windows Azure storage will continue to grow unless the data is groomed. The Management Pack for Windows Azure Fabric provides three rules that control data grooming:

·      Windows Azure Role NT Event Log Grooming

·      Windows Azure Role Performance Counter Grooming

·      Windows Azure Role .NET Trace Grooming

By default, these grooming rules are disabled. If you do not want Operations Manager to periodically groom data from Windows Azure Storage Services, use overrides to enable the rules. By default, the enabled rules run every 24 hours.

You can use the event log on the root management server to track data grooming. Event 34023 is logged when grooming starts. Event 34014 is logged when grooming is completed, and the event includes the count of deleted rows and the time when grooming occurred.

See also the Jump Start on Managing Hybrid Cloud

https://channel9.msdn.com/Series/Moving-to-Hybrid-Cloud-with-Microsoft-Azure/05

Links

The following links connect you to information about common tasks that are associated with System Center management packs:

System Center 2012 – Operations Manager

 

·      Management Pack Life Cycle

·      How to Import a Management Pack

·      Tuning Monitoring by Using Targeting and Overrides

·      How to Create a Run As Account  

·      How to Export a Management Pack

·      How to Remove a Management Pack