Configuring SPF and Windows Azure Pack for IaaS usage and metering

After configuring VMM and OM for IaaS usage and metering in the previous blog post (here) it’s now time to configuring SPF and Windows Azure Pack to import Usage data from Operations Manager Data Warehouse into Windows Azure Pack (WAP) Usage Database.

The series includes the following posts:

• IaaS Usage and Service Reporting using System Center 2012 R2 and Windows Azure Pack
• Configuring VMM and OM for IaaS usage and metering.  
• Configuring SPF and Windows Azure Pack for IaaS usage and metering. (This blog post)
• Installing & configuring Service Reporting for IaaS usage and metering.

Service Provider Foundation has a usage module that WAP uses to collect data from Operations Manager data warehouse. In this blog post we will go over how to configure the following:

• Configure SPF to extract data from Operations Manager Data Warehouse
• Configure WAP to connect to SPF Usage Service
• Verify that Data is being stored in the WAP usage database

The following things must be configured before starting on this guide

• Completing all steps and pre-requisites from the previous blog post: Configuring VMM and OM for IaaS usage and metering

The Environment:

Operations Manager Server:	OM01.contoso.com
Operations Manager Database Server:	DB04.contoso.com
VMM Server	VMM01.contoso.com
SPF Server	VMM01.contoso.com
WAP Server	WAP01.contoso.com
WAP Database Server:	DB02.contoso.com
Service Reporting Server	VMM01.contoso.com
Service Reporting Database	DB06.contoso.com
SharePoint Server	SP01.contoso.com

Configure SPF to extract data from Operations Manager Data Warehouse

The usage module in SPF needs to be configured using PowerShell. With SPF there is a set of commands that enables you to configure settings in SPF. By using these commands you can tell SPF where the Operations Manager Server and Data Warehouse database are located.

To configure SPF Usage do the following:

1. Logon to the SPF server as an Administrator

2. Open IIS Manager Console

3. Select Application Pools

4. Check the identity for the Usage Application (eg. Contoso\!spf)

 

5. Close the IIS console

6. Open Computer Management Console > Local Users Groups > Groups

7. Verify that the user you will later use to query SPF for data from WAP is member of the local SPF_Usage users group, if not add the user to the Group. In this scenario I’m using SPF Usage Application Identity user. Any other user can be used as long as it’s member of the Usage security Group.

   

8. Open PowerShell console with administrative privilege

9. Run the following commands (change values in “<>” with your own values):

• Import-module spfadmin

• # Provide server name to the OM DW SQL instance

• $OMDWSqlServer = "<SCOM DW DB SQL Server>”

• Example: $OMDWSqlServer = "DB04.contoso.com"

• # Provide server name to the OM RMS Server

• $OMServer = "<SCOM RMS Server Name>"

• Example: $OMServer = "OM01.contoso.com"

• # Register the SCOM Data Warehouse instance to SPF usage metering

• $stamp = Get-SCSPFStamp;

• $server = New-SCSPFServer -Name $OMServer -ServerType OMDW -Stamps $stamp[0];

• For SPF Preview use the following command:

• $setting = New-SCSPFSetting -Name $OMDWSqlServer -SettingString "Data Source=$OMDWSqlServer;Initial Catalog=OperationsManagerDW;Integrated Security=True" -SettingType DatabaseConnectionString –Server $server

• For SPF RTM use the following command:

• $setting = New-SCSPFSetting -Name $OMDWSqlServer –Value "Data Source=$OMDWSqlServer;Initial Catalog=OperationsManagerDW;Integrated Security=True" -SettingType DatabaseConnectionString –Server $server

• Example:

• To verify if the setting has been applied to SPF run the following command:

• Get-SCSPFSetting

10. Logon to the SQL Server that hosts the SCOM DW DB as SQL Admin

11. Start SQL Management Studio

12. Select Security > Logins

13. Right Click on Logins folder and select “New Login”

14. Add the SPF Usage Application Identity user for Login name. eg. Contoso\!spf

15. Select User Mapping under “Select a page”

16. Select OperationsManagerDW and select OpsMgrReader under “Database role membership”

17. Click Ok

18. Right Click OperationsManagerDW under Database and select Properties

19. Select Permissions and pick the user just added under User and Roles.

20. Verify that the user has Execute permission on the database, if not assign this permission as shown below.

     

21. Close SQL Management Studio

Configure WAP to connect to SPF Usage Service

The Usage module inside WAP uses SPF to collect data from Operations Manager Data Warehouse. For WAP usage to collect this data we need to tell WAP where the SPF Usage service is located.

To enable this in WAP do the following

1. Start a browser that has access to the WAP Admin Portal and login in as administrator

2. Select VM Clouds from the main menu

3. Select the VM Clouds Configuration from the VM Clouds menu

4. Select Register Service Provider Usage

5. Provide information for each field as given in the example below and Click Ok.

  

Username is the user you added to the SPF_Usage Group under step 7 for Configure SPF to extract data from Operations Manager Data Warehouse.

NB: Remember to add /Usage after the port number for the SPF usage service.

Verify that Data is being stored in the WAP usage database

To verify that IaaS data is being stored in the WAP Usage Database do the following:

1. Logon the WAP Database as Administrator

2. Start SQL Management Studio

3. Select Databases > Tables

4. Right Click on Usage.Records and select “Select Top 1000 Rows”

5. Verify that the result returns Data with ResourceId “VM Utilization”

6. Close the SQL Management Studio Console

Now that we see data in the WAP Usage DB, we have successfully integrated SPF and Windows Azure Pack.

This will now enable us to configure Service Reporting and using billing adapters against Windows Azure Pack.

In the next blog post I will go through how Service Reporting can be configured.

• Installing & configuring Service Reporting for IaaS usage and metering

My colleague KR Kandavel is working on a series of blog posts explaining how you can build a billing adapter.

• Link to blog post coming soon

See you next time for more blogging about Windows Azure Pack & System Center 2012 R2 :-)

Comments

• Anonymous
  January 01, 2003
  Excellent write up. Worked like a charm. Thanks for making this easier then I thought it was going to be.

• Anonymous
  January 01, 2003
  Excellent write up. Worked like a charm. Thanks for making this easier then I thought it was going to be.

• Anonymous
  January 01, 2003
  Thanks for the article Anders!
  I also had a problem of mistyping the usage service URL in the WAP Admin portal.
  
  I got this error when trying to change it: "One or more subscriptions are associated with this resource provider. Any endpoint changes to the resource provider may cause the subscriptions to be unusable.".
  
  The workaround that worked for me was to directly modify SQL database Microsoft.MgmtSvc.Store, table mp.ResourceProviders, cell UsageForwardingAddress for the "systemcenter" resource provider.
  
  Regards,
  Dinko

• Anonymous
  January 01, 2003
  Hold tight this will go out tomorrow. Thanks for reading and asking for more.

• Anonymous
  January 01, 2003
  Hi Anders,I followed your article step by step and everything goes well except the Usage.Records. In my case there is no data in this table.can you please let me know what other thing can I do to check this.Thanks in Advance !!

• Anonymous
  October 09, 2013
  Awaiting for the blog post on "Installing & configuring Service Reporting for IaaS usage and metering ". Please let us know when it will be available.

• Anonymous
  November 01, 2013
  What should i do in order to investigate why Usage.Records table is empty? Get-SCSPFSetting returns results which are similar to yours. OM report 'Virtual Machine Utilization Report' shows that OM see all my three VMs in the cloud. But SpfUsageRecord table of SCSPFDB contain only two VM records.

• Anonymous
  November 05, 2013
  Hi Paul. Can I get you to check the following? Under what user is the SPF Usage APP Pool running? What rights does this user have in OM and OMDW? From your description below it sounds like this part is ok, just want to make sure. Can you also check if you have any errors in the Event log on the SPF Server related to SPF. Lastly what user did you use to register Usage Service with in WAP? And is this user member of SPF_Admins and SPF_usage Groups on the SPF Server? That the last VM is not getting data in SPF sounds wired. It should be all or nothing. Is this VM running and member of a cloud? Was all VMs created by WAP? Anders

• Anonymous
  November 07, 2013
  Yep, Usage.Records table is empty... App pool for usage starts with contosospf contosospf member of all spf_* groups on SPF server and have SQL permissions as shown in post (only OperationsManagerDW db. right?) Event log empty. Get-SCSPFSetting output are similar to yours...

• Anonymous
  November 12, 2013
  Hi ! I also have the same issue ... Followed your step-by-step guide twice (rebuilding from scratch all servers ...) and the same issue ... Data are written in DW DB, SCOM reporting show also VM metrics, VMM to SCOM connection is OK, get-scspfsetting output the same result. What a pain for few graphs ... I just noticed a difference : In Scom diagram, "all Clouds" ressource isn't green checked, only green circled, dunno if it can be the cause ... I also dug into event logs. Nothing in SPF eventlog but in MS > WAP > MgmtSvc-Usage I have lot or errors : Provider read error occurred for provider 'systemcenter' instance id: '{8cece4a2-0321-4445-aa1f-f639d4e3bc1e}'. Error: System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found).   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()   at Microsoft.WindowsAzure.Server.Usage.DefaultUsageRecordsReader.<GetRecordsAsync>d__0.MoveNext() --- End of stack trace from previous location where exception was thrown ---   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()   at Microsoft.WindowsAzure.Server.Usage.CycleRunner.<ProcessProviderInternal>d__19.MoveNext() With instanceID GUID corresponding to systemcenter provider entry in WAP mgmtsvc.usage DB. I really don't understand why in the previous version aka Windows Azure Pack for Windows server, VM reporting was provided by default, without SCOM. Why a such infrastructure "just for graph" ?! Your help to make it work would be appreciated. Thanks anyway for your clear and complete guides !

• Anonymous
  November 12, 2013
  Hi Pierre Could you check what your settings are for the Usage setting in WAP Admin portal: Error 404 normally means that it can't find what it's looking for. So is the URL correct and have you added /usage to the of the web service pointer and doe sit have the right port. It sounds like SPF is working but the error is when WAP is quering the SPF Web service. Anders

• Anonymous
  November 13, 2013
  Hi Anders, Thanks for your quick answer ! I didn't registered the site with FQDN but with short name. When I test it in a browser, shortname = 404, FQDN = forbiden I'm trying to update it in admin site, but I receive the following error : One or more subscriptions are associated with this resource provider. Any endpoint changes to the resource provider may cause the subscriptions to be unusable Do you yjink I need to delete all existing subscriptions to update the serveur URL ? Thnaks again !

• Anonymous
  November 13, 2013
  Hi Pierre I need a little time to investigate this, as I beleve after the first subscription is registered you cannot query the services in the same way, which might be what you are seeing. Again I need to verify this first, so please giv me a little time to investigate this. Anders

• Anonymous
  November 13, 2013
  Hi Pierre Can I get you to check the following: What was the error you were getting below? Can you check that the user you are speficying when configuring the Service Provider Usage is member of SPF_ Usage local Groups on the SPF Server. Also check that you use the right port (8090) and that you specify / usage eg. https://spf01:8090/Usage Your error sounds like it's between WAP and SPF, not further Down in the SPF layer. Let me know if this helps Anders

• Anonymous
  November 14, 2013
  Hi Anders and thanks a lot for your investigation Actually, I figured I filled https://<MyServer>/8090/usage instead of https://<MyServer>:8090/usage ... (/ instead of : ...) So the URL is wrong and it needs to be updated And yes, the account used is the correct one filled in the local group I found few PS commands shipped with azure pack which could update providers but they are not well documented in the help even in technet ... Thanks again !

• Anonymous
  November 14, 2013
  The comment has been removed

• Anonymous
  November 15, 2013
  Hi Anders, I just wanted to keep you informed about the situation I just removed all the subscriptions and I could change the Usage URL. Now the usage.record table is correctly filled with data ! But in the tenant view, VM ressources are still at 0 except for the RAM which keep stuck at the ammount of memory assigned to the VM. Maybe there's a little time for the ressource to be displayed ...

• Anonymous
  December 05, 2013
  Hi Pavel I am experiencing the same problem as you; lastId' query parameter never changed and didn't point to any value (lastId=&batchSize...) If I clear table [scspf].[SpfUsageRecord] table I see data in the browser when I go to https://spfserver:8090/usage/usage?lastID=0&batchSize=1000 However, upon refresh of the browser all data disappears again until I delete the table again - were you able to find a permanent resolution for this problem? Thanks

• Anonymous
  December 19, 2013
  Over the last two “Best Practices” posts, I’ve looked at how to Plan and Build a Hybrid Cloud, and with

• Anonymous
  March 21, 2014
  在上两篇“最佳实践”文章中，我讨论了如何 规划 和 构建 混合云，完成这些技术练习之后，本文将着重探讨部署这一精心设计且构建的混合云的最佳实践。
  本文将介绍一些关键的部署项目

• Anonymous
  May 15, 2014
  I see MYSQL and MSQL records appearing in the usage.records table but there are no VM Utilization records while i do have VM'S allocated.
  
  Any gues on what I'm doing wrong?? :(

• Anonymous
  September 19, 2014
  ,Belstaff Blouson toxic manhttp://www.belstaffsonline.com/Bel Belstaff Blouson toxic man staff-Man-Blouson
  >>>i want to bring in our legal analyst ken Belstaff Olivers Mount leather dall coffee. you heard the jury is asking for clarification on manslaughter

• Anonymous
  September 19, 2014
  The comment has been removed

• Anonymous
  October 31, 2014
  The comment has been removed

• Anonymous
  March 11, 2015
  The comment has been removed

• Anonymous
  May 27, 2015
  ~ Brian McDermott | Escalation Engineer If you use Microsoft Windows Azure Pack and WAP Usage used to

• Anonymous
  June 05, 2015
  The comment has been removed