SAP on Azure: General Update for Customers & Partners April 2017
SAP and Microsoft are continuously adding new features and functionalities to the Azure cloud platform. The objective of the Azure cloud platform is to provide the same performance, product availability support matrix and availability as on-premises solutions with the added flexibility of cloud. This blog includes updates, fixes, enhancements and best practice recommendations collated over recent months.
1. SQL Server Multiple Availability Groups on Azure
The Azure platform requires an Internal Load Balancer (ILB) to support clustering on both Linux or Windows High Availability solutions.
Previously there was a limit of one IP address per ILB. This has been removed and now up to 30 IP addresses can be balanced on a single ILB. There is also a limit of 150 port rules per ILB
This means that it is now possible to consolidate multiple AlwaysOn Listeners onto two or more cluster nodes in the same manner as an on-premises deployment.
Before deploying such a configuration it is highly recommended to make a detailed diagram and plan resources such as:
a. Premium Disk design, Number of SQL Datafiles, NTFS format size and whether to use SQL Datafiles stored directly in Azure blob storage
b. Cluster Quorum model and votes
c. Physical and virtual hostnames and IP addresses
d. AlwaysOn replica configuration (such as auto-failover nodes, synchronous and asynchronous replicas)
e. Document the Port that the SQL Server AlwaysOn Listener will use for each Availability Group
IP Address Name |
IP Address Number |
Hostname |
Port |
Probe Port |
Comment |
Host1 – Physical IP |
xx.xx.xx.10 |
Host1 |
IP assigned to SQL Node 1 |
||
Host2 – Physical IP |
xx.xx.xx.11 |
Host2 |
IP assigned to SQL Node 2 |
||
Host3 – Physical IP |
yy.yy.yy.12 |
Host3 |
IP assigned to SQL Node 3 in DR DC |
||
Virtual IP for SQL Listener 1 |
xx.xx.xx.100 |
SAPDB1 |
56000 |
59998 |
Virtual IP created by cluster for SQL AG #1 in Primary DC [assigned to ILB] |
Virtual IP for SQL Listener 2 |
xx.xx.xx.101 |
SAPDB2 |
56001 |
59999 |
Virtual IP created by cluster for SQL AG #2 in Primary DC [assigned to ILB] |
Virtual IP for Windows Cluster |
xx.xx.xx.1 |
SAPCLUDB1 |
Virtual IP for internal cluster in Primary DC [not assigned to ILB] |
||
Virtual IP for SQL Listener 1 |
yy.yy.yy.100 |
SAPDB1 |
56000 |
Virtual IP created by cluster for SQL AG #1 in DR DC [assigned to ILB] |
|
Virtual IP for SQL Listener 2 |
yy.yy.yy.101 |
SAPDB2 |
56001 |
Virtual IP created by cluster for SQL AG #2 in DR DC [assigned to ILB] |
|
Virtual IP for Windows Cluster |
yy.yy.yy.1 |
SAPCLUDB1 |
Virtual IP for internal cluster in DR DC [not assigned to ILB] |
After careful planning then the ILB configuration PowerShell script can be found in this link
Note: it is recommended to set the SQL Server max memory parameter in this configuration. It is recommended to enable Direct Server Return (called Floating IP in the Portal) in this configuration. Similar functionality to stack multiple instances on a single VM is also available on other DBMS. Probe ports must be unique per ILB, but the same probe port number can be reused on different ILB
2. Multiple ASCS on a Cluster in Azure
Multiple ASCS clusters can be consolidated onto a single cluster. This Multi-SID configuration is explained in detail in this documentation.
It is essential to plan and document the solution before attempting to run the PowerShell Scripts to configure the ASCS
Note: As the same port 445 is shared by multiple Frontend IP addresses Direct Server Return must be enabled in this scenario. Direct Server Return is the PowerShell terminology. Floating IP is the terminology for Direct Server Return in the Azure Portal
IP Address Name |
IP Address Number |
Hostname |
Ports |
Probe Port |
Comment |
Host1 – Physical IP |
xx.xx.xx.20 |
Host1 |
IP assigned to ASCS Node 1 |
||
Host2 – Physical IP |
xx.xx.xx.21 |
Host2 |
IP assigned to ASCS Node 2 |
||
Host3 – Physical IP |
yy.yy.yy.22 |
Host3 |
IP assigned to ASCS Node 3 in DR DC |
||
Virtual IP for ECC ASCS |
xx.xx.xx.200 |
SAPECC |
3200,3300,3600 |
59998 |
Virtual IP created by cluster for ECC ASCS in Primary DC [assigned to ILB] |
Virtual IP for BW ASCS |
xx.xx.xx.201 |
SAPBW |
3201,3301,3601 |
59999 |
Virtual IP created by cluster for BW ASCS in Primary DC [assigned to ILB] |
Virtual IP for Windows Cluster |
xx.xx.xx.2 |
SAPCLUSAP1 |
Virtual IP for internal cluster in Primary DC [not assigned to ILB] |
||
Virtual IP for ECC ASCS |
yy.yy.yy.200 |
SAPECC |
3200,3300,3600 |
Virtual IP created by cluster for ECC ASCS in DR DC [assigned to ILB] |
|
Virtual IP for BW ASCS |
yy.yy.yy.201 |
SAPBW |
3201,3301,3601 |
Virtual IP created by cluster for BW ASCS in DR DC [assigned to ILB] |
|
Virtual IP for Windows Cluster |
yy.yy.yy.2 |
SAPCLUSAP1 |
Virtual IP for internal cluster in DR DC [not assigned to ILB] |
SAP ASCS on Azure Checklist:
1. Ensure the Timeout on the ILB is set to 30 minutes (this is the default in the PowerShell script)
2. Ensure the default.pfl parameter enque/encni/set_so_keepalive = TRUE
3. On Windows set TCP/IP registry values KeepAliveTime and KeepAliveInterval set to 180000 (3 minutes) 1593183 - TCP/IP networking parameters for SQL Server https://launchpad.support.sap.com/#/notes/1593183/E
4. Choose Probe Ports (normally 59999)
5. Set the Windows Cluster timeout
With PowerShell
$cluster = Get-Cluster; $cluster.CrossSubnetDelay = 1500
$cluster = Get-Cluster; $cluster.SameSubnetThreshold = 10
$cluster = Get-Cluster; $cluster.CrossSubnetDelay = 1500
$cluster = Get-Cluster; $cluster.CrossSubnetThreshold = 20
With Cluster Command
cluster /cluster:<ClusterName> /prop SameSubnetDelay=1500
cluster /cluster:<ClusterName> /prop SameSubnetThreshold=10
cluster /cluster:<ClusterName> /prop CrossSubnetDelay=1500
cluster /cluster:<ClusterName> /prop CrossSubnetThreshold=20
On Windows 2016 the default values are already set to the correct values
5. A future blog post will discuss setup and configuration of HA on Suse
See SAP Note 1634991 - How to install (A)SCS instance on more than 2 cluster nodes https://launchpad.support.sap.com/#/notes/0001634991
3. Does the Internal Cluster Virtual IP Need To Be Assigned the Azure Internal Load Balancer (ILB)?
Windows cluster has its own internal Virtual IP and Virtual Hostname. These resources are needed for the operation of the cluster. The virtual IP address of the internal cluster does not need to be added as a Frontend IP address onto the Azure Internal Load Balancer (ILB).
There is no requirement to add the cluster Virtual IP address to the ILB, however this can optionally be done.
4. Useful PowerShell Commands for Azure
A basic level of PowerShell knowledge is typically required to deploy SAP systems on Azure at large scale.
It is possible to perform nearly all activities via the Azure Portal, however it is fast, simple and very repeatable to make PowerShell scripts.
To setup Azure PowerShell Cmdlets:
Make sure to install AzureRM PowerShell Cmdlets while running PowerShell as an Administrator
https://msdn.microsoft.com/en-us/library/mt125356.aspx
On a Windows 10 based console it should be possible to open PowerShell as an Administrator and run:
PS C:\> Install-Module AzureRM
PS C:\> Install-AzureRM
Login using the account provided by the Azure administrator. Typically this is username@domain.com
Login-AzureRmAccount
List the available Azure subscriptions with:
Get-AzureRmSubscription
Set-AzureRmContext -SubscriptionName "<subscription name goes here>"
https://docs.microsoft.com/en-us/powershell/#pivot=main&panel=getstarted
/en-us/powershell/azure/overview?view=azurermps-3.7.0
5. SAP Hana on Azure – Virtual Machines
SAP Hana is certified for production OLAP workloads on Azure VM GS5. SAP BW on Hana and similar applications can be run in production on this VM type
GS5 is not Generally Available for Production OLTP workloads as at April 2017
The GS5 VM is certified for all workloads: Suite on Hana, BW on Hana and S/4 Hana for non-production scenarios
https://global.sap.com/community/ebook/2014-09-02-hana-hardware/enEN/iaas.html
More information about installing SAP applications on Hana on Azure GS5 VM type can be found here
/en-us/azure/virtual-machines/workloads/sap/get-started
1928533 - SAP Applications on Azure: Supported Products and Azure VM types https://launchpad.support.sap.com/#/notes/1928533
2015553 - SAP on Microsoft Azure: Support prerequisites https://launchpad.support.sap.com/#/notes/2015553
6. SAP Hana on Azure – Azure Large Instances
Enterprise customers running large scale Hana scenarios are likely to require more than 2TB of RAM to run large BW on Hana or Suite on Hana scenarios and allow for 1-3 years DB growth. The Azure platform provides the following Large Instances for these scenarios based on Intel E7 Haswell & Broadwell processors:
- SAP HANA on Azure S72 (2-socket, 768GB)
- SAP HANA on Azure S72m (2-socket, 1.5TB)
- SAP HANA on Azure S192 (4-socket, 2.0TB)
- SAP HANA on Azure S192m (4-socket, 4.0TB)
- all use cases are supported, OLAP and OLTP, including BWoH, SoH, S/4H
- in production and non-production
- scale-out configurations are possible on "SAP HANA on Azure S144 (4-socket, 1.5TB)" and on "SAP HANA on Azure S192 (4-socket, 2.0TB)" up to 15+1 nodes (15 active (BW: 1 master, 14 worker) nodes, 1 standby)
Common Question & Answer about Azure Large Instances for Hana:
Q1. Are Large Instances VMs or bare metal? Answer = bare metal Hana TDI certified appliances
Q2. Which HA/DR solutions are supported? Answer = both HSR and storage level replication options are possible
Q3. How to order and provision an Azure Hana Large Instance for Hana? Answer = contact Microsoft Account Team
Q4. What is included in the monthly charge on the Azure Bill? Answer = all compute charges, high speed storage equal to 4 x Hana RAM, network costs between SAP application server VMs and the Hana appliance and any network utilization for storage based replication for DR solutions to another Azure DR peer datacenter are included
Q5. Can Azure Large Instances for Hana be upgraded to a larger size? Answer = Yes
Q6. Are all Hana scenarios such as MCOS and MDC supported? Answer = yes, typically the same functionalities that are available with any other TDI solution are available on Azure Large Instances for Hana
Q7. Does Microsoft resell Hana licenses or provide Hana support? Answer = No, Hana licenses and support is provided by SAP. Microsoft provides IaaS (Infrastructure as a Service) only. Hardware, firmware, storage, networking and an initial installation of Suse for SAP Applications or Redhat is provided. Hana should then be installed by a Hana certified consultant. Customers need to buy a Suse or Redhat license and obtain a support contract for Suse or Redhat
Q8. What is the SLA for Azure Large Instances for Hana? Answer = SLA of 99.99% is described here
Q9. Does Microsoft patch and maintain the Suse or Redhat OS on a Hana Large Instance? Answer = No, Hana Large Instances is a IaaS offering. Layers lower than OS are managed and supported by Microsoft.
Q10. Do Hana Large Instances fully support Suse or Redhat clustering? Answer = Yes
2316233 - SAP HANA on Microsoft Azure (Large Instances) https://launchpad.support.sap.com/#/notes/2316233
Links:
/en-us/azure/virtual-machines/workloads/sap/get-started
HA/DR on Large Instances
Backup/Restore Guide
/en-us/azure/virtual-machines/workloads/sap/sap-hana-backup-guide
/en-us/azure/virtual-machines/workloads/sap/sap-hana-backup-file-level
/en-us/azure/virtual-machines/workloads/sap/sap-hana-backup-storage-snapshots
7. Resource Groups on Azure – What Are They & How Should I Use Them?
Resource Group is a logical collection of Azure objects. The properties of a Resource Group are:
1.All the resources in your group should share the same lifecycle. You deploy, update, and delete them together.
2.deployment cycle it should be in another resource group.
3.Each resource can only exist in one resource group.
4.You can add or remove a resource to a resource group at any time.
5.You can move a resource from one resource group to another group. For more information, see Move resources to new resource group or subscription.
6.A resource group can contain resources that reside in different regions.
7.A resource group can be used to scope access control for administrative actions.
8.A resource can interact with resources in other resource groups. This interaction is common when the two resources are related but do not share the same lifecycle
The question for SAP Basis administrators is: How should I structure resource groups across the Sandbox, Development, QAS and Production environments that make up the entire SAP Landscape?
Feedback so far from actual customer deployments:
1. Small Sandbox or Development systems might all be grouped together into only one Resource Group. A small Development or Sandbox environment might comprise of ECC 6.0, BW, SCM, GRC, PI and a Solman system.
2. Small Development or Sandbox would share a common storage account or use Managed Disks
3. Often a single Vnet or at maximum several Vnets for non-prod and a production (Note: It is possible to deploy VMs or other resources from Resource Group A onto a Vnet in Resource Group B).
4. If there is more than one Vnet within the same datacenter, then Vnet peering is used to reduce latencies
5. If there is one Vnet in Datacenter A for Production and another Vnet in Datacenter B for Non-Production and DR there is a Vnet-2-Vnet gateway setup between these two vnets
6. Network Security Groups are typically setup to only allow SAP specific ports onto the subnets such as 3200-3299, 3300-3399, 3600-3699 etc. A full list of SAP ports can be found here. Windows File Sharing ports 135, 139 and 445 would normally be blocked
7. Prior to the introduction of Managed Disks guidance around storage accounts could be summarize as:
-In all cases Premium Storage should be used for DBMS servers or for standalone engines with high IOPS requirements such as TREX
-Small systems such as Development systems might share one storage account
-Larger QAS systems that might be used for performance testing should ideally have their own storage account
-Large Production SAP applications like ECC or BW should have their own storage account
-Smaller Production systems with low IOPS requirement such as Solman, EP, PI or GRC can share a single storage account
Since the introduction of Managed Disks it is generally recommended to use Managed Disks
8. Some customers are deploying individual SAP applications into their own resource groups in production
8. Azure Managed Disks
Managed Disks is a new feature in Azure Storage. It is recommended to evaluate Managed Disks for new SAP deployments. The benefits of Managed Disks are explained here.
In summary the benefits are:
a. No need to create specific storage accounts and balance the number of VHDs across storage account up to a limit of 10,000 disks per subscription
b. There is no need to manually "pin" specific VHDs to different storage stamps to ensure storage level high availability (for example on AlwaysOn cluster nodes)
c. Integration into Azure Backup
Note: Azure Managed Disks request port 8443 to be opened
https://azure.microsoft.com/en-us/services/managed-disks/
/en-us/azure/storage/storage-managed-disks-overview
https://azure.microsoft.com/en-us/pricing/details/managed-disks/
9. Network Latency on Azure
Azure datacenters are distributed across more geographic locations than any other cloud platform. Network latencies from an Azure datacenter to most worldwide locations are more than acceptable for running SAP applications.
Before deploying SAP on Azure it is recommended to test the network latencies from a customer office to nearby Azure datacenters.
The test below shows a test using https://azurespeed.com/ from Singapore.
A test such as this should be performed on a wired connection. Latencies of geographically nearby datacenters should be 10-50ms and trans-continental could be 100-250ms RTT
If values well in excess of these are seen it is possible that the ISP may have routing problems.
10. SAP Application Server on Azure Public Cloud
When installing SAP application servers on Azure Public Cloud Virtual Machines we recommend following the deployment pattern detailed below:
a. Do not provision additional disks for SAP application servers. Install the following components on the Windows C: drive or Linux Root
-Boot
-OS
-/usr/SAP SAP executable directory
b. Place the OS Pagefile onto the local temporary disk (this is the default for Windows)
c. Deploy the latest OS release possible. As at April 2017 Windows 2016, Suse 12.2 and Redhat 7
d. Linux FS type – review 405827 - Linux: Recommended file systems
e. Use a single virtual network card. If SAP LaMa is used a second virtual network card is recommended
Note: Do not under any circumstances use SAP application servers as file servers for interface files. Create a dedicated Management Station for interface files and SAP DVD Installation Media
11. Windows Dynamic Port Ranges
Windows server uses Dynamic Callback Ports that can overlap with SAP J2EE ports
It is recommended to reserve the ports 50000-50999 for SAP.
The commands below should be run on Windows servers with Java Instances installed:
netsh int ipv4 set dynamicport tcp start=60000 numberofports=5536
netsh int ipv4 show dynamicport tcp
1399935 - Reservation of TCP/UDP ports in Windows https://launchpad.support.sap.com/#/notes/1399935
12. Switch Off SAP Application Server AutoStart During Maintenance
The availability of SAP application servers is improved by configuring Autostart. In a scenario where an Azure component fails and the Azure platform self-heals and moves a VM to another node the impact of this restart is much less if the application server restarts automatically.
Autostart of an SAP instance is configured by adding Autostart = 1 to the SAP default.pfl
Maintenance operations like support packs, upgrades, enhancement packs or kernel updates may assume that the default behavior of the SAP system is not to automatically restart.
Therefore it is generally recommended to comment this profile parameter during such activities
13. DevTest Labs – Great for Building Sandbox Systems
The Azure Portal VM properties page has a feature to automatically shutdown VMs. This is very useful for Sandbox or Upgrade/Support Pack testing machines. This feature allows the BASIS team to provision large and powerful virtual machines for testing but to limit the costs by shutting down these VMs when not in use. This is particularly useful for Hana test machines as Hana needs very large VMs
https://azure.microsoft.com/en-us/services/devtest-lab/
SAP systems that are accessed by end users or SAP functional and ABAP teams must also have an Automatic Start feature in addition to the Automatic Stop feature.
This can be achieved using the following methods:
https://clemmblog.azurewebsites.net/start-stop-windows-azure-vms-according-time-schedule/
14. Disk Cache & Encryption Settings
Azure storage provides multiple options for disk caching and encryption.
General guidance for the use of Premium Azure Storage
disk caching:
Premium Disk Type |
Default Cache Setting |
Recommended Cache Setting for SAP Servers |
OS Disk |
ReadWrite |
ReadWrite |
Data Disk – Write Heavy |
None |
None (for example DB Transaction Log) |
Data Disk – Read Only |
None |
ReadOnly |
Do not use ReadWrite disk cache settings on SAP systems including DBMS disks or TREX
General Guidance for the use of Encryption:
1. Assess the risk profile of the SAP systems and evaluate if Encryption is required or not
2. Azure platform supports Encryption at two different layers
-Storage Account Level
-Disk Level
3. DBMS platforms also support encryption – SQL Server TDE or Hana encryption for example
4. Typically DBMS level encryption has the advantage of also encrypting backup files. Backup files are common attack vector for data theft
5. It is strongly recommended not to use multiple encryption technologies on the same device (for example use DB level encryption, File System level encryption and using an encrypted storage account). This can lead to performance problems
6. A recommended configuration is:
-For DB servers use Disk encryption to protect the OS/Boot disk only. Use native DBMS level encryption to protect the DB files and backups
-For SAP application server use Disk encryption to protect the OS/Boot disk.
Note: some forms of DB level encryption are vulnerable to cloning the entire Virtual Machine and all disks. Using Azure Disk Encryption on the OS/Boot disk prevents cloning an entire VM.
Customer Case Studies on Azure
Large Australian Energy Company Modernizes SAP Applications & Moves to Azure Public Cloud
Zespri https://customers.microsoft.com/en-us/story/kiwi-grower-prunes-costs-defends-business-from-disaste
PACT Building on synergy for a bold growth strategy https://customers.microsoft.com/en-us/story/building-on-synergy-for-a-bold-growth-strategy
AGL Innovation Spotlight: AGL puts energy into action with the Cloud https://customers.microsoft.com/en-us/story/innovation-spotlight-agl-puts-energy-into-action-with
Coats UK https://customers.microsoft.com/en-us/story/coats
The Mosaic Company https://customers.microsoft.com/en-us/story/mosaicandcapgemini
Several new large Enterprise Case Studies and customer go lives will be released on this blog soon
Content from third party websites, SAP and other sources reproduced in accordance with Fair Use criticism, comment, news reporting, teaching, scholarship, and research