Deploy an Azure Batch account and two pools - Terraform

Article
10/24/2024

In this quickstart, you create an Azure Batch account, an Azure Storage account, and two Batch pools using Terraform. Batch is a cloud-based job scheduling service that parallelizes and distributes the processing of large volumes of data across many computers. It's typically used for parametric sweeps, Monte Carlo simulations, financial risk modeling, and other high-performance computing applications. A Batch account is the top-level resource in the Batch service that provides access to pools, jobs, and tasks. The Storage account is used to store and manage all the files that are used and generated by the Batch service, while the two Batch pools are collections of compute nodes that execute the tasks.

Terraform enables the definition, preview, and deployment of cloud infrastructure. Using Terraform, you create configuration files using HCL syntax. The HCL syntax allows you to specify the cloud provider - such as Azure - and the elements that make up your cloud infrastructure. After you create your configuration files, you create an execution plan that allows you to preview your infrastructure changes before they're deployed. Once you verify the changes, you apply the execution plan to deploy the infrastructure.

Specify the required version of Terraform and the required providers.
Define the Azure provider with no additional features.
Define variables for the location of the resource group and the prefix of the resource group name.
Generate a random name for the resource group using the provided prefix.
Create an Azure resource group with the generated name at the specified location.
Generate a random string to be used as the name for the Storage account.
Create a Storage account with the generated name in the created resource group, at the same location, and with a standard account tier and locally redundant Storage replication type.
Generate another random string to be used as the name for the Batch account.
Create a Batch account with the generated name in the created resource group, at the same location, and link it to the created Storage account with Storage keys authentication mode.
Generate a random name for the Batch pool with a "pool" prefix.
Create a Batch pool with a fixed scale using the generated name in the created resource group, linked to the created Batch account, with a standard A1 virtual machine (VM) size, Ubuntu 22.04 node agent SKU, and a start task that echoes 'Hello World from $env' with a maximum of one retry and waits for success.
Create another Batch pool with auto scale, using the same generated name, in the created resource group, linked to the created Batch account, with a standard A1 VM size, Ubuntu 22.04 node agent SKU, and an autoscale formula.
Output the names of the created resource group, Storage account, Batch account, and both Batch pools.

Prerequisites

Create an Azure account with an active subscription. You can create an account for free.
Install and configure Terraform.

Implement the Terraform code

Note

The sample code for this article is located in the Azure Terraform GitHub repo. You can view the log file containing the test results from current and previous versions of Terraform.

See more articles and sample code showing how to use Terraform to manage Azure resources.

Create a directory in which to test and run the sample Terraform code, and make it the current directory.

Create a file named main.tf, and insert the following code:

resource "random_pet" "rg_name" {
  prefix = var.resource_group_name_prefix
}

resource "azurerm_resource_group" "rg" {
  location = var.resource_group_location
  name     = random_pet.rg_name.id
}

resource "random_string" "storage_account_name" {
  length  = 8
  lower   = true
  numeric = false
  special = false
  upper   = false
}

resource "azurerm_storage_account" "example" {
  name                     = random_string.storage_account_name.result
  resource_group_name      = azurerm_resource_group.rg.name
  location                 = azurerm_resource_group.rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "random_string" "batch_account_name" {
  length  = 8
  lower   = true
  numeric = false
  special = false
  upper   = false
}

resource "azurerm_batch_account" "example" {
  name                                = random_string.batch_account_name.result
  resource_group_name                 = azurerm_resource_group.rg.name
  location                            = azurerm_resource_group.rg.location
  storage_account_id                  = azurerm_storage_account.example.id
  storage_account_authentication_mode = "StorageKeys"
}

resource "random_pet" "azurerm_batch_pool_name" {
  prefix = "pool"
}

resource "azurerm_batch_pool" "fixed" {
  name                = "${random_pet.azurerm_batch_pool_name.id}-fixed-pool"
  resource_group_name = azurerm_resource_group.rg.name
  account_name        = azurerm_batch_account.example.name
  display_name        = "Fixed Scale Pool"
  vm_size             = "Standard_A1"
  node_agent_sku_id   = "batch.node.ubuntu 22.04"

  fixed_scale {
    target_dedicated_nodes = 2
    resize_timeout         = "PT15M"
  }

  storage_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "latest"
  }

  start_task {
    command_line       = "echo 'Hello World from $env'"
    task_retry_maximum = 1
    wait_for_success   = true

    common_environment_properties = {
      env = "TEST"
    }

    user_identity {
      auto_user {
        elevation_level = "NonAdmin"
        scope           = "Task"
      }
    }
  }

  metadata = {
    "tagName" = "Example tag"
  }
}

resource "azurerm_batch_pool" "autopool" {
  name                = "${random_pet.azurerm_batch_pool_name.id}-autoscale-pool"
  resource_group_name = azurerm_resource_group.rg.name
  account_name        = azurerm_batch_account.example.name
  display_name        = "Auto Scale Pool"
  vm_size             = "Standard_A1"
  node_agent_sku_id   = "batch.node.ubuntu 22.04"

  auto_scale {
    evaluation_interval = "PT15M"

    formula = <<EOF
      startingNumberOfVMs = 1;
      maxNumberofVMs = 25;
      pendingTaskSamplePercent = $PendingTasks.GetSamplePercent(180 * TimeInterval_Second);
      pendingTaskSamples = pendingTaskSamplePercent < 70 ? startingNumberOfVMs : avg($PendingTasks.GetSample(180 * TimeInterval_Second));
      $TargetDedicatedNodes=min(maxNumberofVMs, pendingTaskSamples);
EOF
  }

  storage_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "latest"
  }
}

Create a file named outputs.tf, and insert the following code:

output "resource_group_name" {
  value = azurerm_resource_group.rg.name
}

output "storage_account_name" {
  value = azurerm_storage_account.example.name
}

output "batch_account_name" {
  value = azurerm_batch_account.example.name
}

output "batch_pool_fixed_name" {
  value = azurerm_batch_pool.fixed.name
}

output "batch_pool_autopool_name" {
  value = azurerm_batch_pool.autopool.name
}

Create a file named providers.tf, and insert the following code:

terraform {
  required_version = ">=1.0"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~>3.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~>3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

Create a file named variables.tf, and insert the following code:

variable "resource_group_location" {
  type        = string
  default     = "eastus"
  description = "Location of the resource group."
}

variable "resource_group_name_prefix" {
  type        = string
  default     = "rg"
  description = "Prefix of the resource group name that's combined with a random ID so name is unique in your Azure subscription."
}

Initialize Terraform

Run terraform init to initialize the Terraform deployment. This command downloads the Azure provider required to manage your Azure resources.

terraform init -upgrade

Key points:

The -upgrade parameter upgrades the necessary provider plugins to the newest version that complies with the configuration's version constraints.

Create a Terraform execution plan

Run terraform plan to create an execution plan.

terraform plan -out main.tfplan

Key points:

The terraform plan command creates an execution plan, but doesn't execute it. Instead, it determines what actions are necessary to create the configuration specified in your configuration files. This pattern allows you to verify whether the execution plan matches your expectations before making any changes to actual resources.
The optional -out parameter allows you to specify an output file for the plan. Using the -out parameter ensures that the plan you reviewed is exactly what is applied.

Apply a Terraform execution plan

Run terraform apply to apply the execution plan to your cloud infrastructure.

terraform apply main.tfplan

Key points:

The example terraform apply command assumes you previously ran terraform plan -out main.tfplan.
If you specified a different filename for the -out parameter, use that same filename in the call to terraform apply.
If you didn't use the -out parameter, call terraform apply without any parameters.

Verify the results

Azure CLI

Run az batch account show to view the Batch account.

az batch account show --name <batch_account_name> --resource-group <resource_group_name>

Replace <batch_account_name> with the name of your Batch account and <resource_group_name> with the name of your resource group.

Clean up resources

When you no longer need the resources created via Terraform, do the following steps:

Run terraform plan and specify the destroy flag.
```
terraform plan -destroy -out main.destroy.tfplan
```
Key points:
- The terraform plan command creates an execution plan, but doesn't execute it. Instead, it determines what actions are necessary to create the configuration specified in your configuration files. This pattern allows you to verify whether the execution plan matches your expectations before making any changes to actual resources.
- The optional -out parameter allows you to specify an output file for the plan. Using the -out parameter ensures that the plan you reviewed is exactly what is applied.
Run terraform apply to apply the execution plan.
```
terraform apply main.destroy.tfplan
```

Troubleshoot Terraform on Azure

Troubleshoot common problems when using Terraform on Azure.

Next steps

See more articles about Batch accounts.

Share via

Deploy an Azure Batch account and two pools - Terraform

Prerequisites

Implement the Terraform code

Initialize Terraform

Create a Terraform execution plan

Apply a Terraform execution plan

Verify the results

Clean up resources

Troubleshoot Terraform on Azure

Next steps

Feedback

Additional resources