Edit

Share via

Deploy an Azure Batch account and two pools with a start task - Terraform

  • Article

In this quickstart, you create an Azure Batch account, an Azure Storage account, and two Batch pools using Terraform. Batch is a cloud-based job scheduling service that parallelizes and distributes the processing of large volumes of data across many computers. It's typically used for tasks like rendering 3D graphics, analyzing large datasets, or processing video. In this case, the resources created include a Batch account (which is the central organizing entity for distributed processing tasks), a Storage account for holding the data to be processed, and two Batch pools, which are groups of virtual machines that execute the tasks.

Terraform enables the definition, preview, and deployment of cloud infrastructure. Using Terraform, you create configuration files using HCL syntax. The HCL syntax allows you to specify the cloud provider - such as Azure - and the elements that make up your cloud infrastructure. After you create your configuration files, you create an execution plan that allows you to preview your infrastructure changes before they're deployed. Once you verify the changes, you apply the execution plan to deploy the infrastructure.

  • Specify the required version of Terraform and the required providers.
  • Define the Azure provider with no additional features.
  • Define variables for the resource group location and name prefix.
  • Generate a random name for the Azure resource group.
  • Create a resource group with the generated name at a specified location.
  • Generate a random string for the Storage account name.
  • Create a Storage account with the generated name in the created resource group.
  • Generate a random string for the Batch account name.
  • Create a Batch account with the generated name in the created resource group and linked to the created Storage account.
  • Generate a random name for the Batch pool.
  • Create a Batch pool with a fixed scale in the created resource group and linked to the created Batch account.
  • Create a Batch pool with autoscale in the created resource group and linked to the created Batch account.
  • Output the names of the created resource group, Storage account, Batch account, and both Batch pools.

Prerequisites

Implement the Terraform code

Note

The sample code for this article is located in the Azure Terraform GitHub repo. You can view the log file containing the test results from current and previous versions of Terraform.

See more articles and sample code showing how to use Terraform to manage Azure resources.

  1. Create a directory in which to test and run the sample Terraform code, and make it the current directory.

  2. Create a file named main.tf, and insert the following code:

    resource "random_pet" "rg_name" {
  prefix = var.resource_group_name_prefix
}

resource "azurerm_resource_group" "rg" {
  location = var.resource_group_location
  name     = random_pet.rg_name.id
}

resource "random_string" "storage_account_name" {
  length  = 8
  lower   = true
  numeric = false
  special = false
  upper   = false
}

resource "azurerm_storage_account" "example" {
  name                     = random_string.storage_account_name.result
  resource_group_name      = azurerm_resource_group.rg.name
  location                 = azurerm_resource_group.rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "random_string" "batch_account_name" {
  length  = 8
  lower   = true
  numeric = false
  special = false
  upper   = false
}

resource "azurerm_batch_account" "example" {
  name                                = random_string.batch_account_name.result
  resource_group_name                 = azurerm_resource_group.rg.name
  location                            = azurerm_resource_group.rg.location
  storage_account_id                  = azurerm_storage_account.example.id
  storage_account_authentication_mode = "StorageKeys"
}

resource "random_pet" "azurerm_batch_pool_name" {
  prefix = "pool"
}

resource "azurerm_batch_pool" "fixed" {
  name                = "${random_pet.azurerm_batch_pool_name.id}-fixed-pool"
  resource_group_name = azurerm_resource_group.rg.name
  account_name        = azurerm_batch_account.example.name
  display_name        = "Fixed Scale Pool"
  vm_size             = "Standard_D4_v3"
  node_agent_sku_id   = "batch.node.ubuntu 22.04"

  fixed_scale {
    target_dedicated_nodes = 2
    resize_timeout         = "PT15M"
  }

  storage_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "latest"
  }

  start_task {
    command_line       = "echo 'Hello World from $env'"
    task_retry_maximum = 1
    wait_for_success   = true

    common_environment_properties = {
      env = "TEST"
    }

    user_identity {
      auto_user {
        elevation_level = "NonAdmin"
        scope           = "Task"
      }
    }
  }

  metadata = {
    "tagName" = "Example tag"
  }
}

resource "azurerm_batch_pool" "autopool" {
  name                = "${random_pet.azurerm_batch_pool_name.id}-autoscale-pool"
  resource_group_name = azurerm_resource_group.rg.name
  account_name        = azurerm_batch_account.example.name
  display_name        = "Auto Scale Pool"
  vm_size             = "Standard_D4_v3"
  node_agent_sku_id   = "batch.node.ubuntu 22.04"

  auto_scale {
    evaluation_interval = "PT15M"

    formula = <<EOF
      startingNumberOfVMs = 1;
      maxNumberofVMs = 25;
      pendingTaskSamplePercent = $PendingTasks.GetSamplePercent(180 * TimeInterval_Second);
      pendingTaskSamples = pendingTaskSamplePercent < 70 ? startingNumberOfVMs : avg($PendingTasks.GetSample(180 * TimeInterval_Second));
      $TargetDedicatedNodes=min(maxNumberofVMs, pendingTaskSamples);
EOF
  }

  storage_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "latest"
  }
}

  3. Create a file named outputs.tf, and insert the following code:

    output "resource_group_name" {
  value = azurerm_resource_group.rg.name
}

output "storage_account_name" {
  value = azurerm_storage_account.example.name
}

output "batch_account_name" {
  value = azurerm_batch_account.example.name
}

output "batch_pool_fixed_name" {
  value = azurerm_batch_pool.fixed.name
}

output "batch_pool_autopool_name" {
  value = azurerm_batch_pool.autopool.name
}

  4. Create a file named providers.tf, and insert the following code:

    terraform {
  required_version = ">=1.0"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~>3.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~>3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

  5. Create a file named variables.tf, and insert the following code:

    variable "resource_group_location" {
  type        = string
  default     = "eastus"
  description = "Location of the resource group."
}

variable "resource_group_name_prefix" {
  type        = string
  default     = "rg"
  description = "Prefix of the resource group name that's combined with a random ID so name is unique in your Azure subscription."
}

Initialize Terraform

Run terraform init to initialize the Terraform deployment. This command downloads the Azure provider required to manage your Azure resources.

terraform init -upgrade

Key points:

  • The -upgrade parameter upgrades the necessary provider plugins to the newest version that complies with the configuration's version constraints.

Create a Terraform execution plan

Run terraform plan to create an execution plan.

terraform plan -out main.tfplan

Key points:

  • The terraform plan command creates an execution plan, but doesn't execute it. Instead, it determines what actions are necessary to create the configuration specified in your configuration files. This pattern allows you to verify whether the execution plan matches your expectations before making any changes to actual resources.
  • The optional -out parameter allows you to specify an output file for the plan. Using the -out parameter ensures that the plan you reviewed is exactly what is applied.

Apply a Terraform execution plan

Run terraform apply to apply the execution plan to your cloud infrastructure.

terraform apply main.tfplan

Key points:

  • The example terraform apply command assumes you previously ran terraform plan -out main.tfplan.
  • If you specified a different filename for the -out parameter, use that same filename in the call to terraform apply.
  • If you didn't use the -out parameter, call terraform apply without any parameters.

Verify the results

Run az batch account show to view the Batch account.

az batch account show --name <batch_account_name> --resource-group <resource_group_name>

In the above command, replace <batch_account_name> with the name of your Batch account and <resource_group_name> with the name of your resource group.

Clean up resources

When you no longer need the resources created via Terraform, do the following steps:

  1. Run terraform plan and specify the destroy flag.

    terraform plan -destroy -out main.destroy.tfplan

    Key points:

    • The terraform plan command creates an execution plan, but doesn't execute it. Instead, it determines what actions are necessary to create the configuration specified in your configuration files. This pattern allows you to verify whether the execution plan matches your expectations before making any changes to actual resources.
    • The optional -out parameter allows you to specify an output file for the plan. Using the -out parameter ensures that the plan you reviewed is exactly what is applied.

  2. Run terraform apply to apply the execution plan.

    terraform apply main.destroy.tfplan

Troubleshoot Terraform on Azure

Troubleshoot common problems when using Terraform on Azure.

Next steps

See more articles about Batch accounts.