Script actions during cluster creation

Note

We will retire Azure HDInsight on AKS on January 31, 2025. Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads. The remaining clusters on your subscription will be stopped and removed from the host.

Only basic support will be available until the retirement date.

Important

This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.

Azure HDInsight on AKS provides a mechanism called Script Actions that invoke custom scripts to customize the cluster. These scripts are used to install additional components and change configuration settings. Script actions can be provisioned only during cluster creation as of now. Post cluster creation, Script Actions are part of the roadmap. This article explains how you can provision script actions when you create an HDInsight on AKS cluster.

Use a script action during cluster creation using Azure portal

1. Upload the script action in a ADLS/WASB storage(doesn't have to be the primary cluster storage). In this example we consider an ADLS storage. To upload a script into your storage, navigate into the target storage and the container where you want to upload it.

   

2. To upload a script into your storage, navigate into the target storage and the container. Click on the upload button and select the script from your local drive. After the script gets uploaded you should be able to see it in the container(see following image).

   

3. Create a new cluster as described here.

4. From the Configuration tab, select + Add script action.

   

   This action opens the Script Action window. Provide the following details:

   

   Property	Description
   Script Action Name	Unique name of the script action.
   Bash Script URL	Location where the script is stored. For example - abfs://<CONTAINER>@<DATALAKESTOREACCOUNTNAME>.dfs.core.windows.net/<file_path>, update the data lake storage name and file path.
   Services	Select the specific service components where the Script Action needs to run.
   Parameters	Specify the parameters, if necessary for the script.
   TimeOutInMinutes	Choose the timeout for each script

   

   Note

   • All the Script Actions persisted.
   • Script actions are available only for Apache Spark cluster type.

5. Select ‘OK’ to save the script.

6. Then you can again use + Add Script Action to add another script if necessary.

   

7. Complete the remaining cluster creation steps to create a cluster.

   Important

   • There's no automatic way to undo the changes made by a script action.
   • Script actions must finish within 40 minutes, or they time out causing cluster creation to fail.
   • During cluster provisioning, the script runs concurrently with other setup and configuration processes.
   • Competition for resources such as CPU time or network bandwidth might cause the script to take longer to finish.
   • To minimize the time it takes to run the script, avoid tasks like downloading and compiling applications from the source. Precompile applications and store the binary in Azure Data Lake Store Gen2.

View the list of Script Actions

1. You can view the list of Script Actions in the "Configuration" tab.