Muokkaa

Jaa

Access Apache Flink® CLI client using Secure Shell (SSH) on HDInsight on AKS clusters with Azure portal

  • Artikkeli

Note

We will retire Azure HDInsight on AKS on January 31, 2025. Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads. The remaining clusters on your subscription will be stopped and removed from the host.

Only basic support will be available until the retirement date.

Important

This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.

This example guides how to enter the Apache Flink CLI client on HDInsight on AKS clusters using SSH on Azure portal, we cover both SQL and Flink DataStream.

Prerequisites

  • You're required to select SSH during creation of Flink Cluster

Connecting to the SSH from Azure portal

Once the Flink cluster is created, you can observe on the left pane the Settings option to access Secure Shell

Screenshot showing How to create POD and connect to webssh.

Connecting to SQL Client

You're required to change directory to /opt/flink-webssh/bin and then execute ./sql-client.sh

Screenshot how to find sql client file.

Screenshot showing how to run SQL client.

You're now on SQL Client on Flink

Refer to this document to perform few more tests.

Flink provides a Command-Line Interface (CLI)  bin/flink to run programs that are packaged as JAR files and to control their execution.

The CLI is part Secure Shell (SSH), and it connects to the running JobManager and use the client configurations specified at conf/flink-conf.yaml.

Submitting a job means to upload the job’s JAR to the SSH pod and initiating the job execution. To illustrate an example for this article, we select a long-running job like examples/streaming/StateMachineExample.jar.

Note

For managing dependencies, the expectation is to build and submit a fat jar for the job.

  • Upload the fat job jar from ABFS to webssh.

  • Based on your use case, you’re required to edit the client configurations using Flink configuration management under flink-client-configs.

  • Let us run StateMachineExample.jar

    ./bin/flink run \
    --detached \
    ./examples/streaming/StateMachineExample.jar

Note

Submitting the job using --detached will make the command return after the submission is done. The output contains the ID of the newly submitted job.

Reference