HDInsight tools for IntelliJ & Eclipse April Updates

 

We are pleased to announce the April updates of HDInsight Tools for IntelliJ & Eclipse. This is a quality milestone and we focus primarily on refactoring the components and fixing bugs. We also added Azure Data Lake Store support and Eclipse local emulator support in this release. The HDInsight Tools for IntelliJ & Eclipse serve the open source community and are of interest to HDInsight Spark developers. The tools run smoothly in Linux, Mac and Windows. 

Summary of Key Updates

Azure Data Lake Store support -

• HDInsight Visual Studio plugin, Eclipse plugin and IntelliJ plugin now support Azure Data Lake Store (ADLS). Users can now view ADLS entities in the service explorer, add ADLS namespace/path in authoring, submit Hive/Spark jobs reading/writing to ADLS in HDInsight cluster.
• To use Azure Data Lake Store, users firstly need to create Azure HDInsight cluster with Data Lake Store as storage, follow the instructions to Create an HDInsight cluster with Data Lake Store using Azure Portal
• As shown below, ADLS entities can be viewed in the service explorer

• By clicking “Explorer” above, users can explore data stored in ADLS, as shown below:

• Users can read/write ADLS data in their Hive/Spark jobs, as shown below
  • If Data Lake Store is the primary storage for the cluster, use adl:/// . This is the root of the cluster storage in Azure Data Lake. This may translate to path of /clusters/CLUSTERNAME in the Data Lake Store account.
  • If Data Lake Store is additional storage for the cluster, use adl://DATALAKEACCOUNT.azuredatalakestore.net/ . The URI specifies the Data Lake Store account the data is written to and data is written starting at the root of the Data Lake Store.

 

• Refer to this page for Use HDInsight Spark cluster to analyze data in Data Lake Store
• Refer to this page for Use Azure Data Lake Store with Apache Storm with HDInsight

Local Emulator for Eclipse Plugin -

• Local emulator is supported before in IntelliJ plugin.
• Now local emulator is also supported in Eclipse plugin, similar functionalities and user experiences as local emulator in IntelliJ
• Refer to this page for more details about local emulator support (/en-us/azure/hdinsight/hdinsight-tools-for-intellij-with-hortonworks-sandbox)

Quality Improvement -

The major improvements are code refactoring and telemetry enhancements. More than forty bugs around job author, submission and job view are fixed to improve the quality of the tools in this release.

Installation

• If you have HDInsight Tools for Visual Studio/Eclipse/IntelliJ installed before, the new bits can be updated in the IDE directly.
• Otherwise please refer to the pages below to download the latest bits or distribute the information to the customers:
  • HDInsight Visual Studio plugin
  • HDInsight Eclipse plugin
  • HDInsight IntelliJ plugin

Upcoming releases

The following features are planned for upcoming release:

• Debuggability: Remote debugging support for Spark application
• Monitoring: Improve Spark application view, job view and job graph
• Usability: Improve installation experience; Integrate into IntelliJ run menu
• Enable Mooncake support

Feedback

We look forward to your comments and feedback. If there is any feature request, customer ask or suggestion, please do send us an email. For bug submission, you can open one using the template here.