Azure Data Lake U-SQL April 25 2017 Updates: Introducing Packages, UNPIVOT INCLUDE NULLS, fast file set preview flag, R extension returns dataframes, exporting your cluster database with sample data to your local run and more!

We have concluded the rollout of our April 2017 refresh to all the regions today.

Here are the April 2017 Updates for Azure Data Lake U-SQL and Developer Tooling!

The main items are the release of the package feature that allows you to bundle assembly reference statements, variable declarations into a shareable package and reduce the amount of typing you have to do when including more complex assembly dependencies, allowing UNPIVOT to include rows with nulls, you can return a dataframe from the R extension, and exporting your cluster database with sample data to your local run environment in VisualStudio. We also publicly document the file set scalability improvement opt-in statement. If you use file sets in your query, this is a must try feature!

Thanks to all of you who continue to volunteer to test the new version of the more scalable file set. Now everyone can do it. Please contact us if you have explicit feedback or want to explore the new flexible-schema feature preview for TVF parameters.

Here is the list of topics with links to the detailed release notes:

• Pending and Upcoming Deprecations
  • DROP CREDENTIAL DDL will start to raise an error an upcoming refresh
  • U-SQL jobs will introduce an upper limit for the number of table-backing files being read
  • Table-valued functions will disallow result variable names to conflict with parameter names
• Breaking Changes
  • This release has no known breaking changes.
• Major U-SQL Bug Fixes, Performance and Scale Improvements
  • Improved data-size dependent selection for default numbers of HASH DISTRIBUTION buckets
  • CREATE STATISTICS Performance Improvements
  • U-SQL increases the supported number of UNION and UNION ALL operators used in a single query expression
  • The R U-SQL Extension now provides the ability to return a data frame instead of a string
  • Improved error reporting in the U-SQL Python Extension
• U-SQL Preview Features
  • Input File Set scales orders of magnitudes better (opt-in statement is now provided!)
  • A limited flexible-schema feature for U-SQL table-valued function parameters is now available for preview (requires opt-in)
• New U-SQL capabilities
  • U-SQL adds the notion of Packages
  • U-SQL's UNPIVOT allows explicit inclusion and exclusion of null values
  • The column alias is no longer required for expressions that end with a property or field access
• Azure Data Lake Tools for Visual Studio New Capabilities
  • New Location of the Data Lake Menu
  • Exporting U-SQL database schemas and sample data and importing that information into your local run environment is now supported
  • Local database credential management is now supported
• Azure Portal Updates
  • Rotate keys used in Azure Data Lake Store encryption
  • Control who has access to debug in Azure Data Lake Analytics

If you want to use the above preview capabilities, please request access by contacting us.

In order to get access to the new syntactic features and new tool capabilities on your local environment, you will need to refresh your ADL Tools. If you use VisualStudio 2013 or 2015, you can download and install them directly from MSDN or use the new Check for Updates menu item mentioned above. If you are using VisualStudio 2017, you currently have to wait for the next VisualStudio 2017 refresh that should occur about every 6 to 8 weeks. Otherwise you will not be able to use the new features during local run and submission to the cluster will give you syntax errors for the new language features (although you can still submit the script anyway).

You can find more details with examples in the April 2017 release notes (or by clicking on the items above) on our GitHub site, where you also can find our previous release notes.