Azure Data Lake U-SQL September 2016 Updates: OUTER UNION, Set operations by name, FILE and PARTITION intrinsic functions and more!

I finally found the time to publish the release notes of the September refresh. My apologies for the delay.

The main take away (and there will be a separate blog post about it as well) is that the October refresh will remove the old deprecated syntax of the items we have announced over the last couple of release notes!

We also are looking for some volunteers to test the new version of more scalable file set. Please contact us if you want to try it and help us validate it.

Here is the list of topics with links to the detailed release notes:

• Pending and Upcoming Deprecations
  • Final Deprecation Warning for old PARTITIONED BY Syntax
  • DateTime file set pattern will require HH instead of hh for the hour pattern to align with 24h clock semantics
  • Final Deprecation Warning for {col:*} File Set pattern
• Breaking Changes
  • ALTER TABLE ADD/DROP COLUMN: The COLUMN keyword is now mandatory
• Major U-SQL Bug Fixes, Performance and Scale Improvements
  • Fixed order of REFERENCE and USING in binding
  • CREATE FUNCTION and CREATE PROCEDURE require fully qualify type names in table type definitions
  • Input File Set scales orders of magnitudes better (requires opt-in)
  • Job manager performance got improved which helps short vertex executions
  • Increase IN elements count max
• New U-SQL capabilities
  • New DECLARE statement options
    • DECLARE EXTERNAL
    • DECLARE CONST
  • New scalar FILE and PARTITION object functions are available to test for presence and other properties
    • FILE object functions
    • PARTITION object function
  • U-SQL introduces set operations by name in addition to the existing positional processing
  • U-SQL introduces OUTER UNION to enable easier handling of semistructured data
  • New constant-foldable expressions in U-SQL
  • U-SQL USING adds support of namespace aliasing
  • Visual Studio Tool Provides Job Diff for Job Performance Debugging
  • Visual Studio Tool Provides Throttling Warning Now preview capabilitiescontacting us
  • U-SQL is starting to increase the limit on files in file set extractors
  • U-SQL now offers a preview of sampling capabilities September release notesGitHub siteprevious release notes
• If you want to use the following
• , please request access by
• .
• You can find more details with examples in the
• on our
• where you also can find our
• .