Share via

Databricks Runtime 16.0 (Beta)

  • Article

Important

Databricks Runtime 16.0 is in Beta. The contents of the supported environments may change during the Beta. Changes can include the list of packages or versions of installed packages.

The following release notes provide information about Databricks Runtime 16.0, powered by Apache Spark 3.5.0.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

Behavioral changes

Improve duplicate match detection in Delta Lake MERGE

In Databricks Runtime 15.4 LTS and below, MERGE operations fail if more than one row in the source table matches the same row in the target table based on the MERGE condition specified in the ON clause. In Databricks Runtime 16.0 and above, MERGE also considers conditions specified in the WHEN MATCHED clause. See Upsert into a Delta Lake table using merge.

Cluster library installation method can no longer be overridden

The Spark configurations spark.databricks.libraries.enableSparkPyPI, spark.databricks.libraries.enableMavenResolution, and spark.databricks.libraries.enableCRANResolutionnow now default to true and cannot be overridden.

JDK 17 is now the default

[Breaking change] In Databricks Runtime 16.0 and above, the default JDK version is switched from JDK 8 to JDK 17. This change is made because of the planned deprecation and end-of-support for JDK 8. This affects the following:

  • Java code running on Azure Databricks compute must be compatible with Java 17.
  • Scala code running in notebooks or Azure Databricks compute must be compatible with Java 17.
  • Java and Scala libraries installed on compute must be compatible with Java 17.
  • Versions of the Apache Hive metastore client below 2.x. Setting the Spark configuration spark.sql.hive.metastore.version to a version lower than 2.x will cause compatibility issues with Java 17 and connection failures to the Hive metastore. Databricks recommends upgrading Hive to a version above 2.0.0.

If you need to revert to Java 8, add the following to the Spark environment variables when you configure your Azure Databricks compute:

JNAME=zulu8-ca-amd64

If you’re using ARM instances, use the following:

JNAME=zulu8-ca-arm64

To learn more about specifying JDK versions with Azure Databricks compute, see Create a cluster that uses JDK 17.

For help with migrating your code from Java 8, see the following guides:

Default timeout of two hours for cluster-scoped library installs

In Databricks Runtime 16.0 and above, cluster-scoped library installation has a default timeout of two hours. Library installations that take longer than this timeout will fail, and the installation is terminated. When configuring a cluster, you can change the timeout period using the Spark configuration spark.databricks.driver.clusterLibraryInstallationTimeoutSec.

Installing libraries from DBFS and setting the spark conf spark.databricks.driver.dbfsLibraryInstallationAllowed are disabled

In Databricks Runtime 16.0 and above, installing libraries from DBFS is fully disabled. This change is made to improve the security of libraries in a Databricks workspace. Additionally, in Databricks Runtime 16.0 and above, you can no longer use the Spark configuration spark.databricks.driver.dbfsLibraryInstallationAllowed.

New features and improvements

More reliable reloading of modified Python modules with improvements to autoreload

In Databricks Runtime 16.0 and above, updates to the autoreload extension improve the safety and reliability of reloading modified Python modules imported from workspace files. With these changes, autoreload, when possible, reloads only the portion of a module that has changed instead of the entire module. Additionally, Azure Databricks now automatically suggests using the autoreload extension if the module has changed since its last import. See Autoreload for Python modules.

Avro support for recursive schema

You can now use the recursiveFieldMaxDepth option with the from_avro function and the avro data source. This option sets the maximum depth for schema recursion on the Avro data source. See Read and write streaming Avro data.

Expanded support for Confluent Schema Registry for Avro

Azure Databricks now supports Avro schema reference with the Confluent Schema Registry. See Authenticate to an external Confluent Schema Registry.

Force reclustering on tables with liquid clustering

In Databricks Runtime 16.0 and above, you can use the OPTIMIZE FULL syntax to force the reclustering of all records in a table with liquid clustering enabled. See Force reclustering for all records.

The Delta APIs for Python and Scala now support identity columns

You can now use the Delta APIs for Python and Scala to create tables with identity columns. See Use identity columns in Delta Lake.

Fine-grained access control on single user compute is generally available

In Databricks Runtime 16.0 and above, fine-grained access control on single user compute is generally available. In workspaces enabled for serverless compute, if a query is run on supported compute such as single user compute and the query accesses any of the following objects, the compute resource passes the query to the serverless compute to run data filtering:

  • Views defined over tables on which the user does not have the SELECT privilege.
  • Dynamic views.
  • Tables with row filters or column masks applied.
  • Materialized views and streaming tables.

Create liquid clustered tables during streaming writes

You can now use clusterBy to enable liquid clustering when creating new tables with Structured Streaming writes. See Enable liquid clustering.

Support for the OPTIMIZE FULL clause

Databricks Runtime 16.0 supports the OPTIMIZE FULL clause. This clause optimizes all records in a table that uses liquid clustering, including data that might have previously been clustered.

New SQL functions

The following SQL functions are added in Databricks Runtime 16.0:

  • try_url_decode

    This function is an error-tolerant version of url_decode. This function returns NULL if the input is not a valid URL-encoded string.

  • zeroifnull

    If the input expression to the zeroifnull() function is NULL, then the function returns 0. Otherwise, the value of the input expression is returned.

  • nullifzero

    Returns NULL if the input is 0 or its input if it is not 0. If the input expression to the nullifzero() function is 0, then the function returns NULL. If the input expression is not 0, the value of the input expression is returned

Other changes

SparkR is now deprecated

In Databricks Runtime 16.0 and above, SparkR in Databricks is deprecated in preparation for its deprecation in the upcoming Spark 4 release. See the Apache Spark Deprecate SparkR thread.

Databricks recommends using sparklyr instead.

Library upgrades

  • Upgraded Python libraries:
    • azure-core from 1.30.2 to 1.31.0
    • azure-storage-blob from 12.19.1 to 12.23.0
    • azure-storage-file-datalake from 12.14.0 to 12.17.0
    • black from 23.3.0 to 24.4.2
    • blinker from 1.4 to 1.7.0
    • boto3 from 1.34.39 to 1.34.69
    • botocore from 1.34.39 to 1.34.69
    • certifi from 2023.7.22 to 2024.6.2
    • cffi from 1.15.1 to 1.16.0
    • click from 8.0.4 to 8.1.7
    • comm from 0.1.2 to 0.2.1
    • contourpy from 1.0.5 to 1.2.0
    • cryptography from 41.0.3 to 42.0.5
    • Cython from 0.29.32 to 3.0.11
    • databricks-sdk from 0.20.0 to 0.30.0
    • dbus-python from 1.2.18 to 1.3.2
    • filelock from 3.13.4 to 3.15.4
    • fonttools from 4.25.0 to 4.51.0
    • GitPython from 3.1.43 to 3.1.37
    • google-api-core from 2.18.0 to 2.20.0
    • google-auth from 2.31.0 to 2.35.0
    • google-cloud-storage from 2.17.0 to 2.18.2
    • google-crc32c from 1.5.0 to 1.6.0
    • google-resumable-media from 2.7.1 to 2.7.2
    • googleapis-common-protos from 1.63.2 to 1.65.0
    • httplib2 from 0.20.2 to 0.20.4
    • idna from 3.4 to 3.7
    • ipykernel from 6.25.1 to 6.28.0
    • ipython from 8.15.0 to 8.25.0
    • jedi from 0.18.1 to 0.19.1
    • jmespath from 0.10.0 to 1.0.1
    • joblib from 1.2.0 to 1.4.2
    • jupyter_client from 7.4.9 to 8.6.0
    • jupyter_core from 5.3.0 to 5.7.2
    • launchpadlib from 1.10.16 to 1.11.0
    • lazr.restfulclient from 0.14.4 to 0.14.6
    • matplotlib from 3.7.2 to 3.8.4
    • mlflow-skinny from 2.11.4 to 2.15.1
    • more-itertools from 8.10.0 to 10.3.0
    • mypy-extensions from 0.4.3 to 1.0.0
    • nest-asyncio from 1.5.6 to 1.6.0
    • numpy from 1.23.5 to 1.26.4
    • oauthlib from 3.2.0 to 3.2.2
    • packaging from 23.2 to 24.1
    • patsy from 0.5.3 to 0.5.6
    • pip from 23.2.1 to 24.2
    • plotly from 5.9.0 to 5.22.0
    • prompt-toolkit from 3.0.36 to 3.0.43
    • pyarrow from 14.0.1 to 15.0.2
    • pydantic from 1.10.6 to 2.8.2
    • PyGObject from 3.42.1 to 3.48.2
    • PyJWT from 2.3.0 to 2.7.0
    • pyodbc from 4.0.38 to 5.0.1
    • python-dateutil from 2.8.2 to 2.9.0.post0
    • python-lsp-jsonrpc from 1.1.1 to 1.1.2
    • pytz from 2022.7 to 2024.1
    • PyYAML from 6.0 to 6.0.1
    • pyzmq from 23.2.0 to 25.1.2
    • requests from 2.31.0 to 2.32.2
    • scikit-learn from 1.3.0 to 1.4.2
    • scipy from 1.11.1 to 1.13.1
    • seaborn from 0.12.2 to 0.13.2
    • setuptools from 68.0.0 to 74.0.0
    • smmap from 5.0.1 to 5.0.0
    • sqlparse from 0.5.0 to 0.5.1
    • statsmodels from 0.14.0 to 0.14.2
    • tornado from 6.3.2 to 6.4.1
    • traitlets from 5.7.1 to 5.14.3
    • typing_extensions from 4.10.0 to 4.11.0
    • ujson from 5.4.0 to 5.10.0
    • virtualenv from 20.24.2 to 20.26.2
    • wheel from 0.38.4 to 0.43.0
    • zipp from 3.11.0 to 3.17.0
  • Upgraded R libraries:
    • arrow from 14.0.0.2 to 16.1.0
    • backports from 1.4.1 to 1.5.0
    • base from 4.3.2 to 4.4.0
    • bitops from 1.0-7 to 1.0-8
    • boot from 1.3-28 to 1.3-30
    • brio from 1.1.4 to 1.1.5
    • broom from 1.0.5 to 1.0.6
    • bslib from 0.6.1 to 0.8.0
    • cachem from 1.0.8 to 1.1.0
    • callr from 3.7.3 to 3.7.6
    • cli from 3.6.2 to 3.6.3
    • clock from 0.7.0 to 0.7.1
    • cluster from 2.1.4 to 2.1.6
    • codetools from 0.2-19 to 0.2-20
    • colorspace from 2.1-0 to 2.1-1
    • compiler from 4.3.2 to 4.4.0
    • crayon from 1.5.2 to 1.5.3
    • curl from 5.2.0 to 5.2.1
    • data.table from 1.15.0 to 1.15.4
    • datasets from 4.3.2 to 4.4.0
    • DBI from 1.2.1 to 1.2.3
    • dbplyr from 2.4.0 to 2.5.0
    • digest from 0.6.34 to 0.6.36
    • downlit from 0.4.3 to 0.4.4
    • evaluate from 0.23 to 0.24.0
    • farver from 2.1.1 to 2.1.2
    • fastmap from 1.1.1 to 1.2.0
    • foreign from 0.8-85 to 0.8-86
    • fs from 1.6.3 to 1.6.4
    • future from 1.33.1 to 1.34.0
    • future.apply from 1.11.1 to 1.11.2
    • gert from 2.0.1 to 2.1.0
    • ggplot2 from 3.4.4 to 3.5.1
    • gh from 1.4.0 to 1.4.1
    • globals from 0.16.2 to 0.16.3
    • graphics from 4.3.2 to 4.4.0
    • grDevices from 4.3.2 to 4.4.0
    • grid from 4.3.2 to 4.4.0
    • gt from 0.10.1 to 0.11.0
    • gtable from 0.3.4 to 0.3.5
    • hardhat from 1.3.1 to 1.4.0
    • highr from 0.10 to 0.11
    • htmltools from 0.5.7 to 0.5.8.1
    • httpuv from 1.6.14 to 1.6.15
    • httr2 from 1.0.0 to 1.0.2
    • ipred from 0.9-14 to 0.9-15
    • KernSmooth from 2.23-21 to 2.23-22
    • knitr from 1.45 to 1.48
    • lattice from 0.21-8 to 0.22-5
    • lava from 1.7.3 to 1.8.0
    • markdown from 1.12 to 1.13
    • MASS from 7.3-60 to 7.3-60.0.1
    • Matrix from 1.5-4.1 to 1.6-5
    • methods from 4.3.2 to 4.4.0
    • mgcv from 1.8-42 to 1.9-1
    • mlflow from 2.10.0 to 2.14.1
    • munsell from 0.5.0 to 0.5.1
    • nlme from 3.1-163 to 3.1-165
    • openssl from 2.1.1 to 2.2.0
    • parallel from 4.3.2 to 4.4.0
    • parallelly from 1.36.0 to 1.38.0
    • pkgbuild from 1.4.3 to 1.4.4
    • pkgdown from 2.0.7 to 2.1.0
    • pkgload from 1.3.4 to 1.4.0
    • processx from 3.8.3 to 3.8.4
    • prodlim from 2023.08.28 to 2024.06.25
    • promises from 1.2.1 to 1.3.0
    • ps from 1.7.6 to 1.7.7
    • ragg from 1.2.7 to 1.3.2
    • Rcpp from 1.0.12 to 1.0.13
    • RcppEigen from 0.3.3.9.4 to 0.3.4.0.0
    • reactR from 0.5.0 to 0.6.0
    • recipes from 1.0.9 to 1.1.0
    • remotes from 2.4.2.1 to 2.5.0
    • reprex from 2.1.0 to 2.1.1
    • rlang from 1.1.3 to 1.1.4
    • rmarkdown from 2.25 to 2.27
    • roxygen2 from 7.3.1 to 7.3.2
    • rpart from 4.1.21 to 4.1.23
    • RSQLite from 2.3.5 to 2.3.7
    • rstudioapi from 0.15.0 to 0.16.0
    • rvest from 1.0.3 to 1.0.4
    • sass from 0.4.8 to 0.4.9
    • shape from 1.4.6 to 1.4.6.1
    • shiny from 1.8.0 to 1.9.1
    • sparklyr from 1.8.4 to 1.8.6
    • spatial from 7.3-15 to 7.3-17
    • splines from 4.3.2 to 4.4.0
    • stats from 4.3.2 to 4.4.0
    • stats4 from 4.3.2 to 4.4.0
    • stringi from 1.8.3 to 1.8.4
    • survival from 3.5-5 to 3.6-4
    • swagger from 3.33.1 to 5.17.14.1
    • systemfonts from 1.0.5 to 1.1.0
    • tcltk from 4.3.2 to 4.4.0
    • testthat from 3.2.1 to 3.2.1.1
    • textshaping from 0.3.7 to 0.4.0
    • tidyselect from 1.2.0 to 1.2.1
    • tinytex from 0.49 to 0.52
    • tools from 4.3.2 to 4.4.0
    • usethis from 2.2.2 to 3.0.0
    • utils from 4.3.2 to 4.4.0
    • uuid from 1.2-0 to 1.2-1
    • V8 from 4.4.1 to 4.4.2
    • withr from 3.0.0 to 3.0.1
    • xfun from 0.41 to 0.46
    • xopen from 1.0.0 to 1.0.1
    • yaml from 2.3.8 to 2.3.10
  • Upgraded Java libraries:
    • com.amazonaws.aws-java-sdk-autoscaling from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudformation from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudfront from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudhsm from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudsearch from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudtrail from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudwatch from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudwatchmetrics from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-codedeploy from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cognitoidentity from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cognitosync from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-config from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-core from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-datapipeline from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-directconnect from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-directory from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-dynamodb from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-ec2 from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-ecs from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-efs from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-elasticache from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-elasticbeanstalk from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-elasticloadbalancing from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-elastictranscoder from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-emr from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-glacier from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-glue from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-iam from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-importexport from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-kinesis from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-kms from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-lambda from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-logs from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-machinelearning from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-opsworks from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-rds from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-redshift from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-route53 from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-s3 from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-ses from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-simpledb from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-simpleworkflow from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-sns from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-sqs from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-ssm from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-storagegateway from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-sts from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-support from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-workspaces from 1.12.610 to 1.12.638
    • com.amazonaws.jmespath-java from 1.12.610 to 1.12.638
    • com.google.protobuf.protobuf-java from 2.6.1 to 3.25.1
    • io.airlift.aircompressor from 0.25 to 0.27
    • io.delta.delta-sharing-client_2.12 from 1.1.2 to 1.2.0
    • io.netty.netty-all from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-buffer from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-codec from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-codec-http from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-codec-http2 from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-codec-socks from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-common from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-handler from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-handler-proxy from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-resolver from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-transport from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-transport-classes-epoll from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-transport-classes-kqueue from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-transport-native-epoll from 4.1.96.Final-linux-x86_64 to 4.1.108.Final-linux-x86_64
    • io.netty.netty-transport-native-kqueue from 4.1.96.Final-osx-x86_64 to 4.1.108.Final-osx-x86_64
    • io.netty.netty-transport-native-unix-common from 4.1.96.Final to 4.1.108.Final
    • org.apache.ivy.ivy from 2.5.1 to 2.5.2
    • org.apache.zookeeper.zookeeper from 3.6.3 to 3.9.2
    • org.apache.zookeeper.zookeeper-jute from 3.6.3 to 3.9.2
    • org.rocksdb.rocksdbjni from 8.11.4 to 9.2.1
    • org.scalactic.scalactic_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-compatible from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-core_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-diagrams_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-featurespec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-flatspec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-freespec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-funspec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-funsuite_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-matchers-core_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-mustmatchers_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-propspec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-refspec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-shouldmatchers_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-wordspec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest_2.12 from 3.2.15 to 3.2.16

Apache Spark

Databricks Runtime 16.0 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 15.4 LTS, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-49898] [DBRRM-1282][SC-178410] Fix documentation and default for event log task metrics accumulator logging flag from SPARK-42204
  • [SPARK-49743] [ES-1260022][BEHAVE-157][SC-177475][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields
  • [SPARK-49816] [SC-177896][SQL] Should only update out-going-ref-count for referenced outer CTE relation
  • [SPARK-48939] [SC-177022][SC-172766][AVRO] Support reading Avro with recursive schema reference
  • [SPARK-49688] [SC-177468][ES-1242349][CONNECT] Fix a data race between interrupt and execute plan
  • [SPARK-49771] [SC-177466][PYTHON] Improve Pandas Scalar Iter UDF error when output rows exceed input rows
  • [SPARK-48866] [SC-170772][SQL] Fix hints of valid charset in the error message of INVALID_PARAMETER_VALUE.CHARSET
  • [SPARK-48195] [FIXFORWARD][SC-177267][CORE] Save and reuse RDD/Broadcast created by SparkPlan
  • [SPARK-49585] [CONNECT] Replace executions map in SessionHolder with operationID set
  • [SPARK-49211] [SC-174257][SQL] V2 Catalog can also support built-in data sources
  • [SPARK-49684] Minimise the lifetime of the session restoration lock
  • [SPARK-48059] [SPARK-48145][SPARK-48134][SPARK-48182][SPARK-48209][SPARK-48291] Structured log framework on the java side
  • [SPARK-48857] [SC-170661][SQL] Restrict charsets in CSVOptions
  • [SPARK-49152] [SC-173690][SQL] V2SessionCatalog should use V2Command
  • [SPARK-42846] [SC-176588][SQL] Remove error condition _LEGACY_ERROR_TEMP_2011
  • [SPARK-48195] [SC-177267][CORE] Save and reuse RDD/Broadcast created by SparkPlan
  • [SPARK-49630] [SC-177379][SS] Add flatten option to process collection types with state data source reader
  • [SPARK-49699] [SC-177154][SS] Disable PruneFilters for streaming workloads
  • [SPARK-48781] [SC-175282][SQL] Add Catalog APIs for loading stored procedures
  • [SPARK-49667] [SC-177068][SQL] Disallow CS_AI collators with expressions that use StringSearch
  • [SPARK-49737] [SC-177207][SQL] Disable bucketing on collated columns in complex types
  • [SPARK-48712] [SC-169794][SQL] Perf Improvement for encode with empty values or UTF-8 charset
  • [SPARK-49038] [SC-173933][SQL] SQLMetric should report the raw value in the accumulator update event
  • [SPARK-48541] [SC-169066][CORE] Add a new exit code for executors killed by TaskReaper
  • [SPARK-48774] [SC-170138][SQL] Use SparkSession in SQLImplicits
  • [SPARK-49719] [SC-177139][SQL] Make UUID and SHUFFLE accept integer seed
  • [SPARK-49713] [SC-177135][PYTHON][CONNECT] Make function count_min_sketch accept number arguments
  • [SPARK-47601] [SC-162499][GRAPHX] Graphx: Migrate logs with variables to structured logging framework
  • [SPARK-49738] [SC-177219][SQL] Endswith bug fix
  • [SPARK-48623] [SC-170822][CORE] Structured logging migrations [Part 3]
  • [SPARK-49677] [SC-177148][SS] Ensure that changelog files are written on commit and forceSnapshot flag is also reset
  • [SPARK-49684] [SC-177040][CONNECT] Remove global locks from session and execution managers
  • [SPARK-48302] [SC-168814][PYTHON] Preserve nulls in map columns in PyArrow Tables
  • [SPARK-48601] [SC-169025][SQL] Give a more user friendly error message when setting a null value for JDBC Option
  • [SPARK-48635] [SC-169263][SQL] Assign classes to join type errors and as-of join error
  • [SPARK-49673] [SC-177032][CONNECT] Increase CONNECT_GRPC_ARROW_MAX_BATCH_SIZE to 0.7 * CONNECT_GRPC_MAX_MESSAGE_SIZE
  • [SPARK-49693] [SC-177071][PYTHON][CONNECT] Refine the string representation of timedelta
  • [SPARK-49687] [SC-176901][SQL] Delay sorting in validateAndMaybeEvolveStateSchema
  • [SPARK-49718] [SC-177112][PS] Switch Scatter plot to sampled data
  • [SPARK-48472] [SC-169044][SQL] Enable reflect expressions with collated strings
  • [SPARK-48484] [SC-167484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts
  • [SPARK-48341] [SC-166560][CONNECT] Allow plugins to use QueryTest in their tests
  • [SPARK-42252] [SC-168723][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer
  • [SPARK-48314] [SC-166565][SS] Don’t double cache files for FileStreamSource using Trigger.AvailableNow
  • [SPARK-49567] [SC-176241][PYTHON] Use classic instead of vanilla from PySpark code base
  • [SPARK-48374] [SC-167596][PYTHON] Support additional PyArrow Table column types
  • [SPARK-48300] [SC-166481][SQL] Codegen Support for from_xml
  • [SPARK-49412] [SC-177059][PS] Compute all box plot metrics in single job
  • [SPARK-49692] [SC-177031][PYTHON][CONNECT] Refine the string representation of literal date and datetime
  • [SPARK-49392] [ES-1130351][SC-176705][SQL] Catch errors when failing to write to external data source
  • [SPARK-48306] [SC-166241][SQL] Improve UDT in error message
  • [SPARK-44924] [SC-166379][SS] Add config for FileStreamSource cached files
  • [SPARK-48176] [SC-165644][SQL] Adjust name of FIELD_ALREADY_EXISTS error condition
  • [SPARK-49691] [SC-176988][PYTHON][CONNECT] Function substring should accept column names
  • [SPARK-49502] [SC-176077][CORE] Avoid NPE in SparkEnv.get.shuffleManager.unregisterShuffle
  • [SPARK-49244] [SC-176703][SQL] Further exception improvements for parser/interpreter
  • [SPARK-48355] [SC-176684][SQL] Support for CASE statement
  • [SPARK-49355] [SC-175121][SQL] levenshtein should check whether the collation values of all parameter types are the same
  • [SPARK-49640] [SC-176953][PS] Apply reservoir sampling in SampledPlotBase
  • [SPARK-49678] [SC-176857][CORE] Support spark.test.master in SparkSubmitArguments
  • [SPARK-49680] [SC-176856][PYTHON] Limit Sphinx build parallelism to 4 by default
  • [SPARK-49396] Revert “[SC-176030][SQL] Modify nullability check for CaseWhen expression”
  • [SPARK-48419] [SC-167443][SQL] Foldable propagation replace foldable column shoul…
  • [SPARK-49556] [SC-176757][SQL] Add SQL pipe syntax for the SELECT operator
  • [SPARK-49438] [SC-175237][SQL] Fix the pretty name of the FromAvro & ToAvro expression
  • [SPARK-49659] [SC-1229924][SQL] Add a nice user-facing error for scalar subqueries inside VALUES clause
  • [SPARK-49646] [SC-176778][SQL] fix subquery decorrelation for union/set operations when parentOuterReferences has references not covered in collectedChildOuterReferences
  • [SPARK-49354] [SC-175034][SQL] split_part should check whether the collation values of all parameter types are the same
  • [SPARK-49478] [SC-175914][CONNECT] Handle null metrics in ConnectProgressExecutionListener
  • [SPARK-48358] [SC-176374][SQL] Support for REPEAT statement
  • [SPARK-49183] [SC-173680][SQL] V2SessionCatalog.createTable should respect PROP_IS_MANAGED_LOCATION
  • [SPARK-49611] [SC-176791][SQL] Introduce TVF collations() & remove the SHOW COLLATIONS command
  • [SPARK-49261] [SC-176589][SQL] Don’t replace literals in aggregate expressions with group-by expressions
  • [SPARK-49099] [SC-173229][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog
  • [SPARK-49594] [SC-176569][SS] Adding check on whether columnFamilies were added or removed to write StateSchemaV3 file
  • [SPARK-49578] [SC-176385][SQL] Remove the ANSI config suggestion in CAST_INVALID_INPUT and CAST_OVERFLOW
  • [SPARK-48882] [SC-174256][SS] Assign names to streaming output mode related error classes
  • [SPARK-49155] [SC-176506][SQL][SS] Use more appropriate parameter type to construct GenericArrayData
  • [SPARK-49519] [SC-176388][SQL] Merge options of table and relation when constructing FileScanBuilder
  • [SPARK-49591] [SC-176587][SQL] Add Logical Type column to variant readme
  • [SPARK-49596] [SC-176423][SQL] Improve performance of FormatString
  • [SPARK-49525] [SC-176044][SS][CONNECT] Minor log improvement to Server Side Streaming Query ListenerBus Listener
  • [SPARK-49583] [SC-176272][SQL] Define the error sub-condition SECONDS_FRACTION for invalid seconds fraction pattern
  • [SPARK-49536] [SC-176242] Handle error in python streaming data source record prefetching
  • [SPARK-49443] [SC-176273][SQL][PYTHON] Implement to_variant_object expression and make schema_of_variant expressions print OBJECT for for Variant Objects
  • [SPARK-49544] [SASP-3990][SC-176557][CONNECT] Replace coarse-locking in SparkConnectExecutionManager with ConcurrentMap
  • [SPARK-49548] [SASP-3990][SC-176556][CONNECT] Replace coarse-locking in SparkConnectSessionManager with ConcurrentMap
  • [SPARK-49551] [SC-176218][SS] Improve RocksDB log for replayChangelog
  • [SPARK-49595] [SC-176396][CONNECT][SQL] Fix DataFrame.unpivot/melt in Spark Connect Scala Client
  • [SPARK-49006] [SC-176162] Implement purging for OperatorStateMetadataV2 and StateSchemaV3 files
  • [SPARK-49600] [SC-176426][PYTHON] Remove Python 3.6 and older-related logic from try_simplify_traceback
  • [SPARK-49303] [SC-176013][SS] Implement TTL for ValueState in transformWithStateInPandas API
  • [SPARK-49191] [SC-176243][SS] Add support for reading transformWithState map state variables with state data source reader
  • [SPARK-49593] [SC-176371][SS] Throw RocksDB exception to the caller on DB close if an error is seen
  • [SPARK-49334] [SC-174803][SQL] str_to_map should check whether the collation values of all parameter types are the same
  • [SPARK-42204] [SC-176126][CORE] Add option to disable redundant logging of TaskMetrics internal accumulators in event logs
  • [SPARK-49575] [SC-176256][SS] Add logging for lock release only if acquiredThreadInfo is not null
  • [SPARK-49539] [SC-176250][SS] Update internal col families start identifier to a different one
  • [SPARK-49205] [SC-173853][SQL] KeyGroupedPartitioning should inherit HashPartitioningLike
  • [SPARK-49396] [SC-176030][SQL] Modify nullability check for CaseWhen expression
  • [SPARK-49476] [SC-175700][SQL] Fix nullability of base64 function
  • [SPARK-47262] [SC-174829][SQL] Assign names to error conditions for parquet conversions
  • [SPARK-47247] [SC-158349][SQL] Use smaller target size when coalescing partitions with exploding joins
  • [SPARK-49501] [SC-176166][SQL] Fix double-escaping of table location
  • [SPARK-49083] [SC-173214][CONNECT] Allow from_xml and from_json to natively work with json schemas
  • [SPARK-49043] [SC-174673][SQL] Fix interpreted codepath group by on map containing collated strings
  • [SPARK-48986] [SC-172265][CONNECT][SQL] Add ColumnNode Intermediate Representation
  • [SPARK-49326] [SC-176016][SS] Classify Error class for Foreach sink user function error
  • [SPARK-48348] [SC-175950][SPARK-48376][SQL] Introduce LEAVE and ITERATE statements
  • [SPARK-49523] [SC-175949][CONNECT] Increase maximum wait time for connect server to come up for testing
  • [SPARK-49000] [BEHAVE-105][ES-1194747][SQL] Fix “select count(distinct 1) from t” where t is empty table by expanding RewriteDistinctAggregates - DBR version 16.x
  • [SPARK-49311] [SC-175038][SQL] Make it possible for large ‘interval second’ values to be cast to decimal
  • [SPARK-49200] [SC-173699][SQL] Fix null type non-codegen ordering exception
  • [SPARK-49467] [SC-176051][SS] Add support for state data source reader and list state
  • [SPARK-47307] [SC-170891][SQL] Add a config to optionally chunk base64 strings
  • [SPARK-49391] [SC-176032][PS] Box plot select outliers by distance from fences
  • [SPARK-49445] [SC-175845][UI] Support show tooltip in the progress bar of UI
  • [SPARK-49451] [SC-175702] Allow duplicate keys in parse_json.
  • [SPARK-49275] [SC-175701][SQL] Fix return type nullness of the xpath expression
  • [SPARK-49021] [SC-175578][SS] Add support for reading transformWithState value state variables with state data source reader
  • [SPARK-49474] [BEHAVE-143][SC-169253][SC-175933][SS] Classify Error class for FlatMapGroupsWithState user function error
  • [SPARK-49408] [SC-175932][SQL] Use IndexedSeq in ProjectingInternalRow
  • [SPARK-49509] [SC-175853][CORE] Use Platform.allocateDirectBuffer instead of ByteBuffer.allocateDirect
  • [SPARK-49382] [SC-175013][PS] Make frame box plot properly render the fliers/outliers
  • [SPARK-49002] [SC-172846][SQL] Consistently handle invalid locations in WAREHOUSE/SCHEMA/TABLE/PARTITION/DIRECTORY
  • [SPARK-49480] [SC-175699][CORE] Fix NullPointerException from SparkThrowableHelper.isInternalError
  • [SPARK-49477] [SC-175828][PYTHON] Improve pandas udf invalid return type error message
  • [SPARK-48693] [SC-169492][SQL] Simplify and unify toString of Invoke and StaticInvoke
  • [SPARK-49441] [SC-175716][ML] StringIndexer sort arrays in executors
  • [SPARK-49347] [SC-175004][R] Deprecate SparkR
  • [SPARK-49357] [SC-175227][CONNECT][PYTHON] Vertically truncate deeply nested protobuf message
  • [SPARK-41982] [SC-120604][SQL] Partitions of type string should not be treated as numeric types
  • [SPARK-48776] [SC-170452][BEHAVE-72] Fix timestamp formatting for json, xml and csv
  • [SPARK-49223] [SC-174800][ML] Simplify the StringIndexer.countByValue with builtin functions
  • [SPARK-49016] Revert “[SC-174663][SQL] Restore the behavior thatqueries from raw CSV files are disallowed when only include corrupt record column and assign name to _LEGACY_ERROR_TEMP_1285”
  • [SPARK-49041] [SC-172392][PYTHON][CONNECT] Raise proper error for dropDuplicates when wrong subset is given
  • [SPARK-49050] [SC-175235] Enabling deleteIfExists operator in TWS with Virtual Column Families
  • [SPARK-49216] [SC-173919][CORE]Fix to not log message context with explicitly LogEntry constructed when Structured Logging conf is off
  • [SPARK-49252] [SC-175596][CORE] MakeTaskSetExcludeList and HeathTracker independent
  • [SPARK-49352] [SC-174971][SQL] Avoid redundant array transform for identical expression
  • [SPARK-42307] [SC-173863][SQL] Assign name for error _LEGACY_ERROR_TEMP_2232
  • [SPARK-49197] [SC-173732][CORE] Redact Spark Command output in launcher module
  • [SPARK-48913] [SC-173934][SQL] Implement IndentingXMLStreamWriter
  • [SPARK-49306] [SC-175363][PYTHON][SQL] Create SQL function aliases for ‘zeroifnull’ and ‘nullifzero’
  • [SPARK-48344] [SQL] SQL Scripting execution (including Spark Connect)
  • [SPARK-49402] [SC-175122][PYTHON] Fix Binder integration in PySpark documentation
  • [SPARK-49017] [SC-174664][SQL] Insert statement fails when multiple parameters are being used
  • [SPARK-49318] [SC-174733][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience
  • [SPARK-49016] [SC-174663][SQL] Restore the behavior that queries from raw CSV files are disallowed when only include corrupt record column and assign name to _LEGACY_ERROR_TEMP_1285
  • [SPARK-49387] [SC-175124][PYTHON] Fix type hint for accuracy in percentile_approx and approx_percentile
  • [SPARK-49131] [SC-174666][SS] TransformWithState should properly set implicit grouping keys even with lazy iterators
  • [SPARK-49301] [SC-174795][SS] Chunk arrow data passed to Python worker
  • [SPARK-49039] [SC-174651][UI] Reset checkbox when executor metrics are loaded in the Stages tab
  • [SPARK-48428] [SC-169806][SQL]: Fix IllegalStateException in NestedColumnAliasing
  • [SPARK-49353] [SC-174830][SQL] Update docs related to UTF-32 encoding/decoding
  • [SPARK-48613] [SC-170966][SQL] SPJ: Support auto-shuffle one side + less join keys than partition keys
  • [SPARK-47473] [SC-160450][BEHAVE-127][SQL] Fix correctness issue of converting postgres INFINITY timestamps
  • [SPARK-49142] [SC-173658][CONNECT][PYTHON] Follow up to revert proto to string performance cost
  • [SPARK-49300] [SC-175008][CORE] Fix Hadoop delegation token leak when tokenRenewalInterval is not set.
  • [SPARK-49367] [SC-175012][PS] Parallelize the KDE computation for multiple columns (plotly backend)
  • [SPARK-49365] [SC-175011][PS] Simplify the bucket aggregation in hist plot
  • [SPARK-49372] [SC-175003][SS] Ensure that latestSnapshot is set to none on close to avoid subsequent use
  • [SPARK-49341] [SC-174785] Remove connector/docker in favor of Apache Spark Operator
  • [SPARK-49344] [SC-174894][PS] Support json_normalize for Pandas API on Spark
  • [SPARK-49306] [SC-174794][SQL] Create new SQL functions ‘zeroifnull’ and ‘nullifzero’
  • [SPARK-48796] [SC-174668][SS] Load Column Family Id from RocksDBCheckpointMetadata for VCF when restarting
  • [SPARK-49342] [SC-174899][SQL] Make TO_AVRO SQL function ‘jsonFormatSchema’ argument optional
  • [SPARK-48628] [SC-174695][CORE] Add task peak on/off heap memory metrics
  • [SPARK-47407] [SC-159379][BEHAVE-126][SQL] Support java.sql.Types.NULL map to NullType
  • [SPARK-48628] [SC-173407][CORE] Add task peak on/off heap memory metrics
  • [SPARK-49166] [SC-173987][SQL] Support OFFSET in correlated subquery
  • [SPARK-49269] [SC-174676][SQL] Eagerly evaluate VALUES() list in AstBuilder
  • [SPARK-49281] [SC-174782][SQL] Optimze parquet binary getBytes with getBytesUnsafe to avoid copy cost
  • [SPARK-49113] [SC-174734] Dont assert on translation bugs - silently swallow the exception
  • [SPARK-49098] [SC-173253][SQL] Add write options for INSERT
  • [SPARK-48638] [SC-174694][FOLLOW][CONNECT] Fix documentation for ExecutionInfo
  • [SPARK-49250] [ES-1222826][SQL] Improve error message for nested UnresolvedWindowExpression in CheckAnalysis
  • [SPARK-48755] [SC-174258][SS][PYTHON] transformWithState pyspark base implementation and ValueState support
  • [SPARK-48966] [SC-174329][SQL] Improve error message with invalid unresolved column reference in UDTF call
  • [SPARK-46590] [SC-154115][SQL] Fix coalesce failed with unexpected partition indeces
  • [SPARK-49235] [SC-174159][SQL] Refactor ResolveInlineTables rule so it doesn’t traverse the whole tree
  • [SPARK-49060] [SC-173107][CONNECT] Clean up Mima rules for SQL-Connect binary compatibility checks
  • [SPARK-48762] [SC-172525][SQL] Introduce clusterBy DataFrameWriter API for Python
  • [SPARK-49207] [SC-173852][SQL] Fix one-to-many case mapping in SplitPart and StringSplitSQL
  • [SPARK-49204] [SC-173850][SQL] Fix surrogate pair handling in StringInstr and StringLocate
  • [SPARK-36680] [SC-170640][SQL] Supports Dynamic Table Options for Spark SQL
  • [SPARK-49204] [SC-173849][SQL] Fix surrogate pair handling in SubstringIndex
  • [SPARK-49204] [SC-173848][SQL] Fix surrogate pair handling in StringTrim
  • [SPARK-48967] [SC-173993]Fix forward SparkConfigOwnershipSuite test for OPTIMIZE_INSERT_INTO_VALUES_PARSER
  • [SPARK-49204] [SC-173851][SQL] Fix surrogate pair handling in StringReplace
  • [SPARK-48967] [SC-173993][SQL][16.x] Improve performance and memory footprint of “INSERT INTO … VALUES” Statements
  • [SPARK-49099] Revert “[SC-173229][SQL] CatalogManager.setCurrent…
  • [SPARK-48347] [SC-173812][SQL] Support for WHILE statement
  • [SPARK-49128] [SC-173344][CORE] Support custom History Server UI title
  • [SPARK-49146] [SC-173825][SS] Move assertion errors related to watermark missing in append mode streaming queries to error framework
  • [SPARK-45787] [SC-172197][SQL] Support Catalog.listColumns for clustering columns
  • [SPARK-49099] [SC-173229][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog
  • [SPARK-49138] [SC-173483][SQL] Fix CollationTypeCasts of several expressions
  • [SPARK-49163] [SC-173666][SQL] Attempt to create table based on broken parquet partition data results should return user-facing error
  • [SPARK-49201] [SC-173793][PS][PYTHON][CONNECT] Reimplement hist plot with Spark SQL
  • [SPARK-49188] [SC-173682][SQL] Internal error on concat_ws called on array of arrays of string
  • [SPARK-49137] [SC-173677][SQL] When the Boolean condition in the if statement is invalid, an exception should be thrown
  • [SPARK-49193] [SC-173672][SQL] Improve the performance of RowSetUtils.toColumnBasedSet
  • [SPARK-46632] [SC-173697][SQL] Fix subexpression elimination when equivalent ternary expressions have different children
  • [SPARK-49078] [SC-173078][SQL] Support show columns syntax in v2 table
  • [SPARK-49141] [SC-173388][SQL] Mark variant as hive incompatible data type
  • [SPARK-49059] [Cherry-Pick][15.x][SC-172528][CONNECT] Move SessionHolder.forTesting(...) to the test package
  • [SPARK-49111] [SC-173661][SQL] Move withProjectAndFilter to the companion object of DataSourceV2Strategy
  • [SPARK-49185] [SC-173688][PS][PYTHON][CONNECT] Reimplement kde plot with Spark SQL
  • [SPARK-49178] [SC-173673][SQL] Optimize performance of Row#getSeq to match the performance when using Spark 3.5 with Scala 2.12
  • [SPARK-49093] [SC-172958][SQL] GROUP BY with MapType nested inside complex type
  • [SPARK-49142] [SC-173469][CONNECT][PYTHON] Lower Spark Connect client log level to debug
  • [SPARK-48761] [SC-172048][SQL] Introduce clusterBy DataFrameWriter API for Scala
  • [SPARK-48346] [SC-173083][SQL] Support for IF ELSE statements in SQL scripts
  • [SPARK-48338] [SC-173112][SQL] Improve exceptions thrown from parser/interpreter
  • [SPARK-48658] [SC-169474][SQL] Encode/Decode functions report coding errors instead of mojibake for unmappable characters
  • [SPARK-49071] [SC-172954][SQL] Remove ArraySortLike trait
  • [SPARK-49107] Revert “Revert “[SC-173103][SQL] ROUTINE_ALREADY_EXISTS supports RoutineType””
  • [SPARK-49070] [SC-172907][SS][SQL] TransformWithStateExec.initialState is rewritten incorrectly to produce invalid query plan
  • [SPARK-49114] [SC-173217] Sub-categorize cannot load state store errors
  • [SPARK-49107] Revert “[SC-173103][SQL] ROUTINE_ALREADY_EXISTS supports RoutineType”
  • [SPARK-49048] [SC-173223][SS] Add support for reading relevant operator metadata at given batch id
  • [SPARK-49094] [SC-173049][SQL] Fix ignoreCorruptFiles non-functioning for hive orc impl with mergeSchema off
  • [SPARK-49108] [SC-173102][EXAMPLE] Add submit_pi.sh REST API example
  • [SPARK-49107] [SC-173103][SQL] ROUTINE_ALREADY_EXISTS supports RoutineType
  • [SPARK-48997] [SC-172484][SS] Implement individual unloads for maintenance thread pool thread failures
  • [SPARK-49063] [SC-173094][SQL] Fix Between with ScalarSubqueries
  • [SPARK-45891] [SC-172305][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec
  • [SPARK-49074] [BEHAVE-110][SC-172815][SQL] Fix variant with df.cache()
  • [SPARK-49003] [SC-172613][SQL] Fix interpreted code path hashing to be collation aware
  • [SPARK-48740] [SC-172430][SQL] Catch missing window specification error early
  • [SPARK-48999] [SC-172245][SS] Divide PythonStreamingDataSourceSimpleSuite
  • [SPARK-49031] [SC-172602] Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2
  • [SPARK-49053] [SC-172494][PYTHON][ML] Make model save/load helper functions accept spark session
  • [SPARK-49032] [Backport][15.x][SS] Add schema path in metadata table entry, verify expected version and add operator metadata related test for operator metadata format v2
  • [SPARK-49034] [SC-172306][CORE] Support server-side sparkProperties replacement in REST Submission API
  • [SPARK-48931] [SC-171895][SS] Reduce Cloud Store List API cost for state store maintenance task
  • [SPARK-48849] [SC-172068][SS]Create OperatorStateMetadataV2 for the TransformWithStateExec operator
  • [SPARK-49013] [SC-172322] Change key in collationsMap for Map and Array types in scala
  • [SPARK-48414] [SC-171884][PYTHON] Fix breaking change in python’s fromJson
  • [SPARK-48910] [SC-171001][SQL] Use HashSet/HashMap to avoid linear searches in PreprocessTableCreation
  • [SPARK-49007] [SC-172204][CORE] Improve MasterPage to support custom title
  • [SPARK-49009] [SC-172263][SQL][PYTHON] Make Column APIs and functions accept Enums
  • [SPARK-49033] [SC-172303][CORE] Support server-side environmentVariables replacement in REST Submission API
  • [SPARK-48363] [SC-166470][SQL] Cleanup some redundant codes in from_xml
  • [SPARK-46743] [SC-170867][SQL][BEHAVE-84] Count bug after ScalarSubqery is folded if it has an empty relation
  • [SPARK-49040] [SC-172351][SQL] Fix doc sql-ref-syntax-aux-exec-imm.md
  • [SPARK-48998] [SC-172212][ML] Meta algorithms save/load model with SparkSession
  • [SPARK-48959] [SC-171708][SQL] Make NoSuchNamespaceException extend NoSuchDatabaseException to restore the exception handling
  • [SPARK-48996] [SC-172130][SQL][PYTHON] Allow bare literals for and and or of Column
  • [SPARK-48990] [SC-171936] Follow up for #101759 - test fix
  • [SPARK-48338] [SC-171912][SQL] Check variable declarations
  • [SPARK-48990] [SC-171936][SQL] Unified variable related SQL syntax keywords
  • [SPARK-48988] [SC-171915][ML] Make DefaultParamsReader/Writer handle metadata with spark session
  • [SPARK-48974] [SC-171978][SQL][SS][ML][MLLIB] Use SparkSession.implicits instead of SQLContext.implicits
  • [SPARK-48760] [SC-170870][SQL] Fix CatalogV2Util.applyClusterByChanges
  • [SPARK-48928] [SC-171956] Log Warning for Calling .unpersist() on Locally Checkpointed RDDs
  • [SPARK-48760] [SC-170139][SQL] Introduce ALTER TABLE … CLUSTER BY SQL syntax to change clustering columns
  • [SPARK-48844] Revert “[SC-170669][SQL] USE INVALID_EMPTY_LOCATION instead of UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY when path is empty”
  • [SPARK-48833] [SC-171914][SQL][VARIANT] Support variant in InMemoryTableScan
  • [SPARK-48975] [SC-171894][PROTOBUF] Remove unnecessary ScalaReflectionLock definition from protobuf
  • [SPARK-48970] [SC-171800][PYTHON][ML] Avoid using SparkSession.getActiveSession in spark ML reader/writer
  • [SPARK-48844] [SC-170669][SQL] USE INVALID_EMPTY_LOCATION instead of UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY when path is empty
  • [SPARK-48714] [SC-170136] Fix failing df.mergeInto tests in PySpark and UC
  • [SPARK-48957] [SC-171797][SS] Return sub-classified error class on state store load for hdfs and rocksdb provider
  • [SPARK-48891] [Backport][15x][SC-171677][SS] Refactor StateSchemaCompatibilityChecker to unify all state schema formats
  • [SPARK-48972] [SC-171795][PYTHON] Unify the literal string handling in functions
  • [SPARK-48388] [SC-171337][SQL] Fix SET statement behavior for SQL Scripts
  • [SPARK-48743] [SC-170552][SQL][SS] MergingSessionIterator should better handle when getStruct returns null
  • [SPARK-48623] [15.x][SC-171322][CORE] Migrate FileAppender logs to structured logging
  • [SPARK-36680] [DBRRM-1123] Revert “[SC-170640][SQL] Supports Dynamic Table Options for Spark SQL”
  • [SPARK-48841] [SC-170868][BEHAVE-83][SQL] Include collationName to sql() of Collate
  • [SPARK-48941] [SC-171703][PYTHON][ML] Replace RDD read / write API invocation with Dataframe read / write API
  • [SPARK-48938] [SC-171577][PYTHON] Improve error messages when registering Python UDTFs
  • [SPARK-48350] [SC-171040][SQL] Introduction of Custom Exceptions for Sql Scripting
  • [SPARK-48907] [SC-171158][SQL] Fix the value explicitTypes in COLLATION_MISMATCH.EXPLICIT
  • [SPARK-48945] [SC-171658][PYTHON] Simplify regex functions with lit
  • [SPARK-48944] [SC-171576][CONNECT] Unify the JSON-format schema handling in Connect Server
  • [SPARK-48836] [SC-171569] Integrate SQL schema with state schema/metadata
  • [SPARK-48946] [SC-171504][SQL] NPE in redact method when session is null
  • [SPARK-48921] [SC-171412][SQL] ScalaUDF encoders in subquery should be resolved for MergeInto
  • [SPARK-45155] [SC-171048][CONNECT] Add API Docs for Spark Connect JVM/Scala Client
  • [SPARK-48900] [SC-171319] Add reason field for cancelJobGroup and cancelJobsWithTag
  • [SPARK-48865] [SC-171154][SQL] Add try_url_decode function
  • [SPARK-48851] [SC-170767][SQL] Change the value of SCHEMA_NOT_FOUND from namespace to catalog.namespace
  • [SPARK-48510] [SC-170893][2/2] Support UDAF toColumn API in Spark Connect
  • [SPARK-45190] [SC-171055][SPARK-48897][PYTHON][CONNECT] Make from_xml support StructType schema
  • [SPARK-48930] [SC-171304][CORE] Redact awsAccessKeyId by including accesskey pattern
  • [SPARK-48909] [SC-171080][ML][MLLIB] Uses SparkSession over SparkContext when writing metadata
  • [SPARK-48883] [SC-171133][ML][R] Replace RDD read / write API invocation with Dataframe read / write API
  • [SPARK-48924] [SC-171313][PS] Add a pandas-like make_interval helper function
  • [SPARK-48884] [SC-171051][PYTHON] Remove unused helper function PythonSQLUtils.makeInterval
  • [SPARK-48817] [SC-170636][SQL] Eagerly execute union multi commands together
  • [SPARK-48896] [SC-171079][ML][MLLIB] Avoid repartition when writing out the metadata
  • [SPARK-48892] [SC-171127][ML] Avoid per-row param read in Tokenizer
  • [SPARK-48927] [SC-171227][CORE] Show the number of cached RDDs in StoragePage
  • [SPARK-48886] [15.x][Backport][SC-171039][SS] Add version info to changelog v2 to allow for easier evolution
  • [SPARK-48903] [SC-171136][SS] Set the RocksDB last snapshot version correctly on remote load
  • [SPARK-48742] [SC-170538][SS] Virtual Column Family for RocksDB
  • [SPARK-48726] [15.x][SC-170753][SS] Create the StateSchemaV3 file format, and write this out for the TransformWithStateExec operator
  • [SPARK-48794] [SC-170882][CONNECT][15.x] df.mergeInto support for Spark Connect (Scala and Python)
  • [SPARK-48714] [SC-170136][PYTHON] Implement DataFrame.mergeInto in PySpark
  • [SPARK-48772] [SC-170642][SS][SQL] State Data Source Change Feed Reader Mode
  • [SPARK-48666] [SC-170887][SQL] Do not push down filter if it contains PythonUDFs
  • [SPARK-48845] [SC-170889][SQL] GenericUDF catch exceptions from children
  • [SPARK-48880] [SC-170974][CORE] Avoid throw NullPointerException if driver plugin fails to initialize
  • [SPARK-48888] [Backport][15x][SC-170973][SS] Remove snapshot creation based on changelog ops size
  • [SPARK-48871] [SC-170876] Fix INVALID_NON_DETERMINISTIC_EXPRESSIONS validation in…
  • [SPARK-48883] [SC-170894][ML][R] Replace RDD read / write API invocation with Dataframe read / write API
  • [SPARK-36680] [SC-170640][SQL] Supports Dynamic Table Options for Spark SQL
  • [SPARK-48804] [SC-170558][SQL] Add classIsLoadable & OutputCommitter.isAssignableFrom check for output committer class configrations
  • [SPARK-46738] [SC-170791][PYTHON] Reenable a group of doctests
  • [SPARK-48858] [SC-170756][PYTHON] Remove deprecated setDaemon method call of Thread in log_communication.py
  • [SPARK-48639] [SC-169801][CONNECT][PYTHON] Add Origin to RelationCommon
  • [SPARK-48863] [SC-170770][ES-1133940][SQL] Fix ClassCastException when parsing JSON with “spark.sql.json.enablePartialResults” enabled
  • [SPARK-48343] [SC-170450][SQL] Introduction of SQL Scripting interpreter
  • [SPARK-48529] [SC-170755][SQL] Introduction of Labels in SQL Scripting
  • [SPARK-45292] Revert “[SC-151609][SQL][HIVE] Remove Guava from shared classes from IsolatedClientLoader”
  • [SPARK-48037] [SC-165330][CORE][3.5] Fix SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
  • [SPARK-48720] [SC-170551][SQL] Align the command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2
  • [SPARK-48485] [SC-167825][CONNECT][SS] Support interruptTag and interruptAll in streaming queries
  • [SPARK-45292] [SC-151609][SQL][HIVE] Remove Guava from shared classes from IsolatedClientLoader
  • [SPARK-48668] [SC-169815][SQL] Support ALTER NAMESPACE … UNSET PROPERTIES in v2
  • [SPARK-47914] [SC-165313][SQL] Do not display the splits parameter in Range
  • [SPARK-48807] [SC-170643][SQL] Binary Support for CSV datasource
  • [SPARK-48220] [SC-167592][PYTHON][15.X] Allow passing PyArrow Table to createDataFrame()
  • [SPARK-48545] [SC-169543][SQL] Create to_avro and from_avro SQL functions to match DataFrame equivalents
  • [SPARK-47577] [SC-168875][SPARK-47579] Correct misleading usage of log key TASK_ID

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

  • Operating System: Ubuntu 24.04.1 LTS
  • Java: Zulu17.50+19-CA
  • Scala: 2.12.18
  • Python: 3.12.3
  • R: 4.4.0
  • Delta Lake: 3.2.1

Installed Python libraries

Library Version Library Version Library Version
annotated-types 0.7.0 asttokens 2.0.5 astunparse 1.6.3
autocommand 2.2.2 azure-core 1.31.0 azure-storage-blob 12.23.0
azure-storage-file-datalake 12.17.0 backports.tarfile 1.2.0 black 24.4.2
blinker 1.7.0 boto3 1.34.69 botocore 1.34.69
cachetools 5.3.3 certifi 2024.6.2 cffi 1.16.0
chardet 4.0.0 charset-normalizer 2.0.4 click 8.1.7
cloudpickle 2.2.1 comm 0.2.1 contourpy 1.2.0
cryptography 42.0.5 cycler 0.11.0 Cython 3.0.11
databricks-sdk 0.30.0 dbus-python 1.3.2 debugpy 1.6.7
decorator 5.1.1 Deprecated 1.2.14 distlib 0.3.8
docstring-to-markdown 0.11 entrypoints 0.4 executing 0.8.3
facets-overview 1.1.1 filelock 3.15.4 fonttools 4.51.0
gitdb 4.0.11 GitPython 3.1.37 google-api-core 2.20.0
google-auth 2.35.0 google-cloud-core 2.4.1 google-cloud-storage 2.18.2
google-crc32c 1.6.0 google-resumable-media 2.7.2 googleapis-common-protos 1.65.0
grpcio 1.60.0 grpcio-status 1.60.0 httplib2 0.20.4
idna 3.7 importlib-metadata 6.0.0 importlib_resources 6.4.0
inflect 7.3.1 ipyflow-core 0.0.198 ipykernel 6.28.0
ipython 8.25.0 ipython-genutils 0.2.0 ipywidgets 7.7.2
isodate 0.6.1 jaraco.context 5.3.0 jaraco.functools 4.0.1
jaraco.text 3.12.1 jedi 0.19.1 jmespath 1.0.1
joblib 1.4.2 jupyter_client 8.6.0 jupyter_core 5.7.2
kiwisolver 1.4.4 launchpadlib 1.11.0 lazr.restfulclient 0.14.6
lazr.uri 1.0.6 matplotlib 3.8.4 matplotlib-inline 0.1.6
mccabe 0.7.0 mlflow-skinny 2.15.1 more-itertools 10.3.0
mypy 1.10.0 mypy-extensions 1.0.0 nest-asyncio 1.6.0
nodeenv 1.9.1 numpy 1.26.4 oauthlib 3.2.2
opentelemetry-api 1.27.0 opentelemetry-sdk 1.27.0 opentelemetry-semantic-conventions 0.48b0
packaging 24.1 pandas 1.5.3 parso 0.8.3
pathspec 0.10.3 patsy 0.5.6 pexpect 4.8.0
pillow 10.3.0 pip 24.2 platformdirs 3.10.0
plotly 5.22.0 pluggy 1.0.0 prompt-toolkit 3.0.43
proto-plus 1.24.0 protobuf 4.24.1 psutil 5.9.0
psycopg2 2.9.3 ptyprocess 0.7.0 pure-eval 0.2.2
pyarrow 15.0.2 pyasn1 0.4.8 pyasn1-modules 0.2.8
pyccolo 0.0.52 pycparser 2.21 pydantic 2.8.2
pydantic_core 2.20.1 pyflakes 3.2.0 Pygments 2.15.1
PyGObject 3.48.2 PyJWT 2.7.0 pyodbc 5.0.1
pyparsing 3.0.9 pyright 1.1.294 python-dateutil 2.9.0.post0
python-lsp-jsonrpc 1.1.2 python-lsp-server 1.10.0 pytoolconfig 1.2.6
pytz 2024.1 PyYAML 6.0.1 pyzmq 25.1.2
requests 2.32.2 rope 1.12.0 rsa 4.9
s3transfer 0.10.2 scikit-learn 1.4.2 scipy 1.13.1
seaborn 0.13.2 setuptools 74.0.0 six 1.16.0
smmap 5.0.0 sqlparse 0.5.1 ssh-import-id 5.11
stack-data 0.2.0 statsmodels 0.14.2 tenacity 8.2.2
threadpoolctl 2.2.0 tokenize-rt 4.2.1 tomli 2.0.1
tornado 6.4.1 traitlets 5.14.3 typeguard 4.3.0
types-protobuf 3.20.3 types-psutil 5.9.0 types-pytz 2023.3.1.1
types-PyYAML 6.0.0 types-requests 2.31.0.0 types-setuptools 68.0.0.0
types-six 1.16.0 types-urllib3 1.26.25.14 typing_extensions 4.11.0
ujson 5.10.0 unattended-upgrades 0.1 urllib3 1.26.16
virtualenv 20.26.2 wadllib 1.3.6 wcwidth 0.2.5
whatthepatch 1.0.2 wheel 0.43.0 wrapt 1.14.1
yapf 0.33.0 zipp 3.17.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot.

Library Version Library Version Library Version
arrow 16.1.0 askpass 1.2.0 assertthat 0.2.1
backports 1.5.0 base 4.4.0 base64enc 0.1-3
bigD 0.2.0 bit 4.0.5 bit64 4.0.5
bitops 1.0-8 blob 1.2.4 boot 1.3-30
brew 1.0-10 brio 1.1.5 broom 1.0.6
bslib 0.8.0 cachem 1.1.0 callr 3.7.6
caret 6.0-94 cellranger 1.1.0 chron 2.3-61
class 7.3-22 cli 3.6.3 clipr 0.8.0
clock 0.7.1 cluster 2.1.6 codetools 0.2-20
colorspace 2.1-1 commonmark 1.9.1 compiler 4.4.0
config 0.3.2 conflicted 1.2.0 cpp11 0.4.7
crayon 1.5.3 credentials 2.0.1 curl 5.2.1
data.table 1.15.4 datasets 4.4.0 DBI 1.2.3
dbplyr 2.5.0 desc 1.4.3 devtools 2.4.5
diagram 1.6.5 diffobj 0.3.5 digest 0.6.36
downlit 0.4.4 dplyr 1.1.4 dtplyr 1.3.1
e1071 1.7-14 ellipsis 0.3.2 evaluate 0.24.0
fansi 1.0.6 farver 2.1.2 fastmap 1.2.0
fontawesome 0.5.2 forcats 1.0.0 foreach 1.5.2
foreign 0.8-86 forge 0.2.0 fs 1.6.4
future 1.34.0 future.apply 1.11.2 gargle 1.5.2
generics 0.1.3 gert 2.1.0 ggplot2 3.5.1
gh 1.4.1 git2r 0.33.0 gitcreds 0.1.2
glmnet 4.1-8 globals 0.16.3 glue 1.7.0
googledrive 2.1.1 googlesheets4 1.1.1 gower 1.0.1
graphics 4.4.0 grDevices 4.4.0 grid 4.4.0
gridExtra 2.3 gsubfn 0.7 gt 0.11.0
gtable 0.3.5 hardhat 1.4.0 haven 2.5.4
highr 0.11 hms 1.1.3 htmltools 0.5.8.1
htmlwidgets 1.6.4 httpuv 1.6.15 httr 1.4.7
httr2 1.0.2 ids 1.0.1 ini 0.3.1
ipred 0.9-15 isoband 0.2.7 iterators 1.0.14
jquerylib 0.1.4 jsonlite 1.8.8 juicyjuice 0.1.0
KernSmooth 2.23-22 knitr 1.48 labeling 0.4.3
later 1.3.2 lattice 0.22-5 lava 1.8.0
lifecycle 1.0.4 listenv 0.9.1 lubridate 1.9.3
magrittr 2.0.3 markdown 1.13 MASS 7.3-60.0.1
Matrix 1.6-5 memoise 2.0.1 methods 4.4.0
mgcv 1.9-1 mime 0.12 miniUI 0.1.1.1
mlflow 2.14.1 ModelMetrics 1.2.2.2 modelr 0.1.11
munsell 0.5.1 nlme 3.1-165 nnet 7.3-19
numDeriv 2016.8-1.1 openssl 2.2.0 parallel 4.4.0
parallelly 1.38.0 pillar 1.9.0 pkgbuild 1.4.4
pkgconfig 2.0.3 pkgdown 2.1.0 pkgload 1.4.0
plogr 0.2.0 plyr 1.8.9 praise 1.0.0
prettyunits 1.2.0 pROC 1.18.5 processx 3.8.4
prodlim 2024.06.25 profvis 0.3.8 progress 1.2.3
progressr 0.14.0 promises 1.3.0 proto 1.0.0
proxy 0.4-27 ps 1.7.7 purrr 1.0.2
R6 2.5.1 ragg 1.3.2 randomForest 4.7-1.1
rappdirs 0.3.3 rcmdcheck 1.4.0 RColorBrewer 1.1-3
Rcpp 1.0.13 RcppEigen 0.3.4.0.0 reactable 0.4.4
reactR 0.6.0 readr 2.1.5 readxl 1.4.3
recipes 1.1.0 rematch 2.0.0 rematch2 2.1.2
remotes 2.5.0 reprex 2.1.1 reshape2 1.4.4
rlang 1.1.4 rmarkdown 2.27 RODBC 1.3-23
roxygen2 7.3.2 rpart 4.1.23 rprojroot 2.0.4
Rserve 1.8-13 RSQLite 2.3.7 rstudioapi 0.16.0
rversions 2.1.2 rvest 1.0.4 sass 0.4.9
scales 1.3.0 selectr 0.4-2 sessioninfo 1.2.2
shape 1.4.6.1 shiny 1.9.1 sourcetools 0.1.7-1
sparklyr 1.8.6 spatial 7.3-17 splines 4.4.0
sqldf 0.4-11 SQUAREM 2021.1 stats 4.4.0
stats4 4.4.0 stringi 1.8.4 stringr 1.5.1
survival 3.6-4 swagger 5.17.14.1 sys 3.4.2
systemfonts 1.1.0 tcltk 4.4.0 testthat 3.2.1.1
textshaping 0.4.0 tibble 3.2.1 tidyr 1.3.1
tidyselect 1.2.1 tidyverse 2.0.0 timechange 0.3.0
timeDate 4032.109 tinytex 0.52 tools 4.4.0
tzdb 0.4.0 urlchecker 1.0.1 usethis 3.0.0
utf8 1.2.4 utils 4.4.0 uuid 1.2-1
V8 4.4.2 vctrs 0.6.5 viridisLite 0.4.2
vroom 1.6.5 waldo 0.5.2 whisker 0.4.1
withr 3.0.1 xfun 0.46 xml2 1.3.6
xopen 1.0.1 xtable 1.8-4 yaml 2.3.10
zeallot 0.1.0 zip 2.3.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.12.638
com.amazonaws aws-java-sdk-cloudformation 1.12.638
com.amazonaws aws-java-sdk-cloudfront 1.12.638
com.amazonaws aws-java-sdk-cloudhsm 1.12.638
com.amazonaws aws-java-sdk-cloudsearch 1.12.638
com.amazonaws aws-java-sdk-cloudtrail 1.12.638
com.amazonaws aws-java-sdk-cloudwatch 1.12.638
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.12.638
com.amazonaws aws-java-sdk-codedeploy 1.12.638
com.amazonaws aws-java-sdk-cognitoidentity 1.12.638
com.amazonaws aws-java-sdk-cognitosync 1.12.638
com.amazonaws aws-java-sdk-config 1.12.638
com.amazonaws aws-java-sdk-core 1.12.638
com.amazonaws aws-java-sdk-datapipeline 1.12.638
com.amazonaws aws-java-sdk-directconnect 1.12.638
com.amazonaws aws-java-sdk-directory 1.12.638
com.amazonaws aws-java-sdk-dynamodb 1.12.638
com.amazonaws aws-java-sdk-ec2 1.12.638
com.amazonaws aws-java-sdk-ecs 1.12.638
com.amazonaws aws-java-sdk-efs 1.12.638
com.amazonaws aws-java-sdk-elasticache 1.12.638
com.amazonaws aws-java-sdk-elasticbeanstalk 1.12.638
com.amazonaws aws-java-sdk-elasticloadbalancing 1.12.638
com.amazonaws aws-java-sdk-elastictranscoder 1.12.638
com.amazonaws aws-java-sdk-emr 1.12.638
com.amazonaws aws-java-sdk-glacier 1.12.638
com.amazonaws aws-java-sdk-glue 1.12.638
com.amazonaws aws-java-sdk-iam 1.12.638
com.amazonaws aws-java-sdk-importexport 1.12.638
com.amazonaws aws-java-sdk-kinesis 1.12.638
com.amazonaws aws-java-sdk-kms 1.12.638
com.amazonaws aws-java-sdk-lambda 1.12.638
com.amazonaws aws-java-sdk-logs 1.12.638
com.amazonaws aws-java-sdk-machinelearning 1.12.638
com.amazonaws aws-java-sdk-opsworks 1.12.638
com.amazonaws aws-java-sdk-rds 1.12.638
com.amazonaws aws-java-sdk-redshift 1.12.638
com.amazonaws aws-java-sdk-route53 1.12.638
com.amazonaws aws-java-sdk-s3 1.12.638
com.amazonaws aws-java-sdk-ses 1.12.638
com.amazonaws aws-java-sdk-simpledb 1.12.638
com.amazonaws aws-java-sdk-simpleworkflow 1.12.638
com.amazonaws aws-java-sdk-sns 1.12.638
com.amazonaws aws-java-sdk-sqs 1.12.638
com.amazonaws aws-java-sdk-ssm 1.12.638
com.amazonaws aws-java-sdk-storagegateway 1.12.638
com.amazonaws aws-java-sdk-sts 1.12.638
com.amazonaws aws-java-sdk-support 1.12.638
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.12.638
com.amazonaws jmespath-java 1.12.638
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks databricks-sdk-java 0.27.0
com.databricks jets3t 0.7.1-0
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.15.2
com.fasterxml.jackson.core jackson-core 2.15.2
com.fasterxml.jackson.core jackson-databind 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-yaml 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-joda 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.16.0
com.fasterxml.jackson.module jackson-module-paranamer 2.15.2
com.fasterxml.jackson.module jackson-module-scala_2.12 2.15.2
com.github.ben-manes.caffeine caffeine 2.9.3
com.github.fommil jniloader 1.1
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java 1.1-natives
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java 1.1-natives
com.github.fommil.netlib netlib-native_ref-linux-x86_64 1.1-natives
com.github.fommil.netlib netlib-native_system-linux-x86_64 1.1-natives
com.github.luben zstd-jni 1.5.5-4
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.10.1
com.google.crypto.tink tink 1.9.0
com.google.errorprone error_prone_annotations 2.10.0
com.google.flatbuffers flatbuffers-java 23.5.26
com.google.guava guava 15.0
com.google.protobuf protobuf-java 3.25.1
com.helger profiler 1.1.1
com.ibm.icu icu4j 75.1
com.jcraft jsch 0.1.55
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.microsoft.sqlserver mssql-jdbc 11.2.2.jre8
com.ning compress-lzf 1.1.2
com.sun.mail javax.mail 1.5.2
com.sun.xml.bind jaxb-core 2.2.11
com.sun.xml.bind jaxb-impl 2.2.11
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.4.3
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.uber h3 3.7.3
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 4.0.3
commons-cli commons-cli 1.5.0
commons-codec commons-codec 1.16.0
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.5
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.13.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 3.0.3
dev.ludovic.netlib blas 3.0.3
dev.ludovic.netlib lapack 3.0.3
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.27
io.delta delta-sharing-client_2.12 1.2.0
io.dropwizard.metrics metrics-annotation 4.2.19
io.dropwizard.metrics metrics-core 4.2.19
io.dropwizard.metrics metrics-graphite 4.2.19
io.dropwizard.metrics metrics-healthchecks 4.2.19
io.dropwizard.metrics metrics-jetty9 4.2.19
io.dropwizard.metrics metrics-jmx 4.2.19
io.dropwizard.metrics metrics-json 4.2.19
io.dropwizard.metrics metrics-jvm 4.2.19
io.dropwizard.metrics metrics-servlets 4.2.19
io.netty netty-all 4.1.108.Final
io.netty netty-buffer 4.1.108.Final
io.netty netty-codec 4.1.108.Final
io.netty netty-codec-http 4.1.108.Final
io.netty netty-codec-http2 4.1.108.Final
io.netty netty-codec-socks 4.1.108.Final
io.netty netty-common 4.1.108.Final
io.netty netty-handler 4.1.108.Final
io.netty netty-handler-proxy 4.1.108.Final
io.netty netty-resolver 4.1.108.Final
io.netty netty-tcnative-boringssl-static 2.0.61.Final
io.netty netty-tcnative-boringssl-static 2.0.61.Final-linux-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-linux-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-osx-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-osx-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-windows-x86_64
io.netty netty-tcnative-classes 2.0.61.Final
io.netty netty-transport 4.1.108.Final
io.netty netty-transport-classes-epoll 4.1.108.Final
io.netty netty-transport-classes-kqueue 4.1.108.Final
io.netty netty-transport-native-epoll 4.1.108.Final
io.netty netty-transport-native-epoll 4.1.108.Final-linux-aarch_64
io.netty netty-transport-native-epoll 4.1.108.Final-linux-riscv64
io.netty netty-transport-native-epoll 4.1.108.Final-linux-x86_64
io.netty netty-transport-native-kqueue 4.1.108.Final-osx-aarch_64
io.netty netty-transport-native-kqueue 4.1.108.Final-osx-x86_64
io.netty netty-transport-native-unix-common 4.1.108.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.11
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.12.1
net.java.dev.jna jna 5.8.0
net.razorvine pickle 1.3
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.9.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.10.11
org.apache.ant ant-jsch 1.10.11
org.apache.ant ant-launcher 1.10.11
org.apache.arrow arrow-format 15.0.0
org.apache.arrow arrow-memory-core 15.0.0
org.apache.arrow arrow-memory-netty 15.0.0
org.apache.arrow arrow-vector 15.0.0
org.apache.avro avro 1.11.3
org.apache.avro avro-ipc 1.11.3
org.apache.avro avro-mapred 1.11.3
org.apache.commons commons-collections4 4.4
org.apache.commons commons-compress 1.23.0
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.6.1
org.apache.commons commons-text 1.10.0
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.datasketches datasketches-java 3.1.0
org.apache.datasketches datasketches-memory 2.0.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-runtime 3.3.6
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.8.1
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.httpcomponents httpclient 4.5.14
org.apache.httpcomponents httpcore 4.4.16
org.apache.ivy ivy 2.5.2
org.apache.logging.log4j log4j-1.2-api 2.22.1
org.apache.logging.log4j log4j-api 2.22.1
org.apache.logging.log4j log4j-core 2.22.1
org.apache.logging.log4j log4j-layout-template-json 2.22.1
org.apache.logging.log4j log4j-slf4j2-impl 2.22.1
org.apache.orc orc-core 1.9.2-shaded-protobuf
org.apache.orc orc-mapreduce 1.9.2-shaded-protobuf
org.apache.orc orc-shims 1.9.2
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.ws.xmlschema xmlschema-core 2.3.0
org.apache.xbean xbean-asm9-shaded 4.23
org.apache.yetus audience-annotations 0.13.0
org.apache.zookeeper zookeeper 3.9.2
org.apache.zookeeper zookeeper-jute 3.9.2
org.checkerframework checker-qual 3.31.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.collections eclipse-collections 11.1.0
org.eclipse.collections eclipse-collections-api 11.1.0
org.eclipse.jetty jetty-client 9.4.52.v20230823
org.eclipse.jetty jetty-continuation 9.4.52.v20230823
org.eclipse.jetty jetty-http 9.4.52.v20230823
org.eclipse.jetty jetty-io 9.4.52.v20230823
org.eclipse.jetty jetty-jndi 9.4.52.v20230823
org.eclipse.jetty jetty-plus 9.4.52.v20230823
org.eclipse.jetty jetty-proxy 9.4.52.v20230823
org.eclipse.jetty jetty-security 9.4.52.v20230823
org.eclipse.jetty jetty-server 9.4.52.v20230823
org.eclipse.jetty jetty-servlet 9.4.52.v20230823
org.eclipse.jetty jetty-servlets 9.4.52.v20230823
org.eclipse.jetty jetty-util 9.4.52.v20230823
org.eclipse.jetty jetty-util-ajax 9.4.52.v20230823
org.eclipse.jetty jetty-webapp 9.4.52.v20230823
org.eclipse.jetty jetty-xml 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-api 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-client 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-common 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-server 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-servlet 9.4.52.v20230823
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.40
org.glassfish.jersey.containers jersey-container-servlet-core 2.40
org.glassfish.jersey.core jersey-client 2.40
org.glassfish.jersey.core jersey-common 2.40
org.glassfish.jersey.core jersey-server 2.40
org.glassfish.jersey.inject jersey-hk2 2.40
org.hibernate.validator hibernate-validator 6.1.7.Final
org.ini4j ini4j 0.5.4
org.javassist javassist 3.29.2-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M11
org.json4s json4s-core_2.12 3.7.0-M11
org.json4s json4s-jackson_2.12 3.7.0-M11
org.json4s json4s-scalap_2.12 3.7.0-M11
org.lz4 lz4-java 1.8.0
org.mlflow mlflow-spark_2.12 2.9.1
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.6.1
org.roaringbitmap RoaringBitmap 0.9.45-databricks
org.roaringbitmap shims 0.9.45-databricks
org.rocksdb rocksdbjni 9.2.1
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.15
org.scala-lang scala-library_2.12 2.12.15
org.scala-lang scala-reflect_2.12 2.12.15
org.scala-lang.modules scala-collection-compat_2.12 2.11.0
org.scala-lang.modules scala-java8-compat_2.12 0.9.1
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.2.16
org.scalanlp breeze-macros_2.12 2.1.0
org.scalanlp breeze_2.12 2.1.0
org.scalatest scalatest-compatible 3.2.16
org.scalatest scalatest-core_2.12 3.2.16
org.scalatest scalatest-diagrams_2.12 3.2.16
org.scalatest scalatest-featurespec_2.12 3.2.16
org.scalatest scalatest-flatspec_2.12 3.2.16
org.scalatest scalatest-freespec_2.12 3.2.16
org.scalatest scalatest-funspec_2.12 3.2.16
org.scalatest scalatest-funsuite_2.12 3.2.16
org.scalatest scalatest-matchers-core_2.12 3.2.16
org.scalatest scalatest-mustmatchers_2.12 3.2.16
org.scalatest scalatest-propspec_2.12 3.2.16
org.scalatest scalatest-refspec_2.12 3.2.16
org.scalatest scalatest-shouldmatchers_2.12 3.2.16
org.scalatest scalatest-wordspec_2.12 3.2.16
org.scalatest scalatest_2.12 3.2.16
org.slf4j jcl-over-slf4j 2.0.7
org.slf4j jul-to-slf4j 2.0.7
org.slf4j slf4j-api 2.0.7
org.slf4j slf4j-simple 1.7.25
org.threeten threeten-extra 1.7.1
org.tukaani xz 1.9
org.typelevel algebra_2.12 2.0.1
org.typelevel cats-kernel_2.12 2.1.1
org.typelevel spire-macros_2.12 0.17.0
org.typelevel spire-platform_2.12 0.17.0
org.typelevel spire-util_2.12 0.17.0
org.typelevel spire_2.12 0.17.0
org.wildfly.openssl wildfly-openssl 1.1.3.Final
org.xerial sqlite-jdbc 3.42.0.0
org.xerial.snappy snappy-java 1.1.10.3
org.yaml snakeyaml 2.0
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.cryptools AmazonCorrettoCryptoProvider 1.6.2-linux-x86_64
stax stax-api 1.0.1