Databricks Runtime 16.1 (Beta)

Important

Databricks Runtime 16.1 is in Beta. The contents of the supported environments may change during the Beta. Changes can include the list of packages or versions of installed packages.

The following release notes provide information about Databricks Runtime 16.1, powered by Apache Spark 3.5.0.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

Behavioral changes

The VARIANT data type can no longer be used with operations that require comparisons

In DBR 16.1 and above, you cannot use the following clauses or operators in queries that include a VARIANT data type:

• DISTINCT
• INTERSECT
• EXCEPT
• UNION
• DISTRIBUTE BY

Additionally, you cannot use these DataFrame functions:

• df.dropDuplicates()
• df.repartition()

These operations perform comparisons, and comparisons that use the VARIANT data type produce undefined results and are not supported in Databricks. If you use the VARIANT type in your Azure Databricks workloads or tables, Databricks recommends the following changes:

• Update queries or expressions to explicitly cast VARIANT values to non-VARIANT data types.
• If you have fields that must be used with any of the above operations, extract those fields from the VARIANT data type and store them using non-VARIANT data types.

To learn more, see Query variant data.

New features and improvements

Support for collations in Apache Spark in Public Preview

You can now assign language-aware, case-insensitive, and access-insensitive collations to STRING columns and expressions. These collations are used in string comparisons, sorting, grouping operations, and many string functions. See Collation.

Support for collations in Delta Lake in Public Preview

You can now define collations for columns when creating or altering a Delta table. See Collation support for Delta Lake.

Lite mode for vacuum in Public Preview

You can now use VACUUM table_name LITE to perform a lighter-weight vacuum operation that leverages metadata in the Delta transaction log. See Full vs. lite mode and VACUUM.

Parameterize USE CATALOG with IDENTIFIER clause support

In Databricks Runtime 16.1 and later the IDENTIFIER clause is supported for the USE CATALOG statement. With this support you can parameterize the current catalog based on a string variable or parameter marker.

COMMENT ON COLUMN support for tables and views

In Databricks Runtime 16.1 and later, the COMMENT ON statement supports altering comments for view and table columns.

New SQL functions

In Databricks Runtime 16.1 and later the following new built-in SQL functions are available:

• dayname(expr) returns the three-letter English acronym for the day of the week for the given date.
• uniform(expr1, expr2 [,seed]) returns a random value with independent and identically distributed values within the specified range of numbers.
• randstr(length) returns a random string of length alpha-numeric characters.

Named parameter invocation for more functions

The following function support named parameter invocation in Databricks Runtime 16.1 and later:

• variant_explode
• variant_explode_outer
• inline
• inline_outer
• posexplode
• posexplode_outer

Hive metastore support for the SYNC METADATA argument of the REPAIR TABLE command

In Databricks Runtime 16.1 and above, you can use the SYNC METADATA argument with REPAIR TABLE to update the metadata of a Hive metastore managed table.

Enhanced data integrity for compressed Apache Arrow batches

In Databricks Runtime 16.1 and above, to further protect against data corruption, every LZ4 compressed Arrow batch now includes the LZ4 content and block checksums. See LZ4 Frame Format Description.

Support added for Scala methods on Unity Catalog shared access mode compute

In Databricks Runtime 16.1 and above, on Unity Catalog shared access mode compute, support is added for the following Scala methods: Dataset.flatMapGroups(), Dataset.mapGroups(), and DataStreamWriter.foreach().

StreamingQueryListener support for Scala

You can now use StreamingQueryListener in Scala on compute configured with shared access mode.

Built-in Oracle JDBC Driver

In Databricks Runtime 16.1 and above, the Oracle JDBC Driver is built into Azure Databricks. If you use a customer-uploaded JDBC driver JAR via DriverManager, you must rewrite scripts to explicitly use the custom JAR. Otherwise, the built-in driver is used.

More detailed errors for Delta tables accessed with paths

A new error message experience for Delta tables accessed using paths is now available. All exceptions are now forwarded to the user. The exception DELTA_MISSING_DELTA_TABLE is now reserved for when underlying files cannot be read as a Delta table.

Other changes

Renamed error codes for the cloudFiles Structured Streaming source

This release includes a change to rename the following error codes:

• _LEGACY_ERROR_TEMP_DBR_0143 is renamed to CF_INCORRECT_STREAM_USAGE.
• _LEGACY_ERROR_TEMP_DBR_0260 is renamed to CF_INCORRECT_BATCH_USAGE .

Bug fixes

Nested types now properly accept NULL constraints

This release fixes a bug affecting some Delta generated columns of nested types, for example, STRUCT. These columns would sometimes incorrectly reject expressions based on NULL or NOT NULL constraints of nested fields. This has been fixed.

Library upgrades

• Upgraded Python libraries:
  • ipyflow-core from 0.0.198 to 0.0.201
  • pyccolo from 0.0.52 to 0.0.65
• Upgraded R libraries:
• Upgraded Java libraries:
  • io.delta.delta-sharing-client_2.12 from 1.2.0 to 1.2.2
  • org.lz4.lz4-java from 1.8.0 to 1.8.0-databricks-1
  • software.amazon.cryptools.AmazonCorrettoCryptoProvider from 1.6.2-linux-x86_64 to 2.4.1-linux-x86_64

Apache Spark

Databricks Runtime 16.1 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 16.0, as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-50482] [SC-182879][CORE] Deprecated no-op spark.shuffle.spill config
• [SPARK-50032] [SC-182706][SQL][16.x] Allow use of fully qualified collation name
• [SPARK-50467] [SC-182823][PYTHON] Add __all__ for builtin functions
• [SPARK-48898] [SC-182828][SQL] Fix Variant shredding bug
• [SPARK-50441] [SC-182668][SQL] Fix parametrized identifiers not working when referencing CTEs
• [SPARK-50446] [SC-182639][PYTHON] Concurrent level in Arrow-optimized Python UDF
• [SPARK-50430] [SC-182536][CORE] Use the standard Properties.clone instead of manual clone
• [SPARK-50471] [SC-182790][PYTHON] Support Arrow-based Python Data Source Writer
• [SPARK-50466] [SC-182791][PYTHON] Refine the docstring for string functions - part 1
• [SPARK-50194] [DBR16.x][SC-182593][SS][PYTHON] Integration of New Timer API and Initial State API with Timer
• [SPARK-50437] [SC-182586][SS] Reduce overhead of creating deserializers in TransformWithStateExec
• [SPARK-49676] [DBR16.x][SC-182538][SS][PYTHON] Add Support for Chaining o…
• [SPARK-49294] [SC-182730][UI] Add width attribute for shuffle-write-time checkbox.
• [SPARK-50426] [SC-182540][PYTHON] Avoid static Python data source lookup when using builtin or Java data sources
• [SPARK-48356] [SC-182603][SQL] Support for FOR statement
• [SPARK-50333] [SC-182136][SQL] Codegen Support for CsvToStructs (by Invoke & RuntimeReplaceable)
• [SPARK-50285] [SC-182575] Metrics for commits to StagedTable instances
• [SPARK-50081] [SC-182344][SQL] Codegen Support for XPath*(by Invoke & RuntimeReplaceable)
• [SPARK-50440] [SC-182592] [SQL] Refactor AttributeSeq.resolveCandidates
• [SPARK-50067] [SC-179648][SQL] Codegen Support for SchemaOfCsv(by Invoke & RuntimeReplaceable)
• [SPARK-49873] [SC-178577][SQL] fix post merge failure on error testing
• [SPARK-50118] [SC-181259][CONNET] Reset isolated state cache when tasks are running
• [SPARK-49873] [SC-178577][SQL] Assign proper error class for _LEGACY_ERROR_TEMP_1325
• [SPARK-50381] [SC-182197][CORE] Support spark.master.rest.maxThreads
• [SPARK-46725] [SC-182448][SQL] Add DAYNAME function
• [SPARK-50270] [SC-181179][SS][PYTHON] Added custom state metrics for TransformWithStateInPandas
• [SPARK-50118] Revert “[SC-181259][CONNET] Reset isolated state cache when tasks are running”
• [SPARK-50075] [SC-181820][SQL][PYTHON][CONNECT] Add DataFrame APIs for table-valued functions
• [SPARK-49470] [SC-175736][UI] Update dataTables stylesheets and javascripts from 1.13.5 to 1.13.11
• [SPARK-50235] Revert “[SC-180786][SQL] Clean up ColumnVector resource after processing all rows in ColumnarToRowExec”
• [SPARK-50324] [SC-182278][PYTHON][CONNECT] Make createDataFrame trigger Config RPC at most once
• [SPARK-50387] [SC-182441][SS] Update condition for timer expiry and relevant test
• [SPARK-50287] [SC-182400][SQL] Merge options of table and relation when creating WriteBuilder in FileTable
• [SPARK-50066] [SC-181484][SQL] Codegen Support for SchemaOfXml (by Invoke & RuntimeReplaceable)
• [SPARK-50092] [SC-181568][SQL] Fix PostgreSQL connector behaviour for multidimensional arrays
• [SPARK-50318] [SC-181641][SQL] Add IntervalUtils.makeYearMonthInterval to deduplicate code between interpreted and codegen
• [SPARK-50312] [SC-181646][SQL] SparkThriftServer createServer parameter passing error when kerberos is true
• [SPARK-50246] [SC-181468][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2167: INVALID_JSON_RECORD_TYPE
• [SPARK-50214] [SC-180692][SQL] From json/xml should not change collations in the given schema
• [SPARK-50250] [SC-181466][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2075: UNSUPPORTED_FEATURE.WRITE_FOR_BINARY_SOURCE
• [SPARK-50248] [SC-181467][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2058: INVALID_PARTITION_VALUE
• [SPARK-50118] [SC-181259][CONNET] Reset isolated state cache when tasks are running
• [SPARK-50235] [SC-180786][SQL] Clean up ColumnVector resource after processing all rows in ColumnarToRowExec
• [SPARK-50156] [SC-180781][SQL] Integrate _LEGACY_ERROR_TEMP_2113 into UNRECOGNIZED_STATISTIC
• [SPARK-50069] [SC-180163][SQL] Integrate _LEGACY_ERROR_TEMP_0028 into UNSUPPORTED_FROM_TO_EXPRESSION
• [SPARK-50154] [SC-180663][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_0043: INVALID_RESET_COMMAND_FORMAT
• [SPARK-49967] [SC-179534][SQL] Codegen Support for StructsToJson(to_json)
• [SPARK-50055] [SC-180978][SQL] Add TryMakeInterval alternative
• [SPARK-50397] [SC-182367][CORE] Remove deprecated --ip and -i arguments from Master/Worker
• [SPARK-50238] [SC-181434][PYTHON] Add Variant Support in PySpark UDFs/UDTFs/UDAFs and Python UC UDFs
• [SPARK-50079] [SC-179830][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2013: NEGATIVE_VALUES_IN_FREQUENCY_EXPRESSION
• [SPARK-50182] [SC-180346][EXAMPLE] Add submit-sql.sh REST API example
• [SPARK-49966] [SC-179501][SQL] Use Invoke to implement JsonToStructs(from_json)
• [SPARK-50302] [SC-182518][SS] Ensure secondary index sizes equal primary index sizes for TransformWithState stateful variables with TTL
• [SPARK-50301] [SC-182241][SS][16.x] Make TransformWithState metrics reflect their intuitive meanings
• [SPARK-50175] [SC-182140][SQL] Change collation precedence calculation
• [SPARK-50148] [SC-180292][SQL] Make StaticInvoke compatible with the method that declare throw exception
• [SPARK-50280] [SC-181214][PYTHON] Refactor result sorting and empty bin filling in compute_hist
• [SPARK-50190] [SC-182458][PYTHON] Remove direct dependency of Numpy from Histogram
• [SPARK-50382] [SC-182368][CONNECT] Add documentation for general information on application development with/extending Spark Connect
• [SPARK-50296] [SC-181464][PYTHON][CONNECT] Avoid using a classproperty in threadpool for Python Connect client
• [SPARK-49566] [SC-182239][SQL] Add SQL pipe syntax for the EXTEND operator
• [SPARK-50036] [SC-179533][CORE][PYTHON] Include SPARK_LOG_SCHEMA in the context of REPL shell
• [SPARK-49859] [SC-178259][CONNECT] Replace multiprocessing.ThreadPool with ThreadPoolExecutor
• [SPARK-50141] [SC-182378][PYTHON] Make lpad and rpad accept Column type arguments
• [SPARK-50379] [SC-182142][SQL] Fix DayTimeIntevalType handling in WindowExecBase
• [SPARK-49954] [SC-179110][SQL] Codegen Support for SchemaOfJson (by Invoke & RuntimeReplaceable)
• [SPARK-50398] [SC-182341][CORE] Use ExitCode 0 for --help usage in Spark scripts
• [SPARK-50377] [SC-182238][SQL] Allow to evaluate foldable RuntimeReplaceable
• [SPARK-50241] [SC-181444][SQL] Replace NullIntolerant Mixin with Expression.nullIntolerant method
• [SPARK-50084] [SC-179672][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_3168: MISSING_TIMEOUT_CONFIGURATION
• [SPARK-50078] [SC-179649][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_0038: DUPLICATED_CTE_NAMES
• [SPARK-50057] [SC-179573][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_1049: INVALID_ATTRIBUTE_NAME_SYNTAX
• [SPARK-50070] [SC-179579][SQL] Integrate _LEGACY_ERROR_TEMP_0039 into UNSUPPORTED_SQL_STATEMENT
• [SPARK-50378] [SC-182235][SS] Add custom metric for tracking spent for proc initial state in transformWithState
• [SPARK-50029] [SC-179531][SQL] Make StaticInvoke compatible with the method that return Any
• [SPARK-49990] [SC-179497][SQL] Improve performance of randStr
• [SPARK-50048] [SC-179528][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2114: UNRECOGNIZED_STATISTIC
• [SPARK-50053] [SC-179532][SQL] Turn _LEGACY_ERROR_TEMP_2104 into INTERNAL_ERROR
• [SPARK-49665] [SC-180054][SQL] Trim collation support for string functions
• [SPARK-48549] [SC-176472][SQL][PYTHON] Improve SQL function sentences
• [SPARK-50022] [SC-179503][CORE][UI] Fix MasterPage to hide App UI links when UI is disabled
• [SPARK-50087] [SC-182152] Robust handling of boolean expressions in CASE WHEN for MsSqlServer and future connectors
• [SPARK-49991] [SC-179481][SQL] Make HadoopMapReduceCommitProtocol respect ‘mapreduce.output.basename’ to generate file names
• [SPARK-50038] [SC-179521][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_0008: MERGE_WITHOUT_WHEN
• [SPARK-50236] [SC-181671][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_1156: COLUMN_NOT_DEFINED_IN_TABLE
• [SPARK-50021] [SC-179500][CORE][UI] Fix ApplicationPage to hide App UI links when UI is disabled
• [SPARK-49911] [SC-179111][SQL] Fix semantic of support binary equality
• [SPARK-50025] [SC-179496][SQL] Integrate _LEGACY_ERROR_TEMP_1253 into EXPECT_VIEW_NOT_TABLE
• [SPARK-49829] [SC-179480][SS] Fix the bug on the optimization on adding input to state store in stream-stream join
• [SPARK-50004] [SC-179499][SQL] Integrate _LEGACY_ERROR_TEMP_3327 into FIELD_NOT_FOUND
• [SPARK-50380] [SC-182210][SQL] ReorderAssociativeOperator should respect the contract in ConstantFolding
• [SPARK-50340] [SC-181859][SQL] Unwrap UDT in INSERT input query
• [SPARK-50237] [SC-181660][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2138-9: CIRCULAR_CLASS_REFERENCE
• [SPARK-50258] [SC-181993][SQL] Fix output column order changed issue after AQE optimization
• [SPARK-49773] [SC-178369][SQL] Uncaught Java exception from make_timestamp() with bad timezone
• [SPARK-49977] [SC-179265][SQL] Use stack-based iterative computation to avoid creating many Scala List objects for deep expression trees
• [SPARK-50153] [SC-181591][SQL] Add name to RuleExecutor to make printing QueryExecutionMetrics’s logs clearer
• [SPARK-50320] [SC-181668][CORE] Make --remote an official option by removing experimental warning
• [SPARK-49909] [SC-179492]Revert “[SQL] Fix the pretty name of some expressions”
• [SPARK-50330] [SC-180720][SC-181764][SQL] Add hints to Sort and Window nodes
• [SPARK-50364] [SC-182003][SQL] Implement serialization for LocalDateTime type in Row.jsonValue
• [SPARK-50016] [SC-182139][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2067: UNSUPPORTED_PARTITION_TRANSFORM
• [SPARK-49899] [SC-181175][PYTHON][SS] Support deleteIfExists for TransformWithStateInPandas
• [SPARK-49757] [SC-177824][SQL] Support IDENTIFIER expression in SET CATALOG statement
• [SPARK-50315] [SC-181886][SQL] Support custom metrics for V1Fallback writes
• [SPARK-42838] [SC-181509][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_2000
• [SPARK-50353] [SC-181985][SQL] Refactor ResolveSQLOnFile
• [SPARK-48344] [SC-181967][SQL] Prepare SQL Scripting for addition of Execution Framework
• [SPARK-49345] [SC-174786][CONNECT] Make sure using the current running Spark Session
• [SPARK-49925] [SC-178882][SQL] Add tests for order by with collated strings
• [SPARK-50167] [SC-181199][PYTHON][CONNECT] Improve PySpark plotting error messages and imports
• [SPARK-49368] [SC-174999][CONNECT] Avoid accessing protobuf lite classes directly
• [SPARK-50056] [SC-181378][SQL] Codegen Support for ParseUrl (by Invoke & RuntimeReplaceable)
• [SPARK-49601] [SC-180770][SS][PYTHON] Support Initial State Handling for TransformWithStateInPandas
• [SPARK-49908] [SC-178768][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_0044
• [SPARK-50144] [SC-180307][SS] Address the limitation of metrics calculation with DSv1 streaming sources
• [SPARK-49874] [SC-178303][SQL] Remove trim and ltrim collation specifiers.
• [SPARK-49513] [SC-180339][SS] Add Support for timer in transformWithStateInPandas API
• [SPARK-49119] [SC-175607][SQL] Fix the inconsistency of syntax show columns between v1 and v2
• [SPARK-49206] [SC-173704][CORE][UI] Add Environment Variables table to Master EnvironmentPage
• [SPARK-49934] [SC-179581][SQL] Add implicit cast for accessing collated map with literal
• [SPARK-50049] [SC-181659][SQL] Support custom driver metrics in writing to v2 table
• [SPARK-50171] [SC-180295][PYTHON] Make numpy optional for KDE plot
• [SPARK-49962] [SC-179320][SQL] Simplify AbstractStringTypes class hierarchy
• [SPARK-50226] [SC-181483][SQL] Correct MakeDTInterval and MakeYMInterval to catch Java exceptions
• [SPARK-48775] [SC-170801][SQL][STS] Replace SQLContext with SparkSession in STS
• [SPARK-49015] [SC-175688][CORE] Connect Server should respect spark.log.structuredLogging.enabled
• [SPARK-50327] [SC-181667][SQL][16.x] Factor out function resolution to be reused in the single-pass Analyzer
• [SPARK-49995] [SC-180762][SQL] Add named argument support to more TVFs
• [SPARK-49268] [SC-174903][CORE] Log IO exceptions in SHS history provider
• [SPARK-48123] [SC-164989][Core] Provide a constant table schema for querying structured logs
• [SPARK-49217] [SC-174904][CORE] Support separate buffer size configuration in UnsafeShuffleWriter
• [SPARK-50325] [SC-181664][SQL][16.x] Factor out alias resolution to be reused in the single-pass Analyzer
• [SPARK-50322] [SC-181665][SQL] Fix parameterized identifier in a sub-query
• [SPARK-48400] [SC-175283][CORE] Promote PrometheusServlet to DeveloperApi
• [SPARK-50118] Revert “[SC-181259][CONNET] Reset isolated state cache when tasks are running”
• [SPARK-50306] [SC-181564][PYTHON][CONNECT] Support Python 3.13 in Spark Connect
• [SPARK-50152] [SC-181264][SS] Support handleInitialState with state data source reader
• [SPARK-50260] [SC-181271][CONNECT] Refactor and optimize Spark C…
• [SPARK-47591] [SC-163090][SQL] Hive-thriftserver: Migrate logInfo with variables to structured logging framework
• [SPARK-49312] [SC-174672][PYTHON] Improve error message for assertSchemaEqual
• [SPARK-49439] [SC-175236][SQL] Fix the pretty name of the FromProtobuf & ToProtobuf expression
• [SPARK-50092] [ES-1258521] Fix PostgreSQL connector behaviour for multidimensional arrays
• [SPARK-49913] [SC-181565][SQL] Add check for unique label names in nested labeled scopes
• [SPARK-49563] [SC-181465][SQL] Add SQL pipe syntax for the WINDOW operator
• [SPARK-49661] [SC-179021][SQL] Implement trim collation hashing and comparison.
• [SPARK-38912] [SC-181543][PYTHON] Remove the comment related to classmethod and property
• [SPARK-49770] [16.x][SC-179802][SC-179270][SS][RocksDB Hardening] Improve RocksDB SST file mapping management, and fix issue with reloading same version with existing snapshot
• [SPARK-49002] Revert “[SC-172846][SQL] Consistently handle invalid locations in WAREHOUSE/SCHEMA/TABLE/PARTITION/DIRECTORY
• [SPARK-50290] [SC-181453][SQL] Add a flag to disable DataFrameQueryContext creation
• [SPARK-49668] [SC-178268][SQL] Implement collation key support for trim collation
• [SPARK-50262] [SC-181193][SQL] Forbid specification complex types during altering collation
• [SPARK-48898] [SC-181435][SQL] Add Variant shredding functions
• [SPARK-48273] [SC-181381]Revert “[SQL] Fix late rewrite of PlanWithUnresolvedIdentifier
• [SPARK-50222] [SC-180706][CORE] Support spark.submit.appName
• [SPARK-50208] [SC-180662][CORE] Support spark.master.useDriverIdAsAppName.enabled
• [SPARK-50224] [SC-180689][SQL] The replacements of IsValidUTF8|ValidateUTF8|TryValidateUTF8|MakeValidUTF8 shall be NullIntolerant
• [SPARK-50247] [SC-180962][CORE] Define BLOCK_MANAGER_REREGISTRATION_FAILED as ExecutorExitCode
• [SPARK-50282] [SC-181221][ML] Simplify TargetEncoderModel.transform
• [SPARK-50112] [SC-180763][SQL] Allowing the TransformWithState operator to use Avro encoding
• [SPARK-50267] [SC-181180][ML] Improve TargetEncoder.fit with DataFrame APIs
• [SPARK-37178] [SC-180939][ML] Add Target Encoding to ml.feature
• [SPARK-50118] [SC-181259][CONNET] Reset isolated state cache when tasks are running
• [SPARK-50085] [BEHAVE-176][SC-179809][PYTHON] Make lit(ndarray) with np.int8 respect the numpy datatype
• [SPARK-50256] [SC-181048][SQL] Add lightweight validation to check if a logical plan becomes unresolved after every optimizer rule
• [SPARK-50196] [SC-180932][CONNECT] Fix Python error context to use a proper context
• [SPARK-50274] [SC-181181][CORE] Guard against use-after-close in DirectByteBufferOutputStream
• [SPARK-49999] [SC-180063][PYTHON][CONNECT] Support optional “column” parameter in box, kde and hist plots
• [SPARK-50273] [SC-181178][SS] Improve logging for RocksDB lock acquire/release cases
• [SPARK-50033] [SC-180720][SC-180659][SQL] Add a hint to logical.Aggregate() node
• [SPARK-50163] [16.x][SC-180201][SC-180664][SS] Fix the RocksDB extra acquireLock release due to the completion listener
• [SPARK-50253] [SC-180969][SS] Stream-Stream Join should not fetch checkpoint ID if not supported.
• [SPARK-50255] [SC-180964][PYTHON] Avoid unnecessary casting in compute_hist
• [SPARK-50228] [SC-180780][SQL] Move the RewriteCollationJoin rule to FinishAnalysis
• [SPARK-50001] [SC-179524][PYTHON][PS][CONNECT] Adjust “precision” to be part of kwargs for box plots
• [SPARK-49637] [SC-180160][SQL] Changed error message for INVALID_FRACTION_OF_SECOND
• [SPARK-49530] [SC-180658][PYTHON] Get active session from dataframes
• [SPARK-50195] [SC-180654][CORE] Fix StandaloneRestServer to propagate spark.app.name to SparkSubmit properly
• [SPARK-50229] [SC-180773] Reduce memory usage on driver for wide schemas by reducing the lifetime of AttributeReference objects created during logical planning
• [SPARK-50231] [SC-180815][PYTHON] Make function instr accept Column substring
• [SPARK-49854] [SC-179812][16.x][SQL] Clone artifact manager during session clone
• [SPARK-50028] [SC-180699][CONNECT] Replace global locks in Spark Connect server listener with fine-grained locks
• [SPARK-49615] [SC-180672] [ML] Make all ML feature transformers dataset schema validation conforming “spark.sql.caseSensitive” config.
• [SPARK-50219] [SC-180694][SQL] Refactor ApplyCharTypePadding so that helper methods can be used in single-pass resolver
• [SPARK-50077] [SC-179827][SQL] Introduce a new pattern object for LogicalRelation to help avoiding default full params pattern
• [SPARK-50128] [Backport][16x][SC-180677][SS] Add stateful processor handle APIs using implicit encoders in Scala
• [SPARK-50061] [SC-179961][SQL] Enable analyze table for collated columns
• [SPARK-49993] [SC-180084][SQL] Improve error messages for Sum and Average
• [SPARK-49638] [SC-179665][SQL] Remove the ANSI config suggestion in INVALID_URL
• [SPARK-50204] [SC-180660][SQL] Factor out HiveTableRelation read path resolution
• [SPARK-50193] [SC-180651][SS] Fix exception handling for validating time modes
• [SPARK-50179] [SC-180342][CORE] Make spark.app.name property optional in REST API
• [SPARK-50068] [SC-180300][SQL] Refactor TypeCoercion and AnsiTypeCoercion to separate single node transformations
• [SPARK-49411] [SC-179483][SS] Communicate State Store Checkpoint ID between driver and stateful operators
• [SPARK-50124] [SC-180294][SQL] LIMIT/OFFSET should preserve data ordering
• [SPARK-49506] [SC-180165][SQL] Optimize ArrayBinarySearch for foldable array
• [SPARK-50097] [SC-179908][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_1248: ALTER_TABLE_SERDE_FOR_DATASOURCE_TABLE
• [SPARK-50071] [SC-180159][SQL][PYTHON] Add try_make_timestamp(_ltz and _ntz) and related tests
• [SPARK-50024] Revert “[SC-179515][PYTHON][CONNECT] Switch to use logger instead of warnings module in client”
• [SPARK-50054] [SC-180228][PYTHON][CONNECT] Support histogram plots
• [SPARK-50015] [SC-179964][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_1125: MISSING_DATABASE_FOR_V1_SESSION_CATALOG
• [SPARK-50155] [SC-180240][3.5] Move scala and java files to their default folders
• [SPARK-49980] [SC-180353][CORE][SQL] Fix potential file stream leaks caused by interruption in canceled tasks
• [SPARK-49010] [SC-172304][SQL][XML] Add unit tests for XML schema inference case sensitivity
• [SPARK-50024] [SC-179515][PYTHON][CONNECT] Switch to use logger instead of warnings module in client
• [SPARK-49562] [SC-180211][SQL] Add SQL pipe syntax for aggregation
• [SPARK-49663] [SC-180239][SQL] Enable RTRIM suggestions in collation expressions
• [SPARK-48965] [SC-175926][SQL] Use the correct schema in Dataset#toJSON
• [SPARK-48493] [SC-175893][PYTHON] Enhance Python Datasource Reader with direct Arrow Batch support for improved performance
• [SPARK-49734] [SC-180226][PYTHON] Add seed argument for function shuffle
• [SPARK-50174] [16.x][SC-180253][SQL] Factor out UnresolvedCatalogRelation resolution
• [SPARK-49989] [SC-179512][PYTHON][CONNECT] Support kde/density plots
• [SPARK-49805] [SC-180218][SQL][ML] Remove private[xxx] functions from function.scala
• [SPARK-49808] [SC-179490][SQL] Fix a deadlock in subquery execution due to lazy vals
• [SPARK-49929] [SC-180144][PYTHON][CONNECT] Support box plots
• [SPARK-50008] [SC-179290][PS][CONNECT] Avoid unnecessary operations in attach_distributed_sequence_column
• [SPARK-49767] [SC-180161][PS][CONNECT] Refactor the internal function invocation
• [SPARK-49683] [SC-178341][SQL] Block trim collation
• [SPARK-49939] [SC-178941][SQL] Codegen Support for json_object_keys (by Invoke & RuntimeReplaceable)
• [SPARK-50031] [SC-179582][SQL] Add the TryParseUrl expression
• [SPARK-49766] [SC-178933][SQL] Codegen Support for json_array_length (by Invoke & RuntimeReplaceable)
• [SPARK-50046] [SC-180026][SS] Use stable order of EventTimeWatermark node to calculate watermark
• [SPARK-49540] [SC-180145][PS] Unify the usage of distributed_sequence_id
• [SPARK-50060] [SC-179965][SQL] Disabled conversion between different collated types in TypeCoercion and AnsiTypeCoercion
• [SPARK-49004] [SC-173244][CONNECT] Use separate registry for Column API internal functions
• [SPARK-49811] [SC-177888][SQL]Rename StringTypeAnyCollation
• [SPARK-49202] [SC-180059][PS] Apply ArrayBinarySearch for histogram
• [SPARK-49203] [SC-175734][SQL] Add expression for java.util.Arrays.binarySearch
• [SPARK-50034] [SC-179816][CORE] Fix Misreporting of Fatal Errors as Uncaught Exceptions in SparkUncaughtExceptionHandler
• [SPARK-50093] [SC-179836][SQL] Collations that use ICU should have the version of the used ICU library
• [SPARK-49985] [SC-179644][SQL] Remove support for interval types in Variant
• [SPARK-49103] [SC-173066][CORE] Support spark.master.rest.filters
• [SPARK-50090] [SC-179819] Refactor ResolveBinaryArithmetic to separate single-node transformation
• [SPARK-49902] [SC-179650][SQL] Catch underlying runtime errors in RegExpReplace
• [SPARK-49126] [SC-173342][CORE] Move spark.history.ui.maxApplications config definition to History.scala
• [SPARK-50094] [SC-179820][PYTHON][CONNECT] Better error message when using memory profiler on editors with no line numbers
• [SPARK-50062] [SC-179835][SQL] Support collations by InSet
• [SPARK-50035] [Backport][16x][SC-179530][SS] Add support for explicit handleExpiredTimer function part of the stateful processor
• [SPARK-49982] [SC-179814][SQL] Fix negative caching in InMemoryRelation
• [SPARK-49082] [SC-173350][SQL] Widening type promotions in AvroDeserializer
• [SPARK-50088] [SC-179680][SQL] Refactor UnresolvedStarBase.expand
• [SPARK-49802] [SC-179482][SS] Add support for read change feed for map and list types used in stateful processors
• [SPARK-49846] [SC-179506][SS] Add numUpdatedStateRows and numRemovedStateRows metrics for use with transformWithState operator
• [SPARK-50050] [SC-179643][PYTHON][CONNECT][16.X] Make lit accept str and bool type numpy ndarray
• [SPARK-49821] [SC-179527][SS][PYTHON] Implement MapState and TTL support for TransformWithStateInPandas
• [SPARK-49558] [SC-179485][SQL] Add SQL pipe syntax for LIMIT/OFFSET and ORDER/SORT/CLUSTER/DISTRIBUTE BY
• [SPARK-48144] [SC-165725][LC-4080][SQL] Fix canPlanAsBroadcastHashJoin to respect shuffle join hints
• [SPARK-50051] [SC-179571][PYTHON][CONNECT] Make lit works with empty numpy ndarray
• [SPARK-50018] [SC-179488][SQL] Make AbstractStringType serializable
• [SPARK-50016] [SC-179491][SQL] Improve explicit collation mismatch error
• [SPARK-50010] [SC-179487][SQL] Expand implicit collation mismatch error
• [SPARK-48749] [SC-170116][SQL] Simplify UnaryPositive and eliminate its Catalyst Rules with RuntimeReplaceable
• [SPARK-49857] [SC-178576][SQL] Add storageLevel to Dataset localCheckpoint API
• [SPARK-50058] [SC-179538][SQL] Factor out plan normalization functions to later use those in single-pass Analyzer testing
• [SPARK-50052] [SC-179535][PYTHON][16.X] Make NumpyArrayConverter support empty str ndarray
• [SPARK-47261] [SC-173665][SQL] Assign better name for errors _LEGACY_ERROR_TEMP_1172, _LEGACY_ERROR_TEMP_1173, and _LEGACY_ERROR_TEMP_1174
• [SPARK-49147] [SC-173471][CORE] Mark KryoRegistrator with DeveloperApi interface
• [SPARK-48949] [SC-173070][SQL] SPJ: Runtime partition filtering
• [SPARK-50044] [SC-179523][PYTHON] Refine the docstring of multiple math functions
• [SPARK-48757] [SC-170113][CORE] Make IndexShuffleBlockResolver have explicit constructors
• [SPARK-50037] [SQL] Refactor AttributeSeq.resolve(…)
• [SPARK-48782] [SC-177056][SQL] Add support for executing procedures in catalogs
• [SPARK-49057] [SC-173081][SQL] Do not block the AQE loop when submitting query stages
• [SPARK-48824] [SC-176772][BEHAVE-167][SQL] Add Identity Column SQL syntax
• [SPARK-48773] [SC-170773] Document config “spark.default.parallelism” by config builder framework
• [SPARK-48735] [SC-169810][SQL] Performance Improvement for BIN function
• [SPARK-48900] [SC-172433] Add reason field for all internal calls for job/stage cancellation
• [SPARK-48488] [SC-167605][CORE] Fix methods log[info|warning|error] in SparkSubmit
• [SPARK-48708] [SC-169809][CORE] Remove three unnecessary type registrations from KryoSerializer
• [SPARK-49958] [SC-179312][PYTHON] Python API for string validation functions
• [SPARK-49979] [SC-179264][SQL] Fix AQE hanging issue when collecting twice on a failed plan
• [SPARK-48729] [SC-169795][SQL] Add a UserDefinedFunction interface to represent a SQL function
• [SPARK-49997] [SC-179279][SQL] Integrate _LEGACY_ERROR_TEMP_2165 into MALFORMED_RECORD_IN_PARSING
• [SPARK-49259] [SC-179271][SS]Size based partition creation during kafka read
• [SPARK-48129] [SC-165006][PYTHON] Provide a constant table schema in PySpark for querying structured logs
• [SPARK-49951] [SC-179259][SQL] Assign proper error condition for LEGACY_ERROR_TEMP(1099|3085)
• [SPARK-49971] [SC-179278][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_1097
• [SPARK-49998] [SC-179277][SQL] Integrate _LEGACY_ERROR_TEMP_1252 into EXPECT_TABLE_NOT_VIEW
• [SPARK-49876] [SC-179262][CONNECT] Get rid of global locks from Spark Connect Service
• [SPARK-49957] [SC-179202][SQL] Scala API for string validation functions
• [SPARK-48480] [SC-173055][SS][CONNECT] StreamingQueryListener should not be affected by spark.interrupt()
• [SPARK-49643] [SC-179239][SQL] Merge _LEGACY_ERROR_TEMP_2042 into ARITHMETIC_OVERFLOW
• [SPARK-49959] [SC-179109][SQL] Fix ColumnarArray.copy() to read nulls from the correct offset
• [SPARK-49956] Revert “[SC-179070] Disabled collations with collect_set expression”
• [SPARK-49987] [SC-179180][SQL] Fix the error prompt when seedExpression is non-foldable in randstr
• [SPARK-49948] [SC-179158][PS][CONNECT] Add parameter “precision” to pandas on Spark box plot
• [SPARK-49970] [SC-179167][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2069
• [SPARK-49916] [SC-179108][SQL] Throw appropriate Exception for type mismatch between ColumnType and data type in some rows
• [SPARK-49956] [SC-179070] Disabled collations with collect_set expression
• [SPARK-49974] [16.x][SC-179071][SQL] Move resolveRelations(…) out of the Analyzer.scala
• [SPARK-47259] [SC-176437][SQL] Assign names to error conditions for interval errors
• [SPARK-47430] [SC-173679][SQL] Rework group by map type to fix bind reference exception
• [SPARK-49067] [SC-172616][SQL] Move utf-8 literal into internal methods of UrlCodec class
• [SPARK-49955] [SC-178989][SQL] null value does not mean corrupted file when parsing JSON string RDD
• [SPARK-49393] [SC-175212][SQL] Fail by default in deprecated catalog plugin APIs
• [SPARK-49952] [SC-178985][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_1142
• [SPARK-49405] [SC-175224][SQL] Restrict charsets in JsonOptions
• [SPARK-49892] [SC-178975][SQL] Assign proper error class for _LEGACY_ERROR_TEMP_1136
• [SPARK-49904] [SC-178973][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2140
• [SPARK-47257] [SC-174244][SQL] Assign names to error classes _LEGACY_ERROR_TEMP_105[3-4] and _LEGACY_ERROR_TEMP_1331
• [SPARK-49915] [SC-178869][SQL] Handle zeros and ones in ReorderAssociativeOperator
• [SPARK-49891] [SC-178970][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2271
• [SPARK-49829] [ES-1254513] Revise the optimization on adding input to state store in stream-stream join (correctness fix)
• [SPARK-49918] [SC-178748][CORE] Use read-only access to conf in SparkContext where appropriate
• [SPARK-49666] [SC-177891][SQL] Add feature flag for trim collation feature
• [SPARK-48885] [SC-171050][SQL] Make some subclasses of RuntimeReplaceable override replacement to lazy val
• [SPARK-49932] [SC-178931][CORE] Use tryWithResource release JsonUtils#toJsonString resources to avoid memory leaks
• [SPARK-49949] [SC-178978][PS] Avoid unnecessary analyze task in attach_sequence_column
• [SPARK-49924] [SC-178935][SQL] Keep containsNull after ArrayCompact replacement
• [SPARK-49905] [ES-1265435] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
• [SPARK-49615] [SC-178877] Bugfix: Make ML column schema validation conforms with spark config spark.sql.caseSensitive.
• [SPARK-49121] [SC-174787][SQL] Support from_protobuf and to_protobuf for SQL functions
• [SPARK-47496] [SC-160529][SQL] Java SPI Support for dynamic JDBC dialect registering
• [SPARK-49359] [SC-174895][SQL] Allow StagedTableCatalog implementations to fall back to non-atomic write
• [SPARK-49895] [SC-178543][SQL][ES-1270338] Improve error when encountering trailing comma in SELECT clause
• [SPARK-47945] [SC-163457][SQL] MsSQLServer: Document Mapping Spark SQL Data Types from Microsoft SQL Server and add tests
• [SPARK-46037] [SC-175495][SQL] Correctness fix for Shuffled Hash Join build left without codegen
• [SPARK-47813] [SC-162615][SQL] Replace getArrayDimension with updateExtraColumnMeta
• [SPARK-49542] [SC-178765][SQL] Partition transform exception evaluate error
• [SPARK-47172] [SC-169537][CORE] Add support for AES-GCM for RPC encryption
• [SPARK-47666] [SC-161476][SQL] Fix NPE when reading mysql bit array as LongType
• [SPARK-48947] [SC-174239][SQL] Use lowercased charset name to decrease cache missing in Charset.forName
• [SPARK-49909] [SC-178552][SQL] Fix the pretty name of some expressions
• [SPARK-47647] [SC-161377][SQL] Make MySQL data source able to read bit(n>1) as BinaryType like Postgres
• [SPARK-46622] [SC-153004][CORE] Override toString method for o.a.s.network.shuffledb.StoreVersion
• [SPARK-48961] [SC-171910][PYTHON] Make the parameter naming of PySparkException consistent with JVM
• [SPARK-49889] [SC-178474][PYTHON] Add argument trim for functionstrim/ltrim/rtrim
• [SPARK-49863] [ES-1184693][SC-178546][SQL] Fix NormalizeFloatingNumbers to preserve nullability of nested structs
• [SPARK-47537] [SC-160747][SQL] Fix error data type mapping on MySQL Connector/J
• [SPARK-47628] [SC-161257][SQL] Fix Postgres bit array issue ‘Cannot cast to boolean’
• [SPARK-49843] [SC-178084][ES-1268164][SQL] Fix change comment on char/varchar columns
• [SPARK-49549] [SC-178764][SQL] Assign a name to the error conditions _LEGACY_ERROR_TEMP_3055, 3146
• [SPARK-49782] [SC-178756][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
• [SPARK-49791] [SC-177730][SQL] Make DelegatingCatalogExtension more extendable
• [SPARK-49559] [SC-178551][SQL] Add SQL pipe syntax for set operations
• [SPARK-49906] [SC-178536][SQL] Introduce and use CONFLICTING_DIRECTORY_STRUCTURES error for PartitioningUtils
• [SPARK-49877] [SC-178392][SQL] Change classifyException function signature: add isRuntime argument
• [SPARK-47501] [SC-160532][SQL] Add convertDateToDate like the existing convertTimestampToTimestamp for JdbcDialect
• [SPARK-49044] [SC-175746][SQL] ValidateExternalType should return child in error
• [SPARK-47462] [SC-160460][SQL] Align mappings of other unsigned numeric types with TINYINT in MySQLDialect
• [SPARK-47435] [SC-160129][SQL] Fix overflow issue of MySQL UNSIGNED TINYINT caused by SPARK-45561
• [SPARK-49398] [SC-176377][SQL] Improve the error for parameters in the query of CACHE TABLE and CREATE VIEW
• [SPARK-47263] [SC-177012][SQL] Assign names to the legacy conditions _LEGACY_ERROR_TEMP_13[44-46]
• [SPARK-49605] [SC-176605][SQL] Fix the prompt when ascendingOrder is DataTypeMismatch in SortArray
• [SPARK-49806] [SC-178304][SQL][CONNECT] Remove redundant blank space after show in Scala and Connect clients
• [SPARK-47258] [SC-175032][SQL] Assign names to error classes _LEGACY_ERROR_TEMP_127[0-5]
• [SPARK-49564] [SC-178461][SQL] Add SQL pipe syntax for the JOIN operator
• [SPARK-49836] [SC-178339][SQL][SS] Fix possibly broken query when window is provided to window/session_window fn
• [SPARK-47390] [SC-163306][SC-159312][SPARK-47396][SC-159376][SPARK-47406] Handle SQL timestamps mapping for Postgres and MySQL
• [SPARK-49776] [SC-177818][PYTHON][CONNECT] Support pie plots
• [SPARK-49824] [SC-178251][SS][CONNECT] Improve logging in SparkConnectStreamingQueryCache
• [SPARK-49894] [SC-178460][PYTHON][CONNECT] Refine the string representation of column field operations
• [SPARK-49764] [SC-177491][PYTHON][CONNECT] Support area plots
• [SPARK-49694] [SC-177376][PYTHON][CONNECT] Support scatter plots
• [SPARK-49744] [SC-178363][SS][PYTHON] Implement TTL support for ListState in TransformWithStateInPandas
• [SPARK-49879] [SC-178334][CORE] Move TransportCipherUtil to a separate file to eliminate Java compilation warnings
• [SPARK-49866] [SC-178353][SQL] Improve the error message for describe table with partition columns
• [SPARK-49867] [SC-178302][ES-1260570][SQL] Improve the error message when index is out of bounds when calling GetColumnByOrdinal
• [SPARK-49607] [SC-177373][PYTHON] Update the sampling approach for sampled based plots
• [SPARK-49626] [SC-177276][PYTHON][CONNECT] Support horizontal and vertical bar plots
• [SPARK-49531] [SC-177770][PYTHON][CONNECT] Support line plot with plotly backend
• [SPARK-49444] [SC-177692][ES-1203248][SQL] Modified UnivocityParser to throw runtime exceptions caused by ArrayIndexOutOfBounds with more user-oriented messages
• [SPARK-49870] [SC-178262][PYTHON] Add Python 3.13 support in Spark Classic
• [SPARK-49560] [SC-178121][SQL] Add SQL pipe syntax for the TABLESAMPLE operator
• [SPARK-49864] [SC-178305][SQL] Improve message of BINARY_ARITHMETIC_OVERFLOW
• [SPARK-48780] [SC-170274][SQL] Make errors in NamedParametersSupport generic to handle functions and procedures
• [SPARK-49358] [SC-178158][SQL] Mode expression for map types with collated strings
• [SPARK-47341] [SC-178157][SQL] Fix inaccurate documentation of RuntimeConfig#get
• [SPARK-48357] [SC-178153][SQL] Support for LOOP statement
• [SPARK-49845] [SC-178059][CORE] Make appArgs and environmentVariables optional in REST API
• [SPARK-49246] [SC-174679][SQL] TableCatalog#loadTable should indicate if it’s for writing
• [SPARK-48048] [SC-177525][SC-164846][CONNECT][SS] Added client side listener support for Scala
• [SPARK-48700] [SC-177978][SQL] Mode expression for complex types (all collations)
• [SPARK-48196] [SC-177899][SQL] Turn QueryExecution lazy val plans into LazyTry
• [SPARK-49749] [16.x][SC-177877][CORE] Change log level to debug in BlockManagerInfo
• [SPARK-49561] [SC-177897][SQL] Add SQL pipe syntax for the PIVOT and UNPIVOT operators
• [SPARK-49823] [SC-177885][SS] Avoid flush during shutdown in rocksdb close path
• [SPARK-49820] [SC-177855] [PYTHON] Change raise IOError to raise OSError
• [SPARK-49653] [SC-177266][SQL] Single join for correlated scalar subqueries
• [SPARK-49552] [SC-177477][PYTHON] Add DataFrame API support for new ‘randstr’ and ‘uniform’ SQL functions
• [SPARK-48303] [16.x][SC-166251][CORE] Reorganize LogKeys
• [SPARK-49656] [16x][Backport][SS] Add support for state variables with value state collection types and read change feed options
• [SPARK-48112] [SC-165129][CONNECT] Expose session in SparkConnectPlanner to plugins
• [SPARK-48126] [16.x][SC-165309][Core] Make spark.log.structuredLogging.enabled effective
• [SPARK-49505] [SC-176873][SQL] Create new SQL functions “randstr” and “uniform” to generate random strings or numbers within ranges
• [SPARK-49463] [SC-177474] Support ListState for TransformWithStateInPandas
• [SPARK-48131] [SC-165007][Core] Unify MDC key mdc.taskName and task_name
• [SPARK-49557] [SC-177227][SQL] Add SQL pipe syntax for the WHERE operator
• [SPARK-49323] [16.x][SC-174689][CONNECT] Move MockObserver from Spark Connect Server’s test folder to the Server’s main folder
• [SPARK-49745] [SC-177501][SS] Add change to read registered timers through state data source reader
• [SPARK-49772] [16.x][SC-177478][SC-177214][SS] Remove ColumnFamilyOptions and add configs directly to dbOptions in RocksDB

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

• Operating System: Ubuntu 24.04.1 LTS
• Java: Zulu17.54+21-CA
• Scala: 2.12.18
• Python: 3.12.3
• R: 4.4.0
• Delta Lake: 3.2.1

Installed Python libraries

Library	Version	Library	Version	Library	Version
annotated-types	0.7.0	asttokens	2.0.5	astunparse	1.6.3
autocommand	2.2.2	azure-core	1.31.0	azure-storage-blob	12.23.0
azure-storage-file-datalake	12.17.0	backports.tarfile	1.2.0	black	24.4.2
blinker	1.7.0	boto3	1.34.69	botocore	1.34.69
cachetools	5.3.3	certifi	2024.6.2	cffi	1.16.0
chardet	4.0.0	charset-normalizer	2.0.4	click	8.1.7
cloudpickle	2.2.1	comm	0.2.1	contourpy	1.2.0
cryptography	42.0.5	cycler	0.11.0	Cython	3.0.11
databricks-sdk	0.30.0	dbus-python	1.3.2	debugpy	1.6.7
decorator	5.1.1	Deprecated	1.2.14	distlib	0.3.8
docstring-to-markdown	0.11	entrypoints	0.4	executing	0.8.3
facets-overview	1.1.1	filelock	3.15.4	fonttools	4.51.0
gitdb	4.0.11	GitPython	3.1.37	google-api-core	2.20.0
google-auth	2.35.0	google-cloud-core	2.4.1	google-cloud-storage	2.18.2
google-crc32c	1.6.0	google-resumable-media	2.7.2	googleapis-common-protos	1.65.0
grpcio	1.60.0	grpcio-status	1.60.0	httplib2	0.20.4
idna	3.7	importlib-metadata	6.0.0	importlib_resources	6.4.0
inflect	7.3.1	ipyflow-core	0.0.201	ipykernel	6.28.0
ipython	8.25.0	ipython-genutils	0.2.0	ipywidgets	7.7.2
isodate	0.6.1	jaraco.context	5.3.0	jaraco.functools	4.0.1
jaraco.text	3.12.1	jedi	0.19.1	jmespath	1.0.1
joblib	1.4.2	jupyter_client	8.6.0	jupyter_core	5.7.2
kiwisolver	1.4.4	launchpadlib	1.11.0	lazr.restfulclient	0.14.6
lazr.uri	1.0.6	matplotlib	3.8.4	matplotlib-inline	0.1.6
mccabe	0.7.0	mlflow-skinny	2.15.1	more-itertools	10.3.0
mypy	1.10.0	mypy-extensions	1.0.0	nest-asyncio	1.6.0
nodeenv	1.9.1	numpy	1.26.4	oauthlib	3.2.2
opentelemetry-api	1.27.0	opentelemetry-sdk	1.27.0	opentelemetry-semantic-conventions	0.48b0
packaging	24.1	pandas	1.5.3	parso	0.8.3
pathspec	0.10.3	patsy	0.5.6	pexpect	4.8.0
pillow	10.3.0	pip	24.2	platformdirs	3.10.0
plotly	5.22.0	pluggy	1.0.0	prompt-toolkit	3.0.43
proto-plus	1.24.0	protobuf	4.24.1	psutil	5.9.0
psycopg2	2.9.3	ptyprocess	0.7.0	pure-eval	0.2.2
pyarrow	15.0.2	pyasn1	0.4.8	pyasn1-modules	0.2.8
pyccolo	0.0.65	pycparser	2.21	pydantic	2.8.2
pydantic_core	2.20.1	pyflakes	3.2.0	Pygments	2.15.1
PyGObject	3.48.2	PyJWT	2.7.0	pyodbc	5.0.1
pyparsing	3.0.9	pyright	1.1.294	python-dateutil	2.9.0.post0
python-lsp-jsonrpc	1.1.2	python-lsp-server	1.10.0	pytoolconfig	1.2.6
pytz	2024.1	PyYAML	6.0.1	pyzmq	25.1.2
requests	2.32.2	rope	1.12.0	rsa	4.9
s3transfer	0.10.2	scikit-learn	1.4.2	scipy	1.13.1
seaborn	0.13.2	setuptools	74.0.0	six	1.16.0
smmap	5.0.0	sqlparse	0.5.1	ssh-import-id	5.11
stack-data	0.2.0	statsmodels	0.14.2	tenacity	8.2.2
threadpoolctl	2.2.0	tokenize-rt	4.2.1	tomli	2.0.1
tornado	6.4.1	traitlets	5.14.3	typeguard	4.3.0
types-protobuf	3.20.3	types-psutil	5.9.0	types-pytz	2023.3.1.1
types-PyYAML	6.0.0	types-requests	2.31.0.0	types-setuptools	68.0.0.0
types-six	1.16.0	types-urllib3	1.26.25.14	typing_extensions	4.11.0
ujson	5.10.0	unattended-upgrades	0.1	urllib3	1.26.16
virtualenv	20.26.2	wadllib	1.3.6	wcwidth	0.2.5
whatthepatch	1.0.2	wheel	0.43.0	wrapt	1.14.1
yapf	0.33.0	zipp	3.17.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2024-08-04.

Library	Version	Library	Version	Library	Version
arrow	16.1.0	askpass	1.2.0	assertthat	0.2.1
backports	1.5.0	base	4.4.0	base64enc	0.1-3
bigD	0.2.0	bit	4.0.5	bit64	4.0.5
bitops	1.0-8	blob	1.2.4	boot	1.3-30
brew	1.0-10	brio	1.1.5	broom	1.0.6
bslib	0.8.0	cachem	1.1.0	callr	3.7.6
caret	6.0-94	cellranger	1.1.0	chron	2.3-61
class	7.3-22	cli	3.6.3	clipr	0.8.0
clock	0.7.1	cluster	2.1.6	codetools	0.2-20
colorspace	2.1-1	commonmark	1.9.1	compiler	4.4.0
config	0.3.2	conflicted	1.2.0	cpp11	0.4.7
crayon	1.5.3	credentials	2.0.1	curl	5.2.1
data.table	1.15.4	datasets	4.4.0	DBI	1.2.3
dbplyr	2.5.0	desc	1.4.3	devtools	2.4.5
diagram	1.6.5	diffobj	0.3.5	digest	0.6.36
downlit	0.4.4	dplyr	1.1.4	dtplyr	1.3.1
e1071	1.7-14	ellipsis	0.3.2	evaluate	0.24.0
fansi	1.0.6	farver	2.1.2	fastmap	1.2.0
fontawesome	0.5.2	forcats	1.0.0	foreach	1.5.2
foreign	0.8-86	forge	0.2.0	fs	1.6.4
future	1.34.0	future.apply	1.11.2	gargle	1.5.2
generics	0.1.3	gert	2.1.0	ggplot2	3.5.1
gh	1.4.1	git2r	0.33.0	gitcreds	0.1.2
glmnet	4.1-8	globals	0.16.3	glue	1.7.0
googledrive	2.1.1	googlesheets4	1.1.1	gower	1.0.1
graphics	4.4.0	grDevices	4.4.0	grid	4.4.0
gridExtra	2.3	gsubfn	0.7	gt	0.11.0
gtable	0.3.5	hardhat	1.4.0	haven	2.5.4
highr	0.11	hms	1.1.3	htmltools	0.5.8.1
htmlwidgets	1.6.4	httpuv	1.6.15	httr	1.4.7
httr2	1.0.2	ids	1.0.1	ini	0.3.1
ipred	0.9-15	isoband	0.2.7	iterators	1.0.14
jquerylib	0.1.4	jsonlite	1.8.8	juicyjuice	0.1.0
KernSmooth	2.23-22	knitr	1.48	labeling	0.4.3
later	1.3.2	lattice	0.22-5	lava	1.8.0
lifecycle	1.0.4	listenv	0.9.1	lubridate	1.9.3
magrittr	2.0.3	markdown	1.13	MASS	7.3-60.0.1
Matrix	1.6-5	memoise	2.0.1	methods	4.4.0
mgcv	1.9-1	mime	0.12	miniUI	0.1.1.1
mlflow	2.14.1	ModelMetrics	1.2.2.2	modelr	0.1.11
munsell	0.5.1	nlme	3.1-165	nnet	7.3-19
numDeriv	2016.8-1.1	openssl	2.2.0	parallel	4.4.0
parallelly	1.38.0	pillar	1.9.0	pkgbuild	1.4.4
pkgconfig	2.0.3	pkgdown	2.1.0	pkgload	1.4.0
plogr	0.2.0	plyr	1.8.9	praise	1.0.0
prettyunits	1.2.0	pROC	1.18.5	processx	3.8.4
prodlim	2024.06.25	profvis	0.3.8	progress	1.2.3
progressr	0.14.0	promises	1.3.0	proto	1.0.0
proxy	0.4-27	ps	1.7.7	purrr	1.0.2
R6	2.5.1	ragg	1.3.2	randomForest	4.7-1.1
rappdirs	0.3.3	rcmdcheck	1.4.0	RColorBrewer	1.1-3
Rcpp	1.0.13	RcppEigen	0.3.4.0.0	reactable	0.4.4
reactR	0.6.0	readr	2.1.5	readxl	1.4.3
recipes	1.1.0	rematch	2.0.0	rematch2	2.1.2
remotes	2.5.0	reprex	2.1.1	reshape2	1.4.4
rlang	1.1.4	rmarkdown	2.27	RODBC	1.3-23
roxygen2	7.3.2	rpart	4.1.23	rprojroot	2.0.4
Rserve	1.8-13	RSQLite	2.3.7	rstudioapi	0.16.0
rversions	2.1.2	rvest	1.0.4	sass	0.4.9
scales	1.3.0	selectr	0.4-2	sessioninfo	1.2.2
shape	1.4.6.1	shiny	1.9.1	sourcetools	0.1.7-1
sparklyr	1.8.6	spatial	7.3-17	splines	4.4.0
sqldf	0.4-11	SQUAREM	2021.1	stats	4.4.0
stats4	4.4.0	stringi	1.8.4	stringr	1.5.1
survival	3.6-4	swagger	5.17.14.1	sys	3.4.2
systemfonts	1.1.0	tcltk	4.4.0	testthat	3.2.1.1
textshaping	0.4.0	tibble	3.2.1	tidyr	1.3.1
tidyselect	1.2.1	tidyverse	2.0.0	timechange	0.3.0
timeDate	4032.109	tinytex	0.52	tools	4.4.0
tzdb	0.4.0	urlchecker	1.0.1	usethis	3.0.0
utf8	1.2.4	utils	4.4.0	uuid	1.2-1
V8	4.4.2	vctrs	0.6.5	viridisLite	0.4.2
vroom	1.6.5	waldo	0.5.2	whisker	0.4.1
withr	3.0.1	xfun	0.46	xml2	1.3.6
xopen	1.0.1	xtable	1.8-4	yaml	2.3.10
zeallot	0.1.0	zip	2.3.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID	Artifact ID	Version
antlr	antlr	2.7.7
com.amazonaws	amazon-kinesis-client	1.12.0
com.amazonaws	aws-java-sdk-autoscaling	1.12.638
com.amazonaws	aws-java-sdk-cloudformation	1.12.638
com.amazonaws	aws-java-sdk-cloudfront	1.12.638
com.amazonaws	aws-java-sdk-cloudhsm	1.12.638
com.amazonaws	aws-java-sdk-cloudsearch	1.12.638
com.amazonaws	aws-java-sdk-cloudtrail	1.12.638
com.amazonaws	aws-java-sdk-cloudwatch	1.12.638
com.amazonaws	aws-java-sdk-cloudwatchmetrics	1.12.638
com.amazonaws	aws-java-sdk-codedeploy	1.12.638
com.amazonaws	aws-java-sdk-cognitoidentity	1.12.638
com.amazonaws	aws-java-sdk-cognitosync	1.12.638
com.amazonaws	aws-java-sdk-config	1.12.638
com.amazonaws	aws-java-sdk-core	1.12.638
com.amazonaws	aws-java-sdk-datapipeline	1.12.638
com.amazonaws	aws-java-sdk-directconnect	1.12.638
com.amazonaws	aws-java-sdk-directory	1.12.638
com.amazonaws	aws-java-sdk-dynamodb	1.12.638
com.amazonaws	aws-java-sdk-ec2	1.12.638
com.amazonaws	aws-java-sdk-ecs	1.12.638
com.amazonaws	aws-java-sdk-efs	1.12.638
com.amazonaws	aws-java-sdk-elasticache	1.12.638
com.amazonaws	aws-java-sdk-elasticbeanstalk	1.12.638
com.amazonaws	aws-java-sdk-elasticloadbalancing	1.12.638
com.amazonaws	aws-java-sdk-elastictranscoder	1.12.638
com.amazonaws	aws-java-sdk-emr	1.12.638
com.amazonaws	aws-java-sdk-glacier	1.12.638
com.amazonaws	aws-java-sdk-glue	1.12.638
com.amazonaws	aws-java-sdk-iam	1.12.638
com.amazonaws	aws-java-sdk-importexport	1.12.638
com.amazonaws	aws-java-sdk-kinesis	1.12.638
com.amazonaws	aws-java-sdk-kms	1.12.638
com.amazonaws	aws-java-sdk-lambda	1.12.638
com.amazonaws	aws-java-sdk-logs	1.12.638
com.amazonaws	aws-java-sdk-machinelearning	1.12.638
com.amazonaws	aws-java-sdk-opsworks	1.12.638
com.amazonaws	aws-java-sdk-rds	1.12.638
com.amazonaws	aws-java-sdk-redshift	1.12.638
com.amazonaws	aws-java-sdk-route53	1.12.638
com.amazonaws	aws-java-sdk-s3	1.12.638
com.amazonaws	aws-java-sdk-ses	1.12.638
com.amazonaws	aws-java-sdk-simpledb	1.12.638
com.amazonaws	aws-java-sdk-simpleworkflow	1.12.638
com.amazonaws	aws-java-sdk-sns	1.12.638
com.amazonaws	aws-java-sdk-sqs	1.12.638
com.amazonaws	aws-java-sdk-ssm	1.12.638
com.amazonaws	aws-java-sdk-storagegateway	1.12.638
com.amazonaws	aws-java-sdk-sts	1.12.638
com.amazonaws	aws-java-sdk-support	1.12.638
com.amazonaws	aws-java-sdk-swf-libraries	1.11.22
com.amazonaws	aws-java-sdk-workspaces	1.12.638
com.amazonaws	jmespath-java	1.12.638
com.clearspring.analytics	stream	2.9.6
com.databricks	Rserve	1.8-3
com.databricks	databricks-sdk-java	0.27.0
com.databricks	jets3t	0.7.1-0
com.databricks.scalapb	scalapb-runtime_2.12	0.4.15-10
com.esotericsoftware	kryo-shaded	4.0.2
com.esotericsoftware	minlog	1.3.0
com.fasterxml	classmate	1.3.4
com.fasterxml.jackson.core	jackson-annotations	2.15.2
com.fasterxml.jackson.core	jackson-core	2.15.2
com.fasterxml.jackson.core	jackson-databind	2.15.2
com.fasterxml.jackson.dataformat	jackson-dataformat-cbor	2.15.2
com.fasterxml.jackson.dataformat	jackson-dataformat-yaml	2.15.2
com.fasterxml.jackson.datatype	jackson-datatype-joda	2.15.2
com.fasterxml.jackson.datatype	jackson-datatype-jsr310	2.16.0
com.fasterxml.jackson.module	jackson-module-paranamer	2.15.2
com.fasterxml.jackson.module	jackson-module-scala_2.12	2.15.2
com.github.ben-manes.caffeine	caffeine	2.9.3
com.github.fommil	jniloader	1.1
com.github.fommil.netlib	native_ref-java	1.1
com.github.fommil.netlib	native_ref-java	1.1-natives
com.github.fommil.netlib	native_system-java	1.1
com.github.fommil.netlib	native_system-java	1.1-natives
com.github.fommil.netlib	netlib-native_ref-linux-x86_64	1.1-natives
com.github.fommil.netlib	netlib-native_system-linux-x86_64	1.1-natives
com.github.luben	zstd-jni	1.5.5-4
com.github.wendykierp	JTransforms	3.1
com.google.code.findbugs	jsr305	3.0.0
com.google.code.gson	gson	2.10.1
com.google.crypto.tink	tink	1.9.0
com.google.errorprone	error_prone_annotations	2.10.0
com.google.flatbuffers	flatbuffers-java	23.5.26
com.google.guava	guava	15.0
com.google.protobuf	protobuf-java	3.25.1
com.helger	profiler	1.1.1
com.ibm.icu	icu4j	75.1
com.jcraft	jsch	0.1.55
com.jolbox	bonecp	0.8.0.RELEASE
com.lihaoyi	sourcecode_2.12	0.1.9
com.microsoft.azure	azure-data-lake-store-sdk	2.3.9
com.microsoft.sqlserver	mssql-jdbc	11.2.2.jre8
com.microsoft.sqlserver	mssql-jdbc	11.2.3.jre8
com.ning	compress-lzf	1.1.2
com.sun.mail	javax.mail	1.5.2
com.sun.xml.bind	jaxb-core	2.2.11
com.sun.xml.bind	jaxb-impl	2.2.11
com.tdunning	json	1.8
com.thoughtworks.paranamer	paranamer	2.8
com.trueaccord.lenses	lenses_2.12	0.4.12
com.twitter	chill-java	0.10.0
com.twitter	chill_2.12	0.10.0
com.twitter	util-app_2.12	7.1.0
com.twitter	util-core_2.12	7.1.0
com.twitter	util-function_2.12	7.1.0
com.twitter	util-jvm_2.12	7.1.0
com.twitter	util-lint_2.12	7.1.0
com.twitter	util-registry_2.12	7.1.0
com.twitter	util-stats_2.12	7.1.0
com.typesafe	config	1.4.3
com.typesafe.scala-logging	scala-logging_2.12	3.7.2
com.uber	h3	3.7.3
com.univocity	univocity-parsers	2.9.1
com.zaxxer	HikariCP	4.0.3
commons-cli	commons-cli	1.5.0
commons-codec	commons-codec	1.16.0
commons-collections	commons-collections	3.2.2
commons-dbcp	commons-dbcp	1.4
commons-fileupload	commons-fileupload	1.5
commons-httpclient	commons-httpclient	3.1
commons-io	commons-io	2.13.0
commons-lang	commons-lang	2.6
commons-logging	commons-logging	1.1.3
commons-pool	commons-pool	1.5.4
dev.ludovic.netlib	arpack	3.0.3
dev.ludovic.netlib	blas	3.0.3
dev.ludovic.netlib	lapack	3.0.3
info.ganglia.gmetric4j	gmetric4j	1.0.10
io.airlift	aircompressor	0.27
io.delta	delta-sharing-client_2.12	1.2.2
io.dropwizard.metrics	metrics-annotation	4.2.19
io.dropwizard.metrics	metrics-core	4.2.19
io.dropwizard.metrics	metrics-graphite	4.2.19
io.dropwizard.metrics	metrics-healthchecks	4.2.19
io.dropwizard.metrics	metrics-jetty9	4.2.19
io.dropwizard.metrics	metrics-jmx	4.2.19
io.dropwizard.metrics	metrics-json	4.2.19
io.dropwizard.metrics	metrics-jvm	4.2.19
io.dropwizard.metrics	metrics-servlets	4.2.19
io.netty	netty-all	4.1.108.Final
io.netty	netty-buffer	4.1.108.Final
io.netty	netty-codec	4.1.108.Final
io.netty	netty-codec-http	4.1.108.Final
io.netty	netty-codec-http2	4.1.108.Final
io.netty	netty-codec-socks	4.1.108.Final
io.netty	netty-common	4.1.108.Final
io.netty	netty-handler	4.1.108.Final
io.netty	netty-handler-proxy	4.1.108.Final
io.netty	netty-resolver	4.1.108.Final
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-db-r16
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-db-r16-linux-aarch_64
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-db-r16-linux-x86_64
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-db-r16-osx-aarch_64
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-db-r16-osx-x86_64
io.netty	netty-tcnative-boringssl-static	2.0.61.Final-db-r16-windows-x86_64
io.netty	netty-tcnative-classes	2.0.61.Final
io.netty	netty-transport	4.1.108.Final
io.netty	netty-transport-classes-epoll	4.1.108.Final
io.netty	netty-transport-classes-kqueue	4.1.108.Final
io.netty	netty-transport-native-epoll	4.1.108.Final
io.netty	netty-transport-native-epoll	4.1.108.Final-linux-aarch_64
io.netty	netty-transport-native-epoll	4.1.108.Final-linux-riscv64
io.netty	netty-transport-native-epoll	4.1.108.Final-linux-x86_64
io.netty	netty-transport-native-kqueue	4.1.108.Final-osx-aarch_64
io.netty	netty-transport-native-kqueue	4.1.108.Final-osx-x86_64
io.netty	netty-transport-native-unix-common	4.1.108.Final
io.prometheus	simpleclient	0.7.0
io.prometheus	simpleclient_common	0.7.0
io.prometheus	simpleclient_dropwizard	0.7.0
io.prometheus	simpleclient_pushgateway	0.7.0
io.prometheus	simpleclient_servlet	0.7.0
io.prometheus.jmx	collector	0.12.0
jakarta.annotation	jakarta.annotation-api	1.3.5
jakarta.servlet	jakarta.servlet-api	4.0.3
jakarta.validation	jakarta.validation-api	2.0.2
jakarta.ws.rs	jakarta.ws.rs-api	2.1.6
javax.activation	activation	1.1.1
javax.el	javax.el-api	2.2.4
javax.jdo	jdo-api	3.0.1
javax.transaction	jta	1.1
javax.transaction	transaction-api	1.1
javax.xml.bind	jaxb-api	2.2.11
javolution	javolution	5.5.1
jline	jline	2.14.6
joda-time	joda-time	2.12.1
net.java.dev.jna	jna	5.8.0
net.razorvine	pickle	1.3
net.sf.jpam	jpam	1.1
net.sf.opencsv	opencsv	2.3
net.sf.supercsv	super-csv	2.2.0
net.snowflake	snowflake-ingest-sdk	0.9.6
net.sourceforge.f2j	arpack_combined_all	0.1
org.acplt.remotetea	remotetea-oncrpc	1.1.2
org.antlr	ST4	4.0.4
org.antlr	antlr-runtime	3.5.2
org.antlr	antlr4-runtime	4.9.3
org.antlr	stringtemplate	3.2.1
org.apache.ant	ant	1.10.11
org.apache.ant	ant-jsch	1.10.11
org.apache.ant	ant-launcher	1.10.11
org.apache.arrow	arrow-format	15.0.0
org.apache.arrow	arrow-memory-core	15.0.0
org.apache.arrow	arrow-memory-netty	15.0.0
org.apache.arrow	arrow-vector	15.0.0
org.apache.avro	avro	1.11.3
org.apache.avro	avro-ipc	1.11.3
org.apache.avro	avro-mapred	1.11.3
org.apache.commons	commons-collections4	4.4
org.apache.commons	commons-compress	1.23.0
org.apache.commons	commons-crypto	1.1.0
org.apache.commons	commons-lang3	3.12.0
org.apache.commons	commons-math3	3.6.1
org.apache.commons	commons-text	1.10.0
org.apache.curator	curator-client	2.13.0
org.apache.curator	curator-framework	2.13.0
org.apache.curator	curator-recipes	2.13.0
org.apache.datasketches	datasketches-java	3.1.0
org.apache.datasketches	datasketches-memory	2.0.0
org.apache.derby	derby	10.14.2.0
org.apache.hadoop	hadoop-client-runtime	3.3.6
org.apache.hive	hive-beeline	2.3.9
org.apache.hive	hive-cli	2.3.9
org.apache.hive	hive-jdbc	2.3.9
org.apache.hive	hive-llap-client	2.3.9
org.apache.hive	hive-llap-common	2.3.9
org.apache.hive	hive-serde	2.3.9
org.apache.hive	hive-shims	2.3.9
org.apache.hive	hive-storage-api	2.8.1
org.apache.hive.shims	hive-shims-0.23	2.3.9
org.apache.hive.shims	hive-shims-common	2.3.9
org.apache.hive.shims	hive-shims-scheduler	2.3.9
org.apache.httpcomponents	httpclient	4.5.14
org.apache.httpcomponents	httpcore	4.4.16
org.apache.ivy	ivy	2.5.2
org.apache.logging.log4j	log4j-1.2-api	2.22.1
org.apache.logging.log4j	log4j-api	2.22.1
org.apache.logging.log4j	log4j-core	2.22.1
org.apache.logging.log4j	log4j-layout-template-json	2.22.1
org.apache.logging.log4j	log4j-slf4j2-impl	2.22.1
org.apache.orc	orc-core	1.9.2-shaded-protobuf
org.apache.orc	orc-mapreduce	1.9.2-shaded-protobuf
org.apache.orc	orc-shims	1.9.2
org.apache.thrift	libfb303	0.9.3
org.apache.thrift	libthrift	0.12.0
org.apache.ws.xmlschema	xmlschema-core	2.3.0
org.apache.xbean	xbean-asm9-shaded	4.23
org.apache.yetus	audience-annotations	0.13.0
org.apache.zookeeper	zookeeper	3.9.2
org.apache.zookeeper	zookeeper-jute	3.9.2
org.checkerframework	checker-qual	3.31.0
org.codehaus.jackson	jackson-core-asl	1.9.13
org.codehaus.jackson	jackson-mapper-asl	1.9.13
org.codehaus.janino	commons-compiler	3.0.16
org.codehaus.janino	janino	3.0.16
org.datanucleus	datanucleus-api-jdo	4.2.4
org.datanucleus	datanucleus-core	4.1.17
org.datanucleus	datanucleus-rdbms	4.1.19
org.datanucleus	javax.jdo	3.2.0-m3
org.eclipse.collections	eclipse-collections	11.1.0
org.eclipse.collections	eclipse-collections-api	11.1.0
org.eclipse.jetty	jetty-client	9.4.52.v20230823
org.eclipse.jetty	jetty-continuation	9.4.52.v20230823
org.eclipse.jetty	jetty-http	9.4.52.v20230823
org.eclipse.jetty	jetty-io	9.4.52.v20230823
org.eclipse.jetty	jetty-jndi	9.4.52.v20230823
org.eclipse.jetty	jetty-plus	9.4.52.v20230823
org.eclipse.jetty	jetty-proxy	9.4.52.v20230823
org.eclipse.jetty	jetty-security	9.4.52.v20230823
org.eclipse.jetty	jetty-server	9.4.52.v20230823
org.eclipse.jetty	jetty-servlet	9.4.52.v20230823
org.eclipse.jetty	jetty-servlets	9.4.52.v20230823
org.eclipse.jetty	jetty-util	9.4.52.v20230823
org.eclipse.jetty	jetty-util-ajax	9.4.52.v20230823
org.eclipse.jetty	jetty-webapp	9.4.52.v20230823
org.eclipse.jetty	jetty-xml	9.4.52.v20230823
org.eclipse.jetty.websocket	websocket-api	9.4.52.v20230823
org.eclipse.jetty.websocket	websocket-client	9.4.52.v20230823
org.eclipse.jetty.websocket	websocket-common	9.4.52.v20230823
org.eclipse.jetty.websocket	websocket-server	9.4.52.v20230823
org.eclipse.jetty.websocket	websocket-servlet	9.4.52.v20230823
org.fusesource.leveldbjni	leveldbjni-all	1.8
org.glassfish.hk2	hk2-api	2.6.1
org.glassfish.hk2	hk2-locator	2.6.1
org.glassfish.hk2	hk2-utils	2.6.1
org.glassfish.hk2	osgi-resource-locator	1.0.3
org.glassfish.hk2.external	aopalliance-repackaged	2.6.1
org.glassfish.hk2.external	jakarta.inject	2.6.1
org.glassfish.jersey.containers	jersey-container-servlet	2.40
org.glassfish.jersey.containers	jersey-container-servlet-core	2.40
org.glassfish.jersey.core	jersey-client	2.40
org.glassfish.jersey.core	jersey-common	2.40
org.glassfish.jersey.core	jersey-server	2.40
org.glassfish.jersey.inject	jersey-hk2	2.40
org.hibernate.validator	hibernate-validator	6.1.7.Final
org.ini4j	ini4j	0.5.4
org.javassist	javassist	3.29.2-GA
org.jboss.logging	jboss-logging	3.3.2.Final
org.jdbi	jdbi	2.63.1
org.jetbrains	annotations	17.0.0
org.joda	joda-convert	1.7
org.jodd	jodd-core	3.5.2
org.json4s	json4s-ast_2.12	3.7.0-M11
org.json4s	json4s-core_2.12	3.7.0-M11
org.json4s	json4s-jackson_2.12	3.7.0-M11
org.json4s	json4s-scalap_2.12	3.7.0-M11
org.lz4	lz4-java	1.8.0-databricks-1
org.mlflow	mlflow-spark_2.12	2.9.1
org.objenesis	objenesis	2.5.1
org.postgresql	postgresql	42.6.1
org.roaringbitmap	RoaringBitmap	0.9.45-databricks
org.roaringbitmap	shims	0.9.45-databricks
org.rocksdb	rocksdbjni	9.2.1
org.rosuda.REngine	REngine	2.1.0
org.scala-lang	scala-compiler_2.12	2.12.15
org.scala-lang	scala-library_2.12	2.12.15
org.scala-lang	scala-reflect_2.12	2.12.15
org.scala-lang.modules	scala-collection-compat_2.12	2.11.0
org.scala-lang.modules	scala-java8-compat_2.12	0.9.1
org.scala-lang.modules	scala-parser-combinators_2.12	1.1.2
org.scala-lang.modules	scala-xml_2.12	1.2.0
org.scala-sbt	test-interface	1.0
org.scalacheck	scalacheck_2.12	1.14.2
org.scalactic	scalactic_2.12	3.2.16
org.scalanlp	breeze-macros_2.12	2.1.0
org.scalanlp	breeze_2.12	2.1.0
org.scalatest	scalatest-compatible	3.2.16
org.scalatest	scalatest-core_2.12	3.2.16
org.scalatest	scalatest-diagrams_2.12	3.2.16
org.scalatest	scalatest-featurespec_2.12	3.2.16
org.scalatest	scalatest-flatspec_2.12	3.2.16
org.scalatest	scalatest-freespec_2.12	3.2.16
org.scalatest	scalatest-funspec_2.12	3.2.16
org.scalatest	scalatest-funsuite_2.12	3.2.16
org.scalatest	scalatest-matchers-core_2.12	3.2.16
org.scalatest	scalatest-mustmatchers_2.12	3.2.16
org.scalatest	scalatest-propspec_2.12	3.2.16
org.scalatest	scalatest-refspec_2.12	3.2.16
org.scalatest	scalatest-shouldmatchers_2.12	3.2.16
org.scalatest	scalatest-wordspec_2.12	3.2.16
org.scalatest	scalatest_2.12	3.2.16
org.slf4j	jcl-over-slf4j	2.0.7
org.slf4j	jul-to-slf4j	2.0.7
org.slf4j	slf4j-api	2.0.7
org.slf4j	slf4j-simple	1.7.25
org.threeten	threeten-extra	1.7.1
org.tukaani	xz	1.9
org.typelevel	algebra_2.12	2.0.1
org.typelevel	cats-kernel_2.12	2.1.1
org.typelevel	spire-macros_2.12	0.17.0
org.typelevel	spire-platform_2.12	0.17.0
org.typelevel	spire-util_2.12	0.17.0
org.typelevel	spire_2.12	0.17.0
org.wildfly.openssl	wildfly-openssl	1.1.3.Final
org.xerial	sqlite-jdbc	3.42.0.0
org.xerial.snappy	snappy-java	1.1.10.3
org.yaml	snakeyaml	2.0
oro	oro	2.0.8
pl.edu.icm	JLargeArrays	1.5
software.amazon.cryptools	AmazonCorrettoCryptoProvider	2.4.1-linux-x86_64
stax	stax-api	1.0.1