Udostępnij za pośrednictwem

Databricks Runtime 16.1

  • Artykuł

The following release notes provide information about Databricks Runtime 16.1, powered by Apache Spark 3.5.0.

Databricks released this version in December 2024.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

Behavioral changes

The VARIANT data type can no longer be used with operations that require comparisons

In DBR 16.1 and above, you cannot use the following clauses or operators in queries that include a VARIANT data type:

  • DISTINCT
  • INTERSECT
  • EXCEPT
  • UNION
  • DISTRIBUTE BY

Additionally, you cannot use these DataFrame functions:

  • df.dropDuplicates()
  • df.repartition()

These operations perform comparisons, and comparisons that use the VARIANT data type produce undefined results and are not supported in Databricks. If you use the VARIANT type in your Azure Databricks workloads or tables, Databricks recommends the following changes:

  • Update queries or expressions to explicitly cast VARIANT values to non-VARIANT data types.
  • If you have fields that must be used with any of the above operations, extract those fields from the VARIANT data type and store them using non-VARIANT data types.

To learn more, see Query variant data.

New features and improvements

Support for collations in Apache Spark is in Public Preview

You can now assign language-aware, case-insensitive, and access-insensitive collations to STRING columns and expressions. These collations are used in string comparisons, sorting, grouping operations, and many string functions. See Collation.

Support for collations in Delta Lake is in Public Preview

You can now define collations for columns when creating or altering a Delta table. See Collation support for Delta Lake.

LITE mode for vacuum is in Public Preview

You can now use VACUUM table_name LITE to perform a lighter-weight vacuum operation that leverages metadata in the Delta transaction log. See Full vs. lite mode and VACUUM.

Support for parameterizing the USE CATALOG with IDENTIFIER clause

In Databricks Runtime 16.1 and later, the IDENTIFIER clause is supported for the USE CATALOG statement. With this support, you can parameterize the current catalog based on a string variable or parameter marker.

COMMENT ON COLUMN support for tables and views

In Databricks Runtime 16.1 and later, the COMMENT ON statement supports altering comments for view and table columns.

New SQL functions

In Databricks Runtime 16.1 and later, the following new built-in SQL functions are available:

  • dayname(expr) returns the three-letter English acronym for the day of the week for the given date.
  • uniform(expr1, expr2 [,seed]) returns a random value with independent and identically distributed values within the specified range of numbers.
  • randstr(length) returns a random string of length alpha-numeric characters.

Named parameter invocation for more functions

In Databricks Runtime 16.1 and later, the following functions support named parameter invocation:

The SYNC METADATA parameter to the REPAIR TABLE command is supported with the Hive metastore

In Databricks Runtime 16.1 and above, you can use the SYNC METADATA parameter with the REPAIR TABLE command to update the metadata of a Hive metastore managed table. See REPAIR TABLE.

Enhanced data integrity for compressed Apache Arrow batches

In Databricks Runtime 16.1 and above, to further protect against data corruption, every LZ4 compressed Arrow batch now includes the LZ4 content and block checksums. See LZ4 Frame Format Description.

Support added for Scala methods on Unity Catalog shared access mode compute

In Databricks Runtime 16.1 and above, on Unity Catalog shared access mode compute, support is added for the following Scala methods: Dataset.flatMapGroups(), Dataset.mapGroups(), and DataStreamWriter.foreach().

StreamingQueryListener support for Scala

You can now use StreamingQueryListener in Scala on compute configured with shared access mode.

Built-in Oracle JDBC Driver

In Databricks Runtime 16.1 and above, the Oracle JDBC Driver is built into Azure Databricks. If you use a customer-uploaded JDBC driver JAR via DriverManager, you must rewrite scripts to explicitly use the custom JAR. Otherwise, the built-in driver is used.

More detailed errors for Delta tables accessed with paths

A new error message experience for Delta tables accessed using paths is now available. All exceptions are now forwarded to the user. The exception DELTA_MISSING_DELTA_TABLE is now reserved for when underlying files cannot be read as a Delta table.

Other changes

Renamed error codes for the cloudFiles Structured Streaming source

This release includes a change to rename the following error codes:

  • _LEGACY_ERROR_TEMP_DBR_0143 is renamed to CF_INCORRECT_STREAM_USAGE.
  • _LEGACY_ERROR_TEMP_DBR_0260 is renamed to CF_INCORRECT_BATCH_USAGE .

Bug fixes

Nested types now properly accept NULL constraints

This release fixes a bug affecting some Delta generated columns of nested types, for example, STRUCT. These columns would sometimes incorrectly reject expressions based on NULL or NOT NULL constraints of nested fields. This has been fixed.

Library upgrades

  • Upgraded Python libraries:
    • ipyflow-core from 0.0.198 to 0.0.201
    • pyccolo from 0.0.52 to 0.0.65
  • Upgraded R libraries:
  • Upgraded Java libraries:
    • io.delta.delta-sharing-client_2.12 from 1.2.0 to 1.2.2
    • org.lz4.lz4-java from 1.8.0 to 1.8.0-databricks-1
    • software.amazon.cryptools.AmazonCorrettoCryptoProvider from 1.6.2-linux-x86_64 to 2.4.1-linux-x86_64

Apache Spark

Databricks Runtime 16.1 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 16.0, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-50482] [SC-182879][CORE] Deprecated no-op spark.shuffle.spill config
  • [SPARK-50032] [SC-182706][SQL][16.x] Allow use of fully qualified collation name
  • [SPARK-50467] [SC-182823][PYTHON] Add __all__ for builtin functions
  • [SPARK-48898] [SC-182828][SQL] Fix Variant shredding bug
  • [SPARK-50441] [SC-182668][SQL] Fix parametrized identifiers not working when referencing CTEs
  • [SPARK-50446] [SC-182639][PYTHON] Concurrent level in Arrow-optimized Python UDF
  • [SPARK-50430] [SC-182536][CORE] Use the standard Properties.clone instead of manual clone
  • [SPARK-50471] [SC-182790][PYTHON] Support Arrow-based Python Data Source Writer
  • [SPARK-50466] [SC-182791][PYTHON] Refine the docstring for string functions - part 1
  • [SPARK-50194] [DBR16.x][SC-182593][SS][PYTHON] Integration of New Timer API and Initial State API with Timer
  • [SPARK-50437] [SC-182586][SS] Reduce overhead of creating deserializers in TransformWithStateExec
  • [SPARK-49676] [DBR16.x][SC-182538][SS][PYTHON] Add Support for Chaining o…
  • [SPARK-49294] [SC-182730][UI] Add width attribute for shuffle-write-time checkbox.
  • [SPARK-50426] [SC-182540][PYTHON] Avoid static Python data source lookup when using builtin or Java data sources
  • [SPARK-48356] [SC-182603][SQL] Support for FOR statement
  • [SPARK-50333] [SC-182136][SQL] Codegen Support for CsvToStructs (by Invoke & RuntimeReplaceable)
  • [SPARK-50285] [SC-182575] Metrics for commits to StagedTable instances
  • [SPARK-50081] [SC-182344][SQL] Codegen Support for XPath*(by Invoke & RuntimeReplaceable)
  • [SPARK-50440] [SC-182592] [SQL] Refactor AttributeSeq.resolveCandidates
  • [SPARK-50067] [SC-179648][SQL] Codegen Support for SchemaOfCsv(by Invoke & RuntimeReplaceable)
  • [SPARK-49873] [SC-178577][SQL] fix post merge failure on error testing
  • [SPARK-50118] [SC-181259][CONNET] Reset isolated state cache when tasks are running
  • [SPARK-49873] [SC-178577][SQL] Assign proper error class for _LEGACY_ERROR_TEMP_1325
  • [SPARK-50381] [SC-182197][CORE] Support spark.master.rest.maxThreads
  • [SPARK-46725] [SC-182448][SQL] Add DAYNAME function
  • [SPARK-50270] [SC-181179][SS][PYTHON] Added custom state metrics for TransformWithStateInPandas
  • [SPARK-50118] Revert “[SC-181259][CONNET] Reset isolated state cache when tasks are running”
  • [SPARK-50075] [SC-181820][SQL][PYTHON][CONNECT] Add DataFrame APIs for table-valued functions
  • [SPARK-49470] [SC-175736][UI] Update dataTables stylesheets and javascripts from 1.13.5 to 1.13.11
  • [SPARK-50235] Revert “[SC-180786][SQL] Clean up ColumnVector resource after processing all rows in ColumnarToRowExec”
  • [SPARK-50324] [SC-182278][PYTHON][CONNECT] Make createDataFrame trigger Config RPC at most once
  • [SPARK-50387] [SC-182441][SS] Update condition for timer expiry and relevant test
  • [SPARK-50287] [SC-182400][SQL] Merge options of table and relation when creating WriteBuilder in FileTable
  • [SPARK-50066] [SC-181484][SQL] Codegen Support for SchemaOfXml (by Invoke & RuntimeReplaceable)
  • [SPARK-50092] [SC-181568][SQL] Fix PostgreSQL connector behaviour for multidimensional arrays
  • [SPARK-50318] [SC-181641][SQL] Add IntervalUtils.makeYearMonthInterval to deduplicate code between interpreted and codegen
  • [SPARK-50312] [SC-181646][SQL] SparkThriftServer createServer parameter passing error when kerberos is true
  • [SPARK-50246] [SC-181468][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2167: INVALID_JSON_RECORD_TYPE
  • [SPARK-50214] [SC-180692][SQL] From json/xml should not change collations in the given schema
  • [SPARK-50250] [SC-181466][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2075: UNSUPPORTED_FEATURE.WRITE_FOR_BINARY_SOURCE
  • [SPARK-50248] [SC-181467][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2058: INVALID_PARTITION_VALUE
  • [SPARK-50118] [SC-181259][CONNET] Reset isolated state cache when tasks are running
  • [SPARK-50235] [SC-180786][SQL] Clean up ColumnVector resource after processing all rows in ColumnarToRowExec
  • [SPARK-50156] [SC-180781][SQL] Integrate _LEGACY_ERROR_TEMP_2113 into UNRECOGNIZED_STATISTIC
  • [SPARK-50069] [SC-180163][SQL] Integrate _LEGACY_ERROR_TEMP_0028 into UNSUPPORTED_FROM_TO_EXPRESSION
  • [SPARK-50154] [SC-180663][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_0043: INVALID_RESET_COMMAND_FORMAT
  • [SPARK-49967] [SC-179534][SQL] Codegen Support for StructsToJson(to_json)
  • [SPARK-50055] [SC-180978][SQL] Add TryMakeInterval alternative
  • [SPARK-50397] [SC-182367][CORE] Remove deprecated --ip and -i arguments from Master/Worker
  • [SPARK-50238] [SC-181434][PYTHON] Add Variant Support in PySpark UDFs/UDTFs/UDAFs and Python UC UDFs
  • [SPARK-50079] [SC-179830][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2013: NEGATIVE_VALUES_IN_FREQUENCY_EXPRESSION
  • [SPARK-50182] [SC-180346][EXAMPLE] Add submit-sql.sh REST API example
  • [SPARK-49966] [SC-179501][SQL] Use Invoke to implement JsonToStructs(from_json)
  • [SPARK-50302] [SC-182518][SS] Ensure secondary index sizes equal primary index sizes for TransformWithState stateful variables with TTL
  • [SPARK-50301] [SC-182241][SS][16.x] Make TransformWithState metrics reflect their intuitive meanings
  • [SPARK-50175] [SC-182140][SQL] Change collation precedence calculation
  • [SPARK-50148] [SC-180292][SQL] Make StaticInvoke compatible with the method that declare throw exception
  • [SPARK-50280] [SC-181214][PYTHON] Refactor result sorting and empty bin filling in compute_hist
  • [SPARK-50190] [SC-182458][PYTHON] Remove direct dependency of Numpy from Histogram
  • [SPARK-50382] [SC-182368][CONNECT] Add documentation for general information on application development with/extending Spark Connect
  • [SPARK-50296] [SC-181464][PYTHON][CONNECT] Avoid using a classproperty in threadpool for Python Connect client
  • [SPARK-49566] [SC-182239][SQL] Add SQL pipe syntax for the EXTEND operator
  • [SPARK-50036] [SC-179533][CORE][PYTHON] Include SPARK_LOG_SCHEMA in the context of REPL shell
  • [SPARK-49859] [SC-178259][CONNECT] Replace multiprocessing.ThreadPool with ThreadPoolExecutor
  • [SPARK-50141] [SC-182378][PYTHON] Make lpad and rpad accept Column type arguments
  • [SPARK-50379] [SC-182142][SQL] Fix DayTimeIntevalType handling in WindowExecBase
  • [SPARK-49954] [SC-179110][SQL] Codegen Support for SchemaOfJson (by Invoke & RuntimeReplaceable)
  • [SPARK-50398] [SC-182341][CORE] Use ExitCode 0 for --help usage in Spark scripts
  • [SPARK-50377] [SC-182238][SQL] Allow to evaluate foldable RuntimeReplaceable
  • [SPARK-50241] [SC-181444][SQL] Replace NullIntolerant Mixin with Expression.nullIntolerant method
  • [SPARK-50084] [SC-179672][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_3168: MISSING_TIMEOUT_CONFIGURATION
  • [SPARK-50078] [SC-179649][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_0038: DUPLICATED_CTE_NAMES
  • [SPARK-50057] [SC-179573][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_1049: INVALID_ATTRIBUTE_NAME_SYNTAX
  • [SPARK-50070] [SC-179579][SQL] Integrate _LEGACY_ERROR_TEMP_0039 into UNSUPPORTED_SQL_STATEMENT
  • [SPARK-50378] [SC-182235][SS] Add custom metric for tracking spent for proc initial state in transformWithState
  • [SPARK-50029] [SC-179531][SQL] Make StaticInvoke compatible with the method that return Any
  • [SPARK-49990] [SC-179497][SQL] Improve performance of randStr
  • [SPARK-50048] [SC-179528][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2114: UNRECOGNIZED_STATISTIC
  • [SPARK-50053] [SC-179532][SQL] Turn _LEGACY_ERROR_TEMP_2104 into INTERNAL_ERROR
  • [SPARK-49665] [SC-180054][SQL] Trim collation support for string functions
  • [SPARK-48549] [SC-176472][SQL][PYTHON] Improve SQL function sentences
  • [SPARK-50022] [SC-179503][CORE][UI] Fix MasterPage to hide App UI links when UI is disabled
  • [SPARK-50087] [SC-182152] Robust handling of boolean expressions in CASE WHEN for MsSqlServer and future connectors
  • [SPARK-49991] [SC-179481][SQL] Make HadoopMapReduceCommitProtocol respect ‘mapreduce.output.basename’ to generate file names
  • [SPARK-50038] [SC-179521][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_0008: MERGE_WITHOUT_WHEN
  • [SPARK-50236] [SC-181671][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_1156: COLUMN_NOT_DEFINED_IN_TABLE
  • [SPARK-50021] [SC-179500][CORE][UI] Fix ApplicationPage to hide App UI links when UI is disabled
  • [SPARK-49911] [SC-179111][SQL] Fix semantic of support binary equality
  • [SPARK-50025] [SC-179496][SQL] Integrate _LEGACY_ERROR_TEMP_1253 into EXPECT_VIEW_NOT_TABLE
  • [SPARK-49829] [SC-179480][SS] Fix the bug on the optimization on adding input to state store in stream-stream join
  • [SPARK-50004] [SC-179499][SQL] Integrate _LEGACY_ERROR_TEMP_3327 into FIELD_NOT_FOUND
  • [SPARK-50380] [SC-182210][SQL] ReorderAssociativeOperator should respect the contract in ConstantFolding
  • [SPARK-50340] [SC-181859][SQL] Unwrap UDT in INSERT input query
  • [SPARK-50237] [SC-181660][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2138-9: CIRCULAR_CLASS_REFERENCE
  • [SPARK-50258] [SC-181993][SQL] Fix output column order changed issue after AQE optimization
  • [SPARK-49773] [SC-178369][SQL] Uncaught Java exception from make_timestamp() with bad timezone
  • [SPARK-49977] [SC-179265][SQL] Use stack-based iterative computation to avoid creating many Scala List objects for deep expression trees
  • [SPARK-50153] [SC-181591][SQL] Add name to RuleExecutor to make printing QueryExecutionMetrics’s logs clearer
  • [SPARK-50320] [SC-181668][CORE] Make --remote an official option by removing experimental warning
  • [SPARK-49909] [SC-179492]Revert “[SQL] Fix the pretty name of some expressions”
  • [SPARK-50330] [SC-180720][SC-181764][SQL] Add hints to Sort and Window nodes
  • [SPARK-50364] [SC-182003][SQL] Implement serialization for LocalDateTime type in Row.jsonValue
  • [SPARK-50016] [SC-182139][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2067: UNSUPPORTED_PARTITION_TRANSFORM
  • [SPARK-49899] [SC-181175][PYTHON][SS] Support deleteIfExists for TransformWithStateInPandas
  • [SPARK-49757] [SC-177824][SQL] Support IDENTIFIER expression in SET CATALOG statement
  • [SPARK-50315] [SC-181886][SQL] Support custom metrics for V1Fallback writes
  • [SPARK-42838] [SC-181509][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_2000
  • [SPARK-50353] [SC-181985][SQL] Refactor ResolveSQLOnFile
  • [SPARK-48344] [SC-181967][SQL] Prepare SQL Scripting for addition of Execution Framework
  • [SPARK-49345] [SC-174786][CONNECT] Make sure using the current running Spark Session
  • [SPARK-49925] [SC-178882][SQL] Add tests for order by with collated strings
  • [SPARK-50167] [SC-181199][PYTHON][CONNECT] Improve PySpark plotting error messages and imports
  • [SPARK-49368] [SC-174999][CONNECT] Avoid accessing protobuf lite classes directly
  • [SPARK-50056] [SC-181378][SQL] Codegen Support for ParseUrl (by Invoke & RuntimeReplaceable)
  • [SPARK-49601] [SC-180770][SS][PYTHON] Support Initial State Handling for TransformWithStateInPandas
  • [SPARK-49908] [SC-178768][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_0044
  • [SPARK-50144] [SC-180307][SS] Address the limitation of metrics calculation with DSv1 streaming sources
  • [SPARK-49874] [SC-178303][SQL] Remove trim and ltrim collation specifiers.
  • [SPARK-49513] [SC-180339][SS] Add Support for timer in transformWithStateInPandas API
  • [SPARK-49119] [SC-175607][SQL] Fix the inconsistency of syntax show columns between v1 and v2
  • [SPARK-49206] [SC-173704][CORE][UI] Add Environment Variables table to Master EnvironmentPage
  • [SPARK-49934] [SC-179581][SQL] Add implicit cast for accessing collated map with literal
  • [SPARK-50049] [SC-181659][SQL] Support custom driver metrics in writing to v2 table
  • [SPARK-50171] [SC-180295][PYTHON] Make numpy optional for KDE plot
  • [SPARK-49962] [SC-179320][SQL] Simplify AbstractStringTypes class hierarchy
  • [SPARK-50226] [SC-181483][SQL] Correct MakeDTInterval and MakeYMInterval to catch Java exceptions
  • [SPARK-48775] [SC-170801][SQL][STS] Replace SQLContext with SparkSession in STS
  • [SPARK-49015] [SC-175688][CORE] Connect Server should respect spark.log.structuredLogging.enabled
  • [SPARK-50327] [SC-181667][SQL][16.x] Factor out function resolution to be reused in the single-pass Analyzer
  • [SPARK-49995] [SC-180762][SQL] Add named argument support to more TVFs
  • [SPARK-49268] [SC-174903][CORE] Log IO exceptions in SHS history provider
  • [SPARK-48123] [SC-164989][Core] Provide a constant table schema for querying structured logs
  • [SPARK-49217] [SC-174904][CORE] Support separate buffer size configuration in UnsafeShuffleWriter
  • [SPARK-50325] [SC-181664][SQL][16.x] Factor out alias resolution to be reused in the single-pass Analyzer
  • [SPARK-50322] [SC-181665][SQL] Fix parameterized identifier in a sub-query
  • [SPARK-48400] [SC-175283][CORE] Promote PrometheusServlet to DeveloperApi
  • [SPARK-50118] Revert “[SC-181259][CONNET] Reset isolated state cache when tasks are running”
  • [SPARK-50306] [SC-181564][PYTHON][CONNECT] Support Python 3.13 in Spark Connect
  • [SPARK-50152] [SC-181264][SS] Support handleInitialState with state data source reader
  • [SPARK-50260] [SC-181271][CONNECT] Refactor and optimize Spark C…
  • [SPARK-47591] [SC-163090][SQL] Hive-thriftserver: Migrate logInfo with variables to structured logging framework
  • [SPARK-49312] [SC-174672][PYTHON] Improve error message for assertSchemaEqual
  • [SPARK-49439] [SC-175236][SQL] Fix the pretty name of the FromProtobuf & ToProtobuf expression
  • [SPARK-50092] [ES-1258521] Fix PostgreSQL connector behaviour for multidimensional arrays
  • [SPARK-49913] [SC-181565][SQL] Add check for unique label names in nested labeled scopes
  • [SPARK-49563] [SC-181465][SQL] Add SQL pipe syntax for the WINDOW operator
  • [SPARK-49661] [SC-179021][SQL] Implement trim collation hashing and comparison.
  • [SPARK-38912] [SC-181543][PYTHON] Remove the comment related to classmethod and property
  • [SPARK-49770] [16.x][SC-179802][SC-179270][SS][RocksDB Hardening] Improve RocksDB SST file mapping management, and fix issue with reloading same version with existing snapshot
  • [SPARK-49002] Revert “[SC-172846][SQL] Consistently handle invalid locations in WAREHOUSE/SCHEMA/TABLE/PARTITION/DIRECTORY
  • [SPARK-50290] [SC-181453][SQL] Add a flag to disable DataFrameQueryContext creation
  • [SPARK-49668] [SC-178268][SQL] Implement collation key support for trim collation
  • [SPARK-50262] [SC-181193][SQL] Forbid specification complex types during altering collation
  • [SPARK-48898] [SC-181435][SQL] Add Variant shredding functions
  • [SPARK-48273] [SC-181381]Revert “[SQL] Fix late rewrite of PlanWithUnresolvedIdentifier
  • [SPARK-50222] [SC-180706][CORE] Support spark.submit.appName
  • [SPARK-50208] [SC-180662][CORE] Support spark.master.useDriverIdAsAppName.enabled
  • [SPARK-50224] [SC-180689][SQL] The replacements of IsValidUTF8|ValidateUTF8|TryValidateUTF8|MakeValidUTF8 shall be NullIntolerant
  • [SPARK-50247] [SC-180962][CORE] Define BLOCK_MANAGER_REREGISTRATION_FAILED as ExecutorExitCode
  • [SPARK-50282] [SC-181221][ML] Simplify TargetEncoderModel.transform
  • [SPARK-50112] [SC-180763][SQL] Allowing the TransformWithState operator to use Avro encoding
  • [SPARK-50267] [SC-181180][ML] Improve TargetEncoder.fit with DataFrame APIs
  • [SPARK-37178] [SC-180939][ML] Add Target Encoding to ml.feature
  • [SPARK-50118] [SC-181259][CONNET] Reset isolated state cache when tasks are running
  • [SPARK-50085] [BEHAVE-176][SC-179809][PYTHON] Make lit(ndarray) with np.int8 respect the numpy datatype
  • [SPARK-50256] [SC-181048][SQL] Add lightweight validation to check if a logical plan becomes unresolved after every optimizer rule
  • [SPARK-50196] [SC-180932][CONNECT] Fix Python error context to use a proper context
  • [SPARK-50274] [SC-181181][CORE] Guard against use-after-close in DirectByteBufferOutputStream
  • [SPARK-49999] [SC-180063][PYTHON][CONNECT] Support optional “column” parameter in box, kde and hist plots
  • [SPARK-50273] [SC-181178][SS] Improve logging for RocksDB lock acquire/release cases
  • [SPARK-50033] [SC-180720][SC-180659][SQL] Add a hint to logical.Aggregate() node
  • [SPARK-50163] [16.x][SC-180201][SC-180664][SS] Fix the RocksDB extra acquireLock release due to the completion listener
  • [SPARK-50253] [SC-180969][SS] Stream-Stream Join should not fetch checkpoint ID if not supported.
  • [SPARK-50255] [SC-180964][PYTHON] Avoid unnecessary casting in compute_hist
  • [SPARK-50228] [SC-180780][SQL] Move the RewriteCollationJoin rule to FinishAnalysis
  • [SPARK-50001] [SC-179524][PYTHON][PS][CONNECT] Adjust “precision” to be part of kwargs for box plots
  • [SPARK-49637] [SC-180160][SQL] Changed error message for INVALID_FRACTION_OF_SECOND
  • [SPARK-49530] [SC-180658][PYTHON] Get active session from dataframes
  • [SPARK-50195] [SC-180654][CORE] Fix StandaloneRestServer to propagate spark.app.name to SparkSubmit properly
  • [SPARK-50229] [SC-180773] Reduce memory usage on driver for wide schemas by reducing the lifetime of AttributeReference objects created during logical planning
  • [SPARK-50231] [SC-180815][PYTHON] Make function instr accept Column substring
  • [SPARK-49854] [SC-179812][16.x][SQL] Clone artifact manager during session clone
  • [SPARK-50028] [SC-180699][CONNECT] Replace global locks in Spark Connect server listener with fine-grained locks
  • [SPARK-49615] [SC-180672] [ML] Make all ML feature transformers dataset schema validation conforming “spark.sql.caseSensitive” config.
  • [SPARK-50219] [SC-180694][SQL] Refactor ApplyCharTypePadding so that helper methods can be used in single-pass resolver
  • [SPARK-50077] [SC-179827][SQL] Introduce a new pattern object for LogicalRelation to help avoiding default full params pattern
  • [SPARK-50128] [Backport][16x][SC-180677][SS] Add stateful processor handle APIs using implicit encoders in Scala
  • [SPARK-50061] [SC-179961][SQL] Enable analyze table for collated columns
  • [SPARK-49993] [SC-180084][SQL] Improve error messages for Sum and Average
  • [SPARK-49638] [SC-179665][SQL] Remove the ANSI config suggestion in INVALID_URL
  • [SPARK-50204] [SC-180660][SQL] Factor out HiveTableRelation read path resolution
  • [SPARK-50193] [SC-180651][SS] Fix exception handling for validating time modes
  • [SPARK-50179] [SC-180342][CORE] Make spark.app.name property optional in REST API
  • [SPARK-50068] [SC-180300][SQL] Refactor TypeCoercion and AnsiTypeCoercion to separate single node transformations
  • [SPARK-49411] [SC-179483][SS] Communicate State Store Checkpoint ID between driver and stateful operators
  • [SPARK-50124] [SC-180294][SQL] LIMIT/OFFSET should preserve data ordering
  • [SPARK-49506] [SC-180165][SQL] Optimize ArrayBinarySearch for foldable array
  • [SPARK-50097] [SC-179908][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_1248: ALTER_TABLE_SERDE_FOR_DATASOURCE_TABLE
  • [SPARK-50071] [SC-180159][SQL][PYTHON] Add try_make_timestamp(_ltz and _ntz) and related tests
  • [SPARK-50024] Revert “[SC-179515][PYTHON][CONNECT] Switch to use logger instead of warnings module in client”
  • [SPARK-50054] [SC-180228][PYTHON][CONNECT] Support histogram plots
  • [SPARK-50015] [SC-179964][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_1125: MISSING_DATABASE_FOR_V1_SESSION_CATALOG
  • [SPARK-50155] [SC-180240][3.5] Move scala and java files to their default folders
  • [SPARK-49980] [SC-180353][CORE][SQL] Fix potential file stream leaks caused by interruption in canceled tasks
  • [SPARK-49010] [SC-172304][SQL][XML] Add unit tests for XML schema inference case sensitivity
  • [SPARK-50024] [SC-179515][PYTHON][CONNECT] Switch to use logger instead of warnings module in client
  • [SPARK-49562] [SC-180211][SQL] Add SQL pipe syntax for aggregation
  • [SPARK-49663] [SC-180239][SQL] Enable RTRIM suggestions in collation expressions
  • [SPARK-48965] [SC-175926][SQL] Use the correct schema in Dataset#toJSON
  • [SPARK-48493] [SC-175893][PYTHON] Enhance Python Datasource Reader with direct Arrow Batch support for improved performance
  • [SPARK-49734] [SC-180226][PYTHON] Add seed argument for function shuffle
  • [SPARK-50174] [16.x][SC-180253][SQL] Factor out UnresolvedCatalogRelation resolution
  • [SPARK-49989] [SC-179512][PYTHON][CONNECT] Support kde/density plots
  • [SPARK-49805] [SC-180218][SQL][ML] Remove private[xxx] functions from function.scala
  • [SPARK-49808] [SC-179490][SQL] Fix a deadlock in subquery execution due to lazy vals
  • [SPARK-49929] [SC-180144][PYTHON][CONNECT] Support box plots
  • [SPARK-50008] [SC-179290][PS][CONNECT] Avoid unnecessary operations in attach_distributed_sequence_column
  • [SPARK-49767] [SC-180161][PS][CONNECT] Refactor the internal function invocation
  • [SPARK-49683] [SC-178341][SQL] Block trim collation
  • [SPARK-49939] [SC-178941][SQL] Codegen Support for json_object_keys (by Invoke & RuntimeReplaceable)
  • [SPARK-50031] [SC-179582][SQL] Add the TryParseUrl expression
  • [SPARK-49766] [SC-178933][SQL] Codegen Support for json_array_length (by Invoke & RuntimeReplaceable)
  • [SPARK-50046] [SC-180026][SS] Use stable order of EventTimeWatermark node to calculate watermark
  • [SPARK-49540] [SC-180145][PS] Unify the usage of distributed_sequence_id
  • [SPARK-50060] [SC-179965][SQL] Disabled conversion between different collated types in TypeCoercion and AnsiTypeCoercion
  • [SPARK-49004] [SC-173244][CONNECT] Use separate registry for Column API internal functions
  • [SPARK-49811] [SC-177888][SQL]Rename StringTypeAnyCollation
  • [SPARK-49202] [SC-180059][PS] Apply ArrayBinarySearch for histogram
  • [SPARK-49203] [SC-175734][SQL] Add expression for java.util.Arrays.binarySearch
  • [SPARK-50034] [SC-179816][CORE] Fix Misreporting of Fatal Errors as Uncaught Exceptions in SparkUncaughtExceptionHandler
  • [SPARK-50093] [SC-179836][SQL] Collations that use ICU should have the version of the used ICU library
  • [SPARK-49985] [SC-179644][SQL] Remove support for interval types in Variant
  • [SPARK-49103] [SC-173066][CORE] Support spark.master.rest.filters
  • [SPARK-50090] [SC-179819] Refactor ResolveBinaryArithmetic to separate single-node transformation
  • [SPARK-49902] [SC-179650][SQL] Catch underlying runtime errors in RegExpReplace
  • [SPARK-49126] [SC-173342][CORE] Move spark.history.ui.maxApplications config definition to History.scala
  • [SPARK-50094] [SC-179820][PYTHON][CONNECT] Better error message when using memory profiler on editors with no line numbers
  • [SPARK-50062] [SC-179835][SQL] Support collations by InSet
  • [SPARK-50035] [Backport][16x][SC-179530][SS] Add support for explicit handleExpiredTimer function part of the stateful processor
  • [SPARK-49982] [SC-179814][SQL] Fix negative caching in InMemoryRelation
  • [SPARK-49082] [SC-173350][SQL] Widening type promotions in AvroDeserializer
  • [SPARK-50088] [SC-179680][SQL] Refactor UnresolvedStarBase.expand
  • [SPARK-49802] [SC-179482][SS] Add support for read change feed for map and list types used in stateful processors
  • [SPARK-49846] [SC-179506][SS] Add numUpdatedStateRows and numRemovedStateRows metrics for use with transformWithState operator
  • [SPARK-50050] [SC-179643][PYTHON][CONNECT][16.X] Make lit accept str and bool type numpy ndarray
  • [SPARK-49821] [SC-179527][SS][PYTHON] Implement MapState and TTL support for TransformWithStateInPandas
  • [SPARK-49558] [SC-179485][SQL] Add SQL pipe syntax for LIMIT/OFFSET and ORDER/SORT/CLUSTER/DISTRIBUTE BY
  • [SPARK-48144] [SC-165725][LC-4080][SQL] Fix canPlanAsBroadcastHashJoin to respect shuffle join hints
  • [SPARK-50051] [SC-179571][PYTHON][CONNECT] Make lit works with empty numpy ndarray
  • [SPARK-50018] [SC-179488][SQL] Make AbstractStringType serializable
  • [SPARK-50016] [SC-179491][SQL] Improve explicit collation mismatch error
  • [SPARK-50010] [SC-179487][SQL] Expand implicit collation mismatch error
  • [SPARK-48749] [SC-170116][SQL] Simplify UnaryPositive and eliminate its Catalyst Rules with RuntimeReplaceable
  • [SPARK-49857] [SC-178576][SQL] Add storageLevel to Dataset localCheckpoint API
  • [SPARK-50058] [SC-179538][SQL] Factor out plan normalization functions to later use those in single-pass Analyzer testing
  • [SPARK-50052] [SC-179535][PYTHON][16.X] Make NumpyArrayConverter support empty str ndarray
  • [SPARK-47261] [SC-173665][SQL] Assign better name for errors _LEGACY_ERROR_TEMP_1172, _LEGACY_ERROR_TEMP_1173, and _LEGACY_ERROR_TEMP_1174
  • [SPARK-49147] [SC-173471][CORE] Mark KryoRegistrator with DeveloperApi interface
  • [SPARK-48949] [SC-173070][SQL] SPJ: Runtime partition filtering
  • [SPARK-50044] [SC-179523][PYTHON] Refine the docstring of multiple math functions
  • [SPARK-48757] [SC-170113][CORE] Make IndexShuffleBlockResolver have explicit constructors
  • [SPARK-50037] [SQL] Refactor AttributeSeq.resolve(…)
  • [SPARK-48782] [SC-177056][SQL] Add support for executing procedures in catalogs
  • [SPARK-49057] [SC-173081][SQL] Do not block the AQE loop when submitting query stages
  • [SPARK-48824] [SC-176772][BEHAVE-167][SQL] Add Identity Column SQL syntax
  • [SPARK-48773] [SC-170773] Document config “spark.default.parallelism” by config builder framework
  • [SPARK-48735] [SC-169810][SQL] Performance Improvement for BIN function
  • [SPARK-48900] [SC-172433] Add reason field for all internal calls for job/stage cancellation
  • [SPARK-48488] [SC-167605][CORE] Fix methods log[info|warning|error] in SparkSubmit
  • [SPARK-48708] [SC-169809][CORE] Remove three unnecessary type registrations from KryoSerializer
  • [SPARK-49958] [SC-179312][PYTHON] Python API for string validation functions
  • [SPARK-49979] [SC-179264][SQL] Fix AQE hanging issue when collecting twice on a failed plan
  • [SPARK-48729] [SC-169795][SQL] Add a UserDefinedFunction interface to represent a SQL function
  • [SPARK-49997] [SC-179279][SQL] Integrate _LEGACY_ERROR_TEMP_2165 into MALFORMED_RECORD_IN_PARSING
  • [SPARK-49259] [SC-179271][SS]Size based partition creation during kafka read
  • [SPARK-48129] [SC-165006][PYTHON] Provide a constant table schema in PySpark for querying structured logs
  • [SPARK-49951] [SC-179259][SQL] Assign proper error condition for LEGACY_ERROR_TEMP(1099|3085)
  • [SPARK-49971] [SC-179278][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_1097
  • [SPARK-49998] [SC-179277][SQL] Integrate _LEGACY_ERROR_TEMP_1252 into EXPECT_TABLE_NOT_VIEW
  • [SPARK-49876] [SC-179262][CONNECT] Get rid of global locks from Spark Connect Service
  • [SPARK-49957] [SC-179202][SQL] Scala API for string validation functions
  • [SPARK-48480] [SC-173055][SS][CONNECT] StreamingQueryListener should not be affected by spark.interrupt()
  • [SPARK-49643] [SC-179239][SQL] Merge _LEGACY_ERROR_TEMP_2042 into ARITHMETIC_OVERFLOW
  • [SPARK-49959] [SC-179109][SQL] Fix ColumnarArray.copy() to read nulls from the correct offset
  • [SPARK-49956] Revert “[SC-179070] Disabled collations with collect_set expression”
  • [SPARK-49987] [SC-179180][SQL] Fix the error prompt when seedExpression is non-foldable in randstr
  • [SPARK-49948] [SC-179158][PS][CONNECT] Add parameter “precision” to pandas on Spark box plot
  • [SPARK-49970] [SC-179167][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2069
  • [SPARK-49916] [SC-179108][SQL] Throw appropriate Exception for type mismatch between ColumnType and data type in some rows
  • [SPARK-49956] [SC-179070] Disabled collations with collect_set expression
  • [SPARK-49974] [16.x][SC-179071][SQL] Move resolveRelations(…) out of the Analyzer.scala
  • [SPARK-47259] [SC-176437][SQL] Assign names to error conditions for interval errors
  • [SPARK-47430] [SC-173679][SQL] Rework group by map type to fix bind reference exception
  • [SPARK-49067] [SC-172616][SQL] Move utf-8 literal into internal methods of UrlCodec class
  • [SPARK-49955] [SC-178989][SQL] null value does not mean corrupted file when parsing JSON string RDD
  • [SPARK-49393] [SC-175212][SQL] Fail by default in deprecated catalog plugin APIs
  • [SPARK-49952] [SC-178985][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_1142
  • [SPARK-49405] [SC-175224][SQL] Restrict charsets in JsonOptions
  • [SPARK-49892] [SC-178975][SQL] Assign proper error class for _LEGACY_ERROR_TEMP_1136
  • [SPARK-49904] [SC-178973][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2140
  • [SPARK-47257] [SC-174244][SQL] Assign names to error classes _LEGACY_ERROR_TEMP_105[3-4] and _LEGACY_ERROR_TEMP_1331
  • [SPARK-49915] [SC-178869][SQL] Handle zeros and ones in ReorderAssociativeOperator
  • [SPARK-49891] [SC-178970][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2271
  • [SPARK-49829] [ES-1254513] Revise the optimization on adding input to state store in stream-stream join (correctness fix)
  • [SPARK-49918] [SC-178748][CORE] Use read-only access to conf in SparkContext where appropriate
  • [SPARK-49666] [SC-177891][SQL] Add feature flag for trim collation feature
  • [SPARK-48885] [SC-171050][SQL] Make some subclasses of RuntimeReplaceable override replacement to lazy val
  • [SPARK-49932] [SC-178931][CORE] Use tryWithResource release JsonUtils#toJsonString resources to avoid memory leaks
  • [SPARK-49949] [SC-178978][PS] Avoid unnecessary analyze task in attach_sequence_column
  • [SPARK-49924] [SC-178935][SQL] Keep containsNull after ArrayCompact replacement
  • [SPARK-49905] [ES-1265435] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
  • [SPARK-49615] [SC-178877] Bugfix: Make ML column schema validation conforms with spark config spark.sql.caseSensitive.
  • [SPARK-49121] [SC-174787][SQL] Support from_protobuf and to_protobuf for SQL functions
  • [SPARK-47496] [SC-160529][SQL] Java SPI Support for dynamic JDBC dialect registering
  • [SPARK-49359] [SC-174895][SQL] Allow StagedTableCatalog implementations to fall back to non-atomic write
  • [SPARK-49895] [SC-178543][SQL][ES-1270338] Improve error when encountering trailing comma in SELECT clause
  • [SPARK-47945] [SC-163457][SQL] MsSQLServer: Document Mapping Spark SQL Data Types from Microsoft SQL Server and add tests
  • [SPARK-46037] [SC-175495][SQL] Correctness fix for Shuffled Hash Join build left without codegen
  • [SPARK-47813] [SC-162615][SQL] Replace getArrayDimension with updateExtraColumnMeta
  • [SPARK-49542] [SC-178765][SQL] Partition transform exception evaluate error
  • [SPARK-47172] [SC-169537][CORE] Add support for AES-GCM for RPC encryption
  • [SPARK-47666] [SC-161476][SQL] Fix NPE when reading mysql bit array as LongType
  • [SPARK-48947] [SC-174239][SQL] Use lowercased charset name to decrease cache missing in Charset.forName
  • [SPARK-49909] [SC-178552][SQL] Fix the pretty name of some expressions
  • [SPARK-47647] [SC-161377][SQL] Make MySQL data source able to read bit(n>1) as BinaryType like Postgres
  • [SPARK-46622] [SC-153004][CORE] Override toString method for o.a.s.network.shuffledb.StoreVersion
  • [SPARK-48961] [SC-171910][PYTHON] Make the parameter naming of PySparkException consistent with JVM
  • [SPARK-49889] [SC-178474][PYTHON] Add argument trim for functionstrim/ltrim/rtrim
  • [SPARK-49863] [ES-1184693][SC-178546][SQL] Fix NormalizeFloatingNumbers to preserve nullability of nested structs
  • [SPARK-47537] [SC-160747][SQL] Fix error data type mapping on MySQL Connector/J
  • [SPARK-47628] [SC-161257][SQL] Fix Postgres bit array issue ‘Cannot cast to boolean’
  • [SPARK-49843] [SC-178084][ES-1268164][SQL] Fix change comment on char/varchar columns
  • [SPARK-49549] [SC-178764][SQL] Assign a name to the error conditions _LEGACY_ERROR_TEMP_3055, 3146
  • [SPARK-49782] [SC-178756][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
  • [SPARK-49791] [SC-177730][SQL] Make DelegatingCatalogExtension more extendable
  • [SPARK-49559] [SC-178551][SQL] Add SQL pipe syntax for set operations
  • [SPARK-49906] [SC-178536][SQL] Introduce and use CONFLICTING_DIRECTORY_STRUCTURES error for PartitioningUtils
  • [SPARK-49877] [SC-178392][SQL] Change classifyException function signature: add isRuntime argument
  • [SPARK-47501] [SC-160532][SQL] Add convertDateToDate like the existing convertTimestampToTimestamp for JdbcDialect
  • [SPARK-49044] [SC-175746][SQL] ValidateExternalType should return child in error
  • [SPARK-47462] [SC-160460][SQL] Align mappings of other unsigned numeric types with TINYINT in MySQLDialect
  • [SPARK-47435] [SC-160129][SQL] Fix overflow issue of MySQL UNSIGNED TINYINT caused by SPARK-45561
  • [SPARK-49398] [SC-176377][SQL] Improve the error for parameters in the query of CACHE TABLE and CREATE VIEW
  • [SPARK-47263] [SC-177012][SQL] Assign names to the legacy conditions _LEGACY_ERROR_TEMP_13[44-46]
  • [SPARK-49605] [SC-176605][SQL] Fix the prompt when ascendingOrder is DataTypeMismatch in SortArray
  • [SPARK-49806] [SC-178304][SQL][CONNECT] Remove redundant blank space after show in Scala and Connect clients
  • [SPARK-47258] [SC-175032][SQL] Assign names to error classes _LEGACY_ERROR_TEMP_127[0-5]
  • [SPARK-49564] [SC-178461][SQL] Add SQL pipe syntax for the JOIN operator
  • [SPARK-49836] [SC-178339][SQL][SS] Fix possibly broken query when window is provided to window/session_window fn
  • [SPARK-47390] [SC-163306][SC-159312][SPARK-47396][SC-159376][SPARK-47406] Handle SQL timestamps mapping for Postgres and MySQL
  • [SPARK-49776] [SC-177818][PYTHON][CONNECT] Support pie plots
  • [SPARK-49824] [SC-178251][SS][CONNECT] Improve logging in SparkConnectStreamingQueryCache
  • [SPARK-49894] [SC-178460][PYTHON][CONNECT] Refine the string representation of column field operations
  • [SPARK-49764] [SC-177491][PYTHON][CONNECT] Support area plots
  • [SPARK-49694] [SC-177376][PYTHON][CONNECT] Support scatter plots
  • [SPARK-49744] [SC-178363][SS][PYTHON] Implement TTL support for ListState in TransformWithStateInPandas
  • [SPARK-49879] [SC-178334][CORE] Move TransportCipherUtil to a separate file to eliminate Java compilation warnings
  • [SPARK-49866] [SC-178353][SQL] Improve the error message for describe table with partition columns
  • [SPARK-49867] [SC-178302][ES-1260570][SQL] Improve the error message when index is out of bounds when calling GetColumnByOrdinal
  • [SPARK-49607] [SC-177373][PYTHON] Update the sampling approach for sampled based plots
  • [SPARK-49626] [SC-177276][PYTHON][CONNECT] Support horizontal and vertical bar plots
  • [SPARK-49531] [SC-177770][PYTHON][CONNECT] Support line plot with plotly backend
  • [SPARK-49444] [SC-177692][ES-1203248][SQL] Modified UnivocityParser to throw runtime exceptions caused by ArrayIndexOutOfBounds with more user-oriented messages
  • [SPARK-49870] [SC-178262][PYTHON] Add Python 3.13 support in Spark Classic
  • [SPARK-49560] [SC-178121][SQL] Add SQL pipe syntax for the TABLESAMPLE operator
  • [SPARK-49864] [SC-178305][SQL] Improve message of BINARY_ARITHMETIC_OVERFLOW
  • [SPARK-48780] [SC-170274][SQL] Make errors in NamedParametersSupport generic to handle functions and procedures
  • [SPARK-49358] [SC-178158][SQL] Mode expression for map types with collated strings
  • [SPARK-47341] [SC-178157][SQL] Fix inaccurate documentation of RuntimeConfig#get
  • [SPARK-48357] [SC-178153][SQL] Support for LOOP statement
  • [SPARK-49845] [SC-178059][CORE] Make appArgs and environmentVariables optional in REST API
  • [SPARK-49246] [SC-174679][SQL] TableCatalog#loadTable should indicate if it’s for writing
  • [SPARK-48048] [SC-177525][SC-164846][CONNECT][SS] Added client side listener support for Scala
  • [SPARK-48700] [SC-177978][SQL] Mode expression for complex types (all collations)
  • [SPARK-48196] [SC-177899][SQL] Turn QueryExecution lazy val plans into LazyTry
  • [SPARK-49749] [16.x][SC-177877][CORE] Change log level to debug in BlockManagerInfo
  • [SPARK-49561] [SC-177897][SQL] Add SQL pipe syntax for the PIVOT and UNPIVOT operators
  • [SPARK-49823] [SC-177885][SS] Avoid flush during shutdown in rocksdb close path
  • [SPARK-49820] [SC-177855] [PYTHON] Change raise IOError to raise OSError
  • [SPARK-49653] [SC-177266][SQL] Single join for correlated scalar subqueries
  • [SPARK-49552] [SC-177477][PYTHON] Add DataFrame API support for new ‘randstr’ and ‘uniform’ SQL functions
  • [SPARK-48303] [16.x][SC-166251][CORE] Reorganize LogKeys
  • [SPARK-49656] [16x][Backport][SS] Add support for state variables with value state collection types and read change feed options
  • [SPARK-48112] [SC-165129][CONNECT] Expose session in SparkConnectPlanner to plugins
  • [SPARK-48126] [16.x][SC-165309][Core] Make spark.log.structuredLogging.enabled effective
  • [SPARK-49505] [SC-176873][SQL] Create new SQL functions “randstr” and “uniform” to generate random strings or numbers within ranges
  • [SPARK-49463] [SC-177474] Support ListState for TransformWithStateInPandas
  • [SPARK-48131] [SC-165007][Core] Unify MDC key mdc.taskName and task_name
  • [SPARK-49557] [SC-177227][SQL] Add SQL pipe syntax for the WHERE operator
  • [SPARK-49323] [16.x][SC-174689][CONNECT] Move MockObserver from Spark Connect Server’s test folder to the Server’s main folder
  • [SPARK-49745] [SC-177501][SS] Add change to read registered timers through state data source reader
  • [SPARK-49772] [16.x][SC-177478][SC-177214][SS] Remove ColumnFamilyOptions and add configs directly to dbOptions in RocksDB

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

  • Operating System: Ubuntu 24.04.1 LTS
  • Java: Zulu17.54+21-CA
  • Scala: 2.12.18
  • Python: 3.12.3
  • R: 4.4.0
  • Delta Lake: 3.2.1

Installed Python libraries

Library Version Library Version Library Version
annotated-types 0.7.0 asttokens 2.0.5 astunparse 1.6.3
autocommand 2.2.2 azure-core 1.31.0 azure-storage-blob 12.23.0
azure-storage-file-datalake 12.17.0 backports.tarfile 1.2.0 black 24.4.2
blinker 1.7.0 boto3 1.34.69 botocore 1.34.69
cachetools 5.3.3 certifi 2024.6.2 cffi 1.16.0
chardet 4.0.0 charset-normalizer 2.0.4 click 8.1.7
cloudpickle 2.2.1 comm 0.2.1 contourpy 1.2.0
cryptography 42.0.5 cycler 0.11.0 Cython 3.0.11
databricks-sdk 0.30.0 dbus-python 1.3.2 debugpy 1.6.7
decorator 5.1.1 Deprecated 1.2.14 distlib 0.3.8
docstring-to-markdown 0.11 entrypoints 0.4 executing 0.8.3
facets-overview 1.1.1 filelock 3.15.4 fonttools 4.51.0
gitdb 4.0.11 GitPython 3.1.37 google-api-core 2.20.0
google-auth 2.35.0 google-cloud-core 2.4.1 google-cloud-storage 2.18.2
google-crc32c 1.6.0 google-resumable-media 2.7.2 googleapis-common-protos 1.65.0
grpcio 1.60.0 grpcio-status 1.60.0 httplib2 0.20.4
idna 3.7 importlib-metadata 6.0.0 importlib_resources 6.4.0
inflect 7.3.1 ipyflow-core 0.0.201 ipykernel 6.28.0
ipython 8.25.0 ipython-genutils 0.2.0 ipywidgets 7.7.2
isodate 0.6.1 jaraco.context 5.3.0 jaraco.functools 4.0.1
jaraco.text 3.12.1 jedi 0.19.1 jmespath 1.0.1
joblib 1.4.2 jupyter_client 8.6.0 jupyter_core 5.7.2
kiwisolver 1.4.4 launchpadlib 1.11.0 lazr.restfulclient 0.14.6
lazr.uri 1.0.6 matplotlib 3.8.4 matplotlib-inline 0.1.6
mccabe 0.7.0 mlflow-skinny 2.15.1 more-itertools 10.3.0
mypy 1.10.0 mypy-extensions 1.0.0 nest-asyncio 1.6.0
nodeenv 1.9.1 numpy 1.26.4 oauthlib 3.2.2
opentelemetry-api 1.27.0 opentelemetry-sdk 1.27.0 opentelemetry-semantic-conventions 0.48b0
packaging 24.1 pandas 1.5.3 parso 0.8.3
pathspec 0.10.3 patsy 0.5.6 pexpect 4.8.0
pillow 10.3.0 pip 24.2 platformdirs 3.10.0
plotly 5.22.0 pluggy 1.0.0 prompt-toolkit 3.0.43
proto-plus 1.24.0 protobuf 4.24.1 psutil 5.9.0
psycopg2 2.9.3 ptyprocess 0.7.0 pure-eval 0.2.2
pyarrow 15.0.2 pyasn1 0.4.8 pyasn1-modules 0.2.8
pyccolo 0.0.65 pycparser 2.21 pydantic 2.8.2
pydantic_core 2.20.1 pyflakes 3.2.0 Pygments 2.15.1
PyGObject 3.48.2 PyJWT 2.7.0 pyodbc 5.0.1
pyparsing 3.0.9 pyright 1.1.294 python-dateutil 2.9.0.post0
python-lsp-jsonrpc 1.1.2 python-lsp-server 1.10.0 pytoolconfig 1.2.6
pytz 2024.1 PyYAML 6.0.1 pyzmq 25.1.2
requests 2.32.2 rope 1.12.0 rsa 4.9
s3transfer 0.10.2 scikit-learn 1.4.2 scipy 1.13.1
seaborn 0.13.2 setuptools 74.0.0 six 1.16.0
smmap 5.0.0 sqlparse 0.5.1 ssh-import-id 5.11
stack-data 0.2.0 statsmodels 0.14.2 tenacity 8.2.2
threadpoolctl 2.2.0 tokenize-rt 4.2.1 tomli 2.0.1
tornado 6.4.1 traitlets 5.14.3 typeguard 4.3.0
types-protobuf 3.20.3 types-psutil 5.9.0 types-pytz 2023.3.1.1
types-PyYAML 6.0.0 types-requests 2.31.0.0 types-setuptools 68.0.0.0
types-six 1.16.0 types-urllib3 1.26.25.14 typing_extensions 4.11.0
ujson 5.10.0 unattended-upgrades 0.1 urllib3 1.26.16
virtualenv 20.26.2 wadllib 1.3.6 wcwidth 0.2.5
whatthepatch 1.0.2 wheel 0.43.0 wrapt 1.14.1
yapf 0.33.0 zipp 3.17.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2024-08-04.

Library Version Library Version Library Version
arrow 16.1.0 askpass 1.2.0 assertthat 0.2.1
backports 1.5.0 base 4.4.0 base64enc 0.1-3
bigD 0.2.0 bit 4.0.5 bit64 4.0.5
bitops 1.0-8 blob 1.2.4 boot 1.3-30
brew 1.0-10 brio 1.1.5 broom 1.0.6
bslib 0.8.0 cachem 1.1.0 callr 3.7.6
caret 6.0-94 cellranger 1.1.0 chron 2.3-61
class 7.3-22 cli 3.6.3 clipr 0.8.0
clock 0.7.1 cluster 2.1.6 codetools 0.2-20
colorspace 2.1-1 commonmark 1.9.1 compiler 4.4.0
config 0.3.2 conflicted 1.2.0 cpp11 0.4.7
crayon 1.5.3 credentials 2.0.1 curl 5.2.1
data.table 1.15.4 datasets 4.4.0 DBI 1.2.3
dbplyr 2.5.0 desc 1.4.3 devtools 2.4.5
diagram 1.6.5 diffobj 0.3.5 digest 0.6.36
downlit 0.4.4 dplyr 1.1.4 dtplyr 1.3.1
e1071 1.7-14 ellipsis 0.3.2 evaluate 0.24.0
fansi 1.0.6 farver 2.1.2 fastmap 1.2.0
fontawesome 0.5.2 forcats 1.0.0 foreach 1.5.2
foreign 0.8-86 forge 0.2.0 fs 1.6.4
future 1.34.0 future.apply 1.11.2 gargle 1.5.2
generics 0.1.3 gert 2.1.0 ggplot2 3.5.1
gh 1.4.1 git2r 0.33.0 gitcreds 0.1.2
glmnet 4.1-8 globals 0.16.3 glue 1.7.0
googledrive 2.1.1 googlesheets4 1.1.1 gower 1.0.1
graphics 4.4.0 grDevices 4.4.0 grid 4.4.0
gridExtra 2.3 gsubfn 0.7 gt 0.11.0
gtable 0.3.5 hardhat 1.4.0 haven 2.5.4
highr 0.11 hms 1.1.3 htmltools 0.5.8.1
htmlwidgets 1.6.4 httpuv 1.6.15 httr 1.4.7
httr2 1.0.2 ids 1.0.1 ini 0.3.1
ipred 0.9-15 isoband 0.2.7 iterators 1.0.14
jquerylib 0.1.4 jsonlite 1.8.8 juicyjuice 0.1.0
KernSmooth 2.23-22 knitr 1.48 labeling 0.4.3
later 1.3.2 lattice 0.22-5 lava 1.8.0
lifecycle 1.0.4 listenv 0.9.1 lubridate 1.9.3
magrittr 2.0.3 markdown 1.13 MASS 7.3-60.0.1
Matrix 1.6-5 memoise 2.0.1 methods 4.4.0
mgcv 1.9-1 mime 0.12 miniUI 0.1.1.1
mlflow 2.14.1 ModelMetrics 1.2.2.2 modelr 0.1.11
munsell 0.5.1 nlme 3.1-165 nnet 7.3-19
numDeriv 2016.8-1.1 openssl 2.2.0 parallel 4.4.0
parallelly 1.38.0 pillar 1.9.0 pkgbuild 1.4.4
pkgconfig 2.0.3 pkgdown 2.1.0 pkgload 1.4.0
plogr 0.2.0 plyr 1.8.9 praise 1.0.0
prettyunits 1.2.0 pROC 1.18.5 processx 3.8.4
prodlim 2024.06.25 profvis 0.3.8 progress 1.2.3
progressr 0.14.0 promises 1.3.0 proto 1.0.0
proxy 0.4-27 ps 1.7.7 purrr 1.0.2
R6 2.5.1 ragg 1.3.2 randomForest 4.7-1.1
rappdirs 0.3.3 rcmdcheck 1.4.0 RColorBrewer 1.1-3
Rcpp 1.0.13 RcppEigen 0.3.4.0.0 reactable 0.4.4
reactR 0.6.0 readr 2.1.5 readxl 1.4.3
recipes 1.1.0 rematch 2.0.0 rematch2 2.1.2
remotes 2.5.0 reprex 2.1.1 reshape2 1.4.4
rlang 1.1.4 rmarkdown 2.27 RODBC 1.3-23
roxygen2 7.3.2 rpart 4.1.23 rprojroot 2.0.4
Rserve 1.8-13 RSQLite 2.3.7 rstudioapi 0.16.0
rversions 2.1.2 rvest 1.0.4 sass 0.4.9
scales 1.3.0 selectr 0.4-2 sessioninfo 1.2.2
shape 1.4.6.1 shiny 1.9.1 sourcetools 0.1.7-1
sparklyr 1.8.6 spatial 7.3-17 splines 4.4.0
sqldf 0.4-11 SQUAREM 2021.1 stats 4.4.0
stats4 4.4.0 stringi 1.8.4 stringr 1.5.1
survival 3.6-4 swagger 5.17.14.1 sys 3.4.2
systemfonts 1.1.0 tcltk 4.4.0 testthat 3.2.1.1
textshaping 0.4.0 tibble 3.2.1 tidyr 1.3.1
tidyselect 1.2.1 tidyverse 2.0.0 timechange 0.3.0
timeDate 4032.109 tinytex 0.52 tools 4.4.0
tzdb 0.4.0 urlchecker 1.0.1 usethis 3.0.0
utf8 1.2.4 utils 4.4.0 uuid 1.2-1
V8 4.4.2 vctrs 0.6.5 viridisLite 0.4.2
vroom 1.6.5 waldo 0.5.2 whisker 0.4.1
withr 3.0.1 xfun 0.46 xml2 1.3.6
xopen 1.0.1 xtable 1.8-4 yaml 2.3.10
zeallot 0.1.0 zip 2.3.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.12.638
com.amazonaws aws-java-sdk-cloudformation 1.12.638
com.amazonaws aws-java-sdk-cloudfront 1.12.638
com.amazonaws aws-java-sdk-cloudhsm 1.12.638
com.amazonaws aws-java-sdk-cloudsearch 1.12.638
com.amazonaws aws-java-sdk-cloudtrail 1.12.638
com.amazonaws aws-java-sdk-cloudwatch 1.12.638
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.12.638
com.amazonaws aws-java-sdk-codedeploy 1.12.638
com.amazonaws aws-java-sdk-cognitoidentity 1.12.638
com.amazonaws aws-java-sdk-cognitosync 1.12.638
com.amazonaws aws-java-sdk-config 1.12.638
com.amazonaws aws-java-sdk-core 1.12.638
com.amazonaws aws-java-sdk-datapipeline 1.12.638
com.amazonaws aws-java-sdk-directconnect 1.12.638
com.amazonaws aws-java-sdk-directory 1.12.638
com.amazonaws aws-java-sdk-dynamodb 1.12.638
com.amazonaws aws-java-sdk-ec2 1.12.638
com.amazonaws aws-java-sdk-ecs 1.12.638
com.amazonaws aws-java-sdk-efs 1.12.638
com.amazonaws aws-java-sdk-elasticache 1.12.638
com.amazonaws aws-java-sdk-elasticbeanstalk 1.12.638
com.amazonaws aws-java-sdk-elasticloadbalancing 1.12.638
com.amazonaws aws-java-sdk-elastictranscoder 1.12.638
com.amazonaws aws-java-sdk-emr 1.12.638
com.amazonaws aws-java-sdk-glacier 1.12.638
com.amazonaws aws-java-sdk-glue 1.12.638
com.amazonaws aws-java-sdk-iam 1.12.638
com.amazonaws aws-java-sdk-importexport 1.12.638
com.amazonaws aws-java-sdk-kinesis 1.12.638
com.amazonaws aws-java-sdk-kms 1.12.638
com.amazonaws aws-java-sdk-lambda 1.12.638
com.amazonaws aws-java-sdk-logs 1.12.638
com.amazonaws aws-java-sdk-machinelearning 1.12.638
com.amazonaws aws-java-sdk-opsworks 1.12.638
com.amazonaws aws-java-sdk-rds 1.12.638
com.amazonaws aws-java-sdk-redshift 1.12.638
com.amazonaws aws-java-sdk-route53 1.12.638
com.amazonaws aws-java-sdk-s3 1.12.638
com.amazonaws aws-java-sdk-ses 1.12.638
com.amazonaws aws-java-sdk-simpledb 1.12.638
com.amazonaws aws-java-sdk-simpleworkflow 1.12.638
com.amazonaws aws-java-sdk-sns 1.12.638
com.amazonaws aws-java-sdk-sqs 1.12.638
com.amazonaws aws-java-sdk-ssm 1.12.638
com.amazonaws aws-java-sdk-storagegateway 1.12.638
com.amazonaws aws-java-sdk-sts 1.12.638
com.amazonaws aws-java-sdk-support 1.12.638
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.12.638
com.amazonaws jmespath-java 1.12.638
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks databricks-sdk-java 0.27.0
com.databricks jets3t 0.7.1-0
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.15.2
com.fasterxml.jackson.core jackson-core 2.15.2
com.fasterxml.jackson.core jackson-databind 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-yaml 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-joda 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.16.0
com.fasterxml.jackson.module jackson-module-paranamer 2.15.2
com.fasterxml.jackson.module jackson-module-scala_2.12 2.15.2
com.github.ben-manes.caffeine caffeine 2.9.3
com.github.fommil jniloader 1.1
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java 1.1-natives
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java 1.1-natives
com.github.fommil.netlib netlib-native_ref-linux-x86_64 1.1-natives
com.github.fommil.netlib netlib-native_system-linux-x86_64 1.1-natives
com.github.luben zstd-jni 1.5.5-4
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.10.1
com.google.crypto.tink tink 1.9.0
com.google.errorprone error_prone_annotations 2.10.0
com.google.flatbuffers flatbuffers-java 23.5.26
com.google.guava guava 15.0
com.google.protobuf protobuf-java 3.25.1
com.helger profiler 1.1.1
com.ibm.icu icu4j 75.1
com.jcraft jsch 0.1.55
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.microsoft.sqlserver mssql-jdbc 11.2.2.jre8
com.microsoft.sqlserver mssql-jdbc 11.2.3.jre8
com.ning compress-lzf 1.1.2
com.sun.mail javax.mail 1.5.2
com.sun.xml.bind jaxb-core 2.2.11
com.sun.xml.bind jaxb-impl 2.2.11
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.4.3
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.uber h3 3.7.3
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 4.0.3
commons-cli commons-cli 1.5.0
commons-codec commons-codec 1.16.0
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.5
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.13.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 3.0.3
dev.ludovic.netlib blas 3.0.3
dev.ludovic.netlib lapack 3.0.3
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.27
io.delta delta-sharing-client_2.12 1.2.2
io.dropwizard.metrics metrics-annotation 4.2.19
io.dropwizard.metrics metrics-core 4.2.19
io.dropwizard.metrics metrics-graphite 4.2.19
io.dropwizard.metrics metrics-healthchecks 4.2.19
io.dropwizard.metrics metrics-jetty9 4.2.19
io.dropwizard.metrics metrics-jmx 4.2.19
io.dropwizard.metrics metrics-json 4.2.19
io.dropwizard.metrics metrics-jvm 4.2.19
io.dropwizard.metrics metrics-servlets 4.2.19
io.netty netty-all 4.1.108.Final
io.netty netty-buffer 4.1.108.Final
io.netty netty-codec 4.1.108.Final
io.netty netty-codec-http 4.1.108.Final
io.netty netty-codec-http2 4.1.108.Final
io.netty netty-codec-socks 4.1.108.Final
io.netty netty-common 4.1.108.Final
io.netty netty-handler 4.1.108.Final
io.netty netty-handler-proxy 4.1.108.Final
io.netty netty-resolver 4.1.108.Final
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16-linux-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16-linux-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16-osx-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16-osx-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16-windows-x86_64
io.netty netty-tcnative-classes 2.0.61.Final
io.netty netty-transport 4.1.108.Final
io.netty netty-transport-classes-epoll 4.1.108.Final
io.netty netty-transport-classes-kqueue 4.1.108.Final
io.netty netty-transport-native-epoll 4.1.108.Final
io.netty netty-transport-native-epoll 4.1.108.Final-linux-aarch_64
io.netty netty-transport-native-epoll 4.1.108.Final-linux-riscv64
io.netty netty-transport-native-epoll 4.1.108.Final-linux-x86_64
io.netty netty-transport-native-kqueue 4.1.108.Final-osx-aarch_64
io.netty netty-transport-native-kqueue 4.1.108.Final-osx-x86_64
io.netty netty-transport-native-unix-common 4.1.108.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.11
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.12.1
net.java.dev.jna jna 5.8.0
net.razorvine pickle 1.3
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.9.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.10.11
org.apache.ant ant-jsch 1.10.11
org.apache.ant ant-launcher 1.10.11
org.apache.arrow arrow-format 15.0.0
org.apache.arrow arrow-memory-core 15.0.0
org.apache.arrow arrow-memory-netty 15.0.0
org.apache.arrow arrow-vector 15.0.0
org.apache.avro avro 1.11.3
org.apache.avro avro-ipc 1.11.3
org.apache.avro avro-mapred 1.11.3
org.apache.commons commons-collections4 4.4
org.apache.commons commons-compress 1.23.0
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.6.1
org.apache.commons commons-text 1.10.0
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.datasketches datasketches-java 3.1.0
org.apache.datasketches datasketches-memory 2.0.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-runtime 3.3.6
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.8.1
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.httpcomponents httpclient 4.5.14
org.apache.httpcomponents httpcore 4.4.16
org.apache.ivy ivy 2.5.2
org.apache.logging.log4j log4j-1.2-api 2.22.1
org.apache.logging.log4j log4j-api 2.22.1
org.apache.logging.log4j log4j-core 2.22.1
org.apache.logging.log4j log4j-layout-template-json 2.22.1
org.apache.logging.log4j log4j-slf4j2-impl 2.22.1
org.apache.orc orc-core 1.9.2-shaded-protobuf
org.apache.orc orc-mapreduce 1.9.2-shaded-protobuf
org.apache.orc orc-shims 1.9.2
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.ws.xmlschema xmlschema-core 2.3.0
org.apache.xbean xbean-asm9-shaded 4.23
org.apache.yetus audience-annotations 0.13.0
org.apache.zookeeper zookeeper 3.9.2
org.apache.zookeeper zookeeper-jute 3.9.2
org.checkerframework checker-qual 3.31.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.collections eclipse-collections 11.1.0
org.eclipse.collections eclipse-collections-api 11.1.0
org.eclipse.jetty jetty-client 9.4.52.v20230823
org.eclipse.jetty jetty-continuation 9.4.52.v20230823
org.eclipse.jetty jetty-http 9.4.52.v20230823
org.eclipse.jetty jetty-io 9.4.52.v20230823
org.eclipse.jetty jetty-jndi 9.4.52.v20230823
org.eclipse.jetty jetty-plus 9.4.52.v20230823
org.eclipse.jetty jetty-proxy 9.4.52.v20230823
org.eclipse.jetty jetty-security 9.4.52.v20230823
org.eclipse.jetty jetty-server 9.4.52.v20230823
org.eclipse.jetty jetty-servlet 9.4.52.v20230823
org.eclipse.jetty jetty-servlets 9.4.52.v20230823
org.eclipse.jetty jetty-util 9.4.52.v20230823
org.eclipse.jetty jetty-util-ajax 9.4.52.v20230823
org.eclipse.jetty jetty-webapp 9.4.52.v20230823
org.eclipse.jetty jetty-xml 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-api 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-client 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-common 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-server 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-servlet 9.4.52.v20230823
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.40
org.glassfish.jersey.containers jersey-container-servlet-core 2.40
org.glassfish.jersey.core jersey-client 2.40
org.glassfish.jersey.core jersey-common 2.40
org.glassfish.jersey.core jersey-server 2.40
org.glassfish.jersey.inject jersey-hk2 2.40
org.hibernate.validator hibernate-validator 6.1.7.Final
org.ini4j ini4j 0.5.4
org.javassist javassist 3.29.2-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M11
org.json4s json4s-core_2.12 3.7.0-M11
org.json4s json4s-jackson_2.12 3.7.0-M11
org.json4s json4s-scalap_2.12 3.7.0-M11
org.lz4 lz4-java 1.8.0-databricks-1
org.mlflow mlflow-spark_2.12 2.9.1
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.6.1
org.roaringbitmap RoaringBitmap 0.9.45-databricks
org.roaringbitmap shims 0.9.45-databricks
org.rocksdb rocksdbjni 9.2.1
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.15
org.scala-lang scala-library_2.12 2.12.15
org.scala-lang scala-reflect_2.12 2.12.15
org.scala-lang.modules scala-collection-compat_2.12 2.11.0
org.scala-lang.modules scala-java8-compat_2.12 0.9.1
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.2.16
org.scalanlp breeze-macros_2.12 2.1.0
org.scalanlp breeze_2.12 2.1.0
org.scalatest scalatest-compatible 3.2.16
org.scalatest scalatest-core_2.12 3.2.16
org.scalatest scalatest-diagrams_2.12 3.2.16
org.scalatest scalatest-featurespec_2.12 3.2.16
org.scalatest scalatest-flatspec_2.12 3.2.16
org.scalatest scalatest-freespec_2.12 3.2.16
org.scalatest scalatest-funspec_2.12 3.2.16
org.scalatest scalatest-funsuite_2.12 3.2.16
org.scalatest scalatest-matchers-core_2.12 3.2.16
org.scalatest scalatest-mustmatchers_2.12 3.2.16
org.scalatest scalatest-propspec_2.12 3.2.16
org.scalatest scalatest-refspec_2.12 3.2.16
org.scalatest scalatest-shouldmatchers_2.12 3.2.16
org.scalatest scalatest-wordspec_2.12 3.2.16
org.scalatest scalatest_2.12 3.2.16
org.slf4j jcl-over-slf4j 2.0.7
org.slf4j jul-to-slf4j 2.0.7
org.slf4j slf4j-api 2.0.7
org.slf4j slf4j-simple 1.7.25
org.threeten threeten-extra 1.7.1
org.tukaani xz 1.9
org.typelevel algebra_2.12 2.0.1
org.typelevel cats-kernel_2.12 2.1.1
org.typelevel spire-macros_2.12 0.17.0
org.typelevel spire-platform_2.12 0.17.0
org.typelevel spire-util_2.12 0.17.0
org.typelevel spire_2.12 0.17.0
org.wildfly.openssl wildfly-openssl 1.1.3.Final
org.xerial sqlite-jdbc 3.42.0.0
org.xerial.snappy snappy-java 1.1.10.3
org.yaml snakeyaml 2.0
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.cryptools AmazonCorrettoCryptoProvider 2.4.1-linux-x86_64
stax stax-api 1.0.1