Databricks Runtime 15.4 LTS
The following release notes provide information about Databricks Runtime 15.4 LTS, powered by Apache Spark 3.5.0.
Databricks released this version in August 2024.
Note
LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.
Tip
To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.
Behavioral changes
- Using a
VARIANT
as the input or output type with a Python UDF, UDAF, or UDTF throws an exception - Change to the default schema binding mode for views
- Disallow using the undocumented
!
syntax instead ofNOT
outside boolean expressions - Disallow undocumented column definition syntax in views
- Consistent error handling for Base64 decoding in Spark and Photon
- Adding a
CHECK
constraint on an invalid column now returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class
Using a VARIANT
as the input or output type with a Python UDF, UDAF, or UDTF throws an exception
[Breaking change] In Databricks Runtime 15.3 and above, calling any Python user-defined function (UDF), user-defined aggregate function (UDAF), or user-defined table function (UDTF) that uses a VARIANT
type as an argument or return value throws an exception. This change is made to prevent issues that might occur because of an invalid value returned by one of these functions. To learn more about the VARIANT
type, see use VARIANTs to store semi-structured data.
Change to the default schema binding mode for views
Views now adapt to schema changes in the underlying query by using schema compensation with regular casting rules. This is a change from the previous default of BINDING
mode, which raised errors when a safe cast could not be performed when referencing the view.
See CREATE VIEW and cast function.
Disallow using the undocumented !
syntax instead of NOT
outside boolean expressions
With this release, the use of !
as a synonym for NOT
outside of boolean expressions is no longer allowed. For example, statements such as the following: CREATE ... IF ! EXISTS
, IS ! NULL, a ! NULL
column or field property, ! IN
and ! BETWEEN, must be replaced with: CREATE ... IF NOT EXISTS
, IS NOT NULL
, a NOT NULL
column or field property, NOT IN
and NOT BETWEEN
.
This change ensures consistency, aligns with the SQL standard, and makes your SQL more portable.
The boolean prefix operator !
(for example, !is_mgr
or !(true AND false)
) is unaffected by this change.
Disallow undocumented column definition syntax in views
Databricks supports CREATE VIEW with named columns and column comments. Previously, the specification of column types, NOT NULL
constraints, or DEFAULT
has been allowed. With this release, you can no longer use this syntax.
This change ensures consistency, aligns with the SQL standard, and supports future enhancements.
Consistent error handling for Base64 decoding in Spark and Photon
This release changes how Photon handles Base64 decoding errors to match the Spark handling of these errors. Before these changes, the Photon and Spark code generation path sometimes failed to raise parsing exceptions, while the Spark interpreted execution correctly raised IllegalArgumentException
or ConversionInvalidInputError
. This update ensures that Photon consistently raises the same exceptions as Spark during Base64 decoding errors, providing more predictable and reliable error handling.
Adding a CHECK
constraint on an invalid column now returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class
To provide more useful error messaging, in Databricks Runtime 15.3 and above, an ALTER TABLE ADD CONSTRAINT
statement that includes a CHECK
constraint referencing an invalid column name returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class. Previously, an INTERNAL_ERROR
was returned.
New features and improvements
- UTF-8 validation functions
- Typed Dataset APIs with Scala UDFs
- Enable UniForm Iceberg using ALTER TABLE
- try_url_decode function
- Optionally allow the optimizer to rely on unenforced foreign key constraints
- Parallelized job runs for selective overwrites
- Improved performance for change data feed with selective overwrites
- Improved query latency for the
COPY INTO
command - Support for dropping the check constraints table feature
- Single user compute supports fine-grained access control, materialized views, and streaming tables (Public Preview)
- Expanded support for Java and Scala libraries
- Expanded support for Scala Dataset operations
- Scala is GA on Unity Catalog shared compute
- Unity Catalog-governed access to external cloud services using service credentials (Public Preview)
UTF-8 validation functions
This release introduces the following functions for validating UTF-8 strings:
- is_valid_utf8 verified whether a string is a valid UTF-8 string.
- make_valid_utf8 converts a potentially invalid UTF-8 string to a valid UTF-8 string using substitution characters
- validate_utf8 raises an error if the input is not a valid UTF-8 string.
- try_validate_utf8 returns
NULL
if the input is not a valid UTF-8 string.
Typed Dataset APIs with Scala UDFs
This release includes added support for typed Dataset APIs with Scala user-defined functions (excluding user-defined aggregate functions) on Unity Catalog-enabled compute with shared access mode. See Typed Dataset APIs.
Enable UniForm Iceberg using ALTER TABLE
You can now enable UniForm Iceberg on existing tables without rewriting data files. See Enable Iceberg reads on an existing table.
try_url_decode function
This release introduces the try_url_decode function, which decodes a URL-encoded string. If the string is not in the correct format, the function returns NULL
instead of raising an error.
Optionally allow the optimizer to rely on unenforced foreign key constraints
To improve query performance, you can now specify the RELY
keyword on FOREIGN KEY
constraints when you CREATE or ALTER a table.
Parallelized job runs for selective overwrites
Selective overwrites using replaceWhere
now run jobs that delete data and insert new data in parallel, improving query performance and cluster utilization.
Improved performance for change data feed with selective overwrites
Selective overwrites using replaceWhere
on tables with change data feed no longer write separate change data files for inserted data. These operations use a hidden _change_type
column present in the underlying Parquet data files to record changes without write amplification.
Improved query latency for the COPY INTO
command
This release includes a change that improves the query latency for the COPY INTO
command. This improvement is implemented by making the loading of state by the RocksDB state store asynchronous. With this change, you should see an improvement in start times for queries with large states, such as queries with a large number of already ingested files.
Support for dropping the check constraints table feature
You can now drop the checkConstraints
table feature from a Delta table using ALTER TABLE table_name DROP FEATURE checkConstraints
. See Disable check constraints.
Single user compute supports fine-grained access control, materialized views, and streaming tables (Public Preview)
When a workspace is enabled for serverless compute, Databricks Runtime 15.4 LTS adds support for fine-grained access control on single user compute. When a query accesses any of the following objects, the single user compute resource on Databricks Runtime 15.4 LTS passes the query to the serverless compute to run data filtering:
- Views defined over tables on which the user does not have the
SELECT
privilege. - Dynamic views.
- Tables with row filters or column masks applied.
- Materialized views and streaming tables.
These queries are unsupported on single user compute running Databricks Runtime 15.3 and below.
For more information, see Fine-grained access control on single user compute.
Expanded support for Java and Scala libraries
Starting in Databricks Runtime 15.4 LTS, all Java and Scala libraries bundled with Databricks Runtime are available in all compute access modes when you use Unity Catalog. To learn more about language support on Unity Catalog-enabled compute, see Compute access mode limitations for Unity Catalog.
Expanded support for Scala Dataset operations
With this release, Unity Catalog-enabled compute resources that use shared access mode support the following Scala Dataset
operations: map
, mapPartitions
, foreachPartition
, flatMap
, reduce
and filter
.
Scala is GA on Unity Catalog shared compute
With this release, Scala is generally available on shared access mode Unity Catalog-enabled compute, including support for scalar user-defined functions (UDFs). Structured Streaming, Hive UDFs, and Hive user-defined aggregate functions are not supported. For a complete list of limitations, see Compute access mode limitations for Unity Catalog.
Unity Catalog-governed access to external cloud services using service credentials (Public Preview)
Service credentials enable simple and secure authentication with your cloud tenant’s services using Azure managed identities (MI) and Unity Catalog. See Manage access to external cloud services using service credentials.
Bug fixes
Library upgrades
- Upgraded Python libraries:
- azure-core from 1.30.1 to 1.30.2
- google-auth from 2.29.0 to 2.31.0
- google-cloud-storage from 2.16.0 to 2.17.0
- google-resumable-media from 2.7.0 to 2.7.1
- googleapis-common-protos from 1.63.0 to 1.63.2
- mlflow-skinny from 2.11.3 to 2.11.4
- proto-plus from 1.23.0 to 1.24.0
- s3transfer from 0.10.1 to 0.10.2
- Upgraded R libraries:
- Upgraded Java libraries:
- com.databricks.databricks-sdk-java from 0.17.1 to 0.27.0
- com.ibm.icu.icu4j from 72.1 to 75.1
- software.amazon.cryptools.AmazonCorrettoCryptoProvider from 1.6.1-linux-x86_64 to 1.6.2-linux-x86_64
Apache Spark
Databricks Runtime 15.4 LTS includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 15.3, and the following additional bug fixes and improvements made to Spark:
- [SPARK-48503] [DBRRM-1150][SC-172196][SQL] Allow grouping on expressions in scalar subqueries, if they are bound to outer rows
- [SPARK-48834] [BEHAVE-79][SC-170972][SQL] Disable variant input/output to python scalar UDFs, UDTFs, UDAFs during query compilation
- [SPARK-48441] [SC-170980][SQL][WARMFIX] Fix StringTrim behaviour for non-UTF8_BINARY collations
- [SPARK-48440] [SC-170895][SQL][WARMFIX] Fix StringTranslate behaviour for non-UTF8_BINARY collations
- [SPARK-48872] [SC-170866][PYTHON] Reduce the overhead of _capture_call_site
- [SPARK-48862] [SC-170845][PYTHON][CONNECT] Avoid calling
_proto_to_string
when INFO level is not enabled - [SPARK-48852] [SC-170837][CONNECT] Fix string trim function in connect
- [SPARK-48791] [SC-170658][CORE] Fix perf regression caused by the accumulators registration overhead using CopyOnWriteArrayList
- [SPARK-48118] [SQL] Support
SPARK_SQL_LEGACY_CREATE_HIVE_TABLE
env variable - [SPARK-48241] [SC-165811][SQL] CSV parsing failure with char/varchar type columns
- [SPARK-48168] [SC-166900][SQL] Add bitwise shifting operators support
- [SPARK-48148] [SC-165630][CORE] JSON objects should not be modified when read as STRING
- [SPARK-46625] [SC-170561] CTE with Identifier clause as reference
- [SPARK-48771] [SC-170546][SQL] Speed up
LogicalPlanIntegrity.validateExprIdUniqueness
for large query plans - [SPARK-48831] [BEHAVE-76][SC-170554][CONNECT] Make default column name of
cast
compatible with Spark Classic - [SPARK-48623] [SC-170544][CORE] Structured logging migrations [Part 2]
- [SPARK-48296] [SC-166138][SQL] Codegen Support for
to_xml
- [SPARK-48027] [SC-165154][SQL] InjectRuntimeFilter for multi-level join should check child join type
- [SPARK-48686] [SC-170365][SQL] Improve performance of ParserUtils.unescapeSQLString
- [SPARK-48798] [SC-170588][PYTHON] Introduce
spark.profile.render
for SparkSession-based profiling - [SPARK-48048] [SC-169099] Revert “[SC-164846][CONNECT][SS] Added client side listener support for Scala”
- [SPARK-47910] [SC-168929][CORE] close stream when DiskBlockObjectWriter closeResources to avoid memory leak
- [SPARK-48816] [SC-170547][SQL] Shorthand for interval converters in UnivocityParser
- [SPARK-48589] [SC-170132][SQL][SS] Add option snapshotStartBatchId and snapshotPartitionId to state data source
- [SPARK-48280] [SC-170293][SQL] Improve collation testing surface area using expression walking
- [SPARK-48837] [SC-170540][ML] In CountVectorizer, only read binary parameter once per transform, not once per row
- [SPARK-48803] [SC-170541][SQL] Throw internal error in Orc(De)serializer to align with ParquetWriteSupport
- [SPARK-48764] [SC-170129][PYTHON] Filtering out IPython-related frames from user stack
- [SPARK-48818] [SC-170414][PYTHON] Simplify
percentile
functions - [SPARK-48479] [SC-169079][SQL] Support creating scalar and table SQL UDFs in parser
- [SPARK-48697] [SC-170122][LC-4703][SQL] Add collation aware string filters
- [SPARK-48800] [SC-170409][CONNECT][SS] Deflake ClientStreamingQuerySuite
- [SPARK-48738] [SC-169814][SQL] Correct since version for built-in func alias
random
,position
,mod
,cardinality
,current_schema
,user
,session_user
,char_length
,character_length
- [SPARK-48638] [SC-169575][CONNECT] Add ExecutionInfo support for DataFrame
- [SPARK-48064] [SC-164697][SQL] Update error messages for routine related error classes
- [SPARK-48810] [CONNECT] Session stop() API should be idempotent and not fail if the session is already closed by the server
- [SPARK-48650] [15.x][PYTHON] Display correct call site from IPython Notebook
- [SPARK-48271] [SC-166076][SQL] Turn match error in RowEncoder into UNSUPPORTED_DATA_TYPE_FOR_ENCODER
- [SPARK-48709] [SC-169603][SQL] Fix varchar type resolution mismatch for DataSourceV2 CTAS
- [SPARK-48792] [SC-170335][SQL] Fix regression for INSERT with partial column list to a table with char/varchar
- [SPARK-48767] [SC-170330][SQL] Fix some error prompts when
variant
type data is invalid - [SPARK-48719] [SC-170339][SQL] Fix the calculation bug of
RegrSlope
&RegrIntercept
when the first parameter is null - [SPARK-48815] [SC-170334][CONNECT] Update environment when stoping connect session
- [SPARK-48646] [SC-169020][PYTHON] Refine Python data source API docstring and type hints
- [SPARK-48806] [SC-170310][SQL] Pass actual exception when url_decode fails
- [SPARK-47777] [SC-168818] fix python streaming data source connect test
- [SPARK-48732] [SC-169793][SQL] Cleanup deprecated api usage related to
JdbcDialect.compileAggregate
- [SPARK-48675] [SC-169538][SQL] Fix cache table with collated column
- [SPARK-48623] [SC-169034][CORE] Structured Logging Migrations
- [SPARK-48655] [SC-169542][SQL] SPJ: Add tests for shuffle skipping for aggregate queries
- [SPARK-48586] [SC-169808][SS] Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()
- [SPARK-48799] [Backport][15.x][SC-170283][SS] Refactor versioning for operator metadata read/write and callers
- [SPARK-48808] [SC-170309][SQL] Fix NPE when connecting thriftserver through Hive 1.2.1 and the result schema is empty
- [SPARK-48715] [SC-170291][SQL] Integrate UTF8String validation into collation-aware string function implementations
- [SPARK-48747] [SC-170120][SQL] Add code point iterator to UTF8String
- [SPARK-48748] [SC-170115][SQL] Cache numChars in UTF8String
- [SPARK-48744] [SC-169817][Core] Log entry should be constructed only once
- [SPARK-46122] [SC-164313][SQL] Set
spark.sql.legacy.createHiveTableByDefault
tofalse
by default - [SPARK-48765] [SC-170119][DEPLOY] Enhance default value evaluation for SPARK_IDENT_STRING
- [SPARK-48759] [SC-170128][SQL] Add migration doc for CREATE TABLE AS SELECT behavior change behavior change since Spark 3.4
- [SPARK-48598] [SC-169484][PYTHON][CONNECT] Propagate cached schema in dataframe operations
- [SPARK-48766] [SC-170126][PYTHON] Document the behavior difference of
extraction
betweenelement_at
andtry_element_at
- [SPARK-48768] [SC-170124][PYTHON][CONNECT] Should not cache
explain
- [SPARK-48770] [Backport][15.x][SC-170133][SS] Change to read operator metadata once on driver to check if we can find info for numColsPrefixKey used for session window agg queries
- [SPARK-48656] [SC-169529][CORE] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in
CartesianRDD.getPartitions
- [SPARK-48597] [SC-168817][SQL] Introduce a marker for isStreaming property in text representation of logical plan
- [SPARK-48472] [SC-169044][SQL] Enable reflect expressions with collated strings
- [SPARK-48699] [SC-169597][SQL] Refine collation API
- [SPARK-48682] [SC-169812][SQL][BEHAVE-58] Use ICU in InitCap expression for UTF8_BINARY strings
- [SPARK-48282] [SC-169813][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringReplace, FindInSet)
- [SPARK-47353] [SC-169599][SQL] Enable collation support for the Mode expression
- [SPARK-48320] [SPARK-48490] Sync the latest Logging trait and test cases from OSS Spark
- [SPARK-48629] [SC-169479] Migrate the residual code to structured logging framework
- [SPARK-48681] [SC-169469][SQL][BEHAVE-58] Use ICU in Lower/Upper expressions for UTF8_BINARY strings
- [SPARK-48573] [15.x][SC-169582][SQL] Upgrade ICU version
- [SPARK-48687] [Backport][15.x][SS] Add change to perform state schema validation and update on driver for stateful queries
- [SPARK-47579] [15.x][SC-167310][CORE][PART4] Migrate logInfo with variables to structured logging framework
- [SPARK-48008] [SC-167363][1/2] Support UDAFs in Spark Connect
- [SPARK-48578] [SC-169505][SQL] add UTF8 string validation related functions
- [SPARK-48670] [SC-169598][SQL] Providing suggestion as part of error message when invalid collation name is given
- [SPARK-48059] [SPARK-48145][SPARK-48134][SPARK-48182][SPARK-48209][SPA… …RK-48291] Structured log framework on the java side
- [SPARK-47599] [15.x][SC-166000][MLLIB] MLLib: Migrate logWarn with variables to structured logging framework
- [SPARK-48706] [SC-169589][PYTHON] Python UDF in higher order functions should not throw internal error
- [SPARK-48498] [BEHAVE-38][SC-168060][SQL] Always do char padding in predicates
- [SPARK-48662] [SC-169533][SQL] Fix StructsToXml expression with collations
- [SPARK-48482] [SC-167702][PYTHON][15.x] dropDuplicates and dropDuplicatesWIthinWatermark should accept variable length args
- [SPARK-48678] [SC-169463][CORE] Performance optimizations for SparkConf.get(ConfigEntry)
- [SPARK-48576] [SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE
- [SPARK-47927] [SC-164123][SQL]: Fix nullability attribute in UDF decoder
- [SPARK-47579] [SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework (new)
- [SPARK-48695] [SC-169473][PYTHON]
TimestampNTZType.fromInternal
not use the deprecated methods - [SPARK-48431] [SC-167290][LC-4066][SQL] Do not forward predicates on collated columns to file readers
- [SPARK-47579] Revert “[SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework”
- [SPARK-47585] [SC-164306][SQL] SQL core: Migrate logInfo with variables to structured logging framework
- [SPARK-48466] [SC-169042][SQL] Create dedicated node for EmptyRelation in AQE
- [SPARK-47579] [SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework
- [SPARK-48410] [SC-168320][SQL] Fix InitCap expression for UTF8_BINARY_LCASE & ICU collations
- [SPARK-48318] [SC-167709][SQL] Enable hash join support for all collations (complex types)
- [SPARK-48435] [SC-168128][SQL] UNICODE collation should not support binary equality
- [SPARK-48555] [SC-169041][SQL][PYTHON][CONNECT] Support using Columns as parameters for several functions in pyspark/scala
- [SPARK-48591] [SC-169081][PYTHON] Add a helper function to simplify
Column.py
- [SPARK-48574] [SC-169043][SQL] Fix support for StructTypes with collations
- [SPARK-48305] [SC-166390][SQL] Add collation support for CurrentLike expressions
- [SPARK-48342] [SC-168941][SQL] Introduction of SQL Scripting Parser
- [SPARK-48649] [SC-169024][SQL] Add “ignoreInvalidPartitionPaths” and “spark.sql.files.ignoreInvalidPartitionPaths” configs to allow ignoring invalid partition paths
- [SPARK-48000] [SC-167194][SQL] Enable hash join support for all collations (StringType)
- [SPARK-48459] [SC-168947][CONNECT][PYTHON] Implement DataFrameQueryContext in Spark Connect
- [SPARK-48602] [SC-168692][SQL] Make csv generator support different output style with spark.sql.binaryOutputStyle
- [SPARK-48283] [SC-168129][SQL] Modify string comparison for UTF8_BINARY_LCASE
- [SPARK-48610] [SC-168830][SQL] refactor: use auxiliary idMap instead of OP_ID_TAG
- [SPARK-48634] [SC-169021][PYTHON][CONNECT] Avoid statically initialize threadpool at ExecutePlanResponseReattachableIterator
- [SPARK-47911] [SC-164658][SQL] Introduces a universal BinaryFormatter to make binary output consistent
- [SPARK-48642] [SC-168889][CORE] False SparkOutOfMemoryError caused by killing task on spilling
- [SPARK-48572] [SC-168844][SQL] Fix DateSub, DateAdd, WindowTime, TimeWindow and SessionWindow expressions
- [SPARK-48600] [SC-168841][SQL] Fix FrameLessOffsetWindowFunction expressions implicit casting
- [SPARK-48644] [SC-168933][SQL] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in Hex.hex
- [SPARK-48587] [SC-168824][VARIANT] Avoid storage amplification when accessing a sub-Variant
- [SPARK-48647] [SC-168936][PYTHON][CONNECT] Refine the error message for
YearMonthIntervalType
indf.collect
- [SPARK-48307] [SC-167802][SQL] InlineCTE should keep not-inlined relations in the original WithCTE node
- [SPARK-48596] [SC-168581][SQL] Perf improvement for calculating hex string for long
- [SPARK-48621] [SC-168726][SQL] Fix Like simplification in Optimizer for collated strings
- [SPARK-47148] [SC-164179][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation
- [SPARK-48584] [SC-168579][SQL] Perf improvement for unescapePathName
- [SPARK-48281] [SC-167260][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringInStr, SubstringIndex)
- [SPARK-48577] [SC-168826][SQL] Invalid UTF-8 byte sequence replacement
- [SPARK-48595] [SC-168580][CORE] Cleanup deprecated api usage related to
commons-compress
- [SPARK-48030] [SC-164303][SQL] SPJ: cache rowOrdering and structType for InternalRowComparableWrapper
- [SPARK-48004] [SC-164005][SQL] Add WriteFilesExecBase trait for v1 write
- [SPARK-48551] [SC-168438][SQL] Perf improvement for escapePathName
- [SPARK-48565] [SC-168437][UI] Fix thread dump display in UI
- [SPARK-48364] [SC-166782][SQL] Add AbstractMapType type casting and fix RaiseError parameter map to work with collated strings
- [SPARK-48421] [SC-168689][SQL] SPJ: Add documentation
- [SPARK-48604] [SC-168698][SQL] Replace deprecated
new ArrowType.Decimal(precision, scale)
method call - [SPARK-46947] [SC-157561][CORE] Delay memory manager initialization until Driver plugin is loaded
- [SPARK-48411] [SC-168576][SS][PYTHON] Add E2E test for DropDuplicateWithinWatermark
- [SPARK-48543] [SC-168697][SS] Track state row validation failures using explicit error class
- [SPARK-48221] [SC-167143][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (Contains, StartsWith, EndsWith, StringLocate)
- [SPARK-47415] [SC-168441][SQL] Add collation support for Levenshtein expression
- [SPARK-48593] [SC-168719][PYTHON][CONNECT] Fix the string representation of lambda function
- [SPARK-48622] [SC-168710][SQL] get SQLConf once when resolving column names
- [SPARK-48594] [SC-168685][PYTHON][CONNECT] Rename
parent
field tochild
inColumnAlias
- [SPARK-48403] [SC-168319][SQL] Fix Lower & Upper expressions for UTF8_BINARY_LCASE & ICU collations
- [SPARK-48162] [SC-166062][SQL] Add collation support for MISC expressions
- [SPARK-48518] [SC-167718][CORE] Make LZF compression be able to run in parallel
- [SPARK-48474] [SC-167447][CORE] Fix the class name of the log in
SparkSubmitArguments
&SparkSubmit
- [SPARK-48012] [SC-168267][SQL] SPJ: Support Transfrom Expressions for One Side Shuffle
- [SPARK-48552] [SC-168212][SQL] multi-line CSV schema inference should also throw FAILED_READ_FILE
- [SPARK-48560] [SC-168268][SS][PYTHON] Make StreamingQueryListener.spark settable
- [SPARK-48569] [SC-168321][SS][CONNECT] Handle edge cases in query.name
- [SPARK-47260] [SC-167323][SQL] Assign name to error class _LEGACY_ERROR_TEMP_3250
- [SPARK-48564] [SC-168327][PYTHON][CONNECT] Propagate cached schema in set operations
- [SPARK-48155] [SC-165910][SQL] AQEPropagateEmptyRelation for join should check if remain child is just BroadcastQueryStageExec
- [SPARK-48506] [SC-167720][CORE] Compression codec short names are case insensitive except for event logging
- [SPARK-48447] [SC-167607][SS] Check state store provider class before invoking the constructor
- [SPARK-47977] [SC-167650] DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd should not throw INTERNAL_ERROR exception
- [SPARK-48513] [Backport][15.x][SC-168085][SS] Add error class for state schema compatibility and minor refactoring
- [SPARK-48413] [SC-167669][SQL] ALTER COLUMN with collation
- [SPARK-48561] [SC-168250][PS][CONNECT] Throw
PandasNotImplementedError
for unsupported plotting functions - [SPARK-48465] [SC-167531][SQL] Avoid no-op empty relation propagation
- [SPARK-48553] [SC-168166][PYTHON][CONNECT] Cache more properties
- [SPARK-48540] [SC-168069][CORE] Avoid ivy output loading settings to stdout
- [SPARK-48535] [SC-168057][SS] Update config docs to indicate possibility of data loss/corruption issue if skip nulls for stream-stream joins config is enabled
- [SPARK-48536] [SC-168059][PYTHON][CONNECT] Cache user specified schema in applyInPandas and applyInArrow
- [SPARK-47873] [SC-163473][SQL] Write collated strings to Hive metastore using the regular string type
- [SPARK-48461] [SC-167442][SQL] Replace NullPointerExceptions with error class in AssertNotNull expression
- [SPARK-47833] [SC-163191][SQL][CORE] Supply caller stackstrace for checkAndGlobPathIfNecessary AnalysisException
- [SPARK-47898] [SC-163146][SQL] Port HIVE-12270: Add DBTokenStore support to HS2 delegation token
- [SPARK-47578] [SC-167497][R] Migrate RPackageUtils with variables to structured logging framework
- [SPARK-47875] [SC-162935][CORE] Remove
spark.deploy.recoverySerializer
- [SPARK-47552] [SC-160880][CORE] Set
spark.hadoop.fs.s3a.connection.establish.timeout
to 30s if missing - [SPARK-47972] [SC-167692][SQL] Restrict CAST expression for collations
- [SPARK-48430] [SC-167489][SQL] Fix map value extraction when map contains collated strings
- [SPARK-47318] [SC-162712][CORE][3.5] Adds HKDF round to AuthEngine key derivation to follow standard KEX practices
- [SPARK-48503] [BEHAVE-29][ES-1135236][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed
- [SPARK-48508] [SC-167695][CONNECT][PYTHON] Cache user specified schema in
DataFrame.{to, mapInPandas, mapInArrow}
- [SPARK-23015] [SC-167188][WINDOWS] Fix bug in Windows where starting multiple Spark instances within the same second causes a failure
- [SPARK-45891] [SC-167608]Revert ” Describe shredding scheme for Variant”
- [SPARK-48391] [SC-167554][CORE]Using addAll instead of add function in fromAccumulatorInfos method of TaskMetrics Class
- [SPARK-48496] [SC-167600][CORE] Use static regex Pattern instances in JavaUtils.timeStringAs and JavaUtils.byteStringAs
- [SPARK-48476] [SC-167488][SQL] fix NPE error message for null delmiter csv
- [SPARK-48489] [SC-167598][SQL] Throw a better, user-facing error when reading illegal schema from text Data Source
- [SPARK-48471] [SC-167324][CORE] Improve documentation and usage guide for history server
- [SPARK-45891] [SC-167597] Describe shredding scheme for Variant
- [SPARK-47333] [SC-159043][SQL] Use checkInputDataTypes to check the parameter types of the function
to_xml
- [SPARK-47387] [SC-159310][SQL] Remove some unused error classes
- [SPARK-48265] [ES-1131440][SQL] Infer window group limit batch should do constant folding
- [SPARK-47716] [SC-167444][SQL] Avoid view name conflict in SQLQueryTestSuite semantic sort test case
- [SPARK-48159] [SC-167141][SQL] Extending support for collated strings on datetime expressions
- [SPARK-48462] [SC-167271][SQL][Tests] Use withSQLConf in tests: Refactor HiveQuerySuite and HiveTableScanSuite
- [SPARK-48392] [SC-167195][CORE] Also load
spark-defaults.conf
when provided--properties-file
- [SPARK-48468] [SC-167417] Add LogicalQueryStage interface in catalyst
- [SPARK-47578] [SC-164988][CORE] Manual backport for Spark PR #46309: Migrate logWarning with variables to structured logging framework
- [SPARK-48415] [SC-167321]Revert “[PYTHON] Refactor TypeName to support parameterized datatypes”
- [SPARK-46544] [SC-151361][SQL] Support v2 DESCRIBE TABLE EXTENDED with table stats
- [SPARK-48325] [SC-166963][CORE] Always specify messages in ExecutorRunner.killProcess
- [SPARK-46841] [SC-167139][SQL] Add collation support for ICU locales and collation specifiers
- [SPARK-47221] [SC-157870][SQL] Uses signatures from CsvParser to AbstractParser
- [SPARK-47246] [SC-158138][SQL] Replace
InternalRow.fromSeq
withnew GenericInternalRow
to save a collection conversion - [SPARK-47597] [SC-163932][STREAMING] Manual backport for Spark PR #46192: Streaming: Migrate logInfo with variables to structured logging framework
- [SPARK-48415] [SC-167130][PYTHON] Refactor
TypeName
to support parameterized datatypes - [SPARK-48434] [SC-167132][PYTHON][CONNECT] Make
printSchema
use the cached schema - [SPARK-48432] [ES-1097114][SQL] Avoid unboxing integers in UnivocityParser
- [SPARK-47463] [SC-162840][SQL] Use V2Predicate to wrap expression with return type of boolean
- [SPARK-47781] [SC-162293][SQL] Handle negative scale decimals for JDBC data sources
- [SPARK-48394] [SC-166966][CORE] Cleanup mapIdToMapIndex on mapoutput unregister
- [SPARK-47072] [SC-156933][SQL] Fix supported interval formats in error messages
- [SPARK-47001] [SC-162487][SQL] Pushdown verification in optimizer
- [SPARK-48335] [SC-166387][PYTHON][CONNECT] Make
_parse_datatype_string
compatible with Spark Connect - [SPARK-48329] [SC-166518][SQL] Enable
spark.sql.sources.v2.bucketing.pushPartValues.enabled
by default - [SPARK-48412] [SC-166898][PYTHON] Refactor data type json parse
- [SPARK-48215] [SC-166781][SQL] Extending support for collated strings on date_format expression
- [SPARK-45009] [SC-166873][SQL][FOLLOW UP] Add error class and tests for decorrelation of predicate subqueries in join condition which reference both join child
- [SPARK-47960] [SC-165295][SS][15.x] Allow chaining other stateful operators after transformWithState operator.
- [SPARK-48340] [SC-166468][PYTHON] Support TimestampNTZ infer schema miss prefer_timestamp_ntz
- [SPARK-48157] [SC-165902][SQL] Add collation support for CSV expressions
- [SPARK-48158] [SC-165652][SQL] Add collation support for XML expressions
- [SPARK-48160] [SC-166064][SQL] Add collation support for XPATH expressions
- [SPARK-48229] [SC-165901][SQL] Add collation support for inputFile expressions
- [SPARK-48367] [SC-166487][CONNECT] Fix lint-scala for scalafmt to detect files to format properly
- [SPARK-47858] [SC-163095][SPARK-47852][PYTHON][SQL] Refactoring the structure for DataFrame error context
- [SPARK-48370] [SC-166787][CONNECT] Checkpoint and localCheckpoint in Scala Spark Connect client
- [SPARK-48247] [SC-166028][PYTHON] Use all values in a dict when inferring MapType schema
- [SPARK-48395] [SC-166794][PYTHON] Fix
StructType.treeString
for parameterized types - [SPARK-48393] [SC-166784][PYTHON] Move a group of constants to
pyspark.util
- [SPARK-48372] [SC-166776][SPARK-45716][PYTHON] Implement
StructType.treeString
- [SPARK-48258] [SC-166467][PYTHON][CONNECT] Checkpoint and localCheckpoint in Spark Connect
See Databricks Runtime 15.4 LTS maintenance updates.
System environment
- Operating System: Ubuntu 22.04.4 LTS
- Java: Zulu 8.78.0.19-CA-linux64
- Scala: 2.12.18
- Python: 3.11.0
- R: 4.3.2
- Delta Lake: 3.2.0
Installed Python libraries
Library | Version | Library | Version | Library | Version |
---|---|---|---|---|---|
asttokens | 2.0.5 | astunparse | 1.6.3 | azure-core | 1.30.2 |
azure-storage-blob | 12.19.1 | azure-storage-file-datalake | 12.14.0 | backcall | 0.2.0 |
black | 23.3.0 | blinker | 1.4 | boto3 | 1.34.39 |
botocore | 1.34.39 | cachetools | 5.3.3 | certifi | 2023.7.22 |
cffi | 1.15.1 | chardet | 4.0.0 | charset-normalizer | 2.0.4 |
click | 8.0.4 | cloudpickle | 2.2.1 | comm | 0.1.2 |
contourpy | 1.0.5 | cryptography | 41.0.3 | cycler | 0.11.0 |
Cython | 0.29.32 | databricks-sdk | 0.20.0 | dbus-python | 1.2.18 |
debugpy | 1.6.7 | decorator | 5.1.1 | distlib | 0.3.8 |
entrypoints | 0.4 | executing | 0.8.3 | facets-overview | 1.1.1 |
filelock | 3.13.4 | fonttools | 4.25.0 | gitdb | 4.0.11 |
GitPython | 3.1.43 | google-api-core | 2.18.0 | google-auth | 2.31.0 |
google-cloud-core | 2.4.1 | google-cloud-storage | 2.17.0 | google-crc32c | 1.5.0 |
google-resumable-media | 2.7.1 | googleapis-common-protos | 1.63.2 | grpcio | 1.60.0 |
grpcio-status | 1.60.0 | httplib2 | 0.20.2 | idna | 3.4 |
importlib-metadata | 6.0.0 | ipyflow-core | 0.0.198 | ipykernel | 6.25.1 |
ipython | 8.15.0 | ipython-genutils | 0.2.0 | ipywidgets | 7.7.2 |
isodate | 0.6.1 | jedi | 0.18.1 | jeepney | 0.7.1 |
jmespath | 0.10.0 | joblib | 1.2.0 | jupyter_client | 7.4.9 |
jupyter_core | 5.3.0 | keyring | 23.5.0 | kiwisolver | 1.4.4 |
launchpadlib | 1.10.16 | lazr.restfulclient | 0.14.4 | lazr.uri | 1.0.6 |
matplotlib | 3.7.2 | matplotlib-inline | 0.1.6 | mlflow-skinny | 2.11.4 |
more-itertools | 8.10.0 | mypy-extensions | 0.4.3 | nest-asyncio | 1.5.6 |
numpy | 1.23.5 | oauthlib | 3.2.0 | packaging | 23.2 |
pandas | 1.5.3 | parso | 0.8.3 | pathspec | 0.10.3 |
patsy | 0.5.3 | pexpect | 4.8.0 | pickleshare | 0.7.5 |
Pillow | 9.4.0 | pip | 23.2.1 | platformdirs | 3.10.0 |
plotly | 5.9.0 | prompt-toolkit | 3.0.36 | proto-plus | 1.24.0 |
protobuf | 4.24.1 | psutil | 5.9.0 | psycopg2 | 2.9.3 |
ptyprocess | 0.7.0 | pure-eval | 0.2.2 | pyarrow | 14.0.1 |
pyasn1 | 0.4.8 | pyasn1-modules | 0.2.8 | pyccolo | 0.0.52 |
pycparser | 2.21 | pydantic | 1.10.6 | Pygments | 2.15.1 |
PyGObject | 3.42.1 | PyJWT | 2.3.0 | pyodbc | 4.0.38 |
pyparsing | 3.0.9 | python-dateutil | 2.8.2 | python-lsp-jsonrpc | 1.1.1 |
pytz | 2022.7 | PyYAML | 6.0 | pyzmq | 23.2.0 |
requests | 2.31.0 | rsa | 4.9 | s3transfer | 0.10.2 |
scikit-learn | 1.3.0 | scipy | 1.11.1 | seaborn | 0.12.2 |
SecretStorage | 3.3.1 | setuptools | 68.0.0 | six | 1.16.0 |
smmap | 5.0.1 | sqlparse | 0.5.0 | ssh-import-id | 5.11 |
stack-data | 0.2.0 | statsmodels | 0.14.0 | tenacity | 8.2.2 |
threadpoolctl | 2.2.0 | tokenize-rt | 4.2.1 | tornado | 6.3.2 |
traitlets | 5.7.1 | typing_extensions | 4.10.0 | tzdata | 2022.1 |
ujson | 5.4.0 | unattended-upgrades | 0.1 | urllib3 | 1.26.16 |
virtualenv | 20.24.2 | wadllib | 1.3.6 | wcwidth | 0.2.5 |
wheel | 0.38.4 | zipp | 3.11.0 |
Installed R libraries
R libraries are installed from the Posit Package Manager CRAN snapshot.
Library | Version | Library | Version | Library | Version |
---|---|---|---|---|---|
arrow | 14.0.0.2 | askpass | 1.2.0 | assertthat | 0.2.1 |
backports | 1.4.1 | base | 4.3.2 | base64enc | 0.1-3 |
bigD | 0.2.0 | bit | 4.0.5 | bit64 | 4.0.5 |
bitops | 1.0-7 | blob | 1.2.4 | boot | 1.3-28 |
brew | 1.0-10 | brio | 1.1.4 | broom | 1.0.5 |
bslib | 0.6.1 | cachem | 1.0.8 | callr | 3.7.3 |
caret | 6.0-94 | cellranger | 1.1.0 | chron | 2.3-61 |
class | 7.3-22 | cli | 3.6.2 | clipr | 0.8.0 |
clock | 0.7.0 | cluster | 2.1.4 | codetools | 0.2-19 |
colorspace | 2.1-0 | commonmark | 1.9.1 | compiler | 4.3.2 |
config | 0.3.2 | conflicted | 1.2.0 | cpp11 | 0.4.7 |
crayon | 1.5.2 | credentials | 2.0.1 | curl | 5.2.0 |
data.table | 1.15.0 | datasets | 4.3.2 | DBI | 1.2.1 |
dbplyr | 2.4.0 | desc | 1.4.3 | devtools | 2.4.5 |
diagram | 1.6.5 | diffobj | 0.3.5 | digest | 0.6.34 |
downlit | 0.4.3 | dplyr | 1.1.4 | dtplyr | 1.3.1 |
e1071 | 1.7-14 | ellipsis | 0.3.2 | evaluate | 0.23 |
fansi | 1.0.6 | farver | 2.1.1 | fastmap | 1.1.1 |
fontawesome | 0.5.2 | forcats | 1.0.0 | foreach | 1.5.2 |
foreign | 0.8-85 | forge | 0.2.0 | fs | 1.6.3 |
future | 1.33.1 | future.apply | 1.11.1 | gargle | 1.5.2 |
generics | 0.1.3 | gert | 2.0.1 | ggplot2 | 3.4.4 |
gh | 1.4.0 | git2r | 0.33.0 | gitcreds | 0.1.2 |
glmnet | 4.1-8 | globals | 0.16.2 | glue | 1.7.0 |
googledrive | 2.1.1 | googlesheets4 | 1.1.1 | gower | 1.0.1 |
graphics | 4.3.2 | grDevices | 4.3.2 | grid | 4.3.2 |
gridExtra | 2.3 | gsubfn | 0.7 | gt | 0.10.1 |
gtable | 0.3.4 | hardhat | 1.3.1 | haven | 2.5.4 |
highr | 0.10 | hms | 1.1.3 | htmltools | 0.5.7 |
htmlwidgets | 1.6.4 | httpuv | 1.6.14 | httr | 1.4.7 |
httr2 | 1.0.0 | ids | 1.0.1 | ini | 0.3.1 |
ipred | 0.9-14 | isoband | 0.2.7 | iterators | 1.0.14 |
jquerylib | 0.1.4 | jsonlite | 1.8.8 | juicyjuice | 0.1.0 |
KernSmooth | 2.23-21 | knitr | 1.45 | labeling | 0.4.3 |
later | 1.3.2 | lattice | 0.21-8 | lava | 1.7.3 |
lifecycle | 1.0.4 | listenv | 0.9.1 | lubridate | 1.9.3 |
magrittr | 2.0.3 | markdown | 1.12 | MASS | 7.3-60 |
Matrix | 1.5-4.1 | memoise | 2.0.1 | methods | 4.3.2 |
mgcv | 1.8-42 | mime | 0.12 | miniUI | 0.1.1.1 |
mlflow | 2.10.0 | ModelMetrics | 1.2.2.2 | modelr | 0.1.11 |
munsell | 0.5.0 | nlme | 3.1-163 | nnet | 7.3-19 |
numDeriv | 2016.8-1.1 | openssl | 2.1.1 | parallel | 4.3.2 |
parallelly | 1.36.0 | pillar | 1.9.0 | pkgbuild | 1.4.3 |
pkgconfig | 2.0.3 | pkgdown | 2.0.7 | pkgload | 1.3.4 |
plogr | 0.2.0 | plyr | 1.8.9 | praise | 1.0.0 |
prettyunits | 1.2.0 | pROC | 1.18.5 | processx | 3.8.3 |
prodlim | 2023.08.28 | profvis | 0.3.8 | progress | 1.2.3 |
progressr | 0.14.0 | promises | 1.2.1 | proto | 1.0.0 |
proxy | 0.4-27 | ps | 1.7.6 | purrr | 1.0.2 |
R6 | 2.5.1 | ragg | 1.2.7 | randomForest | 4.7-1.1 |
rappdirs | 0.3.3 | rcmdcheck | 1.4.0 | RColorBrewer | 1.1-3 |
Rcpp | 1.0.12 | RcppEigen | 0.3.3.9.4 | reactable | 0.4.4 |
reactR | 0.5.0 | readr | 2.1.5 | readxl | 1.4.3 |
recipes | 1.0.9 | rematch | 2.0.0 | rematch2 | 2.1.2 |
remotes | 2.4.2.1 | reprex | 2.1.0 | reshape2 | 1.4.4 |
rlang | 1.1.3 | rmarkdown | 2.25 | RODBC | 1.3-23 |
roxygen2 | 7.3.1 | rpart | 4.1.21 | rprojroot | 2.0.4 |
Rserve | 1.8-13 | RSQLite | 2.3.5 | rstudioapi | 0.15.0 |
rversions | 2.1.2 | rvest | 1.0.3 | sass | 0.4.8 |
scales | 1.3.0 | selectr | 0.4-2 | sessioninfo | 1.2.2 |
shape | 1.4.6 | shiny | 1.8.0 | sourcetools | 0.1.7-1 |
sparklyr | 1.8.4 | spatial | 7.3-15 | splines | 4.3.2 |
sqldf | 0.4-11 | SQUAREM | 2021.1 | stats | 4.3.2 |
stats4 | 4.3.2 | stringi | 1.8.3 | stringr | 1.5.1 |
survival | 3.5-5 | swagger | 3.33.1 | sys | 3.4.2 |
systemfonts | 1.0.5 | tcltk | 4.3.2 | testthat | 3.2.1 |
textshaping | 0.3.7 | tibble | 3.2.1 | tidyr | 1.3.1 |
tidyselect | 1.2.0 | tidyverse | 2.0.0 | timechange | 0.3.0 |
timeDate | 4032.109 | tinytex | 0.49 | tools | 4.3.2 |
tzdb | 0.4.0 | urlchecker | 1.0.1 | usethis | 2.2.2 |
utf8 | 1.2.4 | utils | 4.3.2 | uuid | 1.2-0 |
V8 | 4.4.1 | vctrs | 0.6.5 | viridisLite | 0.4.2 |
vroom | 1.6.5 | waldo | 0.5.2 | whisker | 0.4.1 |
withr | 3.0.0 | xfun | 0.41 | xml2 | 1.3.6 |
xopen | 1.0.0 | xtable | 1.8-4 | yaml | 2.3.8 |
zeallot | 0.1.0 | zip | 2.3.1 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
Group ID | Artifact ID | Version |
---|---|---|
antlr | antlr | 2.7.7 |
com.amazonaws | amazon-kinesis-client | 1.12.0 |
com.amazonaws | aws-java-sdk-autoscaling | 1.12.610 |
com.amazonaws | aws-java-sdk-cloudformation | 1.12.610 |
com.amazonaws | aws-java-sdk-cloudfront | 1.12.610 |
com.amazonaws | aws-java-sdk-cloudhsm | 1.12.610 |
com.amazonaws | aws-java-sdk-cloudsearch | 1.12.610 |
com.amazonaws | aws-java-sdk-cloudtrail | 1.12.610 |
com.amazonaws | aws-java-sdk-cloudwatch | 1.12.610 |
com.amazonaws | aws-java-sdk-cloudwatchmetrics | 1.12.610 |
com.amazonaws | aws-java-sdk-codedeploy | 1.12.610 |
com.amazonaws | aws-java-sdk-cognitoidentity | 1.12.610 |
com.amazonaws | aws-java-sdk-cognitosync | 1.12.610 |
com.amazonaws | aws-java-sdk-config | 1.12.610 |
com.amazonaws | aws-java-sdk-core | 1.12.610 |
com.amazonaws | aws-java-sdk-datapipeline | 1.12.610 |
com.amazonaws | aws-java-sdk-directconnect | 1.12.610 |
com.amazonaws | aws-java-sdk-directory | 1.12.610 |
com.amazonaws | aws-java-sdk-dynamodb | 1.12.610 |
com.amazonaws | aws-java-sdk-ec2 | 1.12.610 |
com.amazonaws | aws-java-sdk-ecs | 1.12.610 |
com.amazonaws | aws-java-sdk-efs | 1.12.610 |
com.amazonaws | aws-java-sdk-elasticache | 1.12.610 |
com.amazonaws | aws-java-sdk-elasticbeanstalk | 1.12.610 |
com.amazonaws | aws-java-sdk-elasticloadbalancing | 1.12.610 |
com.amazonaws | aws-java-sdk-elastictranscoder | 1.12.610 |
com.amazonaws | aws-java-sdk-emr | 1.12.610 |
com.amazonaws | aws-java-sdk-glacier | 1.12.610 |
com.amazonaws | aws-java-sdk-glue | 1.12.610 |
com.amazonaws | aws-java-sdk-iam | 1.12.610 |
com.amazonaws | aws-java-sdk-importexport | 1.12.610 |
com.amazonaws | aws-java-sdk-kinesis | 1.12.610 |
com.amazonaws | aws-java-sdk-kms | 1.12.610 |
com.amazonaws | aws-java-sdk-lambda | 1.12.610 |
com.amazonaws | aws-java-sdk-logs | 1.12.610 |
com.amazonaws | aws-java-sdk-machinelearning | 1.12.610 |
com.amazonaws | aws-java-sdk-opsworks | 1.12.610 |
com.amazonaws | aws-java-sdk-rds | 1.12.610 |
com.amazonaws | aws-java-sdk-redshift | 1.12.610 |
com.amazonaws | aws-java-sdk-route53 | 1.12.610 |
com.amazonaws | aws-java-sdk-s3 | 1.12.610 |
com.amazonaws | aws-java-sdk-ses | 1.12.610 |
com.amazonaws | aws-java-sdk-simpledb | 1.12.610 |
com.amazonaws | aws-java-sdk-simpleworkflow | 1.12.610 |
com.amazonaws | aws-java-sdk-sns | 1.12.610 |
com.amazonaws | aws-java-sdk-sqs | 1.12.610 |
com.amazonaws | aws-java-sdk-ssm | 1.12.610 |
com.amazonaws | aws-java-sdk-storagegateway | 1.12.610 |
com.amazonaws | aws-java-sdk-sts | 1.12.610 |
com.amazonaws | aws-java-sdk-support | 1.12.610 |
com.amazonaws | aws-java-sdk-swf-libraries | 1.11.22 |
com.amazonaws | aws-java-sdk-workspaces | 1.12.610 |
com.amazonaws | jmespath-java | 1.12.610 |
com.clearspring.analytics | stream | 2.9.6 |
com.databricks | Rserve | 1.8-3 |
com.databricks | databricks-sdk-java | 0.27.0 |
com.databricks | jets3t | 0.7.1-0 |
com.databricks.scalapb | compilerplugin_2.12 | 0.4.15-10 |
com.databricks.scalapb | scalapb-runtime_2.12 | 0.4.15-10 |
com.esotericsoftware | kryo-shaded | 4.0.2 |
com.esotericsoftware | minlog | 1.3.0 |
com.fasterxml | classmate | 1.3.4 |
com.fasterxml.jackson.core | jackson-annotations | 2.15.2 |
com.fasterxml.jackson.core | jackson-core | 2.15.2 |
com.fasterxml.jackson.core | jackson-databind | 2.15.2 |
com.fasterxml.jackson.dataformat | jackson-dataformat-cbor | 2.15.2 |
com.fasterxml.jackson.dataformat | jackson-dataformat-yaml | 2.15.2 |
com.fasterxml.jackson.datatype | jackson-datatype-joda | 2.15.2 |
com.fasterxml.jackson.datatype | jackson-datatype-jsr310 | 2.16.0 |
com.fasterxml.jackson.module | jackson-module-paranamer | 2.15.2 |
com.fasterxml.jackson.module | jackson-module-scala_2.12 | 2.15.2 |
com.github.ben-manes.caffeine | caffeine | 2.9.3 |
com.github.fommil | jniloader | 1.1 |
com.github.fommil.netlib | native_ref-java | 1.1 |
com.github.fommil.netlib | native_ref-java | 1.1-natives |
com.github.fommil.netlib | native_system-java | 1.1 |
com.github.fommil.netlib | native_system-java | 1.1-natives |
com.github.fommil.netlib | netlib-native_ref-linux-x86_64 | 1.1-natives |
com.github.fommil.netlib | netlib-native_system-linux-x86_64 | 1.1-natives |
com.github.luben | zstd-jni | 1.5.5-4 |
com.github.wendykierp | JTransforms | 3.1 |
com.google.code.findbugs | jsr305 | 3.0.0 |
com.google.code.gson | gson | 2.10.1 |
com.google.crypto.tink | tink | 1.9.0 |
com.google.errorprone | error_prone_annotations | 2.10.0 |
com.google.flatbuffers | flatbuffers-java | 23.5.26 |
com.google.guava | guava | 15.0 |
com.google.protobuf | protobuf-java | 2.6.1 |
com.helger | profiler | 1.1.1 |
com.ibm.icu | icu4j | 75.1 |
com.jcraft | jsch | 0.1.55 |
com.jolbox | bonecp | 0.8.0.RELEASE |
com.lihaoyi | sourcecode_2.12 | 0.1.9 |
com.microsoft.azure | azure-data-lake-store-sdk | 2.3.9 |
com.microsoft.sqlserver | mssql-jdbc | 11.2.2.jre8 |
com.ning | compress-lzf | 1.1.2 |
com.sun.mail | javax.mail | 1.5.2 |
com.sun.xml.bind | jaxb-core | 2.2.11 |
com.sun.xml.bind | jaxb-impl | 2.2.11 |
com.tdunning | json | 1.8 |
com.thoughtworks.paranamer | paranamer | 2.8 |
com.trueaccord.lenses | lenses_2.12 | 0.4.12 |
com.twitter | chill-java | 0.10.0 |
com.twitter | chill_2.12 | 0.10.0 |
com.twitter | util-app_2.12 | 7.1.0 |
com.twitter | util-core_2.12 | 7.1.0 |
com.twitter | util-function_2.12 | 7.1.0 |
com.twitter | util-jvm_2.12 | 7.1.0 |
com.twitter | util-lint_2.12 | 7.1.0 |
com.twitter | util-registry_2.12 | 7.1.0 |
com.twitter | util-stats_2.12 | 7.1.0 |
com.typesafe | config | 1.4.3 |
com.typesafe.scala-logging | scala-logging_2.12 | 3.7.2 |
com.uber | h3 | 3.7.3 |
com.univocity | univocity-parsers | 2.9.1 |
com.zaxxer | HikariCP | 4.0.3 |
commons-cli | commons-cli | 1.5.0 |
commons-codec | commons-codec | 1.16.0 |
commons-collections | commons-collections | 3.2.2 |
commons-dbcp | commons-dbcp | 1.4 |
commons-fileupload | commons-fileupload | 1.5 |
commons-httpclient | commons-httpclient | 3.1 |
commons-io | commons-io | 2.13.0 |
commons-lang | commons-lang | 2.6 |
commons-logging | commons-logging | 1.1.3 |
commons-pool | commons-pool | 1.5.4 |
dev.ludovic.netlib | arpack | 3.0.3 |
dev.ludovic.netlib | blas | 3.0.3 |
dev.ludovic.netlib | lapack | 3.0.3 |
info.ganglia.gmetric4j | gmetric4j | 1.0.10 |
io.airlift | aircompressor | 0.25 |
io.delta | delta-sharing-client_2.12 | 1.1.1 |
io.dropwizard.metrics | metrics-annotation | 4.2.19 |
io.dropwizard.metrics | metrics-core | 4.2.19 |
io.dropwizard.metrics | metrics-graphite | 4.2.19 |
io.dropwizard.metrics | metrics-healthchecks | 4.2.19 |
io.dropwizard.metrics | metrics-jetty9 | 4.2.19 |
io.dropwizard.metrics | metrics-jmx | 4.2.19 |
io.dropwizard.metrics | metrics-json | 4.2.19 |
io.dropwizard.metrics | metrics-jvm | 4.2.19 |
io.dropwizard.metrics | metrics-servlets | 4.2.19 |
io.netty | netty-all | 4.1.96.Final |
io.netty | netty-buffer | 4.1.96.Final |
io.netty | netty-codec | 4.1.96.Final |
io.netty | netty-codec-http | 4.1.96.Final |
io.netty | netty-codec-http2 | 4.1.96.Final |
io.netty | netty-codec-socks | 4.1.96.Final |
io.netty | netty-common | 4.1.96.Final |
io.netty | netty-handler | 4.1.96.Final |
io.netty | netty-handler-proxy | 4.1.96.Final |
io.netty | netty-resolver | 4.1.96.Final |
io.netty | netty-tcnative-boringssl-static | 2.0.61.Final-linux-aarch_64 |
io.netty | netty-tcnative-boringssl-static | 2.0.61.Final-linux-x86_64 |
io.netty | netty-tcnative-boringssl-static | 2.0.61.Final-osx-aarch_64 |
io.netty | netty-tcnative-boringssl-static | 2.0.61.Final-osx-x86_64 |
io.netty | netty-tcnative-boringssl-static | 2.0.61.Final-windows-x86_64 |
io.netty | netty-tcnative-classes | 2.0.61.Final |
io.netty | netty-transport | 4.1.96.Final |
io.netty | netty-transport-classes-epoll | 4.1.96.Final |
io.netty | netty-transport-classes-kqueue | 4.1.96.Final |
io.netty | netty-transport-native-epoll | 4.1.96.Final |
io.netty | netty-transport-native-epoll | 4.1.96.Final-linux-aarch_64 |
io.netty | netty-transport-native-epoll | 4.1.96.Final-linux-x86_64 |
io.netty | netty-transport-native-kqueue | 4.1.96.Final-osx-aarch_64 |
io.netty | netty-transport-native-kqueue | 4.1.96.Final-osx-x86_64 |
io.netty | netty-transport-native-unix-common | 4.1.96.Final |
io.prometheus | simpleclient | 0.7.0 |
io.prometheus | simpleclient_common | 0.7.0 |
io.prometheus | simpleclient_dropwizard | 0.7.0 |
io.prometheus | simpleclient_pushgateway | 0.7.0 |
io.prometheus | simpleclient_servlet | 0.7.0 |
io.prometheus.jmx | collector | 0.12.0 |
jakarta.annotation | jakarta.annotation-api | 1.3.5 |
jakarta.servlet | jakarta.servlet-api | 4.0.3 |
jakarta.validation | jakarta.validation-api | 2.0.2 |
jakarta.ws.rs | jakarta.ws.rs-api | 2.1.6 |
javax.activation | activation | 1.1.1 |
javax.el | javax.el-api | 2.2.4 |
javax.jdo | jdo-api | 3.0.1 |
javax.transaction | jta | 1.1 |
javax.transaction | transaction-api | 1.1 |
javax.xml.bind | jaxb-api | 2.2.11 |
javolution | javolution | 5.5.1 |
jline | jline | 2.14.6 |
joda-time | joda-time | 2.12.1 |
net.java.dev.jna | jna | 5.8.0 |
net.razorvine | pickle | 1.3 |
net.sf.jpam | jpam | 1.1 |
net.sf.opencsv | opencsv | 2.3 |
net.sf.supercsv | super-csv | 2.2.0 |
net.snowflake | snowflake-ingest-sdk | 0.9.6 |
net.sourceforge.f2j | arpack_combined_all | 0.1 |
org.acplt.remotetea | remotetea-oncrpc | 1.1.2 |
org.antlr | ST4 | 4.0.4 |
org.antlr | antlr-runtime | 3.5.2 |
org.antlr | antlr4-runtime | 4.9.3 |
org.antlr | stringtemplate | 3.2.1 |
org.apache.ant | ant | 1.10.11 |
org.apache.ant | ant-jsch | 1.10.11 |
org.apache.ant | ant-launcher | 1.10.11 |
org.apache.arrow | arrow-format | 15.0.0 |
org.apache.arrow | arrow-memory-core | 15.0.0 |
org.apache.arrow | arrow-memory-netty | 15.0.0 |
org.apache.arrow | arrow-vector | 15.0.0 |
org.apache.avro | avro | 1.11.3 |
org.apache.avro | avro-ipc | 1.11.3 |
org.apache.avro | avro-mapred | 1.11.3 |
org.apache.commons | commons-collections4 | 4.4 |
org.apache.commons | commons-compress | 1.23.0 |
org.apache.commons | commons-crypto | 1.1.0 |
org.apache.commons | commons-lang3 | 3.12.0 |
org.apache.commons | commons-math3 | 3.6.1 |
org.apache.commons | commons-text | 1.10.0 |
org.apache.curator | curator-client | 2.13.0 |
org.apache.curator | curator-framework | 2.13.0 |
org.apache.curator | curator-recipes | 2.13.0 |
org.apache.datasketches | datasketches-java | 3.1.0 |
org.apache.datasketches | datasketches-memory | 2.0.0 |
org.apache.derby | derby | 10.14.2.0 |
org.apache.hadoop | hadoop-client-runtime | 3.3.6 |
org.apache.hive | hive-beeline | 2.3.9 |
org.apache.hive | hive-cli | 2.3.9 |
org.apache.hive | hive-jdbc | 2.3.9 |
org.apache.hive | hive-llap-client | 2.3.9 |
org.apache.hive | hive-llap-common | 2.3.9 |
org.apache.hive | hive-serde | 2.3.9 |
org.apache.hive | hive-shims | 2.3.9 |
org.apache.hive | hive-storage-api | 2.8.1 |
org.apache.hive.shims | hive-shims-0.23 | 2.3.9 |
org.apache.hive.shims | hive-shims-common | 2.3.9 |
org.apache.hive.shims | hive-shims-scheduler | 2.3.9 |
org.apache.httpcomponents | httpclient | 4.5.14 |
org.apache.httpcomponents | httpcore | 4.4.16 |
org.apache.ivy | ivy | 2.5.1 |
org.apache.logging.log4j | log4j-1.2-api | 2.22.1 |
org.apache.logging.log4j | log4j-api | 2.22.1 |
org.apache.logging.log4j | log4j-core | 2.22.1 |
org.apache.logging.log4j | log4j-layout-template-json | 2.22.1 |
org.apache.logging.log4j | log4j-slf4j2-impl | 2.22.1 |
org.apache.orc | orc-core | 1.9.2-shaded-protobuf |
org.apache.orc | orc-mapreduce | 1.9.2-shaded-protobuf |
org.apache.orc | orc-shims | 1.9.2 |
org.apache.thrift | libfb303 | 0.9.3 |
org.apache.thrift | libthrift | 0.12.0 |
org.apache.ws.xmlschema | xmlschema-core | 2.3.0 |
org.apache.xbean | xbean-asm9-shaded | 4.23 |
org.apache.yetus | audience-annotations | 0.13.0 |
org.apache.zookeeper | zookeeper | 3.6.3 |
org.apache.zookeeper | zookeeper-jute | 3.6.3 |
org.checkerframework | checker-qual | 3.31.0 |
org.codehaus.jackson | jackson-core-asl | 1.9.13 |
org.codehaus.jackson | jackson-mapper-asl | 1.9.13 |
org.codehaus.janino | commons-compiler | 3.0.16 |
org.codehaus.janino | janino | 3.0.16 |
org.datanucleus | datanucleus-api-jdo | 4.2.4 |
org.datanucleus | datanucleus-core | 4.1.17 |
org.datanucleus | datanucleus-rdbms | 4.1.19 |
org.datanucleus | javax.jdo | 3.2.0-m3 |
org.eclipse.collections | eclipse-collections | 11.1.0 |
org.eclipse.collections | eclipse-collections-api | 11.1.0 |
org.eclipse.jetty | jetty-client | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-continuation | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-http | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-io | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-jndi | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-plus | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-proxy | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-security | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-server | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-servlet | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-servlets | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-util | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-util-ajax | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-webapp | 9.4.52.v20230823 |
org.eclipse.jetty | jetty-xml | 9.4.52.v20230823 |
org.eclipse.jetty.websocket | websocket-api | 9.4.52.v20230823 |
org.eclipse.jetty.websocket | websocket-client | 9.4.52.v20230823 |
org.eclipse.jetty.websocket | websocket-common | 9.4.52.v20230823 |
org.eclipse.jetty.websocket | websocket-server | 9.4.52.v20230823 |
org.eclipse.jetty.websocket | websocket-servlet | 9.4.52.v20230823 |
org.fusesource.leveldbjni | leveldbjni-all | 1.8 |
org.glassfish.hk2 | hk2-api | 2.6.1 |
org.glassfish.hk2 | hk2-locator | 2.6.1 |
org.glassfish.hk2 | hk2-utils | 2.6.1 |
org.glassfish.hk2 | osgi-resource-locator | 1.0.3 |
org.glassfish.hk2.external | aopalliance-repackaged | 2.6.1 |
org.glassfish.hk2.external | jakarta.inject | 2.6.1 |
org.glassfish.jersey.containers | jersey-container-servlet | 2.40 |
org.glassfish.jersey.containers | jersey-container-servlet-core | 2.40 |
org.glassfish.jersey.core | jersey-client | 2.40 |
org.glassfish.jersey.core | jersey-common | 2.40 |
org.glassfish.jersey.core | jersey-server | 2.40 |
org.glassfish.jersey.inject | jersey-hk2 | 2.40 |
org.hibernate.validator | hibernate-validator | 6.1.7.Final |
org.ini4j | ini4j | 0.5.4 |
org.javassist | javassist | 3.29.2-GA |
org.jboss.logging | jboss-logging | 3.3.2.Final |
org.jdbi | jdbi | 2.63.1 |
org.jetbrains | annotations | 17.0.0 |
org.joda | joda-convert | 1.7 |
org.jodd | jodd-core | 3.5.2 |
org.json4s | json4s-ast_2.12 | 3.7.0-M11 |
org.json4s | json4s-core_2.12 | 3.7.0-M11 |
org.json4s | json4s-jackson_2.12 | 3.7.0-M11 |
org.json4s | json4s-scalap_2.12 | 3.7.0-M11 |
org.lz4 | lz4-java | 1.8.0 |
org.mlflow | mlflow-spark_2.12 | 2.9.1 |
org.objenesis | objenesis | 2.5.1 |
org.postgresql | postgresql | 42.6.1 |
org.roaringbitmap | RoaringBitmap | 0.9.45-databricks |
org.roaringbitmap | shims | 0.9.45-databricks |
org.rocksdb | rocksdbjni | 8.11.4 |
org.rosuda.REngine | REngine | 2.1.0 |
org.scala-lang | scala-compiler_2.12 | 2.12.15 |
org.scala-lang | scala-library_2.12 | 2.12.15 |
org.scala-lang | scala-reflect_2.12 | 2.12.15 |
org.scala-lang.modules | scala-collection-compat_2.12 | 2.11.0 |
org.scala-lang.modules | scala-parser-combinators_2.12 | 1.1.2 |
org.scala-lang.modules | scala-xml_2.12 | 1.2.0 |
org.scala-sbt | test-interface | 1.0 |
org.scalacheck | scalacheck_2.12 | 1.14.2 |
org.scalactic | scalactic_2.12 | 3.2.15 |
org.scalanlp | breeze-macros_2.12 | 2.1.0 |
org.scalanlp | breeze_2.12 | 2.1.0 |
org.scalatest | scalatest-compatible | 3.2.15 |
org.scalatest | scalatest-core_2.12 | 3.2.15 |
org.scalatest | scalatest-diagrams_2.12 | 3.2.15 |
org.scalatest | scalatest-featurespec_2.12 | 3.2.15 |
org.scalatest | scalatest-flatspec_2.12 | 3.2.15 |
org.scalatest | scalatest-freespec_2.12 | 3.2.15 |
org.scalatest | scalatest-funspec_2.12 | 3.2.15 |
org.scalatest | scalatest-funsuite_2.12 | 3.2.15 |
org.scalatest | scalatest-matchers-core_2.12 | 3.2.15 |
org.scalatest | scalatest-mustmatchers_2.12 | 3.2.15 |
org.scalatest | scalatest-propspec_2.12 | 3.2.15 |
org.scalatest | scalatest-refspec_2.12 | 3.2.15 |
org.scalatest | scalatest-shouldmatchers_2.12 | 3.2.15 |
org.scalatest | scalatest-wordspec_2.12 | 3.2.15 |
org.scalatest | scalatest_2.12 | 3.2.15 |
org.slf4j | jcl-over-slf4j | 2.0.7 |
org.slf4j | jul-to-slf4j | 2.0.7 |
org.slf4j | slf4j-api | 2.0.7 |
org.slf4j | slf4j-simple | 1.7.25 |
org.threeten | threeten-extra | 1.7.1 |
org.tukaani | xz | 1.9 |
org.typelevel | algebra_2.12 | 2.0.1 |
org.typelevel | cats-kernel_2.12 | 2.1.1 |
org.typelevel | spire-macros_2.12 | 0.17.0 |
org.typelevel | spire-platform_2.12 | 0.17.0 |
org.typelevel | spire-util_2.12 | 0.17.0 |
org.typelevel | spire_2.12 | 0.17.0 |
org.wildfly.openssl | wildfly-openssl | 1.1.3.Final |
org.xerial | sqlite-jdbc | 3.42.0.0 |
org.xerial.snappy | snappy-java | 1.1.10.3 |
org.yaml | snakeyaml | 2.0 |
oro | oro | 2.0.8 |
pl.edu.icm | JLargeArrays | 1.5 |
software.amazon.cryptools | AmazonCorrettoCryptoProvider | 1.6.2-linux-x86_64 |
software.amazon.ion | ion-java | 1.0.2 |
stax | stax-api | 1.0.1 |