Κοινή χρήση μέσω

Databricks Runtime 16.2 (Beta)

  • Άρθρο

Important

Databricks Runtime 16.2 is in Beta. The contents of the supported environments may change during the Beta. Changes can include the list of packages or versions of installed packages.

The following release notes provide information about Databricks Runtime 16.2, powered by Apache Spark 3.5.0.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

New features and improvements

Use the timestampdiff & timestampadd in generated column expressions

In Databricks Runtime 16.2 and above, you can use the timestampdiff and timestampadd functions in Delta Lake generated column expressions. See Delta Lake generated columns.

Library upgrades

  • Upgraded Python libraries:
  • Upgraded R libraries:
  • Upgraded Java libraries:
  • Upgraded Java libraries:
    • org.json4s.json4s-ast_2.12 from 3.7.0-M11 to 4.0.7
    • org.json4s.json4s-core_2.12 from 3.7.0-M11 to 4.0.7
    • org.json4s.json4s-jackson_2.12 from 3.7.0-M11 to 4.0.7
    • org.json4s.json4s-scalap_2.12 from 3.7.0-M11 to 4.0.7

Apache Spark

Databricks Runtime 16.2 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 16.1, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-50904] [SC-186976][SQL] Fix collation expression walker query execution
  • [SPARK-49666] [SQL] Enable trimming tests for InSet expression
  • [SPARK-50669] [SC-184566][ES-1327450][SQL] Change the signature of TimestampAdd expression
  • [SPARK-50795] [16.x][SC-186719][SQL] Store timestamp as long type in describe LinkedHashMap
  • [SPARK-50596] [SC-184060][PYTHON] Upgrade Py4J from 0.10.9.7 to 0.10.9.8
  • [SPARK-50818] Revert “Revert “Revert “[SC-186458][PYTHON] Replace has_numpy with have_numpy”””
  • [SPARK-50870] [SC-186950][SQL] Add the timezone when casting to timestamp in V2ScanRelationPushDown
  • [SPARK-50735] [SC-186944][CONNECT] Failure in ExecuteResponseObserver results in infinite reattaching requests
  • [SPARK-50522] [SC-186500][SQL] Support for indeterminate collation
  • [SPARK-50525] [SC-186058][SQL] Define InsertMapSortInRepartitionExpressions Optimizer Rule
  • [SPARK-50679] [SC-184572][SQL] Duplicated common expressions in different With should be projected only once
  • [SPARK-50847] [SC-186916] [SQL] Deny ApplyCharTypePadding from applying on specific In expressions
  • [SPARK-50714] [SC-186786][SQL][SS] Enable schema evolution for TransformWithState when Avro encoding is used
  • [SPARK-50818] Revert “Revert “[SC-186458][PYTHON] Replace has_numpy with have_numpy””
  • [SPARK-50795] [SC-186390][SQL] Display all DESCRIBE AS JSON dates in ISO-8601 format and types as dataType.simpleString
  • [SPARK-50561] [SC-185924][SQL] DBR 16.x cherrypick: Improve type coercion and boundary checking for UNIFORM SQL function
  • [SPARK-50700] [SC-184845][SQL] spark.sql.catalog.spark_catalog supports builtin magic value
  • [SPARK-50831] [SC-186736][BEHAVE-222][SQL] Enable trimming collation by default
  • [SPARK-50263] [SC-186793][CONNECT] Replace System.currentTimeMillis with System.nanoTime
  • [SPARK-50818] Revert “[SC-186458][PYTHON] Replace has_numpy with have_numpy”
  • [SPARK-48730] [SC-184926][SQL] Implement CreateSQLFunctionCommand for SQL Scalar and Table Functions
  • [SPARK-50830] [SC-186718] [SQL] Return single-pass result as the dual run analysis result
  • [SPARK-50707] [SC-186098][SQL] Enable casting to/from char/varchar
  • [SPARK-49490] [SC-182137][SQL] Add benchmarks for initCap
  • [SPARK-50529] [SC-184535][SQL] Change char/varchar behavior under the spark.sql.preserveCharVarcharTypeInfo config
  • [SPARK-49632] [SC-184179][SQL] Remove the ANSI config suggestion in CANNOT_PARSE_TIMESTAMP
  • [SPARK-50815] [SC-186487][PYTHON][SQL] Fix bug where passing null Variants in createDataFrame causes it to fail and add Variant support in createDataFrame in Spark Connect
  • [SPARK-50828] [SC-186493][PYTHON][ML][CONNECT] Deprecate pyspark.ml.connect
  • [SPARK-50600] [SC-186057][CONNECT][SQL] Set analyzed on analysis failure
  • [SPARK-50824] [SC-186472][PYTHON] Avoid importing optional Python packages for checking
  • [SPARK-50818] [SC-186458][PYTHON] Replace has_numpy with have_numpy
  • [SPARK-50722] [SC-185523][SQL] Detect self-contained WITH nodes
  • [SPARK-50755] [SC-185521][SQL] Pretty plan display for InsertIntoHiveTable
  • [SPARK-50756] [SC-185552]Revert “[SQL] Use error class for exceptions in SparkConf.validateSettings”
  • [SPARK-50789] [SC-186312][CONNECT] The inputs for typed aggregations should be analyzed
  • [SPARK-50791] [SC-185867][SC-186338][SQL] Fix NPE in State Store error handling
  • [SPARK-50801] [SC-186365] [SQL] Improve PlanLogger.logPlanResolution so it shows just unresolved and resolved plans
  • [SPARK-50749] [SC-185925][SQL] Fix ordering bug in CommutativeExpression.gatherCommutative method
  • [SPARK-50783] [SC-186347] Canonicalize JVM profiler results file name and layout on DFS
  • [SPARK-50790] [SC-186354][PYTHON] Implement parse json in pyspark
  • [SPARK-50738] [SC-184856][PYTHON] Upgrade black to 23.12.1
  • [SPARK-50764] [SC-185930][PYTHON] Refine the docstring of xpath related methods
  • [SPARK-50798] [SC-186277][SQL] Improve NormalizePlan
  • [SPARK-49883] [SC-183787][SS] State Store Checkpoint Structure V2 Integration with RocksDB and RocksDBFileManager
  • [SPARK-50779] [SC-186062][SQL] Adding feature flag for object level collations
  • [SPARK-50778] [SC-186183][PYTHON] Add metadataColumn to PySpark DataFrame
  • [SPARK-49565] [SC-186056][SQL] DBR 16.x cherrypick: Improve auto-generated expression aliases with pipe SQL operators
  • [SPARK-50541] [16.x][SC-184937] Describe Table As JSON
  • [SPARK-50772] [SC-185923][SQL] DBR 16.x cherrypick: Retain table aliases after SET, EXTEND, DROP operators
  • [SPARK-50769] [SC-185919][SQL] Fix ClassCastException in HistogramNumeric
  • [SPARK-49025] [SC-174667] Sync Delta code diff to DBR
  • [SPARK-50752] [SC-185547][PYTHON][SQL] Introduce configs for tuning Python UDF without Arrow
  • [SPARK-50705] [SC-185944][SQL] Make QueryPlan lock-free
  • [SPARK-50690] [16.x][SC-184640][BEHAVE-211][SQL] Fix discrepancy in DESCRIBE TABLE view query output columns quoting
  • [SPARK-50746] [SC-184932][SQL] Replace Either with VariantPathSegment.
  • [SPARK-50715] [SC-185546][PYTHON][CONNECT] SparkSession.Builder sets the configs in batch
  • [SPARK-50480] [SC-183359][SQL] Extend CharType and VarcharType from StringType
  • [SPARK-50675] [SC-184539][SQL] Table and view level collations support
  • [SPARK-50409] [BEHAVE-194][SC-184516][SQL] Fix set statement to ignore ; at the end of SET;, SET -v; and SET key;
  • [SPARK-50743] [SC-185528][SQL] Normalize CTERelationDef and CTERelationRef IDs
  • [SPARK-50756] [SC-185520][SQL] Use error class for exceptions in SparkConf.validateSettings
  • [SPARK-50693] [SC-184684][CONNECT] The inputs for TypedScalaUdf should be analyzed
  • [SPARK-50744] [SC-184929][SQL] Add a test case for view/CTE name resolution precedence
  • [SPARK-50710] [SC-184767][CONNECT] Add support for optional client reconnection to sessions after release
  • [SPARK-50703] [SC-184843][PYTHON] Refine the docstring of regexp_replace, regexp_substr, and regexp_instr
  • [SPARK-50716] [SC-184823][CORE] Fix the cleanup logic for symbolic links in JavaUtils.deleteRecursivelyUsingJavaIO method
  • [SPARK-50630] [SC-184443][SQL] Fix GROUP BY ordinal support for pipe SQL AGGREGATE operators
  • [SPARK-50614] [SC-184729][SQL] Add Variant shredding support for Parquet
  • [SPARK-50661] [SASP-4936] Add backward-compatibility for old client FEB.
  • [SPARK-50676] [SC-184641][SQL] Remove unused private lazy val mapValueContainsNull from ElementAt
  • [SPARK-50515] [SC-183813][CORE] Add read-only interface to SparkConf
  • [SPARK-50697] [SC-184702][SQL] Enable tail-recursion wherever possible
  • [SPARK-50642] [SC-184726][SC-183517][CONNECT][SS][2/N][16.x]Fix the state schema for FlatMapGroupsWithState in spark connect when there is no initial state
  • [SPARK-50701] [SC-184704][PYTHON] Make plotting require the minimum plotly version
  • [SPARK-50702] [SC-184727][PYTHON] Refine the docstring of regexp_count, regexp_extract and regexp_extract_all
  • [SPARK-50499] [SC-184177][PYTHON] Expose metrics from BasePythonRunner
  • [SPARK-50692] [SC-184705][16.x] Add RPAD pushdown support
  • [SPARK-50682] [SC-184579][SQL] Inner Alias should be canonicalized
  • [SPARK-50699] [SC-184695][PYTHON] Parse and generate DDL string with a specified session
  • [SPARK-50573] [SC-184568][SS] Adding State Schema ID to State Rows to schema evolution
  • [SPARK-50661] [SC-184639][CONNECT][SS][SASP-4936] Fix Spark Connect Scala foreachBatch impl. to support Dataset[T].
  • [SPARK-50689] [SC-184591][SQL] Enforce deterministic ordering in LCA project lists
  • [SPARK-50696] [SC-184667][PYTHON] Optimize Py4J call for DDL parse method
  • [SPARK-49670] [SC-182902][SQL] Enable trim collation for all passthrough expressions
  • [SPARK-50673] [SC-184565][ML] Avoid traversing model coefficients twice in Word2VecModel constructor
  • [SPARK-50310] [SC-184663]Revert “[CONNECT][PYTHON] Call with_origin_to_class when the Column initializing”
  • [SPARK-50687] [SC-184588][PYTHON] Optimize the logic to get stack traces for DataFrameQueryContext
  • [SPARK-50681] [SC-184662][PYTHON][CONNECT] Cache the parsed schema for MapInXXX and ApplyInXXX
  • [SPARK-50674] [SC-184589][PYTHON] Fix check for ‘terminate’ method existence in UDTF evaluation
  • [SPARK-50684] [SC-184582][PYTHON] Improve Py4J performance in DataFrameQueryContext
  • [SPARK-50578] [DBR16.x][SC-184559][PYTHON][SS] Add support for new version of state metadata for TransformWithStateInPandas
  • [SPARK-50602] [SC-184439][SQL] Fix transpose to show a proper error message when invalid index columns are specified
  • [SPARK-50650] [SC-184532][SQL] Improve logging in single-pass Analyzer
  • [SPARK-50665] [SC-184533][SQL] Substitute LocalRelation with ComparableLocalRelation in NormalizePlan
  • [SPARK-50644] [SC-184486][SQL] Read variant struct in Parquet reader.
  • [SPARK-49636] [SC-184089][SQL] Remove the ANSI config suggestion in INVALID_ARRAY_INDEX and INVALID_ARRAY_INDEX_IN_ELEMENT_AT
  • [SPARK-50659] [SC-184514][SQL] Refactor Union output computation out to reuse it in the single-pass Analyzer
  • [SPARK-50659] [SC-184512][SQL] Move Union-related errors to QueryCompilationErrors
  • [SPARK-50530] [SC-183419][SQL] Fix bad implicit string type context calculation
  • [SPARK-50546] [SC-183807][SQL] Add subquery cast support to collation type coercion
  • [SPARK-50405] [SC-182889][SQL] Handle collation type coercion of complex data types properly
  • [SPARK-50637] [SC-184434][SQL] Fix code style for the single-pass Analyzer
  • [SPARK-50638] [SC-184435][SQL] Refactor the view resolution into the separate file to reuse it in the single-pass Analyzer
  • [SPARK-50615] [SC-184298][SQL] Push variant into scan.
  • [SPARK-50619] [SC-184210][SQL] Refactor VariantGet.cast to pack the cast arguments
  • [SPARK-50599] [SC-184058][SQL] Create the DataEncoder trait that allows for Avro and UnsafeRow encoding
  • [SPARK-50076] [SC-183809] Fix logkeys
  • [SPARK-50597] [SC-183972][SQL] Refactor batch construction in Optimizer.scala and SparkOptimizer.scala
  • [SPARK-50339] [SC-183063][SPARK-50360][SS] Enable changelog to store lineage information
  • [SPARK-50526] [SC-183811][SS] Add store encoding format conf into offset log and block non supported stateful operators from using avro
  • [SPARK-50540] [SC-183810][PYTHON][SS] Fix string schema for StatefulProcessorHandle
  • [SPARK-50157] [SC-183789][SQL] Using SQLConf provided by SparkSession first.
  • [SPARK-48898] [SC-183793][SQL] Set nullability correctly in the Variant schema
  • [SPARK-50559] [SC-183814][SQL] Store Except, Intersect and Union’s outputs as lazy vals
  • [SPARK-48416] [SC-183643][SQL] Support nested correlated With expression
  • [SPARK-50428] [SC-183566][SS][PYTHON] Support TransformWithStateInPandas in batch queries
  • [SPARK-50063] [SC-183350][SQL][CONNECT] Add support for Variant in the Spark Connect Scala client
  • [SPARK-50544] [SC-183569][PYTHON][CONNECT] Implement StructType.toDDL
  • [SPARK-50443] [SC-182590][SS] Fixing Maven build errors introduced by Guava cache in RocksDBStateStoreProvider
  • [SPARK-50491] [SC-183434][SQL] Fix bug where empty BEGIN END blocks throw an error
  • [SPARK-50536] [SC-183443][CORE] Log downloaded archive file sizes in SparkContext and Executor
  • [SPARK-45891] [SC-183439][SQL] Rebuild variant binary from shredded data.
  • [SPARK-49565] [SC-183465][SQL] Add SQL pipe syntax for the FROM operator
  • [SPARK-50497] [SC-183338][SQL] Fail queries with proper message if MultiAlias contains non-generator function
  • [SPARK-50460] [SC-183375][PYTHON][CONNECT] Generalize and simplify Connect exception handling
  • [SPARK-50537] [SC-183452][CONNECT][PYTHON] Fix compression option being overwritten in df.write.parquet
  • [SPARK-50329] [SC-183358][SQL] fix InSet$toString
  • [SPARK-50524] [SC-183364][SQL] Lower RowBasedKeyValueBatch.spill warning message to debug level
  • [SPARK-50528] [SC-183385][CONNECT] Move InvalidCommandInput to common module
  • [SPARK-50017] [SC-182438][SS] Support Avro encoding for TransformWithState operator
  • [SPARK-50310] [SC-182660][PYTHON] Add spark.python.sql.dataFrameDebugging.enabled to allow list
  • [SPARK-50463] [SC-182833][SQL] Fix ConstantColumnVector with Columnar to Row conversion
  • [SPARK-50235] [SC-180786][SQL] Clean up ColumnVector resource after processing all rows in ColumnarToRowExec
  • [SPARK-50310] [SC-182660][PYTHON] Add a flag to disable DataFrameQueryContext for PySpark
  • [SPARK-50516] [SC-183279][SS][MINOR] Fix the init state related test to use StreamManualClock
  • [SPARK-50478] [SC-183188][SQL] Fix StringType matching
  • [SPARK-50492] [SC-183177][SS] Fix java.util.NoSuchElementException when event time column is dropped after dropDuplicatesWithinWatermark
  • [SPARK-49566] [SC-182589][SQL] Add SQL pipe syntax for the SET operator
  • [SPARK-50449] [SC-183178][SQL] Fix SQL Scripting grammar allowing empty bodies for loops, IF and CASE
  • [SPARK-50251] [SC-180970][PYTHON] Add getSystemProperty to PySpark SparkContext
  • [SPARK-50421] [SC-183091][CORE] Fix executor related memory config incorrect when multiple resource profiles worked
  • [SPARK-49461] [SC-179572][SS] Persistent Checkpoint ID to commit logs and read it back
  • [SPARK-50343] [SC-183119][SPARK-50344][SQL] Add SQL pipe syntax for the DROP and AS operators
  • [SPARK-50481] [SC-182880][CORE] Improve SortShuffleManager.unregisterShuffle to skip checksum file logic if checksum is disabled
  • [SPARK-50498] [SC-183090][PYTHON] Avoid unnecessary py4j call in listFunctions
  • [SPARK-50489] [SC-183053][SQL][PYTHON] Fix self-join after applyInArrow
  • [SPARK-49695] [SC-182967][SC-176968][BEHAVE-198][SQL] Postgres fix xor push-down

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

  • Operating System: Ubuntu 24.04.1 LTS
  • Java: Zulu17.54+21-CA
  • Scala: 2.12.18
  • Python: 3.12.3
  • R: 4.4.0
  • Delta Lake: 3.3.0

Installed Python libraries

Library Version Library Version Library Version
annotated-types 0.7.0 asttokens 2.0.5 astunparse 1.6.3
autocommand 2.2.2 azure-core 1.31.0 azure-storage-blob 12.23.0
azure-storage-file-datalake 12.17.0 backports.tarfile 1.2.0 black 24.4.2
blinker 1.7.0 boto3 1.34.69 botocore 1.34.69
cachetools 5.3.3 certifi 2024.6.2 cffi 1.16.0
chardet 4.0.0 charset-normalizer 2.0.4 click 8.1.7
cloudpickle 2.2.1 comm 0.2.1 contourpy 1.2.0
cryptography 42.0.5 cycler 0.11.0 Cython 3.0.11
databricks-sdk 0.30.0 dbus-python 1.3.2 debugpy 1.6.7
decorator 5.1.1 Deprecated 1.2.14 distlib 0.3.8
docstring-to-markdown 0.11 entrypoints 0.4 executing 0.8.3
facets-overview 1.1.1 filelock 3.15.4 fonttools 4.51.0
gitdb 4.0.11 GitPython 3.1.37 google-api-core 2.20.0
google-auth 2.35.0 google-cloud-core 2.4.1 google-cloud-storage 2.18.2
google-crc32c 1.6.0 google-resumable-media 2.7.2 googleapis-common-protos 1.65.0
grpcio 1.60.0 grpcio-status 1.60.0 httplib2 0.20.4
idna 3.7 importlib-metadata 6.0.0 importlib_resources 6.4.0
inflect 7.3.1 ipyflow-core 0.0.201 ipykernel 6.28.0
ipython 8.25.0 ipython-genutils 0.2.0 ipywidgets 7.7.2
isodate 0.6.1 jaraco.context 5.3.0 jaraco.functools 4.0.1
jaraco.text 3.12.1 jedi 0.19.1 jmespath 1.0.1
joblib 1.4.2 jupyter_client 8.6.0 jupyter_core 5.7.2
kiwisolver 1.4.4 launchpadlib 1.11.0 lazr.restfulclient 0.14.6
lazr.uri 1.0.6 matplotlib 3.8.4 matplotlib-inline 0.1.6
mccabe 0.7.0 mlflow-skinny 2.15.1 more-itertools 10.3.0
mypy 1.10.0 mypy-extensions 1.0.0 nest-asyncio 1.6.0
nodeenv 1.9.1 numpy 1.26.4 oauthlib 3.2.2
opentelemetry-api 1.27.0 opentelemetry-sdk 1.27.0 opentelemetry-semantic-conventions 0.48b0
packaging 24.1 pandas 1.5.3 parso 0.8.3
pathspec 0.10.3 patsy 0.5.6 pexpect 4.8.0
pillow 10.3.0 pip 24.2 platformdirs 3.10.0
plotly 5.22.0 pluggy 1.0.0 prompt-toolkit 3.0.43
proto-plus 1.24.0 protobuf 4.24.1 psutil 5.9.0
psycopg2 2.9.3 ptyprocess 0.7.0 pure-eval 0.2.2
pyarrow 15.0.2 pyasn1 0.4.8 pyasn1-modules 0.2.8
pyccolo 0.0.65 pycparser 2.21 pydantic 2.8.2
pydantic_core 2.20.1 pyflakes 3.2.0 Pygments 2.15.1
PyGObject 3.48.2 PyJWT 2.7.0 pyodbc 5.0.1
pyparsing 3.0.9 pyright 1.1.294 python-dateutil 2.9.0.post0
python-lsp-jsonrpc 1.1.2 python-lsp-server 1.10.0 pytoolconfig 1.2.6
pytz 2024.1 PyYAML 6.0.1 pyzmq 25.1.2
requests 2.32.2 rope 1.12.0 rsa 4.9
s3transfer 0.10.2 scikit-learn 1.4.2 scipy 1.13.1
seaborn 0.13.2 setuptools 74.0.0 six 1.16.0
smmap 5.0.0 sqlparse 0.5.1 ssh-import-id 5.11
stack-data 0.2.0 statsmodels 0.14.2 tenacity 8.2.2
threadpoolctl 2.2.0 tokenize-rt 4.2.1 tomli 2.0.1
tornado 6.4.1 traitlets 5.14.3 typeguard 4.3.0
types-protobuf 3.20.3 types-psutil 5.9.0 types-pytz 2023.3.1.1
types-PyYAML 6.0.0 types-requests 2.31.0.0 types-setuptools 68.0.0.0
types-six 1.16.0 types-urllib3 1.26.25.14 typing_extensions 4.11.0
ujson 5.10.0 unattended-upgrades 0.1 urllib3 1.26.16
virtualenv 20.26.2 wadllib 1.3.6 wcwidth 0.2.5
whatthepatch 1.0.2 wheel 0.43.0 wrapt 1.14.1
yapf 0.33.0 zipp 3.17.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2024-08-04.

Library Version Library Version Library Version
arrow 16.1.0 askpass 1.2.0 assertthat 0.2.1
backports 1.5.0 base 4.4.0 base64enc 0.1-3
bigD 0.2.0 bit 4.0.5 bit64 4.0.5
bitops 1.0-8 blob 1.2.4 boot 1.3-30
brew 1.0-10 brio 1.1.5 broom 1.0.6
bslib 0.8.0 cachem 1.1.0 callr 3.7.6
caret 6.0-94 cellranger 1.1.0 chron 2.3-61
class 7.3-22 cli 3.6.3 clipr 0.8.0
clock 0.7.1 cluster 2.1.6 codetools 0.2-20
colorspace 2.1-1 commonmark 1.9.1 compiler 4.4.0
config 0.3.2 conflicted 1.2.0 cpp11 0.4.7
crayon 1.5.3 credentials 2.0.1 curl 5.2.1
data.table 1.15.4 datasets 4.4.0 DBI 1.2.3
dbplyr 2.5.0 desc 1.4.3 devtools 2.4.5
diagram 1.6.5 diffobj 0.3.5 digest 0.6.36
downlit 0.4.4 dplyr 1.1.4 dtplyr 1.3.1
e1071 1.7-14 ellipsis 0.3.2 evaluate 0.24.0
fansi 1.0.6 farver 2.1.2 fastmap 1.2.0
fontawesome 0.5.2 forcats 1.0.0 foreach 1.5.2
foreign 0.8-86 forge 0.2.0 fs 1.6.4
future 1.34.0 future.apply 1.11.2 gargle 1.5.2
generics 0.1.3 gert 2.1.0 ggplot2 3.5.1
gh 1.4.1 git2r 0.33.0 gitcreds 0.1.2
glmnet 4.1-8 globals 0.16.3 glue 1.7.0
googledrive 2.1.1 googlesheets4 1.1.1 gower 1.0.1
graphics 4.4.0 grDevices 4.4.0 grid 4.4.0
gridExtra 2.3 gsubfn 0.7 gt 0.11.0
gtable 0.3.5 hardhat 1.4.0 haven 2.5.4
highr 0.11 hms 1.1.3 htmltools 0.5.8.1
htmlwidgets 1.6.4 httpuv 1.6.15 httr 1.4.7
httr2 1.0.2 ids 1.0.1 ini 0.3.1
ipred 0.9-15 isoband 0.2.7 iterators 1.0.14
jquerylib 0.1.4 jsonlite 1.8.8 juicyjuice 0.1.0
KernSmooth 2.23-22 knitr 1.48 labeling 0.4.3
later 1.3.2 lattice 0.22-5 lava 1.8.0
lifecycle 1.0.4 listenv 0.9.1 lubridate 1.9.3
magrittr 2.0.3 markdown 1.13 MASS 7.3-60.0.1
Matrix 1.6-5 memoise 2.0.1 methods 4.4.0
mgcv 1.9-1 mime 0.12 miniUI 0.1.1.1
mlflow 2.14.1 ModelMetrics 1.2.2.2 modelr 0.1.11
munsell 0.5.1 nlme 3.1-165 nnet 7.3-19
numDeriv 2016.8-1.1 openssl 2.2.0 parallel 4.4.0
parallelly 1.38.0 pillar 1.9.0 pkgbuild 1.4.4
pkgconfig 2.0.3 pkgdown 2.1.0 pkgload 1.4.0
plogr 0.2.0 plyr 1.8.9 praise 1.0.0
prettyunits 1.2.0 pROC 1.18.5 processx 3.8.4
prodlim 2024.06.25 profvis 0.3.8 progress 1.2.3
progressr 0.14.0 promises 1.3.0 proto 1.0.0
proxy 0.4-27 ps 1.7.7 purrr 1.0.2
R6 2.5.1 ragg 1.3.2 randomForest 4.7-1.1
rappdirs 0.3.3 rcmdcheck 1.4.0 RColorBrewer 1.1-3
Rcpp 1.0.13 RcppEigen 0.3.4.0.0 reactable 0.4.4
reactR 0.6.0 readr 2.1.5 readxl 1.4.3
recipes 1.1.0 rematch 2.0.0 rematch2 2.1.2
remotes 2.5.0 reprex 2.1.1 reshape2 1.4.4
rlang 1.1.4 rmarkdown 2.27 RODBC 1.3-23
roxygen2 7.3.2 rpart 4.1.23 rprojroot 2.0.4
Rserve 1.8-13 RSQLite 2.3.7 rstudioapi 0.16.0
rversions 2.1.2 rvest 1.0.4 sass 0.4.9
scales 1.3.0 selectr 0.4-2 sessioninfo 1.2.2
shape 1.4.6.1 shiny 1.9.1 sourcetools 0.1.7-1
sparklyr 1.8.6 spatial 7.3-17 splines 4.4.0
sqldf 0.4-11 SQUAREM 2021.1 stats 4.4.0
stats4 4.4.0 stringi 1.8.4 stringr 1.5.1
survival 3.6-4 swagger 5.17.14.1 sys 3.4.2
systemfonts 1.1.0 tcltk 4.4.0 testthat 3.2.1.1
textshaping 0.4.0 tibble 3.2.1 tidyr 1.3.1
tidyselect 1.2.1 tidyverse 2.0.0 timechange 0.3.0
timeDate 4032.109 tinytex 0.52 tools 4.4.0
tzdb 0.4.0 urlchecker 1.0.1 usethis 3.0.0
utf8 1.2.4 utils 4.4.0 uuid 1.2-1
V8 4.4.2 vctrs 0.6.5 viridisLite 0.4.2
vroom 1.6.5 waldo 0.5.2 whisker 0.4.1
withr 3.0.1 xfun 0.46 xml2 1.3.6
xopen 1.0.1 xtable 1.8-4 yaml 2.3.10
zeallot 0.1.0 zip 2.3.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.12.638
com.amazonaws aws-java-sdk-cloudformation 1.12.638
com.amazonaws aws-java-sdk-cloudfront 1.12.638
com.amazonaws aws-java-sdk-cloudhsm 1.12.638
com.amazonaws aws-java-sdk-cloudsearch 1.12.638
com.amazonaws aws-java-sdk-cloudtrail 1.12.638
com.amazonaws aws-java-sdk-cloudwatch 1.12.638
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.12.638
com.amazonaws aws-java-sdk-codedeploy 1.12.638
com.amazonaws aws-java-sdk-cognitoidentity 1.12.638
com.amazonaws aws-java-sdk-cognitosync 1.12.638
com.amazonaws aws-java-sdk-config 1.12.638
com.amazonaws aws-java-sdk-core 1.12.638
com.amazonaws aws-java-sdk-datapipeline 1.12.638
com.amazonaws aws-java-sdk-directconnect 1.12.638
com.amazonaws aws-java-sdk-directory 1.12.638
com.amazonaws aws-java-sdk-dynamodb 1.12.638
com.amazonaws aws-java-sdk-ec2 1.12.638
com.amazonaws aws-java-sdk-ecs 1.12.638
com.amazonaws aws-java-sdk-efs 1.12.638
com.amazonaws aws-java-sdk-elasticache 1.12.638
com.amazonaws aws-java-sdk-elasticbeanstalk 1.12.638
com.amazonaws aws-java-sdk-elasticloadbalancing 1.12.638
com.amazonaws aws-java-sdk-elastictranscoder 1.12.638
com.amazonaws aws-java-sdk-emr 1.12.638
com.amazonaws aws-java-sdk-glacier 1.12.638
com.amazonaws aws-java-sdk-glue 1.12.638
com.amazonaws aws-java-sdk-iam 1.12.638
com.amazonaws aws-java-sdk-importexport 1.12.638
com.amazonaws aws-java-sdk-kinesis 1.12.638
com.amazonaws aws-java-sdk-kms 1.12.638
com.amazonaws aws-java-sdk-lambda 1.12.638
com.amazonaws aws-java-sdk-logs 1.12.638
com.amazonaws aws-java-sdk-machinelearning 1.12.638
com.amazonaws aws-java-sdk-opsworks 1.12.638
com.amazonaws aws-java-sdk-rds 1.12.638
com.amazonaws aws-java-sdk-redshift 1.12.638
com.amazonaws aws-java-sdk-route53 1.12.638
com.amazonaws aws-java-sdk-s3 1.12.638
com.amazonaws aws-java-sdk-ses 1.12.638
com.amazonaws aws-java-sdk-simpledb 1.12.638
com.amazonaws aws-java-sdk-simpleworkflow 1.12.638
com.amazonaws aws-java-sdk-sns 1.12.638
com.amazonaws aws-java-sdk-sqs 1.12.638
com.amazonaws aws-java-sdk-ssm 1.12.638
com.amazonaws aws-java-sdk-storagegateway 1.12.638
com.amazonaws aws-java-sdk-sts 1.12.638
com.amazonaws aws-java-sdk-support 1.12.638
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.12.638
com.amazonaws jmespath-java 1.12.638
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks databricks-sdk-java 0.27.0
com.databricks jets3t 0.7.1-0
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.15.2
com.fasterxml.jackson.core jackson-core 2.15.2
com.fasterxml.jackson.core jackson-databind 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-yaml 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-joda 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.16.0
com.fasterxml.jackson.module jackson-module-paranamer 2.15.2
com.fasterxml.jackson.module jackson-module-scala_2.12 2.15.2
com.github.ben-manes.caffeine caffeine 2.9.3
com.github.fommil jniloader 1.1
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java 1.1-natives
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java 1.1-natives
com.github.fommil.netlib netlib-native_ref-linux-x86_64 1.1-natives
com.github.fommil.netlib netlib-native_system-linux-x86_64 1.1-natives
com.github.luben zstd-jni 1.5.5-4
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.10.1
com.google.crypto.tink tink 1.9.0
com.google.errorprone error_prone_annotations 2.10.0
com.google.flatbuffers flatbuffers-java 23.5.26
com.google.guava guava 15.0
com.google.protobuf protobuf-java 3.25.1
com.helger profiler 1.1.1
com.ibm.icu icu4j 75.1
com.jcraft jsch 0.1.55
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.microsoft.sqlserver mssql-jdbc 11.2.2.jre8
com.microsoft.sqlserver mssql-jdbc 11.2.3.jre8
com.ning compress-lzf 1.1.2
com.sun.mail javax.mail 1.5.2
com.sun.xml.bind jaxb-core 2.2.11
com.sun.xml.bind jaxb-impl 2.2.11
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.4.3
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.uber h3 3.7.3
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 4.0.3
commons-cli commons-cli 1.5.0
commons-codec commons-codec 1.16.0
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.5
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.13.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 3.0.3
dev.ludovic.netlib blas 3.0.3
dev.ludovic.netlib lapack 3.0.3
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.27
io.delta delta-sharing-client_2.12 1.2.2
io.dropwizard.metrics metrics-annotation 4.2.19
io.dropwizard.metrics metrics-core 4.2.19
io.dropwizard.metrics metrics-graphite 4.2.19
io.dropwizard.metrics metrics-healthchecks 4.2.19
io.dropwizard.metrics metrics-jetty9 4.2.19
io.dropwizard.metrics metrics-jmx 4.2.19
io.dropwizard.metrics metrics-json 4.2.19
io.dropwizard.metrics metrics-jvm 4.2.19
io.dropwizard.metrics metrics-servlets 4.2.19
io.netty netty-all 4.1.108.Final
io.netty netty-buffer 4.1.108.Final
io.netty netty-codec 4.1.108.Final
io.netty netty-codec-http 4.1.108.Final
io.netty netty-codec-http2 4.1.108.Final
io.netty netty-codec-socks 4.1.108.Final
io.netty netty-common 4.1.108.Final
io.netty netty-handler 4.1.108.Final
io.netty netty-handler-proxy 4.1.108.Final
io.netty netty-resolver 4.1.108.Final
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16-linux-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16-linux-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16-osx-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16-osx-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-db-r16-windows-x86_64
io.netty netty-tcnative-classes 2.0.61.Final
io.netty netty-transport 4.1.108.Final
io.netty netty-transport-classes-epoll 4.1.108.Final
io.netty netty-transport-classes-kqueue 4.1.108.Final
io.netty netty-transport-native-epoll 4.1.108.Final
io.netty netty-transport-native-epoll 4.1.108.Final-linux-aarch_64
io.netty netty-transport-native-epoll 4.1.108.Final-linux-riscv64
io.netty netty-transport-native-epoll 4.1.108.Final-linux-x86_64
io.netty netty-transport-native-kqueue 4.1.108.Final-osx-aarch_64
io.netty netty-transport-native-kqueue 4.1.108.Final-osx-x86_64
io.netty netty-transport-native-unix-common 4.1.108.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.11
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.12.1
net.java.dev.jna jna 5.8.0
net.razorvine pickle 1.3
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.9.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.10.11
org.apache.ant ant-jsch 1.10.11
org.apache.ant ant-launcher 1.10.11
org.apache.arrow arrow-format 15.0.0
org.apache.arrow arrow-memory-core 15.0.0
org.apache.arrow arrow-memory-netty 15.0.0
org.apache.arrow arrow-vector 15.0.0
org.apache.avro avro 1.11.3
org.apache.avro avro-ipc 1.11.3
org.apache.avro avro-mapred 1.11.3
org.apache.commons commons-collections4 4.4
org.apache.commons commons-compress 1.23.0
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.6.1
org.apache.commons commons-text 1.10.0
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.datasketches datasketches-java 3.1.0
org.apache.datasketches datasketches-memory 2.0.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-runtime 3.3.6
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.8.1
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.httpcomponents httpclient 4.5.14
org.apache.httpcomponents httpcore 4.4.16
org.apache.ivy ivy 2.5.2
org.apache.logging.log4j log4j-1.2-api 2.22.1
org.apache.logging.log4j log4j-api 2.22.1
org.apache.logging.log4j log4j-core 2.22.1
org.apache.logging.log4j log4j-layout-template-json 2.22.1
org.apache.logging.log4j log4j-slf4j2-impl 2.22.1
org.apache.orc orc-core 1.9.2-shaded-protobuf
org.apache.orc orc-mapreduce 1.9.2-shaded-protobuf
org.apache.orc orc-shims 1.9.2
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.ws.xmlschema xmlschema-core 2.3.0
org.apache.xbean xbean-asm9-shaded 4.23
org.apache.yetus audience-annotations 0.13.0
org.apache.zookeeper zookeeper 3.9.2
org.apache.zookeeper zookeeper-jute 3.9.2
org.checkerframework checker-qual 3.31.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.collections eclipse-collections 11.1.0
org.eclipse.collections eclipse-collections-api 11.1.0
org.eclipse.jetty jetty-client 9.4.52.v20230823
org.eclipse.jetty jetty-continuation 9.4.52.v20230823
org.eclipse.jetty jetty-http 9.4.52.v20230823
org.eclipse.jetty jetty-io 9.4.52.v20230823
org.eclipse.jetty jetty-jndi 9.4.52.v20230823
org.eclipse.jetty jetty-plus 9.4.52.v20230823
org.eclipse.jetty jetty-proxy 9.4.52.v20230823
org.eclipse.jetty jetty-security 9.4.52.v20230823
org.eclipse.jetty jetty-server 9.4.52.v20230823
org.eclipse.jetty jetty-servlet 9.4.52.v20230823
org.eclipse.jetty jetty-servlets 9.4.52.v20230823
org.eclipse.jetty jetty-util 9.4.52.v20230823
org.eclipse.jetty jetty-util-ajax 9.4.52.v20230823
org.eclipse.jetty jetty-webapp 9.4.52.v20230823
org.eclipse.jetty jetty-xml 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-api 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-client 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-common 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-server 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-servlet 9.4.52.v20230823
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.40
org.glassfish.jersey.containers jersey-container-servlet-core 2.40
org.glassfish.jersey.core jersey-client 2.40
org.glassfish.jersey.core jersey-common 2.40
org.glassfish.jersey.core jersey-server 2.40
org.glassfish.jersey.inject jersey-hk2 2.40
org.hibernate.validator hibernate-validator 6.1.7.Final
org.ini4j ini4j 0.5.4
org.javassist javassist 3.29.2-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 4.0.7
org.json4s json4s-core_2.12 4.0.7
org.json4s json4s-jackson-core_2.12 4.0.7
org.json4s json4s-jackson_2.12 4.0.7
org.json4s json4s-scalap_2.12 4.0.7
org.lz4 lz4-java 1.8.0-databricks-1
org.mlflow mlflow-spark_2.12 2.9.1
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.6.1
org.roaringbitmap RoaringBitmap 0.9.45-databricks
org.roaringbitmap shims 0.9.45-databricks
org.rocksdb rocksdbjni 9.2.1
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.15
org.scala-lang scala-library_2.12 2.12.15
org.scala-lang scala-reflect_2.12 2.12.15
org.scala-lang.modules scala-collection-compat_2.12 2.11.0
org.scala-lang.modules scala-java8-compat_2.12 0.9.1
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.2.16
org.scalanlp breeze-macros_2.12 2.1.0
org.scalanlp breeze_2.12 2.1.0
org.scalatest scalatest-compatible 3.2.16
org.scalatest scalatest-core_2.12 3.2.16
org.scalatest scalatest-diagrams_2.12 3.2.16
org.scalatest scalatest-featurespec_2.12 3.2.16
org.scalatest scalatest-flatspec_2.12 3.2.16
org.scalatest scalatest-freespec_2.12 3.2.16
org.scalatest scalatest-funspec_2.12 3.2.16
org.scalatest scalatest-funsuite_2.12 3.2.16
org.scalatest scalatest-matchers-core_2.12 3.2.16
org.scalatest scalatest-mustmatchers_2.12 3.2.16
org.scalatest scalatest-propspec_2.12 3.2.16
org.scalatest scalatest-refspec_2.12 3.2.16
org.scalatest scalatest-shouldmatchers_2.12 3.2.16
org.scalatest scalatest-wordspec_2.12 3.2.16
org.scalatest scalatest_2.12 3.2.16
org.slf4j jcl-over-slf4j 2.0.7
org.slf4j jul-to-slf4j 2.0.7
org.slf4j slf4j-api 2.0.7
org.slf4j slf4j-simple 1.7.25
org.threeten threeten-extra 1.7.1
org.tukaani xz 1.9
org.typelevel algebra_2.12 2.0.1
org.typelevel cats-kernel_2.12 2.1.1
org.typelevel spire-macros_2.12 0.17.0
org.typelevel spire-platform_2.12 0.17.0
org.typelevel spire-util_2.12 0.17.0
org.typelevel spire_2.12 0.17.0
org.wildfly.openssl wildfly-openssl 1.1.3.Final
org.xerial sqlite-jdbc 3.42.0.0
org.xerial.snappy snappy-java 1.1.10.3
org.yaml snakeyaml 2.0
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.cryptools AmazonCorrettoCryptoProvider 2.4.1-linux-x86_64
stax stax-api 1.0.1