INCONSISTENT_BEHAVIOR_CROSS_VERSION error class
You may get a different result due to the upgrading to
DATETIME_PATTERN_RECOGNITION
Spark >= 3.0:
Fail to recognize <pattern>
pattern in the DateTimeFormatter.
- You can set
<config>
to “LEGACY
” to restore the behavior before Spark 3.0. - You can form a valid datetime pattern with the guide from ‘
<docroot>
/sql-ref-datetime-pattern.html’.
DATETIME_WEEK_BASED_PATTERN
Spark >= 3.0:
All week-based patterns are unsupported since Spark 3.0, detected week-based character: <c>
.
Please use the SQL function EXTRACT
instead.
PARSE_DATETIME_BY_NEW_PARSER
Spark >= 3.0:
Fail to parse <datetime>
in the new parser.
You can set <config>
to “LEGACY
” to restore the behavior before Spark 3.0, or set to “CORRECTED
” and treat it as an invalid datetime string.
READ_ANCIENT_DATETIME
Spark >= 3.0:
reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z
from <format>
files can be ambiguous, as the files may be written by
Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar
that is different from Spark 3.0+’s Proleptic Gregorian calendar.
See more details in SPARK
-31404. You can set the SQL config <config>
or
the datasource option <option>
to “LEGACY
” to rebase the datetime values
w.r.t. the calendar difference during reading. To read the datetime values
as it is, set the SQL config <config>
or the datasource option <option>
to “CORRECTED
”.
TBD
Spark >= <sparkVersion>
: <details>
WRITE_ANCIENT_DATETIME
Spark >= 3.0:
writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into <format>
files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive later, which uses a legacy hybrid calendar that is different from Spark 3.0+’s Proleptic Gregorian calendar.
See more details in SPARK
-31404.
You can set <config>
to “LEGACY
” to rebase the datetime values w.r.t. the calendar difference during writing, to get maximum interoperability.
Or set the config to “CORRECTED
” to write the datetime values as it is, if you are sure that the written files will only be read by Spark 3.0+ or other systems that use Proleptic Gregorian calendar.