Known issue - Unsupported error for legacy timestamp in Fabric Runtime 1.3
When using the native execution engine in Fabric Runtime 1.3, you might encounter an error if your data contains legacy timestamps. This issue arises due to compatibility challenges introduced when Spark 3.0 transitioned to the Java 8 date/time API, which uses the Proleptic Gregorian calendar (SQL ISO standard). Earlier Spark versions utilized a hybrid Julian-Gregorian calendar, resulting in potential discrepancies when processing timestamp data created by different Spark versions.
Status: Open
Product Experience: Data Engineering
Symptoms
When using legacy timestamp support in native execution engine for Fabric Runtime 1.3, you receive an error. The error message is similar to: Error Source: USER. Error Code: UNSUPPORTED. Reason: Reading legacy timestamp is not supported.
Solutions and workarounds
For more information about the feature that addresses this known issue, see the blog post on legacy timestamp support. To activate the feature, add the following to your Spark session: SET spark.gluten.legacy.timestamp.rebase.enabled = true. Dates that are post-1970 are unaffected, ensuring consistency without extra steps.