ADF Dataflow Derive Column transformation toTimestamp is working for constant timezone but not from column

Question

I am finding Azure Data Factory Dataflow Dataflow Derive Column transformation toTimestamp is working for constant timezone but not from column. I want each column to output the date with time adjusted based on the timezone UTC offset. Any suggestions on a solution?

The constant works with this expression:

toTimestamp(toString(toTimestamp('2022-05-23', 'yyyy-MM-dd'), 'yyyy-MM-dd'), 'yyyy-MM-dd', 'EST')

Data preview:

But this column expression does not:

toTimestamp(toString(toTimestamp('2022-05-23', 'yyyy-MM-dd'), 'yyyy-MM-dd'), 'yyyy-MM-dd', TIMEZONENAME)

Data preview:

We've previously done something like this, but found that it does not work for some Windows time zones and IANA time zones.

toTimestamp(concat(toString(toTimestamp('2022-05-23', 'yyyy-MM-dd'), 'yyyy-MM-dd'), ' ', TIMEZONENAME), 'yyyy-MM-dd zzzz')

Answer 1

@David Buterbaugh

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue:

ADF Dataflow Derive Column transformation toTimestamp is working for constant timezone but not from column

Solution:

The toTimestamp function doesn't support column timezones, but it seems that toUTC does.

toUTC() supporting dynamic time zones is a game-changer, especially when dealing with daylight savings time adjustments.

Expression:

toUTC(toTimestamp(toString(toTimestamp('2022-05-23', 'yyyy-MM-dd'), 'yyyy-MM-dd'), 'yyyy-MM-dd'), TIMEZONENAME)

Dynamically resolves TIMEZONENAME without needing a constant value.

Properly adjusts for daylight savings time, unlike static UTC offsets.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

As Chandra Boorla mentions, toTimestamp doesn't support column timezones. But it appears toUTC does. This appears to meet my need.

toUTC(toTimestamp(toString(toTimestamp('2022-05-23', 'yyyy-MM-dd'), 'yyyy-MM-dd'), 'yyyy-MM-dd'), TIMEZONENAME)

Note that using an offset doesn't really work well with daylight savings time as that offset may change depending on the calendar date.