A Brief History of DateTime [Anthony Moore]

For the .NET 3.5 release, the BCL team has been trying to solve some problems related to dates, times and time zones that customers having been asking us to address for some time. We’ve invested in two significant features:

• TimeZoneInfo: Extended Time Zone Support. This allows for enumeration, conversion and serialization of a time zones other than a machine’s current time zone. This includes support for time zones that change Daylight Saving Time rules over time, such as the recent 2007 change in the USA.
• DateTimeOffset: A DateTime that includes a UTC Offset. This allows a local time zone to map to an absolute UTC time and preserves the local time even if the instance is moved from one machine to another.

DateTimeOffset is a completely new DateTime type that stores the UTC Offset in addition to the DateTime. In presenting the design of this type, there are a couple of questions that keep coming up that we would like to address. The full rationale behind these provisional decisions requires going back through some of the history of DateTime over the last 8 years.

Common Questions

In .NET 3.5 there will be a new DateTime type called DateTimeOffset. The name of this type is an interesting topic in itself, but current plans are that this new type will complement rather than fully replace the existing DateTime type, and thus it will not be called DateTime2. A couple of questions keep coming up about this which I will address in this post by covering some relevant history:

1. Why can’t the existing DateTime type simply be extended to have the UTC offset in it?
2. If (1) is not possible, why can’t the new type be called DateTime2 and completely replace all the usage scenarios of the existing DateTime?

It is necessary to start with the original rationale and design issues surrounding the original DateTime type.

In the Beginning (V1.0)

The original DateTime type was mostly designed in 1998-1999, as part of the the new .NET Platform type system. Up until that time, almost all platforms (e.g. Win32, C, VB, COM, databases) represented dates and times with a combined date and time type that had no time zone information. The time zone context was outside the type’s data and in the application’s context, in the same way that using Decimal to represent money involves the context of the currency unit living outside the Decimal instance. The option of a DateTime with a UTC offset or a time zone ID was discussed and there was a lot of design back and forth. There was a lot of desire to keep the type system simple, and while a time zone aware DateTime was better at representing absolute points in time such as a file time-stamp, it got awkward when doing certain things:

• Interoperating with database, COM or legacy systems where the time zone could not be reliably inferred.
• Representing genuinely abstract dates and time, e.g. the opening times for stores of a company spanning time zones.
• Representing whole dates, e.g. the type for a calendar control to return. (Obviously a “Date” type is an even better way of representing this, and many people have asked for separate Date and Time types, but in V1.0 there was a desire to keep the number of base types small to limit complexity of the platform. This feature request is definitely still on the table, although not planned for .NET 3.5).

Another factor was that many prior platforms had lived with DateTime types like this without running into serious issues. In the end, the desire to keep the platform simple won out, and the DateTime was a date and a time and nothing else, although support for conversion between local and UTC was provided using the machine’s local time zone.

Once this was shipped, a number of serious problems became clear:

• Data loss during DST rollback. Unfortunately the decision to use a simple DateTime type was combined with the convention of defaulting to returning local instances from almost all .NET Framework APIs. Of 84 APIs returning “absolute” points in time in the framework, 81 returned local instances. This resulted in data loss during the repeated hour when the clock was rolled back coming out of DST. Win32 did not have this problem because APIs defaulted to UTC. It was desirable for the .NET Framework to return more user-friendly local times, but the reliability implications were not fully evaluated before shipping. This was definitely an unfortunate mistake.
• Arithmetic errors during DST transitions. As above, the prevalent use of local DateTime instances created a usability problem in that people would do arithmetic with local instances, which was wrong more often than not because of changes to the clock during transitions in and out of DST. Again, Win32 did not have this problem because of the default to UTC. This can be worked around, although the data loss problem above meant that it could not be fully worked around.
• Serialization in XML defaulted to assuming all DateTime instances were local, which prevented correct serialization of UTC instances, abstract instances or date-only instances. It would have been better to not assume anything about the time zone since the instance had no such information.
• DateTime could not deal with the server-side scenario of dealing with local instances from time zones other than the current machine. Correctly handled this required using two fields to represent a DateTime, which was decidedly clumsy.

Changing the Existing DateTime (v2.0)

In V2.0, when the severity of these issues was fully understood, design began on a solution. There was actually a fully developed specification to address the above issues by adding a UTC offset to the existing DateTime type. This was going to be a challenging operation since this was a significant change to what the type was. The only way to keep a reasonable compatibility guarantee was to make the time zone offset optional. However, this proposed change simply did not stand up to design scrutiny and implementation trials, and we discovered that our options of changing the existing type were significantly more limited than we had hoped. Here were some of the problems that emerged:

• Compatibility was the main issue. The only way to have stayed completely compatible would have been for existing APIs to have never set or consumed the offset, even for existing APIs on the DateTime class itself. The optional offset was fine for new code, but having something like DateTime.Now actually set the local time offset, which would have been required to address the data loss problem, would break any code that decided to act differently in the presence of the offset. So you can either have DateTime respect the new offset in existing APIs, but then not allow any existing APIs to return an instance with the offset, or you allow existing APIs to return them, but have DateTime only opt-in to actually using it. Neither option actually addressed the data loss in the existing APIs and would have required them to have been deprecated. There is a case described below that provides a concrete example of the compatibility issues even after the proposed solution was significantly scaled back.
• A lot of systems special cased the binary layout of base types such as DateTime. The design required making the DateTime at least 2 bytes larger. It is not clear if this would have been practical from a compatibility standpoint either.
• If we were to deprecate the existing APIs taking or return DateTime instances, adding a completely new type would have been a much clearer and more compatible way to do this anyway, as it would have been unclear what name the new versions of the existing APIs, and there would have been sets of APIs with identical signatures that differed only by the type of instance of DateTime they took or returned.
• The optional UTC offset created a lot of design consistency problems, because instances without an offset might be used with instances with one. It is not clear what comparison should do. You would either have to disallow comparison, in which case it might as well be a new type anyway, or you could have a compatible comparison that would have odd behavior, such as intransitive comparison. This also creates issues with extra argument validation as most APIs would want to take either absolute or floating instances and not either one.

This was a difficult situation. It was extremely undesirable to leave every existing API taking DateTime and accepting or returning local instances with unreliable behavior around DST transitions. But a comprehensive change to DateTime was not possible in a compatible way either. It was decided to do a very minimal change to DateTime by using some unused bits in the structure to indicate if an instance was UTC or Local, and using an extra state to disambiguate the hour repeated during DST transition to solve the reliability problem. The DateTime.Kind property was introduced to solve the reliability problem without doing a full deprecation of usage of the type. However, it was not a comprehensive solution because to maintain compatibility, the Kind could only be very minimally used in existing APIs. It was decided that we would try to make it possible for existing APIs to use the Kind, but make any behavior changes on DateTime itself opt-in as much as possible. Thus the Kind is used in the critical conversions to and from Local and UTC, but not when DateTime instances are compared or formatted by default.

Even this minimal change presented significant compatibility challenges. An example of this was DateTime.ToString. If the instance was marked with DateTimeKind.Utc, it would makes sense to output its time zone offset as “+00:00” instead of pulling in the machine’s local time which only makes sense for local instances. At one point the code actually did this more correct behavior. However, we found that a significant number of users of DateTime were taking a dependency on this incorrect behavior and were broken by this change. They were serializing UTC instances in XML, and because it defaulted to formatting as a local time and because Parse converted back to the local time zone by default, their code was working most of the time, even though the persisted string would have been incorrect if parsed by a 3^rd party system. To maintain compatibility even this change had to be backed out, and correct formatting of the time zone offset had to be made opt-in via the new “o” and “K” format options.

The DateTimeKind solved the most pressing issues with DateTime, but still left the space with a lot of unresolved issues. It was still very difficult to use correctly, and it could not deal well with the server scenarios where instances from different machines in different time zones needed to be handled.

So, this should address the first question of why we are not currently pursuing further the option of adding an optional offset to the existing DateTime type.

The Comprehensive Solution

After .NET 2.0 it became clear that there was still considerable demand for a type with DateTime and a UTC offset combined. This is the type we are delivering in .NET 3.5 (DateTimeOffset). The second question then comes up as to why we are planning to a new type that complements DateTime instead of creating a complete replacement and calling it DateTime2. It is definitely compelling to have a single new type so that there does not have to be complex guidance around which of the types to use for what scenarios.

Even with compatibility taken out of the picture a lot of the reasoning above still makes it compelling to have new type that is always absolute. First, consider the landscape of date and time related scenarios:

1. Legacy DateTime Interop (databases, COM, Win32, etc).
2. Abstract DateTime scenarios (e.g. store starting time).
3. Date-only scenarios.
4. Time-only scenarios.
5. Absolute time scenarios. (e.g. file time).
6. Server scenarios with dates from multiple time zones where the original local time must be retained.

(Scenarios C-D might be even better handled by dedicated Date and Time types. These options are still on the table, although they will not be in the .NET 3.5 release).

The existing DateTime can do scenarios A-E. But it does a mediocre job of E, because you have to convert it to and from UTC and Local depending on whether you want human-readable display or accurate arithmetic. For scenario F, you need to store an additional field to make it work. The proposed new DateTime with UTC offset is significantly better for scenario E and enables scenarios F with a single type instance. However, it is not as good for scenarios A-D where it would have to default to some arbitrary time zone offset. As a point of reference, the majority of DateTime scenarios involve E, although A and C are fairly common also, and F becomes more important when richer Time Zone support is available.

It is possible to create a new type that addresses scenarios A-D as well by making the UTC Offset optional rather than required. But this creates a lot of the same problems encountered during the aborted attempt to create such a type in .NET 2.0:

• The design consistency problems mentioned above related to having interaction of abstract and absolute instances. With an optional offset, you either get behavior different from DateTime, intransitive comparison or the inability to compare or subtract the two instance types at all.
• Most APIs taking DateTime as input will want something that is always abstract or always absolute and would be unlikely to need either at runtime. As such, most APIs would have to do extra argument validation to special case the less desirable behavior. There is value in having the type always constrained to represent an absolute and unambiguous point in time.
• Deprecating DateTime by adding DateTime2 would imply a second-hand deprecation of any API that did take or return DateTime. While the new type is definitely better for some scenarios, DateTime is working just fine in the majority of existing uses, and is no longer subject to data loss in the V2.0 timeframe forward.
• The majority of APIs on the new type would end up having dual behaviors depending on the mode, which would make the class difficult to understand and use.
• Databases that do support dates and times with UTC offsets also do so by having a separate column type that always has an offset. Breaking from this pattern would make moving data to and from a database more complex.
• The main benefit of a single type is to avoid having to describe or understand which type to use for which scenario. However if there was such a combined type with or without a UTC offset, you would just have to transfer the complex explanation from the selection of the type to the use of the type.

Retrospective Thoughts

An interesting question that is always useful to ask about a design is: Given what we know now, what would we do differently if we could start again?

It would be hard to get broad consensus on this point. My personal opinion is that the date and time space is too complicated to address with a single type. If I could start again, I would have something like the V1.0 DateTime, but have a separate DateTimeOffset type also. The main difference to where we will land with the current plan of record is that we would not need the DateTimeKind that was added to DateTime in V2.0. This was really a band-aid that would not have been needed if a DateTimeOffset had been available in V1. That being said, my own opinion is that the DateTimeKind solution was critical to salvage reliable usage of the existing APIs given that there was no better type to use in V1.0.

Summing Up

1. Why can’t the existing DateTime type simply be extended to have the offset in it and enable these new scenarios? The option of doing this has been thoroughly explored, and there is no practical way to do it without breaking compatibility.
2. If (1) is not possible, why can’t the new type be called DateTime2 and completely replace all the usage scenarios of the existing DateTime? The new type is less effective for scenarios when there is genuinely no time zone such as dealing with DateTime instances from legacy system, dealing with abstract instances, dealing with date-only instances and dealing with time-only instances. Having it support these scenarios as well and fully deprecating the existing DateTime has issues that may not be outweighed by the benefits.

Comments

• Anonymous
  June 18, 2007
  PingBack from http://msdnrss.thecoderblogs.com/2007/06/18/a-brief-history-of-datetime-anthony-moore/

• Anonymous
  June 18, 2007
  You've been kicked (a good thing) - Trackback from DotNetKicks.com

• Anonymous
  June 18, 2007
  Great post Anthony, thanks! Thoughts: (1) << DateTime is working just fine in the majority of existing uses>> I don't think anyone thinks this is really the case. I think we'd all like to see APIs amended as soon as possible to support the new type. (2) In the summary description of DateTime for intellisense, please can we have words to the effect that DateTime is effectively deprecated for many scenarios in favour of the new type. This would go a long way to preventing any more buggy .NET date/time code. (3) Was there any consideration given in the design process around making DateTimeOffset an (immutable) reference type? Pete.

• Anonymous
  June 18, 2007

1. What about System.TypeCode support? Are there any plans for TypeCode.DateTimeOffset ?
2. Are there any plans to support DateTimeOffset as a column type in SQL 2008?
3. Will DateTimeOffset be supported in XSD schemas and System.IO classes? What can be possible issues with mass replacing DateTime to DateTimeOffset ?

• Anonymous
  June 18, 2007
  While I appreciate your efforts, these solutions do not truly address the real problems. I am also a customer and have been reporting my concerns and issues for many years. I need a UTC-DateTime that is a DateTime. If I use DateTimeOffset, I will still need to convert it to DateTime when passing it as a parameter to existing methods. This introduces an opportunity for errors. > "there is no practical way to do > it without breaking compatibility." Being a structure, extension is difficult. Could it be promoted to a class? Probably not a very good idea either. Ideally I would prefer an IDateTime interface, but then many existing signatures would need to be changed or it would be useless. Those changes could be done slowly overtime. But at least give us the interface and start addressing the real problem here. As it is, we now filter through all source code searching for DateTime usage. Every instance is flagged as a potential bug (and usually is) needing special review. It is very timely and costly. Depreciating this poorly designed type would be very (very) appreciated. And depreciating it would also allow you to update signatures to use a new IDateTime interface. I am also dissatisfied with TimeZoneInfo. It is a wrapper around OS timezone info. However, that OS support needs a manual update to be installed, and will need future updates as well. I can not assume that customers will have it or future updates installed. A better solution is to wrap something like http://www.twinsun.com/tz/tz-link.htm . Caching the results for an appropriate amount of time should suffice. New data can be periodically downloaded. The user need not install anything, or even know anything about it. This is something that I can (and do) do on my own. However, integrating it into the framework library is difficult without any interfaces... Please, design to interfaces. It would really solve so many issues.

• Anonymous
  June 18, 2007
  A well detailed and explanatory post that adds clarity to the DateTime enhancement and will help us full understand and apply the new type where required. I think that you have made the right design decisions - not to break with backwards compatibility and to leave the existing type in place to serve in most applications. The new type will be welcomed however by developers building systems that run across time zones.

• Anonymous
  June 19, 2007
  This is one of the best posts I've ever read on MSDN blogs.  It's extremely well written and informative.  Thanks, Anthony! FWIW, I totally agree that "DateTime is working just fine in the majority of existing uses".  It works fine for my use cases, but it's good to know that the BCL team is working hard to handle other use cases as well.

• Anonymous
  June 19, 2007
  >> Anthony Moore Everyone here writes about IDateTime interface. BUT, everyone should be also aware that with current DateTime being Value Type, there would be a performance penalty caused by (un)boxing with methods expecting ref type (IDateTime) instead...

• Anonymous
  June 19, 2007
  I share the question by an earlier poster: "Are there any plans to support DateTimeOffset as a column type in SQL 2008?" Any thoughts on this? If not, what is the preferred way of storing and converting back to DateTimeOffset from a SQL column?!

• Anonymous
  June 19, 2007
  Pete,

1. We will incrementally start to add new APIs that use the new type and will consider changing existing APIs where appropriate.
2. This is a good idea that we will definitely consider.
3. Not that I'm aware of.  I think we always had planned on the new type being a value type.  One of the main goals we had when designing DateTimeOffset was for it to be largely substitutable with DateTime. lexp,
4. There are no plans to add this to System.TypeCode.  Could you explain your scenario for needing this?
5. DateTimeOffset is indeed a new column type in SQL Server 2008!  We worked closely with the SQL team and both types are called DateTimeOffset and have the same internal layout.
6. DateTimeOffset will be supported in XSD schemas, although, I'm not sure whether or not this made it into .NET 3.5.  Which System.IO classes are you referring to?
7. We've designed DateTimeOffset's members to be as similar to DateTime as possible, so it shouldn't be too hard to mass replace. chronos, You can create custom TimeZoneInfo instances with the static TimeZoneInfo.CreateCustomTimeZone method filling it with data from a copy of the Olson time zone database if necessary. Anders, As I mentioned above DateTimeOffset is indeed a new column type in SQL Server 2008!  We worked closely with the SQL team and both type are called DateTimeOffset and have the same internal layout. Thanks, Justin Van Patten CLR Program Manager

• Anonymous
  June 19, 2007
  Tom, > BUT, everyone should be also aware > that with current DateTime being > Value Type, there would be a > performance penalty caused by >(un)boxing with methods expecting > ref type (IDateTime) instead... Of course performance is important. However, more important than that is correctability and lack of bugs. DateTime as is broken. Anyone suggesting otherwise either has not worked with the type for long or is in denial. The problems are clear and have been written about extensively. That is why various solutions are being attempted. However, none of those solutions integrate well into the existing framework. For that to happen, there are only two options: 1) inheritance or 2) interfaces. Option 1 is not really likely for technical reasons. Options 2, while potentially being less performant, offers a solution without sacrificing backward compatibility. For those who are happy with the current DateTime, they may continue to use it without any performance lose. (I truly think they will end up regretting it one day when the bugs become known, though.) This is not a solution. It only changes the problem a little.

• Anonymous
  June 19, 2007
  Justin Van Patten, > You can create custom TimeZoneInfo > instances with the static > TimeZoneInfo.CreateCustomTimeZone > method filling it with data from a > copy of the Olson time zone database > if necessary. I know. That was mentioned a few posts back. However, the real issue: how could it not be necessary? I can not assume that all users will have past and future updates installed. Therefore, I can not depend on the correct TZ information to be present on the system. Thus, I must ignore your data and always import the data via CreateCustomTimeZone. Except for integration with TimeZoneInfo, I have been doing exactly that for a while. Once Orcas is released, I will need to port that over to your TimeZoneInfo implementation. Can the type be extended? Are there any interfaces to implement? Or will you tie my hands like with DateTime and absolutely force me to use CreateCustomTimeZone? I can probably guess the answers.

• Anonymous
  June 19, 2007
  Magari non è uno dei massimi problemi per chi lavora in Europa e generalmente ha a che fare con un solo

• Anonymous
  June 20, 2007
  Java had these problems 10 years ago, and you guys did not learn anything from it.  They ended up deprecating most functionality in Date and moving it to Calendar.  Why couldn't you have a DateTime object that optionaly has a null TimeZone property?

• Anonymous
  June 20, 2007
  Great post and great title... ;)

• Anonymous
  June 21, 2007
  Great, insightful post about a difficult subject. Andrew McCallum is right that Java had similar problems with their Date-Class. But worse they did it wrong the second time with the ugly Calendar class, I've never seen such an unintuitive Date-API. Luckily they started JSR 310 and I really hope they will integrate most of the design of the great Joda-Library.

• Anonymous
  June 21, 2007
  The comment has been removed

• Anonymous
  June 21, 2007
  Exactly my point, Stephen Channell. While we can work with DateTimeOffset up to a point, it will need to be converted to a DateTime to work with existing APIs in the framework. It will be difficult if not impossible to avoid that conversion, which invites an opportunity for error and hence bugs. What we really need is a UtcDateTime that is a DateTime. That may not be possible for technical reasons (value type), but if an IDateTime interface were to be introduced and all framework signatures were updated to use the interface, both backward compatibility could be preserved and those needing a UtcDateTime would be free to use it without need for conversion. It would integrate well with the framework, too.

• Anonymous
  June 28, 2007
  I'd have to support chronos and Stephen Channell in that DateTimeOffset seems like a non-solution to the problem. UTCDateTime is an intuitive concept and IDateTime would solve compatibility issues for future revisions.

• Anonymous
  June 30, 2007
  I must agree with chronos, Stephen Channel, and Morten Mertner. DateTimeOffset is not a solution to the real problem. Backward compatibility can be preserved with the addition of an IDateTime interface while also providing a path towards a real solution: UtcDateTime, which would implement IDateTime.

• Anonymous
  August 13, 2007
  Based on the feedback from my last CLR week , I think one CLR week a year is about right, Welcome to

• Anonymous
  August 13, 2007
  CLR Week: A new annual tradition.

• Anonymous
  August 14, 2007
  Have you ever had to deal with consumer and provider of webservices residing in different time zones

• Anonymous
  September 02, 2007
  皆さん、こんにちは 前回お約束した通り、少しずつではありますがTechEdでのDEMOのフォローをしていきたいと思います。 今回は.NET Framework 3.5 で改善された TimeZone への対応について紹介したいと思います。今回、ご紹介するのは

• Anonymous
  September 02, 2007
  皆さん、こんにちは 前回お約束した通り、少しずつではありますがTechEdでのDEMOのフォローをしていきたいと思います。 今回は.NET Framework 3.5 で改善された TimeZone への対応について紹介したいと思います

• Anonymous
  July 11, 2008
  Celexa and aalcohol. Celexa and provigil. Celexa poids. Celexa and birthcontrol.

• Anonymous
  October 05, 2008
  This is the first of three postings I'm writing to follow up on the MSDN Events sessions I delivered