Transparency in the design process

Today marks the first day of engineering for the ADO.NET Entity Framework V2.0. V2 of the product is a combination of a continued investment in our greater data platform vision as well as a focus on new and expanded ORM scenarios. Some of the things that we are trying to do in this release are as follows:

• Persistence Ignorance : We are looking at ways to introduce a full POCO solution for state management and interaction with the ObjectContext.
• N-Tier Support : Today we support Data Contract serialization of entities or exposing entities via Astoria, in V2 we would like to expand to a DataSet like experience where one can remote graphs and changes to the graphs across the wire using standard WCF services.
• Code-First : We want to enable a convention based, code only experience with EF wherein one may start with classes and opt-in to database generation and deployment. We expect that we would provide attributes and external mapping capabilities for people who wanted something beyond the convention based mapping.
• TDD Scenarios: With the introduction of POCO classes some of the TDD scenarios get a lot easier, and we are looking at incorporating some other asks to better fill out the scenario, such as making our ObjectQuery<T> and other generated members of our context and classes virtual.
• FK's : Today we support bi-directional relationships, and we are looking at introducing model concepts to facilitate the definition of FK like experiences in the model or in one's POCO classes.
• Lazy Loading: Today we support explicit lazy loading (calling .Load), and we are looking at various options around LoadOptions as well as outright implicit lazy loading.
• Query Tree Re-Writing: This allows framework develolpers to contextualy, vertically and horizontally filter query results.
• ...

We have been looking at various options to ensure that the design of Entity Framework V2 truly reflects the requirements of the day to day challenges that our developer community faces when building real-world applications and services. We would like to start by being as transparent as possible in the design process. The approach we take will be similar to what you have seen on Astoria Team blog (ADO.NET Data Services), but I wanted to briefly describe how we’ll go about it so you know exactly what you are looking at when reading one of our design-related posts.

How did we get here? Version 1 of the ADO.NET Entity Framework is set to ship as a part of Visual Studio 2008 SP1 and the .NET Framework 3.5 SP1. This release has been the result of a number of years of effort. During V1 the team was focused on implementing an initial set of scenarios as well as the groundwork necessary to continue to build out the Entity Framework in future versions.

Where are we now? Today marks the first day of engineering for ADO.NET Entity Framework V2.0. What this means in practice is that we have a team with developers, testers, program managers, etc., that we have regular design meetings, and that we have a timeline (or more accurately, that we’re required to have a timeline but of course we’re still working on it :).

Transparency in the design? Over the years Microsoft has been opening up the engineering processes incrementally. Long ago there were only betas, and that was the only chance to see and give feedback about a product before it shipped. Then we started to do Community Tech Previews (CTPs). CTPs enabled us to provide bits more often and gather feedback frequently. The goal with increasing the transparency of design is to take this one step further: we would like to enable folks that are interested in Entity Framework version 2 to follow the design topics as we discuss them, and have the opportunity to provide feedback right during the time where we are actively discussing a certain aspect and before we have made a final decision.

What exactly would we make visible? In short, our design process. To be more concrete, I’m not talking about some fancy set of specifications. What I mean is that as we go through the detailed design of the various aspects of Entity Framework V2, we would post to this blog a) the meeting notes from our design meetings (the team has a design meeting twice a week, plus a lot of impromptu hallway chats), and b) deeper write-ups of specific design challenges where we’d like folks to understand how we’re approaching a problem and provide a channel for deep, detailed feedback.

How transparent is transparent?  We want to be completely clear about the scope of the information we are sharing. One of the things we need to learn both from the Microsoft side and from the community side is whether the model works within a practical set of restrictions. We will post as much of our discussions as it is practically possible. However, we have to make sure we don’t compromise the interests of Microsoft as a company. There are certain things that can range from ideas to specific implementation details that we could consider trade secrets, high-value Microsoft intellectual property or something along those lines. It *will* happen that in some cases we will not discuss a topic publicly, either for a certain term (e.g. until a proper IP protection mechanism is in place) or until we ship or ever. This is nothing new, but I haven’t seen folks from large companies discuss this explicitly before, so we wanted to make sure it is clear here.

About your feedback: We would love to hear your thoughts, be it comments, suggestions, ideas or anything else. However, in the end we are designing a commercial Microsoft product. So we’ll happily take your feedback but you need to understand that by providing us feedback in any form you are agreeing that we may use it to develop our product, that others may use it in connection with the product and that you will not be compensated for any of these things. We may incorporate ideas or make changes based on the comments you make, or we may make changes to the product that are indirectly influenced by discussions that we have with you and other folks in the community. Again, this is nothing new, but instead of having some fancy statement written in legal lingo we wanted to be upfront about this here in this first post on the topic. Of course our legal folks looked at this, and they were cool enough to understand that the informal nature of the process is what makes it work, and they let us get away with this statement in which I think we clearly delineate what will happen with whatever feedback you send our way.

We will start posting design notes and challenges soon and tweak the process as we go.

Tim Mallalieu
Program Manager, Entity Framework

Comments

• Anonymous
  June 23, 2008
  Today marks the first day of engineering for the ADO.NET Entity Framework V2.0, and the team has been

• Anonymous
  June 23, 2008
  Great news! Will there be more bits for early download for MVPs?

• Anonymous
  June 23, 2008
  Entity Framework 2.0 development starts today

• Anonymous
  June 23, 2008
  i think this and this is good news: MS takes a second try to make the ADO.NET Entity Framework a really

• Anonymous
  June 24, 2008
  Good news! What do You mean with "FK like experiences in the model"? What about payload attributes in relationships? What about n-ary relationships (n <> 2)? What about multi valued complex types? What about static class members? These are the things I would like to see in EF V2. No competitor (AFAIK) can provide such and realization of the former will make EF the "best in class". What about the conceptional asymmetry between fetching (LINQ-like) and storing (procedural)? What about the fact, that neither the class model nor the database schema can contain as much information as the conceptual model. So deriving the classes from the database (the only thing one has is a relation! hence the name) can't succede in general (what are entities, what are relationships, what are complex types, what are enums, where to get the behaviour). And deriving the db schema from the classes (where to store the static members, how to map inheritance) can't succede either without additional information. The missing info is stored in the conceptual model (CM)! The only way to infer either classes and db schema is deriving both from the CM! Deriving the artifacts is a PROJECTION! So please leave the straight (not to say trivial models, with convention over configuration) to "LINQ to SQL" and please provide a mature designer for the more complex scenarios, where one needs to store complex mapping which can't be derived from either artifact. For updating the CM out of changed artifacts, please consider instrumenting the generated assemblies with the necessary info out of the CM and find a way to incorporate this info into the db (independent of vendor!). This may reduce the effect of the projection but leads to exposition of conceptional info to end users. Thanks for Your patience. Kind regards Martin

• Anonymous
  June 24, 2008
  Of course this post leads to the question...When is EF v1 going to be released?

• Anonymous
  June 24, 2008
  Here is a raw cut and paste for our POCO 1-Pager. We are currently working through the design and have

• Anonymous
  June 24, 2008
  The comment has been removed

• Anonymous
  June 24, 2008
  The comment has been removed

• Anonymous
  June 24, 2008
  ADO.NET Entity Framework Vote of No Confidence

• Anonymous
  June 24, 2008
  Great news, great idea! Thank you! I think the most important thing is to make the designer features equal to engine features. I wont use stuff in my real world application if I have to write a tons of imperspicuous XML code to have these. A question: will EF v2 be part of .NET 4.0 or there will be an other .NET 3.5 SP release? Regards, Gabor Mezo

• Anonymous
  June 24, 2008
  General Responses: >> Early Bits Yes we will have early bits, outside of the CTP Rhythm. We are still working on the logistics. We will likely make these more broadly available via something like code gallery than restricting just to MVP's. We need to close on the details before makign any full commitment. >> FK experiences We are working on being able to reason about 1-way references instead of bi-directional references. We will drop design notes on this once we have something people can throw darts at >> More relationships (n-ary, rel'ns with payload...) We have been looking at these, they were part of the original data model but cut due to schedule and priority. In the fullness of time we will add these. I am not sure that they will land in V2 as a lot of the focus will be on the types of things outlined in the post. >> assymetry between fetching and storing.... Not sure what this meant. >> designer Yes we need to work on the designer. Yes, the transformation from the classes to the model and from the model to the store are projections. That is a key foundation upon which many of our services will be built in the future.

• Anonymous
  June 24, 2008
  The comment has been removed

• Anonymous
  June 25, 2008
  I had intended to be happy simply being a signatory of ADO .NET Entity Framework Vote of No Confidence

• Anonymous
  June 25, 2008
  This is fabulous news.  Thanks!

• Anonymous
  June 26, 2008
  I made the mistake recently of reading some of the responses to the &#39;Entity Framework Vote of No

• Anonymous
  June 26, 2008
  The comment has been removed

• Anonymous
  June 27, 2008
  If you've been paying attention to what the Entity Framework team has been saying. You will remember

• Anonymous
  June 30, 2008
  .Net -i arendajate kommuuni poolt on postitatud väga asjalik "Vote of no confidence" artikkel ADO.NET

• Anonymous
  July 01, 2008
  No início do ano eu postei um artigo chamado 2008 – O ano para alavancar sua carreira , que foi muito

• Anonymous
  July 02, 2008
  While I have finished my series on LINQ to SQL I wanted to talk about some of the reaction. In his summary

• Anonymous
  July 02, 2008
  While I have finished my series on LINQ to SQL I wanted to talk about some of the reaction. In his summary

• Anonymous
  July 06, 2008
  The comment has been removed

• Anonymous
  July 06, 2008
  The comment has been removed

• Anonymous
  July 08, 2008
  The ADO.NET Entity Framework is very strategic to Microsoft - but a) it is a V1.0 technology (although

• Anonymous
  July 09, 2008
  The Entity Framework's provider model makes it possible for it to work over different database's. The

• Anonymous
  July 21, 2008
  V1 of the Entity Framework allows you to use stored procedures in two main ways: Mapping Create, Update

• Anonymous
  August 01, 2008
  Evolving an API to support new requirements, like POCO, while maintaining backward compatibility is challenging.

• Anonymous
  August 12, 2008
  In V1 of the Entity Framework it is possible to annotate a schema using attributes declared in another

• Anonymous
  August 29, 2008
  Well, I've tried Ideablade's DevForce EF, it is already much more advanced than EF v1 and probably more than EF v2 will be. How difficult can it be for big Microsoft to develop a real enterprise solution with code-generator, model designers, best practices and patterns... everything combined. MM.Net from Mere Mortals, CSLA.Net, DevForce from IdeaBlase, ... They are all much more advanced than everything MS has to offer nowadays. And please, don't talk me about EntLib 4. It doesn't have the designers and felixibility 3rd party vendors has. You want a good example, look as XAF from DevExpress and where they are heading... I'm sorry, I don't believe in this v2 so fast, it will take months before the 'final' version is released and it will be dropped in the next year or 2. Adopt CSLA.Net or XPO or so, make a superb generator, wizard thingy for it with perfect N-tier samples. Then we, developers, would have something to rely on for more than 2 or 3 years.

• Anonymous
  September 10, 2008
  One of the most painful omissions from the Entity Framework V1 was Model First, which basically means

• Anonymous
  October 08, 2008
  In this post Colin Meek and Diego Vega delve into some enhancements we are planning that to LINQ to Entities,

• Anonymous
  October 08, 2008
  In this post Colin Meek and Diego Vega delve into some enhancements we are planning that to LINQ to Entities,

• Anonymous
  October 17, 2008
  Three key things are missing from the entity framework for me, you can work around then with a bit of coding but for me these are important for a project to be expandable and for the EF to become very flexible. First, Model inheritance, you should be able to create models that inherit from other models and can have associations with objects from other models. This would be usefull for applications that use a common base database and build on this structure. Second, Xml datatype. Third, dynamic columns in tables.

• Anonymous
  October 17, 2008
  It would be great if we could known what kind of WCF Service API would be choosen to re-attach a graph of entities in a 3-tiers dettached scenario. So we could already use this API with a custom implementation and switch to EFv2 implementation latter without changing the API. For example, is the API will looks like EntityBag<T> API ? It will also be great if the API would not be too "Microsoft" oriented but rather technology agnostic so we could use it in an interrop scenario between a JAVA Server/.NET Client OR .NET Server/ JAVA Client

• Anonymous
  October 19, 2008
  I'm pretty sure you know very well that ,

• Anonymous
  October 27, 2008
  If you are reading this, you have probably heard by now about the so called impedance mismatch between

• Anonymous
  November 20, 2008
  The first version of Entity Framework provides convenient ways to load, manipulate and persist objects

• Anonymous
  December 02, 2008
  Unfortunately in the current version of the Entity Framework, which ships in .NET 3.5 SP1, we don't make

• Anonymous
  December 03, 2008
  good news this blog. we waiting v2 early. good job.

• Anonymous
  January 08, 2009
  Today the Entity Framework, and more specifically the Entity Data Model, have a limited notion of Functions.

• Anonymous
  January 12, 2009
  A while back I wrote a post that introduced the concept of Computed Properties . Since that time we’ve

• Anonymous
  January 22, 2009
  When VS 2010 ships it will include some significant improvements to our code generation story for the

• Anonymous
  February 17, 2009
  After a year of working with LINQ to SQL, I strongly belivev that LINQ to SQL and Entity Framework (EF)

• Anonymous
  March 16, 2009
  Background A number of months ago we asked whether Foreign Keys (FKs) in Conceptual and Object models

• Anonymous
  March 23, 2009
  Background One of the biggest pieces of feedback we received from the N-Tier Improvements for Entity

• Anonymous
  April 17, 2009
  a nice feature will be working with several edmx: entity1 map in in file1.edmx can inherit from entity2 map in file2.edmx. So you can work with several models !

• Anonymous
  June 10, 2009
  There are currently two ways to get Entity Framework apps up and running, we call these Database First

• Anonymous
  July 31, 2009
  I make a test for this Walkthrough by using code only Context, and i receive this error "underlying connection was closed: The connection was closed unexpectedly" I think that there is an incompatobility when use T4 SelftTracking with CodeOnly approche. unfortunately for us, it's an other limitation of EF

• Anonymous
  July 31, 2009
  Could you please share the code that you tried either here or by sending email to dsimmons@microsoft.com? These two features are not intended to be incompatible, and if there is some issue we will certainly fix it before final release.

• Danny

• Anonymous
  July 31, 2009
  Think you Daniel, I made some diagnostics, and I suspect that the problem lies in the client side when reading the response message. "HandleReturnMessage Void (System.Runtime.Remoting.Messaging.IMessage ....." I make the query to the  model directly from a console application and it works correctly. The service also works correctly in this case:        public Post GetPostWithComments(int postID)        {            var builder = new ContextBuilder<BloggingStoreContext>();            builder.RegisterKey((Entry c) => c.ID);            builder.RegisterKey((Person p) => p.ID);            builder.RegisterKey((Blog b) => b.ID);            SqlConnection connection = new SqlConnection(@"Data Source=LABSQLEXPRESS;Initial Catalog=BloggingCode;Integrated Security=SSPI;MultipleActiveResultSets=True;");            using (var context = builder.Create(connection))            {                if (!context.DatabaseExists())                    context.CreateDatabase();                Post post = context.Entries.OfType<Post>().Include("Comments").SingleOrDefault(e => e.ID == postID);                Post copy = new Post(); //Just for test succesfull returned object                copy.Title = post.Title;                return copy;            }        } but I receice an error "underlying connection was closed: The connection was closed unexpectedly" when make this test        public Post GetPostWithComments(int postID)        {            var builder = new ContextBuilder<BloggingStoreContext>();            builder.RegisterKey((Entry c) => c.ID);            builder.RegisterKey((Person p) => p.ID);            builder.RegisterKey((Blog b) => b.ID);            SqlConnection connection = new SqlConnection(@"Data Source=LABSQLEXPRESS;Initial Catalog=BloggingCode;Integrated Security=SSPI;MultipleActiveResultSets=True;");            using (var context = builder.Create(connection))            {                if (!context.DatabaseExists())                    context.CreateDatabase();               return context.Entries.OfType<Post>().Include("Comments").SingleOrDefault(e => e.ID == postID);            }        } I will send you the refactored files. Nabil

• Anonymous
  March 12, 2010
  It would be really good if xml entity mapping (as an option) for Code-Only is allowed.

• Anonymous
  March 13, 2010
  @Kevin, Can you elaborate a bit?  What do you mean by xml entity mapping?  Are you saying you want to pass some xml to a code-only method which describes the mapping, or are you saying you want to map from properties in the database to an xml property on your entity class, or are you saying you want to map from an xml property in the database to more than one regular property on your entity class, or something else? Thanks, Danny

• Anonymous
  April 06, 2010
  Hi Tim, Sorry, if this is not the right forum to ask this question, but still: Several papers and documentation on Entity Framework are describing how to generate the 'first model' or create the 'first mapping'. I miss the discussion of how to handle an 'application change' or 'the application change iteration'. Ongoing applications with long life cycles, several iterations, several customers, several dbms's often include consecutive changes to the database which require changes to the entity model and changes to the mapping. Where are the 'automatic' or the 'reversing?' detecting of such changes addressed? Since it is not addressed, is it because EVERY application change is surrounded with a NEW database including data transfer from the old one? Thanks. Thehim.

• Anonymous
  April 07, 2010
  Hello Thehim, In general, using EF will not really force you to create a new database or to transfer the data for every change that you make. It is even possible in some cases to alter the database schema (i.e. using DDL commands or the tools provided by the database) and then adjust the storage model definition and the mapping specification without affecting the existing conceptual model, and vice-versa: make changes to the conceptual model and your classes that do not require changes in the database schema, just in the mapping.  Of course, some types of changes (i.e. add a new non-nullable column) will have to be propagated to the other side. Currently, the process of propagating the changes in most cases is not automatic, though. How you keep the database, the model and your entity types in sync depends heavily on how you started your application and how you want to make the changes:

• Changes in the database can be propagated to the model: o Semi-automatically to the model using the “Updated model from the database” option in the EDM Designer (but this may overwrite your customization), or o Manually using the designer to edit the storage  and the mapping information o Manually editing the XML artifacts directly
• Changes in the model can be propagated to the database: o Manually issuing DDL commands or using the tools provided by the database o Optionally, you can regenerate the database completely using Model-First tooling (but this will overwrite the exiting database)
• Changes in the model can be propagated to the entity types: o Automatically if you are using any of the code-generation options o Manually if you have handwritten POCO classes
• If you are using Code-First in the Feature CTP, you can propagate changes to the database: o Manually issuing DDL commands or using the tools provided by the database o Optionally, you can regenerate the database completely using ObjectContext.CreateDatabase (but this will overwrite the existing database) We are currently considering alternative designs for generating automatic migration scripts for some of these scenarios in future versions of EF. It would be great to hear more about how you would use this feature and what you would expect to be able to do. Thanks, Diego

• Anonymous
  April 13, 2010
  The comment has been removed

• Anonymous
  April 13, 2010
  ...sorry, should have been exciting, not exiting, huh?

• Anonymous
  December 09, 2010
  We are up to CTP 5 and I still don't see much improvement from  early releases it is still clunky to work with. http://dotbloat.blogspot.com