Codename "Astoria": Data Services for the Web

The "data programmability" team at Microsoft is responsible for the various technologies developers use in applications to access and manipulate data. One of the topics we have been looking at lately is how "new" internet applications use data in the web environment. Project codename "Astoria" reflects our current thinking on the topic. Instead of telling "what we are going to build", we decided to make the bits and infrastructure public for the development community to look at and give us feedback.

Today we are making a very early experimental release of Project Codename "Astoria". This Community Tech Preview (CTP) release has a dual nature; we are shipping both CTP bits you can download, install and run on your own systems; we are also making available an experimental online service that we hope will help better understand the requirements and use cases of data interfaces in the web.

So, what does it do?

Astoria exposes "data services" that enable applications to access and manipulate data over regular HTTP connections, using URIs to identify pieces of information within the data service, and simple payload formats such as XML and JSON to represent the data exchanged between the client and the server.

What is it useful for?

There are a number of scenarios that we think are interesting for this technology. Most of the scenarios are related to the way new web applications are built.

If you look at AJAX-based web applications, one interesting aspect of the way they are delivered to the client is that typically the presentation and client-side behavior is delivered on the initial hit to a given web page, and then the code in that page (typically Javascript) turns back and fetches data as the user interacts with the user interface. This brings a strong separation between presentation and data. What is more, this means that now we require a server-side piece that the javascript code can "talk to". Of course, you can always roll you own server-side entry points for data access; however, not only that is expensive but also it greatly reduces the chances to build tooling and user-interface controls that can work generically on any data access entry point into the server.

This separation is even stronger in rich internet applications built on technologies such as Silverlight and Flash. The code that is delivered to the client when a user hits the web application is pre-compiled and it contains no data at all other than any hardcoded information. Again, these applications talk back to their servers to retrieve data, manipulate it and push changes back into the service for storage.

Astoria data services are designed to address this space. They present data in a uniform way that can be consumed by any client that can connect over HTTP and parse XML or JSON. The uniform URI and payload format patterns mean that user-interface widgets can be built so they work against any particular shape of data (“schema” if you will) that is exposed through the service.

Where is it?

The main site is here: https://astoria.mslivelabs.com

This document introduces the concepts and motivations for Astoria.

You can also see and interact with several sample data-sets that are already available on the experimental online service. Click on the "Online Service" link in the link bar.
 

Looking forward to hearing your feedback.

-pablo

Comments

• Anonymous
  April 30, 2007
  You've been kicked (a good thing) - Trackback from DotNetKicks.com

• Anonymous
  April 30, 2007
  Hey Pablo! Congratulations for your new blog and for Astoria! Now I know why you were missing lately ;)

• Anonymous
  April 30, 2007
  Microsoft has just announced two new projects at MIX '07: Jasper and Astoria (codenames). Both tools

• Anonymous
  April 30, 2007
  MIX07 Announcements

• Anonymous
  April 30, 2007
  Pablo Castro has announced a very cool project , Microsoft Codename "Astoria": Data Services for the

• Anonymous
  April 30, 2007
  Funny... I was in Astoria yesterday.

• Anonymous
  April 30, 2007
  Pablo, this sounds great. I've been thinking for a while that this is one of the missing pieces in Microsoft's Web strategy, especially in light of the coming Silverlight 1.1. I've been doing exactly what you mention - constantly rebuilding a data interface on the server and grabbing that data. It works well enough but there's plenty of work to do to get this set up for every project. I noticed Julie mentioned that this what tied to the Entity framework? I hope this will be a bit more generic than that <s>...

• Anonymous
  April 30, 2007
  Where will the business rules go? What differs Astoria from WCF REST/POX/JSON endpoints that exposes EDM?

• Anonymous
  May 01, 2007
  I guess that my thinking is similar to Patrik's (above). My full writeup is here: http://udidahan.weblogs.us/2007/05/01/astoria-sdo-and-irrelevance/ As a generic data fetching mechanism, I guess that it's OK, but you seem to pushing the entire REST model which includes data updates - something that absolutely requires business rules. Also, how does this jive with the Service Data Objects work being done elsewhere (IBM et al.)? If all we're really interested in is bringing data closer to the web client, why not just use XML files that are HTTP cached? That is probably the most scalable solution, designed for growth even to things like the Akamai network. I'd really like to see some more architecture/vision style things so that I can understand how this fits in an overal solution. -- Udi Dahan - The Software Simplist MVP Solutions Architect

• Anonymous
  May 01, 2007
  Well, this one snuck up on me. I hadn't heard about this project before Mix, unlike the Silverlight &amp;

• Anonymous
  May 01, 2007
  Hey Pablo forgive me but am I right that "Astoria" is the long awaited  new technology called HTTP2SQL2XML ? ;-)

• Anonymous
  May 01, 2007
  I've been sifting throught the docs and have collected my thoughts here: http://www.base4.net/blog.aspx?ID=394 Keen to here your answers to my questions ;)

• Anonymous
  May 02, 2007
  Patrik: regarding where do the business rules go, this is the way I picture apps would expose their data services: for the subset of the data that is not heavy in semantics would just be exposed directly using Astoria URIs (using policies as needed to enforce read-only/read-write access, as well as auth/anonymous access requirements); for the other parts of the data that need business logic enforced, there are a few mechanisms, and you'd choose depending or your needs: a) you can use regular WCF service operations; these are nice when you want to take an arbitrary input and return an arbitrary output. b) you can use "data aware" WCF service operations, these are the ones that return ObjectQuery<T>. This are great when you want to provide controlled access to the data (e.g. restrict how much data you can see, apply custom policies, validate parameters, etc.). The great part about these ones is that they still get some of the URI patterns (e.g. you can sort, page, expand, etc.) c) if you are ok with flat access but want to control/validate/etc. the entities that enter and leave the system, there are "interceptors" (i'm looking for a better word :) that you can register; you can think of these as middle-tier triggers or something like that. Your code gets called when creating/updating/deleting entities as well as when sending them to the client, and you can make checks, modify the entity or abort the operation. As in how is this different from WCF entry points with EDM entities: this is actually a special form of WCF entry points that exposes data in a URI space automatically, without having to deal with data shaping, URI syntax for sorting/paging/filtering/etc. yourself. -pablo

• Anonymous
  May 02, 2007
  Udi: I just posted some comments regarding Patrik's and your points around business logic. I agree that in most real apps, a large portion of the data in the system cannot be exposed "directly", and that is way Astoria supports various mechanisms for running code in the service, including WCF service operations, "data aware" service operations and also interceptors that run on both entity fetching and during entity updates. I am planning on writing a more elaborate "app architecture using Astoria" post at some point...let's say Mix has kept us busy lately :), but I'll get to it soon. -pablo

• Anonymous
  May 02, 2007
  Adrian: :) We borrowed a lot from SQLXML. In fact, Andy Conrad, one of the folks in the Data Programmability team, used to be part of the SQLXML team. We learned a lot from SQLXML, and also the environment has changed since then. Astoria takes a lot of the lessons (good and bad) and is also designed to live in the current state of the web. There are some aspects such as instance-based updates to give it a more REST-like feel, using managed code in the middle tier for business rules, support for JSON, and others, that were necessary in my opinion and weren't in SQLXML back then. I talk often with various folks that built SQLXML and exchange notes to make sure I "reuse" good ideas and avoid mistakes. -pablo

• Anonymous
  May 02, 2007
  At 1999/2000 people start to do queries with URL and QueryStrings to get XML from a data source, to make it possible to ask for specific data and get a result that was platform independent. Then later we got FOR XML added to SQL Server and also a way to run query directly against the database with URLs. We also got SOAP Web Services. Now we have WCF. But suddenly we are back again to the 1999/2000 where we do queries over URL’s ;)

• Anonymous
  May 02, 2007
  A few months ago, Alex Barnett excitedly told me about a project he was working on with Pablo Castro

• Anonymous
  May 02, 2007
  Hi, are the above links down ? i am getting 'host not found' regards, vikas

• Anonymous
  May 03, 2007
  Links to the video of Pablo's XD006 session are at http://oakleafblog.blogspot.com/2007/05/mix07-session-videos-with-linq-ef-or.html More background on LINQ and Entity Framework related MIX07 sessions is at http://oakleafblog.blogspot.com/2007/04/linq-related-sessions-at-mix.html --rj

• Anonymous
  May 03, 2007
  Thanks for your answer Pablo. It does clear some clouds but I still don't see where Astoria fits into the puzzle and excels over existing technology. I'm looking forward to your architecture post. [OT]:: Btw, can you confirm that you pulled EF so you could implement Model first as an approach in the tools?

• Anonymous
  May 03, 2007
  vikas: I just tested both the website and the online service and they all worked from here. Please try again if you have a chance...if this keeps happening let me know and I'll take another look. -pablo

• Anonymous
  May 03, 2007
  Thanks Pablo .. there seems to be some problem with one of the proxy in my corporate .. i am able to access using other. regards, vikas

• Anonymous
  May 05, 2007
  At MIX07 I've seen a presentation by Pablo Castro from the ADO.NET team on a new project codenamed "Astoria".

• Anonymous
  May 09, 2007
  Pablo, The problem I have is with the generic "put" which leaves you with simplistic optimistic locking. I need a business level API for passing along the client's intent. Check out Realistic Concurrency here: http://udidahan.weblogs.us/2007/01/22/realistic-concurrency/

• Anonymous
  May 09, 2007
  Last week at Mix07 Microsoft announced a project code named Astoria. The technology code named Astoria

• Anonymous
  May 11, 2007
  This was one of several projects that did not surface much at MIX due to the silverlight hype.. I have

• Anonymous
  May 12, 2007
  Hypothetical Performance Question... Pablo, I would like to know if Astoria can solve a hypothetical data broadcasting performance issue. We have a client who needs to broadcast, virtually simultaneously, a small amount of data -- typically a single integer -- to as many as tens of thousands of client applications. The client applications need to register for the broadcast; wait; receive the data within the narrowest possible timespan; and react to the data immediately. We need to know what the "narrowest possible timespan" for such an Astoria broadcast would be... and of course whether Astoria is a candidate technology for such an implementation.

• Anonymous
  May 14, 2007
  Last week at MIX07 Microsoft announced a project code named Astoria. The technology code named Astoria

• Anonymous
  May 20, 2007
  ASTORIA Data Services

• Anonymous
  May 21, 2007
  The comment has been removed

• Anonymous
  June 18, 2007
  Project Codename “ Astoria” - http://astoria.mslivelabs.com . Microsoft Codename Astoria enables applications

• Anonymous
  July 11, 2007
  Quick update on what Ive been looking at recently.. (the on-going list of technolgies and things-to-learn

• Anonymous
  July 12, 2007
  The Astoria Team is happy to announce the availability of an add-on to the Astoria May CTP that contains

• Anonymous
  August 03, 2007
  Back in April, the Data Programmability team at Microsoft announced "Astoria": Data Services for the

• Anonymous
  August 07, 2007
  I've seen a few instances on posts to forums or blogs asking whether Astoria continues to be an experiment

• Anonymous
  August 07, 2007
  I&#39;ve seen a few instances on posts to forums or blogs asking whether Astoria continues to be an experiment

• Anonymous
  November 10, 2007
  12 or 24 templates two start projects.And will there bee a way use office live work space and listas.