Creating a Data Service Provider – Part 9 – Un-typed

Article
03/02/2010

Background info: This post builds on Parts 1 thru 8 which teach you how to create a typed Data Service Provider.

But now the part we’ve all been waiting for- an un-typed DSP – which is what I expect most real world DSP implementations will be based on.

What do we mean by Un-typed?

An un-typed DSP is needed whenever there isn’t a matching CLR class for each ResourceType. Typically you there is some general purpose class used for every ResourceType, something like a Dictionary for instance.

Over the course of the rest of this Post we will look at converting our DSP to expose product and category data stored in two Lists of Dictionaries, i.e. a List for each ResourceSet/Feed and a Dictionary for each Resource.

Why Un-typed?

Un-typed DSPs have a lot of interesting advantages:

You don’t need CLR classes for all the different ResourceTypes we want to expose
You don’t need to recompile / restart the service if the ResourceTypes changes in some way, i.e. you add a new Property.
You can store a pointer to the metadata for the ResourceType directly in the ResourceType itself.
You can store extra data along with a Resource very easily, which potentially allows us to take advantage of Data Services support for OpenTypes and OpenProperties.
You can evolve you model at runtime to expose more ResourceTypes/Sets

So now we know why, lets get started.

Metadata Changes:

The first thing we need to do create all our ResourceTypes:

var productType = new ResourceType(
// CLR type backing this Resource
typeof(Dictionary<string,object>),
// Entity, ComplexType etc
ResourceTypeKind.EntityType,
null, // BaseType
"Namespace", // Namespace
"Product", // Name
false // Abstract?
);
productType.CanReflectOnInstanceType = false;

Note there are two big differences between this and how we did things for Strongly Typed Resource Types (see part 3):

We tell Data Service that Products are stored in a Dictionary<string,object>
We tell Data Service that Dictionary<string,object> doesn’t have the same CLR inheritance hierarchy as the ResourceType itself – via CanReflectOnInstanceType.
This means whenever Casts are required rather than injecting standard CLR casts into the generated Query Expression, Data Services will inject methods calls the underlying injecting query provider is expected to translate.

Next we create our properties just like before:

var prodKey = new ResourceProperty(
"ProdKey",
ResourcePropertyKind.Key |
ResourcePropertyKind.Primitive,
ResourceType.GetPrimitiveResourceType(typeof(int))
);

But we add one extra call:

prodKey.CanReflectOnInstanceTypeProperty = false;

This tells DataServices that the declaring type – in this case Dictionary<string,object> - doesn’t actually have a “ProdKey” property, so a PropertyAccessor expression can’t be used.

With this setup this:

GET ~/Products(2)

Will no longer produce an (pseudo) expression like this:

from p in _queryProvider.GetQueryRootForResourceSet(productsSet)
where p.ProdKey == 2
select p;

Which would be invalid because ProdKey is not a property of Dictionary<string,object>, instead it will produce something like this:

from p in _queryProvider.GetQueryRootForResourceSet(productsSet)
where ((int) DataServiceProviderMethods.GetValue(p,"ProdKey"))
== 2
select p;

Which gives your query provider the opportunity to re-write the expression as:

from p in _queryProvider.GetQueryRootForResourceSet(productsSet)
where ((int) p["ProdKey"]) == 2
select p;

Which for example LINQ to Objects can handle.

NOTE: If your data isn’t in memory – maybe it is in a database – you still have to translate this expression into something like SQL, so it might just be easier to do that in one visit.

Once you’ve updated all your types accordingly you should have no problem getting ~/root.svc or ~/root.svc/$metadata working.

Data Source Changes:

Next we need to make some changes to our Data Sources so that the data is stored in Dictionaries.

If you remember we had this abstract base class from which our Data Source should inherit:

public abstract class DSPContext
{
public abstract IQueryable GetQueryable(
ResourceSet resourceSet);

public abstract object CreateResource(
ResourceType resourceType);

public abstract void AddResource(
ResourceSet resourceSet, object resource);

public abstract void DeleteResource(
object resource);

public abstract void SaveChanges();
}

Here is an implementation that works over dictionaries:

public class SampleUntypedContext: DSPContext
{
public const string RESOURCE_TYPE_KEY = "_ResourceType";

static Dictionary<string, List<Dictionary<string, object>>>
dataSources =
new Dictionary<string, List<Dictionary<string, object>>>{
{"Products", new List<Dictionary<string,object>>()},
{"Categories", new List<Dictionary<string,object>>()}
};

public override IQueryable GetQueryable(
ResourceSet resourceSet)
{
return GetDataSource(resourceSet.Name).AsQueryable();
}

public override void AddResource(
ResourceSet resourceSet, object resource)
{
var resourceInstance = GetResourceInstance(resource);
var instanceTypeName = GetResourceType(resourceInstance);
if (instanceTypeName != resourceSet.ResourceType.FullName)
throw new InvalidOperationException("Unexpected Resource Type");

var dataSource = GetDataSource(resourceSet.Name);
dataSource.Add(resourceInstance);
}

public override void DeleteResource(object resource)
{
var resourceInstance = GetResourceInstance(resource);
var resourceInstanceType = GetResourceType(resource);

List<Dictionary<string,object>> datasource = null;
if (resourceInstanceType == "Product")
datasource = GetDataSource("Products");
else if (resourceInstanceType == "Category")
datasource = GetDataSource("Categories");
else
throw new Exception("ResourceSet not found");
datasource.Add(resourceInstance);
}
public override void SaveChanges()
{
var products = GetDataSource("Products");
var categories = GetDataSource("Categories");

var prodKey = products.Max(p => (int)p["ProdKey"]);
foreach (var prod in products
.Where(p => ((int) p["ProdKey"])== 0))
{
prod["ProdKey"] = ++prodKey;
}

public override object CreateResource(
ResourceType resourceType)
{
var instance = new Dictionary<string, object> {
{ RESOURCE_TYPE_KEY, resourceType.FullName }
};
foreach (var property in resourceType.Properties)
{
instance[property.Name] = GetDefaultValue(property);
}
return instance;
}

public static List<Dictionary<string, object>> GetDataSource(
string resourceSetName)
{
if (!dataSources.ContainsKey(resourceSetName))
throw new Exception("ResourceSet not found");
return dataSources[resourceSetName];
}

private object GetDefaultValue(ResourceProperty prop)
{
if ((prop.Kind & ResourcePropertyKind.Primitive)
== ResourcePropertyKind.Primitive)
{
if (!prop.ResourceType.InstanceType.IsValueType)
return null;
else
return Activator.CreateInstance(
prop.ResourceType.InstanceType
);
}
else if (
prop.Kind==ResourcePropertyKind.ResourceSetReference
){
return new List<Dictionary<string, object>>();
}
else if (
prop.Kind== ResourcePropertyKind.ResourceReference
){
return null;
}
throw new NotSupportedException();
}
}

Again you can see that this has some repetitive code that we could be generalize.

We hold our Lists of Dictionary<string,object> in a Dictionary keyed on the ResourceSet name.

And as before our SaveChanges() function mimic server generated values for primary keys.

Perhaps the most interesting function is the implementation of CreateResource which adds all the properties to the Dictionary and assigns default values for each property, mimicking a strongly typed class constructor.

Now that we have a Data Source we need to fill it with some data. In the typed example we did this by overriding CreateDataSource, so no surprises, we need to do that again:

protected override SampleUntypedContext CreateDataSource()
{
var products = SampleUntypedContext
.GetDataSource("Products");
var categories = SampleUntypedContext
.GetDataSource("Categories");
if (products.Count == 0)
{
var bovril = new Dictionary<string,object>
{
{"_ResourceType", "Namespace.Product"},
{"ProdKey", 1},
{"Name", "Bovril"},
{"Cost", 4.35M},
{"Price", 6.49M}
};
var marmite = new Dictionary<string, object>
{
{"_ResourceType", "Namespace.Product"},
{"ProdKey", 2},
{"Name", "Marmite"},
{"Cost", 4.97M},
{"Price", 7.21M}
};
var food = new Dictionary<string, object>
{
{"_ResourceType", "Namespace.Category"},
{"ID", 1},
{"Name", "Food"}
};
// Build bi-directional relationships
food["Products"] = new List<Dictionary<string, object>> {
marmite,
bovril
};
marmite["Category"] = food;
bovril["Category"] = food;

// Store resources
products.Add(marmite);
products.Add(bovril);
categories.Add(food);
}
return base.CreateDataSource();
}

IDataServiceMetadataProvider changes:

There are no real changes required here, because essentially this is just returning metadata, and the same code we wrote for the Typed provider will work still.

IDataServiceQueryProvider changes:

Here we need to change three things.

The way we work out the ResourceType for a resource:

public ResourceType GetResourceType(object target)
{
return GetResourceType(GetResource(target));
}

private Dictionary<string, object> GetResource(object target)
{
var dict = target as Dictionary<string, object>;
if (dict == null)
{
throw new InvalidCastException(
"Resource is not a dictionary"
);
}
return dict;
}

private ResourceType GetResourceType(
Dictionary<string, object> resource)
{
ResourceType type = null;
if (!resource.ContainsKey("_ResourceType"))
throw new InvalidOperationException(
"ResourceType not known"
);

if (!_metadata.TryResolveResourceType(
resource["_ResourceType"].ToString(),
out type)
)
throw new InvalidOperationException(
"ResourceType not found"
);
return type;
}

The way we get property values:

public object GetPropertyValue(
object target,
ResourceProperty resourceProperty)
{
var resource = GetResource(target);
if (resource.ContainsKey(resourceProperty.Name))
return resource[resourceProperty.Name];
throw new InvalidOperationException("Property not found");
}

And finally the way we return query roots.

If we continue to do what we did in part 5:

public IQueryable GetQueryRootForResourceSet(
ResourceSet resourceSet)
{
return _currentDataSource.GetQueryable(resourceSet);
}

It will work for things URLs like
~/Products
~/Categories

But as soon as we do anything interesting like

~/Products(1)
~/Products(1)/Category
~/Categories(1)/Products
~/Products/?$filter=Name eq 'Marmite'

It will all fall over in a crashing heap.

Because we are Un-typed, DataServices will inject calls to DataServiceProviderMethods.XXX into the query, and then LINQ to Objects will just blindly call those methods – which unfortunately can’t be called directly.

We need to write a wrapped IQueryable that wraps the LINQ to Objects queryable and rewrites the expression so LINQ to Objects can deal with it.

Turns out I wrote a blog post just yesterday on this :)

In that blog post I created an InterceptedQuery<> and InterceptingProvider that wrap an IQueryable and allow you to visit and alter the expression – which is exactly what we need to do – before it passing it on to the underlying query provider.

So our method now looks like this:

public IQueryable GetQueryRootForResourceSet(
ResourceSet resourceSet)
{
var underlying = _currentDataSource.GetQueryable(resourceSet);
return InterceptingProvider.Intercept(underlying, dspVisitor);
}

Visitor Implementation

The interesting piece of course is dspVisitor. This is an instance of a class derived from System.Linq.Expressions.ExpressionVisitor that looks like this:

public class DSPExpressionVisitor: ExpressionVisitor
{
static readonly MethodInfo GetValueMethodInfo =
typeof(DataServiceProviderMethods)
.GetMethod(
"GetValue",
BindingFlags.Static | BindingFlags.Public,
null,
new Type[] {
typeof(object),
typeof(ResourceProperty)
},
null
);

static readonly MethodInfo GetSequenceValueMethodInfo =
typeof(DataServiceProviderMethods)
.GetMethod(
"GetSequenceValue",
BindingFlags.Static | BindingFlags.Public,
null,
new Type[] {
typeof(object),
typeof(ResourceProperty)
},
null
);

static readonly MethodInfo ConvertMethodInfo =
typeof(DataServiceProviderMethods)
.GetMethod(
"Convert",
BindingFlags.Static | BindingFlags.Public
);

static readonly MethodInfo TypeIsMethodInfo =
typeof(DataServiceProviderMethods)
.GetMethod(
"TypeIs",
BindingFlags.Static | BindingFlags.Public
);

// Replace calls to DataServiceProviderMethods.GetValue
// with this expression.
// we cast to a dictionary, and then use the dictionary
// item accessor to get the property.
static readonly Expression<Func<object, ResourceProperty, object>> GetValueReplacement =
(o, rp) => (o as Dictionary<string,object>)[rp.Name];

// To check that something is a particular type we cast to
// Dictionary and compare the _ResourceType field with the
// fullname of the ResourceType.
// Obviously this approach won’t handle inheritance.
static readonly Expression<Func<object, ResourceType, bool>> TypeIsReplacement =
(o, rt) => ((string)(o as Dictionary<string, object>)["_ResourceType"]) == rt.FullName;

protected override Expression VisitMethodCall(
MethodCallExpression node
)
{
if (node.Method == GetValueMethodInfo)
{
// Arguments[0] - the resource to get property from
// Arguments[1] - the ResourceProperty to get
// Invoke the replacement expression, passing the
// appropriate parameters.
return Expression.Invoke(
Expression.Quote(GetValueReplacement),
node.Arguments[0],
node.Arguments[1]
);
}
else if (node.Method.IsGenericMethod &&
node.Method.GetGenericMethodDefinition() ==
GetSequenceValueMethodInfo)
{
// Arguments[0] - the resource
// Arguments[1] - the Property that is a sequence
// Just call the GetValueReplacement(0,1) and
// cast it to IEnumerable<T> which is the
// correct return type
return Expression.Convert(
Expression.Invoke(
Expression.Quote(GetValueReplacement),
node.Arguments[0],
node.Arguments[1]
),
node.Method.ReturnType);
}
else if (node.Method == TypeIsMethodInfo)
{
// Arguments[0] – the resource
// Arguments[1] – the ResourceType
// Invoke the TypeIsReplacement expression
// binding to the resource & resourceType
return Expression.Invoke(
Expression.Quote(TypeIsReplacement),
node.Arguments[0],
node.Arguments[1]
);
}
else if (node.Method == ConvertMethodInfo)
{
// Arguments[0] – the resource
// Arguments[0] – the ResourceType
// no need to do anything, so just
// return the argument
return this.Visit(node.Arguments[0]);
}
return base.VisitMethodCall(node);
}
}

This visitor class grabs hold of MethodInfo’s for each of the DataServiceProviderMethods methods, and stores them once for performance reasons.

Then it overrides VisitMethodCall looking for each of those MethodInfo’s in turn, replacing them with an expression that LINQ to Objects can handle.

To do this we use a little trick to feed in our replacementExpressions:

Expression.Invoke(
Expression.Quote(replacementLambda),
argument0,
argument1
)

Using this approach we don’t have to manually construct the lambda expression, which I think makes our visitor easier to read.

Remember while you are doing this translation / replacement that you need to create expression that your underlying IQueryable provider can actually handle.

For example LINQ to Objects can handle these expressions without any problem, but the Entity Framework would choke because it can’t translate Invoke Expressions in to SQL.

NOTE: if you using .NET 3.5 – where System.Linq.Expressions.ExpressionVisitor isn’t available – you can still use a visitor derived IQToolkit’s base visitor or from this one.

IDataServiceUpdateProvider Changes:

This one is not too tricky: you just need to change each methods so they work against dictionaries rather than strongly typed classes.

Basically we need to take the IDataServiceUpdateProvider implementation in part 7 and update each method in turn to work against a dictionary.

Generally this actually makes things easier (and faster) because we no longer need to use reflection.

For example SetValue is a lot easier now:

public void SetValue(
object targetResource,
string propertyName,
object propertyValue)
{
try
{
_actions.Add(() => ReallySetValue(
targetResource,
propertyName,
propertyValue)
);
}
catch { }
}

public void ReallySetValue(object targetResource, string propertyName, object propertyValue)
{
var resource = targetResource as Dictionary<string, object>;
resource[propertyName] = propertyValue;
}

Putting it altogether

If you make all these changes, you should now have a fully functional, Read/Write Un-typed Data Service Provider!

Congratulations, the world is now officially your oyster…

Comments

Anonymous
April 02, 2010
The comment has been removed
Anonymous
April 03, 2010
Matthew,Assuming you don't have a CLR type for every table then you need to rework the IQToolkit to work untyped.I.e. you need to make it so you can have multiple IQueryable<DSPResource> all pointing at different tables etc.Alex
Anonymous
April 30, 2010
Hey Alex, there are unanswered questions since Dec 09 on this post of yours: can you please take a look?http://blogs.msdn.com/adonet/archive/2009/07/22/customizing-t4-templates.aspxWe are trying to use DataAnnotations in our t4 templates. It seems like a common requirement - surely there must be full implementation available somewhere?Thanks
Anonymous
May 08, 2010
Just wondering, if you might keep your sources somewhere available for this "lab"? Svn/git/...I'd love to follow along with the series and staying in sync :)
Anonymous
June 07, 2010
I'm generating the return results on the fly in code for an un-typed provider, but I'm having trouble with the relationships.returning a List<dictionary<string,object>>() which works fine for ~/Products and ~/Products(1)but how would I return the relationship data for ~/Products(1)/CategoryI triedentity["Category"] = new Dictionary<string,object>()and then populated the categoryand alsoentity["Category"] = new List<Dictionary<string,object>>()and then added a dictionary containing the Categorybut with both I get a NullRefferenceException somewhere in the internals of DataServiceHow am I suppose to populate the relationships?
Anonymous
June 14, 2010
Hi Marius,Could you please post your question on our forum: social.msdn.microsoft.com/.../threads and please include the callstack (even with the DataService internals) of the failure. Ideally if you have a repro project, you can sedn it to me.Thanks,
Anonymous
June 24, 2010
chinese versionwww.cnblogs.com/.../DSP9.html
Anonymous
July 24, 2010
Alex can you add an example of client call to untyped provider? what do i get on the client side?Do i get a dictionary?Thanks in advance
Anonymous
October 31, 2010
Alex, Thanks for your series of WCF dataservice, Can you help to give me some instruction as request below:I want to write a data service with a custom datacontext, data will be queried on the fly (as Object Context), I dont wanna call the CreateDataSource to input data as sample data. I want to override the way of querying data as each time has the request I will build a command to work with data in DB. How can I work on this?
Anonymous
March 16, 2012
Hi Alex, would you like to tell me where can I get the code example for this series ?

Partager via