Tip 55 - How to extend an IQueryable by wrapping it.

Over the last couple of years I’ve found myself in lots of situations where I’ve wanted to get ‘under the hood’ and see what is happening inside an IQueryable, but I haven’t had an easy solution, at least until now.

Getting down and dirty like this is interesting because it means you can:

• Log Queries before they are executed
• Rewrite expressions, for example replacing expressions that are not supported by a provider - EF, LINQ to SQL, LINQ to Objects etc - with expressions that are.

Anyway, I was very pleased a while back when reviewing some sample code by Vitek, who has a new blog by the way, and I realized I could generalize it to create an InterceptedQuery<> and an InterceptingProvider.

The basic idea is that you can use it like this:

public IQueryable<Customer> Customers
{
get{
return InterceptingProvider.CreateQuery(_ctx.Customers, visitor);
}
}

Here visitor, is an ExpressionVisitor that will visit, and potentially rewrite, any queries composed with Customers before it is handed off to the underlying queryable (in this case the Entity Framework).

Implementation

Thanks to Vitek, the implementation is actually pretty simple.

Lets start with the implementation of the InterceptedQuery<> which is trivial:

public class InterceptedQuery<T> : IOrderedQueryable<T>
{
private Expression _expression;
private InterceptingProvider _provider;

     public InterceptedQuery( 
       InterceptingProvider provider, 
       Expression expression) 
    { 
        this._provider = provider; 
        this._expression = expression; 
    }  
    public IEnumerator<T> GetEnumerator() 
    { 
        return this._provider.ExecuteQuery<T>(this._expression); 
    }  
    IEnumerator IEnumerable.GetEnumerator() 
    { 
        return this._provider.ExecuteQuery<T>(this._expression); 
    }  
    public Type ElementType 
    { 
        get { return typeof(T); } 
    }  
    public Expression Expression 
    { 
        get { return this._expression; } 
    }  
    public IQueryProvider Provider 
    { 
        get { return this._provider; } 
    } 
}

Next lets look at InterceptingProvider which is much more interesting:

public class InterceptingProvider : IQueryProvider
{
private IQueryProvider _underlyingProvider;
private Func<Expression,Expression>[] _visitors;

    private InterceptingProvider(
IQueryProvider underlyingQueryProvider,
params Func<Expression,Expression>[] visitors)
{
this._underlyingProvider = underlyingQueryProvider;
this._visitors = visitors;
}

    public static IQueryable<T> Intercept<T>(
IQueryable<T> underlyingQuery,
params ExpressionVisitor[] visitors)
{
Func<Expression, Expression>[] visitFuncs =
visitors
.Select(v => (Func<Expression, Expression>) v.Visit)
.ToArray();
return Intercept<T>(underlyingQuery, visitFuncs);
}

    public static IQueryable<T> Intercept<T>(
IQueryable<T> underlyingQuery,
params Func<Expression,Expression>[] visitors)
{
InterceptingProvider provider = new InterceptingProvider(
underlyingQuery.Provider,
visitors
);
return provider.CreateQuery<T>(
underlyingQuery.Expression);
}
public IEnumerator<TElement> ExecuteQuery<TElement>(
Expression expression)
{
return _underlyingProvider.CreateQuery<TElement>(
InterceptExpr(expression)
).GetEnumerator();
}
public IQueryable<TElement> CreateQuery<TElement>(
Expression expression)
{
return new InterceptedQuery<TElement>(this, expression);
}
public IQueryable CreateQuery(Expression expression)
{
Type et = TypeHelper.FindIEnumerable(expression.Type);
Type qt = typeof(InterceptedQuery<>).MakeGenericType(et);
object[] args = new object[] { this, expression };

        ConstructorInfo ci = qt.GetConstructor(
BindingFlags.NonPublic | BindingFlags.Instance,
null,
new Type[] {
typeof(InterceptingProvider),
typeof(Expression)
},
null);

        return (IQueryable)ci.Invoke(args);
}
public TResult Execute<TResult>(Expression expression)
{
return this._underlyingProvider.Execute<TResult>(
InterceptExpr(expression)
);
}
public object Execute(Expression expression)
{
return this._underlyingProvider.Execute(
InterceptExpr(expression)
);
}
private Expression InterceptExpr(Expression expression)
{
Expression exp = expression;
foreach (var visitor in _visitors)
exp = visitor(exp);
return exp;
}
}

Notice that whenever the Query is executed, we Intercept the current Expression, which involves calling all registered ‘visitors’ in turn, and then executing the resulting expression against the underlying provider.

Implementation Notes:

At its foundation our implementation uses Func<Expression,Expression> rather than .NET 4.0’s System.Linq.Expressions.ExpressionVisitor, mainly because .NET was a little late to the party, so there are lots of visitors that don’t derive from System.Linq.Expressions.ExpressionVisitor today.

You need look no further than Matt Warren’s excellent IQToolkit, for many such examples.

Nevertheless we want to encourage the use of System.Linq.Expressions.ExpressionVisitor so there is a convenience overload for that too.

Remember also that if you are wrapping the Entity Framework and doing any non-trivial expression rewrites you have to avoid any Invoke Expressions - see Colin’s blog post.

One of IQToolkit useful visitors is called ExpressionWriter and with a few minor mods – making both its constructor and the base Visit method public – you can use it to write the expression to the console before any Entity Framework query is executed:

CustomersContext _ctx = new CustomersContext();
ExpressionWriter _writer = new ExpressionWriter(Console.Out);

public IQueryable<Customer> Customers{
get{
return InterceptingProvider.Intercept(_ctx.Customers, _writer.Visit);
}
} 

You’ll also notice that we are using the IQToolkit’s useful TypeHelper class in our untyped CreateQuery method, it really helps us create the correct generic InterceptedQuery<> instance.

Thanks again Matt!

Putting it all together:

Lets show an end to end example.

Here I mimic what a WCF/ADO.NET Data Service has to do to handle this request:

GET ~/People/?$filter=Surname eq ‘James’

if it is backed by an un-typed DSP.

// Create some data
List<Dictionary<string, object>> data = new List<Dictionary<string, object>>();
data.Add(
new Dictionary<string, object>{{"Surname", "James"}, {"Firstname", "Alex"}}
);
data.Add(
new Dictionary<string, object>{{"Surname", "Guard"}, {"Firstname", "Damien"}}
);
data.Add(
new Dictionary<string, object>{{"Surname", "Meek"}, {"Firstname", "Colin"}}
);
data.Add(
new Dictionary<string, object>{{"Surname", "Karas"}, {"Firstname", "Vitek"}}
);
data.Add(
new Dictionary<string, object>{{"Surname", "Warren"}, {"Firstname", "Matt"}}
);

// Create a couple of visitors
var writer = new ExpressionWriter(Console.Out);
var dspVisitor = new DSPExpressionVisitor();

// Intercept queries to the L2O IQueryable
var queryRoot = InterceptingProvider.Intercept(
data.AsQueryable(), // L2O’s iqueryable
writer.Visit, // What does the expression look like first?
dspVisitor.Visit, // Replace GetValue().
writer.Visit // What does the expression look like now?
);

// Create a Data Services handle for the Surname property
ResourceProperty surname = new ResourceProperty(
"Surname",
ResourcePropertyKind.Primitive,
ResourceType.GetPrimitiveResourceType(typeof(string))
);

// Create a query without knowing how to access the Surname
// from x.
var query =
from x in queryRoot
where ((string) DataServiceProviderMethods.GetValue(x, surname))
== "James"
select x;

// Execute the query and print some results
foreach (var x in query)
Console.WriteLine("Found Match:{0}",
x["Firstname"].ToString()
);

As can see we have some People data in a list of dictionaries, and we are trying to find just the People with a Surname of ‘James’.

The problem is that Data Services doesn’t know how to get the Surname from a Dictionary. So it injects a call to DataServiceProviderMethods.GetValue(..).

Fine.

Unfortunately at this point the LINQ to Objects query provider doesn’t have enough context to process this query - blindly calling GetValue like we are doing in this query will fail.

So we intercept the query, and the DSPExpressionVisitor (which I won’t go into here) simply replaces things like this:

DataServiceProviderMethods.GetValue(x, surname)

with this

x[surname.Name]

which if you look at surname you can see is the same as:

x["Surname"]

So when the whole expression is visited you end up with something like this:

var query =
from x in queryRoot
where ((string) x[surname.Name]) == "James"
select x;

Which Linq to Objects can handle just fine!

Summary

This is a general purpose solution that allows you to layer one IQueryable over another, and translate / rewrite / log the query expression before it is passed to the underlying provider.

Enjoy.

Comments

• Anonymous
  March 26, 2010
  My brain hurts. I think I'll have to reread this post a couple of times before I'll be able to fully understand it but it seams really cool. I have to try this.Thanks, great post.
• Anonymous
  September 08, 2010
  Hi Alex,I'm having a little trouble identifying where I would find DSPExpressionVisitor(). Could you point me in the right direction?Cheers,-Joe
• Anonymous
  September 08, 2010
  JoeIt is basically the same as the DSPMethodTranslatingVisitor in the OData Provider toolkit in the OData-SDK www.odata.org/.../odata-sdkLet me know if that helpsAlex
• Anonymous
  September 08, 2010
  Also for more information check out blogs.msdn.com/.../creating-a-data-service-provider-part-9-un-typed.aspxWhich explains why it is necessary etc.
• Anonymous
  November 14, 2011
  Wow, thanks for the post it's exactly what I need. I've implemented this solution in a Repository<T> pattern to abstract the underlying ORM and it works like a charm! Saved me hours of work! ;)