Data Services Streaming Provider Series: Implementing a Streaming Provider (Part 1)

Article
08/04/2010

The Open Data Protocol (OData) enables you to define data feeds that also make binary large object (BLOB) data, such as photos, videos, and documents, available to client applications that consume OData feeds. These BLOBs are not returned within the feed itself (for obvious serialization, memory consumption and performance reasons). Instead, this binary data, called a media resource (MR), is requested from the data service separately from the entry in the feed to which it belongs, called a media link entry (MLE) . An MR cannot exist without a related MLE, and each MLE has a reference to the related MR. (OData inherits this behavior from the AtomPub protocol.) If you are interested in the details and representation of an MLE in an OData feed, see Representing Media Link Entries (either AtomPub or JSON) in the OData Protocol documentation.

To support these behaviors, WCF Data Services defines an IDataServiceStreamProvider interface that, when implemented, is used by the data service runtime to access the Stream that it uses to return or save the MR.

What We Will Cover in this Series

Because it is the most straight-forward way to implement a streaming provider, this initial post in the series demonstrates an IDataServiceStreamProvider implementation that reads binary data from and writes binary data to files stored in the file system as a FileStream. MLE data is stored in a SQL Server database by using the Entity Framework provider. (If you are not already familiar with how to create an OData service by using WCF Data Services, you should first read Getting Started with WCF Data Services and the WCF Data Service quickstart in the MSDN documentation.) Subsequent posts will discuss other strategies and considerations for implementing the IDataServiceStreamProvider interface, such as storing the MR in the database (along with the MLE) and handling concurrency, as well as how to use the WCF Data Services client to consume an MR as a stream in a client application.

Steps Required to Implement a Streaming Provider

This initial blog post will cover the basic requirements for creating a streaming data service, which are:

Create the ASP.NET application.
Define the data provider.
Create the data service.
Implement IDataServiceStreamProvider .
Implement IServiceProvider.
Attribute the model metadata.
Enable large data streams in the ASP.NET application.
Grant the service access to the image file storage location and to the database.

Now, let’s take a look at the data service that we will use in this blog series.

The PhotoData Sample Data Service

This blog series features a sample photo data service that implements a streaming provider to store and retrieve image files, along with information about each photo. The following represents the PhotoInfo entity, which is the MLE in this sample data service:

This data is stored in a single PhotoInfo table in a database, which is accessed by using the Entity Framework provider. The following code defines the data service:

// This method is called only once to initialize service-wide policies.
public static void InitializeService(DataServiceConfiguration config)
{
config.SetEntitySetAccessRule("PhotoInfo",
EntitySetRights.ReadMultiple |
EntitySetRights.ReadSingle |
EntitySetRights.AllWrite);

config.DataServiceBehavior.MaxProtocolVersion =
DataServiceProtocolVersion.V2;
}

You can download the complete source code for this sample data service (and client application) from the MSDN Code Gallery.

Implementing IDataServiceStreamProvider

The WCF Data Services runtime relies on the IDataServiceStreamProvider implementation to get and set the stream that contains MR data. The methods of IDataServiceStreamProvider that are used to get and set the MR stream are GetReadStream and GetWriteStream, each of which return a Stream .

Returning a Data Stream: the GetReadStream Method

The GetReadStream method is called by the data service runtime to get the stream that contains the MR that it returns to the requesting client. The entry parameter supplied by the runtime is the MLE, and data from this entity is used to retrieve the related MR data. In our implementation, the image file is retrieved from the app_data directory by using a file name that is based on the key property of the supplied entity, as seen below:

public Stream GetReadStream(object entity, string etag, bool?
checkETagForEquality, DataServiceOperationContext operationContext)
{
if (checkETagForEquality != null)
{
// This stream provider implementation does not support
// ETag headers for media resources. This means that we do not track
// concurrency for a media resource, and last-in wins on updates.
        throw new DataServiceException(400,
"This sample service does not support the ETag header for a media resource.");
}

    PhotoInfo image = entity as PhotoInfo;
if (image == null)
{
throw new DataServiceException(500, "Internal Server Error.");
}

    // Build the full path to the stored image file, which includes the entity key.
string fullImageFilePath = imageFilePath + "image" + image.PhotoId;

    if (!File.Exists(fullImageFilePath))
{
throw new DataServiceException(500, "The image file could not be found.");
}

    // Return a stream that contains the requested file.
return new FileStream(fullImageFilePath, FileMode.Open);
}

In this implementation, a FileStream is created with read access to the stored image file, and this stream is returned by the data service. In this version of the photo service sample, we are not checking for concurrency in the MR, so we don’t need to worry about the etag or checkETagforEquality parameters. The operationContext parameter provides access to information about the request, including message headers.

Storing the Media Resource from a Filestream: the GetWriteStream Method

As you would expect, the GetWriteStream method is called by the data service to get a stream that is used to store a request from the client that contains an MR. There are two kinds of requests for which the data service uses this implementation:

HTTP POST – when a new MR is inserted, along with its associated MLE.
HTTP PUT – when the MR and MLE already exists, but the MR data is being replaced.

The GetWriteStream method implementation must support both of these write operations. While the HTTP PUT case turns out to be fairly straightforward, the HTTP POST operation is a bit more complex, for the following reasons:

A protocol requirement (inherited from AtomPub) that the MR be created before the associated MLE.
Maintaining transactional integrity between creation of the MR and the MLE.

Because of these complexities, it is helpful to understand the process by which the data service runtime handles a POST request. A new MR/MLE is created by the following process:

The client sends a POST request that contains only the media resource; such a request looks like the following:

POST /PhotoService/PhotoData.svc/PhotoInfo HTTP/1.1
User-Agent: Microsoft ADO.NET Data Services
DataServiceVersion: 1.0;NetFx
MaxDataServiceVersion: 2.0;NetFx
Accept: application/atom+xml,application/xml
Accept-Charset: UTF-8
Content-Type: image/jpeg
Slug: myphoto.jpg
Host: localhost
Transfer-Encoding: chunked
Expect: 100-continue

<<Binary media resource data…>>
The data service creates a new entity object with null properties and calls the GetWriteStream method, passing this object to the entry parameter.
Note: When using the Entity Framework provider, your implementation must set any required properties of this entity instance, which can be done either by using data from the slug message header or by setting default values. In our implementation, we use the contents of the Slug header to set the FileName property.
The data service writes the binary data received in the POST request to the stream returned by the GetWriteStream implementation.
The data service inserts the new entity into the data source. (Since we are storing PhotoInfo object data in a database, the Entity Framework provider is used to insert the row.)
The data service calls IDisposable.Dispose (if implemented) on the IDataServiceStreamProvider instance.
The data service returns a response that includes the newly created MLE, including any server-generated values such as identities. A response to the POST request in step 1 looks like the following:

HTTP/1.1 201 Created
Cache-Control: no-cache
Content-Length: 2048
Content-Type: application/atom+xml;charset=utf-8
Location: https://localhost/PhotoService/PhotoData.svc/PhotoInfo(4)
Server: Microsoft-IIS/7.0
X-AspNet-Version: 4.0.30319
DataServiceVersion: 1.0;
X-Powered-By: ASP.NET
Date: Tue, 27 Jul 2010 06:51:27 GMT

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<entry xml:base=https://localhost/PhotoService/PhotoData.svc/
xmlns:d=https://schemas.microsoft.com/ado/2007/08/dataservices
xmlns:m=https://schemas.microsoft.com/ado/2007/08/dataservices/metadata
xmlns="https://www.w3.org/2005/Atom">
<id>https://localhost/PhotoService/PhotoData.svc/PhotoInfo(4)</id>
<title type="text"></title>
<updated>2010-07-27T06:51:27Z</updated>
<author>
<name />
</author>
<link rel="edit-media" title="PhotoInfo" href="PhotoInfo(4)/$value" />
<link rel="edit" title="PhotoInfo" href="PhotoInfo(4)" />
<category term="PhotoData.PhotoInfo" scheme="https://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
<content type="image/jpeg" src="PhotoInfo(4)/$value" />
<m:properties xmlns:m=https://schemas.microsoft.com/ado/2007/08/dataservices/metadata
xmlns:d="https://schemas.microsoft.com/ado/2007/08/dataservices">
<d:PhotoId m:type="Edm.Int32">4</d:PhotoId>
<d:FileName>myphoto.jpg</d:FileName>
<d:FileSize m:type="Edm.Int32" m:null="true"></d:FileSize>
<d:DateTaken m:type="Edm.DateTime" m:null="true"></d:DateTaken>
<d:TakenBy m:null="true"></d:TakenBy>
<d:DateAdded m:type="Edm.DateTime">2010-07-26T00:00:00-07:00</d:DateAdded>
<d:DateModified m:type="Edm.DateTime">2010-07-26T00:00:00-07:00</d:DateModified>
<d:Comments m:null="true"></d:Comments>
<d:ContentType>image/jpeg</d:ContentType>
<d:Exposure m:type="PhotoData.Exposure">
<d:Aperature m:type="Edm.Decimal" m:null="true"></d:Aperature>
<d:ShutterSpeed m:type="Edm.Int16" m:null="true"></d:ShutterSpeed>
<d:FilmSpeed m:type="Edm.Int16" m:null="true"></d:FilmSpeed>
</d:Exposure>
<d:Dimensions m:type="PhotoData.Dimensions">
<d:Height m:type="Edm.Int16" m:null="true"></d:Height>
<d:Width m:type="Edm.Int16" m:null="true"></d:Width>
</d:Dimensions>
</m:properties>
(Optional) The WCF Data Services client sends a new MERGE or PUT request (depending on the SaveChangesOptions.ReplaceOnUpdate setting) to update the newly created MLE in the data service with additional values from the client’s copy of the entity. This merge overwrites any property values set by the server, except for the entity key, and returns a 204 (No Content) response.

For a POST request, our implementation of GetWriteStream sets any non-nullable property of PhotoInfo (otherwise the Entity Framework provider raises an error) and returns a new FileStream based on a temp file.

public Stream GetWriteStream(object entity, string etag, bool?
checkETagForEquality, DataServiceOperationContext operationContext)
{
if (checkETagForEquality != null)
{
// This stream provider implementation does not support
// ETags associated with BLOBs. This means that we do not
// track concurrency for a media resource and last-in wins on updates.
throw new DataServiceException(400,
"This demo does not support ETags associated with BLOBs");
}

    PhotoInfo image = entity as PhotoInfo;

    if (image == null)
{
throw new DataServiceException(500, "Internal Server Error: "
+ "the Media Link Entry could not be determined.");
}

    // Handle the POST request.
if (operationContext.RequestMethod == "POST")
{
// Set the file name from the Slug header; if we don't have a
// Slug header, just set a temporary name which is overwritten
// by the subsequent MERGE request from the client.
image.FileName =
operationContext.RequestHeaders["Slug"] ?? "newFile";

        // Set the required DateTime values.
image.DateModified = DateTime.Today;
image.DateAdded = DateTime.Today;

        // Set the content type, which cannot be null.
image.ContentType =
operationContext.RequestHeaders["Content-Type"];

        // Cache the current entity to enable us to both create a
// key-based storage file name and to maintain transactional
// integrity in the disposer; we do this only for a POST.
cachedEntity = image;

        return new FileStream(tempFile, FileMode.Open);
}
// Handle the PUT request
else
{
// Return a stream to write to an existing file.
return new FileStream(imageFilePath + "image"
+ image.PhotoId.ToString(),
FileMode.Open, FileAccess.Write);
}
}

Note that the portion of this implementation that handles the PUT request simply returns a FileStream that accesses the image file based on the key value.

After the POST operation succeeds, the temp file created by GetWriteStream is renamed to include the key value and moved to the app_data folder in the disposer, as shown by the following:

public void Dispose()
{
// If we have a cached entity, it must be a POST request.
if (cachedEntity != null)
{

        // Get the new entity from the Entity Framework object state manager.

        ObjectStateEntry entry =
this.context.ObjectStateManager.GetObjectStateEntry(cachedEntity);

        if (entry.State == System.Data.EntityState.Unchanged)
{
// Since the entity was created successfully, move the temp file into the
// storage directory and rename the file based on the new entity key.
File.Move(tempFile, imageFilePath + "image"
+ cachedEntity.PhotoId.ToString());


            // Delete the temp file.
File.Delete(tempFile);
}
else
{
// A problem must have occurred when saving the entity to the
// database, so we should delete the entity and temp file.
context.DeleteObject(cachedEntity);
File.Delete(tempFile);

            throw new DataServiceException("An error occurred. "
+ "The photo could not be saved.");
}
}
}

This also enables us to maintain some level of transactional consistency between creation of the MR and MLE from a POST request when using the Entity Framework provider; the photo service deletes the new image file if the MLE fails to be created.

Other Implementation Details

While GetReadStream and GetWriteStream are the primary methods that we implement, the IDataServiceStreamProvider interface contains several other members that are also required.

DeleteStream – this method is called by the data service to delete an MR. In our data service, this method implementation uses the File.Delete method to delete the image file, as seen in the following code:

public void DeleteStream(object entity,
DataServiceOperationContext operationContext)
{
PhotoInfo image = entity as PhotoInfo;

if (image == null)
{
throw new DataServiceException(500, "Internal Server Error.");
}

try
{
// Delete the requested file by using the key value.
File.Delete(imageFilePath + "image" + image.PhotoId.ToString());
}
catch (IOException ex)
{
throw new DataServiceException("The image could not be found.", ex);
}
}
GetStreamContentType – this method is called by the data service to determine the value of the Content-Type header to set on the response message. In our implementation, the content type is stored in the entity itself:

public string GetStreamContentType(object entity,
DataServiceOperationContext operationContext)
{
// Get the PhotoInfo entity instance.
PhotoInfo image = entity as PhotoInfo;

if (image == null)
{
throw new DataServiceException(500,
"Internal Server Error.");
}

return image.ContentType;
}
ResolveType – returns the namespace-qualified name of the type that is the MLE, which in our implementation looks like this (since we only handle one MLE type):

public string ResolveType(string entitySetName,
DataServiceOperationContext operationContext)
{
// We should only be handling PhotoInfo types.
if (entitySetName == "PhotoInfo")
{
return "PhotoService.PhotoInfo";
}
else
{
// This will raise an DataServiceException.
return null;
}
}
GetReadStreamUri – we return null to let the data service provide the default URI for the MR.
GetStreamETag – this version of the sample does not support the ETag header, which means that we do not track concurrency in the MR. This means that last-in always wins on updates.
StreamBufferSize – this property returns the size of the buffer to use for stream operations; our sample always returns 64000 (for a 64K stream buffer).

For more information about implementing this interface, see Streaming Provider (WCF Data Services) in the MSDN documentation.

Implementing IServiceProvider

Whenever a custom data service provider (of which IDataServiceStreamProvider is one) is implemented in a WCF Data Service, the IServiceProvider interface should also be implemented. IServiceProvider provides the data service runtime with a specific provider implementation, in our case the IDataServiceStreamProvider implementation that returns a PhotoServiceStreamProvider instance. The following code implements the GetService method required by IServiceProvider :

public object GetService(Type serviceType)
{
if (serviceType == typeof(IDataServiceStreamProvider))
{
// Return the stream provider to the data service.
return new PhotoServiceStreamProvider(this.CurrentDataSource);
}
return null;
}

Attributing the Model Metadata

In order for the data service to return MR data as a stream, it has to know for which entity to invoke IDataServiceStreamProvider. An MLE is identified by using the HasStream attribute. When using the reflection provider or a custom data service provider, HasStreamAttribute is applied to the entity type that is the MLE. Because our sample data service uses the Entity Framework provider, we must instead manually apply the HasStream attribute to the PhotoInfo entity in the CSDL portion of the .edmx file that represents our data model, as seen in the following XML fragment:

For more information about using the Entity Framework data provider, see Entity Framework Provider (WCF Data Services) in the MSDN documentation.

The HasStream attribute is also included in the model metadata returned by the service to client applications. When the WCF Data Services client finds the HasStream attribute on the PhotoInfo entity, it also applies HasStreamAttribute to the generated PhotoInfo class on the client.

Enabling Large Data Streams

The WCF Data Services runtime hosts a basic implementation of Windows Communication Foundation (WCF) that it uses for HTTP messaging. Because WCF limits the size of data streams, we must configure the data service endpoint to enable large streams. The following element configures our photo service to receive images up to 500KB:

<system.serviceModel>
<serviceHostingEnvironment aspNetCompatibilityEnabled="true"/>
<services>

<service name="PhotoService.PhotoData">

<endpoint binding="webHttpBinding"
bindingConfiguration="higherMessageSize"
contract="System.Data.Services.IRequestHandler">
</endpoint>
</service>
</services>
<bindings>
<webHttpBinding>

<binding name="higherMessageSize" maxReceivedMessageSize="500000"/>
</webHttpBinding>
</bindings>
</system.serviceModel>

To accept binary streams larger than 500KB, you must set a larger value for the maxReceivedMessageSize attribute.

Granting Access to the IIS Process

The IDataServiceStreamProvider implementation uses a FileStream to access image files from both the temp file directory and the Web site’s app_data folder. We must grant at least modify access to the the process under which the data service runs to access those folders. This same access must be granted in the database itself by creating a new login for this account, and this login must be granted access to the database. The SQL script included in the streaming sample project creates the necessary logins in the server and grants access to the database.

Accessing a Photo as a Binary Stream from the Photo Data Service

At this point, our photo data service is ready to return image files as a stream. We can access the PhotoInfo feed in a Web browser. Sending a GET request to the URI https://localhost/PhotoService/PhotoData.svc/PhotoInfo returns the following feed (with feed-reading disabled in the browser):

Note that the PhotoInfo(1) entry has a Content element of type image/jpeg and an edit-media link, both of which reference the relative URI of the media resource (PhotoInfo(1)/$value).

When we browse to this URI, the data service returns the related MR file as a stream, which is displayed as a JPEG image in the Web browser, as follows:

In the next post in the series, we will show how to use the WCF Data Services client to not only access media resources as streams, but to also create and update image files by generating POST and PUT requests against this same data service.

Glenn Gailey
Senior Programming Writer
WCF Data Services

Comments

Anonymous
August 31, 2010
Can the media be stored in Azure storage as a BLOB?
Anonymous
September 09, 2010
Yes, the IDataServiceStreamProvider implementation is storage independent. We will have an Azure BLOB storage example later on in the series (but it may be a while before we get to it).
Anonymous
January 13, 2011
In the last section, when I browse to http://localhost/PhotoService/PhotoData.svc/PhotoInfo I get an internal server error (500). The service is running as the same URL (minus /PhotoInfo) returns an XML schema, as does /$metadata. The database created fine and I added permissions to the NETWORK SERVICE to the project directory. Web.config is also configured correct. Any tips what I am missing? Thanks!
Anonymous
January 13, 2011
Checking the application event logs, showed a sql connection error. I found that by default, the IIS ASP.NET Application pool Identity is set to ApplicationPoolIdentity by default (under Advanced Settings). Changing it to NetworkService and restarting IIS solved this.
Anonymous
April 21, 2011
I downloaded your code, I believe it has some bugs. You set the SQL access connection string using Windows Authentication, but you let the web application to run in the IIS server, but not the Cassini. Obviously the connection string does not work! After I resolved it, I tried upload a file to save, but I get a run time error, and so on .... I think the readers will appreciate if you can simplify you demo code and make it easily runnable in the readers' dev environment ....
Anonymous
April 22, 2011
How I can implement Basic Custom authentication with Stream Provider.
Anonymous
May 21, 2011
Can you explain in what circumstances it is better to provide access to binary data via resource streams, rather than Binary fields? For example, I have to very quickly add photo uploading to a data service - can I store the photo data as a Binary property of an entity until I get the time to implement a Resource Stream? So far this seems to work for very small images, but fails with large ones - are there inherent limits to the size of data that can be stored in a Binary property? Another question - is is possible to implement the stream provider in such a way that the binary resources are still in the database (in some kind of dedicated table) and then return them as a stream - meaning that database backups etc etc also affect that binary resources and we don't have to keep track of two data stores? I think the streaming providers are a superb feature - are we likely to see any kind of integration of the 'HasStream' attribute into the editor for the EMDX file? Thanks for the article, it looks like this feature is crucial to me now as I develop ODATA services that integrate with Windows Phone 7.
Anonymous
February 15, 2012
I get a HTTP 500 error accessing URI .../PhotoInfo(1)/$value in the browser. And the client app does not display any picture and does not throw an exception. I did unblock each picture in its property page and NETWORK SERVICE has full access rights to the directories. How do I see the account under which IIS is running? Or what else could be the reason for the failure?
Anonymous
June 13, 2012
To get this to work I had to:

Go into IIS Manager
Click on Application Pools
Click on the ASP.NET v4.0
Click on Advanced Settings...
Change Identity from "ApplicationPoolIdentity" to NetworkService

Anonymous
October 21, 2012
Very nice example... Except that, if i work with DbContext instead of ObjectContext i cannot understand how to get entity in Dispose() ? If i try to use DbEntityEntry entry = context.Entry(cachedEntity); instead of ObjectStateEntry entry = this.context.ObjectStateManager.GetObjectStateEntry(cachedEntity); i got exception DbContext is disposed. So when i work with DbContext what is the way to get new entity from the Entity Framework object state manager? I triead almost any way that i know, but without any success/
Anonymous
October 23, 2012
to: Nedelcho Hello. I've got similar question once. DbContext is a wrapper around ObjectContext. So you can just have a link on ObjectContext in you StreamProvider, that link could be retrieved as objectContext=((IObjectContextAdapter)dbContext).ObjectContext. When Dispose() is called objectContext would not be disposed, so you can use exactly the code shown in post.
Anonymous
October 23, 2012
"6. The data service returns a response that includes the newly created MLE, including any server-generated values such as identities..." What could be the reason of response not including server-generated values for MLE? I receive empty ID fields in MLE, that I expect to be filled. When I add simple entity to context and make SaveChanges() I get all the things in place. MLE however is saved to DB so it has all needed values set, moreover response Identity include correct MLE id.

Share via