Calling SharePoint Lists Web Service Using WCF
WCF makes it easy to call the SharePoint web services using WCF. In this post, I’ll show how to call the Lists.asmx web service and show the few things you need to take into account.
Creating the Proxy
Just like if you were using ASMX web services, the place to start is to create a reference to the SharePoint Lists.asmx web service. This is done by appending “/_vti_bin/Lists.asmx” to the end of a site name. This is the first important thing to realize: the Lists.asmx web service is relative to a specific site. If I have a site collection with a top-level site at https://sharepoint, and a child site https://sharepoint/demo, you will run into issues if you use https://sharepoint/_vti_bin/Lists.asmx and try to retrieve data from a list on the child site.
Now that you know how to choose the appropriate URL, pop open a Visual Studio 2008 project and use the Add Service Reference dialog. Enter the URL into the Address box, and click Go. That will query the WSDL for the Lists.asmx service, the representation of which is shown below.
You’ve now created the proxy, but you’re not done yet. Next step… authentication.
Authentication
Once you’ve created the proxy, the next step is to configure the security to be able to call the service methods. By default, SharePoint uses NTLM for authentication. There’s not really a way for the WSDL to express the fact that the web application requires Windows Integrated Authentication using NTLM or Kerberos, so we need to fill in that detail. We can do this by altering the bindingConfiguration to indicate the transport uses NTLM authentication.
1: <?xml version="1.0" encoding="utf-8" ?>
2: <configuration>
3: <system.serviceModel>
4: <bindings>
5: <basicHttpBinding>
6: <binding name="ListsSoap">
7: <security mode="TransportCredentialOnly">
8: <transport clientCredentialType="Ntlm" />
9: </security>
10: </binding>
11: </basicHttpBinding>
12: </bindings>
13: <client>
14: <endpoint
15: address="https://sharepoint/sites/HSC/_vti_bin/lists.asmx"
16: binding="basicHttpBinding"
17: bindingConfiguration="ListsSoap"
18: contract="ServiceReference1.ListsSoap"
19: name="ListsSoap" />
20: </client>
21: </system.serviceModel>
22: </configuration>
The biggest point to note is represented on lines 7 and 8, where we specify the security mode (TransportCredentialOnly) and the type of credential (Ntlm). This tells WCF to use integrated authentication using NTLM.
Since we’ve specified NTLM as the security mechanism, we need to supply the correct credentials to WCF. To use the credentials of the user running the client application, the following works fine.
ServiceReference1.ListsSoapClient proxy = new ConsoleApplication4.ServiceReference1.ListsSoapClient();
proxy.ClientCredentials.Windows.ClientCredential = new NetworkCredential();
XmlElement lists = proxy.GetListCollection();
Now that we’ve successfully created a proxy and configured the security correctly, the next step is to start calling methods on the server... which leads to our next configuration item.
Returning Lots of Data
When you call the services for SharePoint, they can return a very large amount of data. This amount of data can represent the schema for a list that contains many columns, a list with many columns that contains a lot of data, or any combination. There are 2 things you need to configure. The first is the potential size of the entire message. WCF throttles this way down to help protect you against denial of service attacks caused by enormous payloads. You can increase the maxReceivedMessageSize to some number. I have it exorbitantly high here only for illustration… the recommendation is to throttle it to some sensible number. Similarly, the data returned from the ASMX service can contain really long attribute names, causing WCF to reject the message because it exceeds the default policy. Again, I have the readerQuotas set ridiculously high for illustration, you should set it to something more reasonable. Change the app.config to look like this:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
</configSections>
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="ListsSoap" maxReceivedMessageSize="9999999">
<readerQuotas maxBytesPerRead="9999999" />
<security mode="TransportCredentialOnly">
<transport clientCredentialType="Ntlm" />
</security>
</binding>
</basicHttpBinding>
</bindings>
<client>
<endpoint
address="https://sharepoint/sites/HSC/_vti_bin/lists.asmx"
binding="basicHttpBinding"
bindingConfiguration="ListsSoap"
contract="ServiceReference1.ListsSoap"
name="ListsSoap" />
</client>
</system.serviceModel>
</configuration>
OK, we’ve configured for security and potentially huge messages. Now, let’s look at the data that comes back from one of these services. The return formats for methods in the Lists service are documented in the MSDN online SDK, but let’s take a look at one of them. Go to the Visual Studio 2008 Tools directory in your programs start menu, and look for the Visual Studio 2008 Command Prompt. Open that up and type “svcconfigeditor.exe”, and then open up the app.config file for your application. This is an incredibly handy utility to set properties of your WCF client or service. I find it especially handy to configure logging. Go to the Diagnostics node, and you’ll see a screen that looks like this.
Turn on “Log Auto Flush” and “Message Logging” as shown in the picture. Now, go to the child node in the treeview called “Message Logging”, we’ll configure WCF to log the entire message so we can have a peek at it.
You should be at a point where you can simply run the 3 lines of code that we wrote earlier in this post. Once it runs without errors, you should be able to see a new file called app_messages.svclog. Open that file using the utility “svctraceviewer.exe” (run the Visual Studio 2008 Command Prompt again). You should see 2 messages, the request and the response. The complete XML payload for mine looks like:
I couldn’t post the actual payload, because it’s pretty huge even for the few number of lists that we have here. Hence why we needed to increase the readerQuotas and maxReceivedMessageSize earlier in the post.
Working with the Returned XML Data
The returned data is XML, and some of it is downright unfriendly to work with (a lot of it uses the old ADODB.Recordset XML format). Second, the returned data is represented as an XmlElement (you all know my thoughts on this one, but it’s there nonetheless). So, you need to learn how to coerce the XML out of the XmlElement type into something useful.
The first way is to simply query it using XPath and iterate over it. Here’s where the XPathNavigator type comes in handy.
void DumpListCollection(ServiceReference1.ListsSoapClient proxy, System.IO.TextWriter writer)
{
XmlNode node = proxy.GetListCollection();
XPathNavigator nav = node.CreateNavigator();
XPathNodeIterator iter = nav.SelectDescendants("List", "https://schemas.microsoft.com/sharepoint/soap/", false);
while (iter.MoveNext())
{
string title = iter.Current.GetAttribute("Title", string.Empty);
string id = iter.Current.GetAttribute("ID", string.Empty);
writer.WriteLine("{0}\t\t{1}", title, id);
}
writer.Flush();
}
It’s simple, really. Create an XPathNavigator and choose the XPath statement that you want to work with. The XPathNavigator also exposes helpful methods (SelectChildren, SelectAncestors, SelectDescendants, etc) that make the XPath a little easier if you’re not one of those who well up with pride at convoluted XPath queries.
OK, that’s easy enough, just iterate over the nodes. Now, let’s look at the return data for one of those old ADODB.Recordset schema methods that I mentioned before, which is what the GetListItems method returns.
<?xml version="1.0" encoding="utf-8" ?>
<listitems xmlns="https://schemas.microsoft.com/sharepoint/soap/">
<rs:data ItemCount="1" xmlns:rs="urn:schemas-microsoft-com:rowset">
<z:row ows_ContentTypeId="0x010400FC18B450FF380C439C2CDDF2ED7A29F1"
ows_Title="Get Started with Windows SharePoint Services!"
ows_LinkTitleNoMenu="Get Started with Windows SharePoint Services!"
ows_LinkTitle="Get Started with Windows SharePoint Services!"
ows_Body="<div class=ExternalClass6C9D36A90B784F649E081ED819ED11F4>Microsoft Windows SharePoint Services helps you to be more effective by connecting people, information, and documents. For information on getting started, see Help.</div>"
ows_Expires="2009-03-07 06:47:57"
ows_ID="1"
ows_ContentType="Announcement"
ows_Modified="2009-03-07 06:47:57"
ows_Created="2009-03-07 06:47:57"
...
xmlns:z="#RowsetSchema" />
</rs:data>
</listitems>
I elided quite a bit for brevity’s sake here. When confronted with this XML that only it’s author could possibly love (and that is even questionable), how are you expected to work with this data? Turns out to be incredibly easy if you use a System.Data.DataTable that reads from an XmlNodeReader.
void DumpListItems(ServiceReference1.ListsSoapClient proxy, System.IO.TextWriter writer)
{
XmlDocument xmlDoc = new System.Xml.XmlDocument();
XmlElement ndQuery = xmlDoc.CreateElement("Query");
XmlElement ndViewFields = xmlDoc.CreateElement("ViewFields");
XmlElement ndQueryOptions = xmlDoc.CreateElement("QueryOptions");
XmlElement items = proxy.GetListItems("Announcements", null, ndQuery, ndViewFields, null, ndQueryOptions, null);
using (DataSet ds = new DataSet())
{
ds.ReadXml(new XmlNodeReader(items));
ds.WriteXml(writer);
}
}
This is cool because it makes data binding incredibly easy using the resulting DataSet. You could, for instance, bind the DataSet to a grid and display the data. You could also limit the number of fields that are returned to make the payload smaller and more relevant. See the example for the GetListItems method to understand the query, viewfields, and queryoptions parameters. It’s also helpful to see my example above because you don’t have to provide extensive CAML for the 3 parameters, you only need to pass in an XmlElement with the correct element name.
But Kirk, this is Old Skool XML stuff! We wanna see LINQ to XML!!!
I agree, LINQ to XML is hotness, and makes XML programming incredibly easy. However, getting to the XML data using LINQ to XML can be frustrating because there’s not an IEnumerable collection to iterate over. Instead of building integration into the existing System.Xml datatypes, the LINQ to XML folks left that as an exercise to the reader. Suppose you want to query the returned data and stuff it into a generic list. Turns out to be pretty easy, you just have to get creative with the LINQ syntax.
public IEnumerable<SPList> GetLists(ServiceReference1.ListsSoapClient proxy)
{
XmlElement listNodes = proxy.GetListCollection();
var q = from c in listNodes.ChildNodes.Cast<XmlNode>()
select new SPList()
{
ListID = c.Attributes["ID"].Value,
Title = c.Attributes["Title"].Value
};
return q;
}
public class SPList
{
public string ListID { get; set; }
public string Title { get; set; }
public override string ToString()
{
return string.Format("{0} - {1}", Title, ListID);
}
}
In order to access the XmlNode type using LINQ, you can cast the XmlNodeList type, allowing you to query it! Pretty slick little trick, and enables you to use LINQ over the boring ol’ System.Xml namespace.
For More Information
Methods for the Lists service in SharePoint
Using LINQ to XML (ScottGu’s blog)
Super Easy Way to Add WCF to SharePoint 2007 (Sahil Malik has done a tremendous job fighting WCF and SharePoint and has written some great articles and code to help us mere mortals)
SharePoint and Web Services (one of the first articles I ever read on SharePoint’s web services, and still a great read on the topic)
Example for the GetListItems method (help you understand the CAML that *can* be sent, but don’t have to).
Comments
Anonymous
March 10, 2009
PingBack from http://www.clickandsolve.com/?p=20876Anonymous
March 16, 2009
Hey Kirk, Thanks for the linkage. I feel WCF makes SharePoint a tonne much more flexible and better. Would love to have a multi-beer conversation with you on this at some point. Regards, SahilAnonymous
April 08, 2009
Introduction Integrating external applications with SharePoint data and functionality is pretty easy,Anonymous
April 09, 2009
So this is a great blog post and it answers a lot of the questions I had. However, tried to apply this to custom list in my farm and I am having a hard time getting the linq statement to work. Is the syntax different when you are pull information from a particular list using GetListItems?Anonymous
April 28, 2009
Part 4 of the SharePoint for Developers screencast series has been posted to Channel9… this one focusing