Connecting to Exchange using JAX-WS, part 1

Artykuł
07/29/2008

Yes, it is possible to Connect to Exchange using JAX-WS

The fear and trepidation I had when considering the effort to connect a Java application to Exchange Server 2007 was well founded. It wasn't easy. I'm going to spoil the ending and tell you, yes, it is possible to connect Java apps to Exchange Server via Web services, and you can do it securely and with good performance and reliability. But it took some starts and stops to make it happen.

Warning: Old Guy Reminiscing Ahead

Back a few years, I was a technical marketing guy for the Microsoft platform, and my main role Was talking to customers. One of the things I talked with customers about often, was interop. Pretty much every customer I talked to already had a signficant investment in information technology, they had a strategy that was in place and working, they had systems in place, working. They had off-the-shelf apps, they had some custom-built apps, and they had skillz. They were looking to make adjustments, to evolve their systems, to improve. It reminded me of the principles of Agile Development.

Do you remember XP? Do you remember how trendy that was, like 3 or 4 years ago. Everyone was talking about XP, everyone was looking to apply XP principles into their approaches. That seems a long time ago. XP is so 2004.

The approach that companies took with their information systems architecture was something like what developers did in a different domain with XP. Lots of little iterations, constant improvement, and steadily you get closer to your goal.

I remember talking to customers about web services technology, about using standard XML-based protocols to enable better connections between disparate information systems. Yes, Virginia, this was actually a theme of my conversations back only a few years ago. Now it seems so obvious. Now that everything speaks XML and interconnection is fairly easy, the wisdom of using standard protocols to gain efficiencies in IT is apparent. But only a short time ago this issue was not so clearly understood.

To make my case that web services and XML based protocols were the future, I'd show customers the investments that Microsoft was making into them. I'd tell them about all the standards was Microsoft working on. But the thing that customers wanted was hard evidence - not standards or papers. They wanted proof of XML usage in Microsoft products. So I'd talk about all the XML-based protocols that were already in the existing Microsoft systems they had: the web services libraries in the .NET Framework, the tooling in Visual Studio (but these were fairly obvious). The XML shredding capability that was introduced in SQL 2000 and improved in SQL 2005. Actually with the FOR XML clause you can do some pretty nifty things with the data that is already in your database. The MSXML component - now sort of just accepted as part of the landscape, a key, if almost forgotten enabler of AJAX - but 6 or 7 years ago it was undergoing rapid development.

These are all pretty good examples of Microsoft's investment in XML and WS-*. But one of the things that really surprised people was the web services capabilities we were introducing into Office (with WordML and ExcelML and so on), and the Exchange Server.

Exchange Server 2007 and Web services

Exchange Server 2003 had some limited web services capabilities. With Exchange Server 2007, Microsoft delivered an email and collab server that exposed all of its features through standard web services interfaces. Today if you have an Exchange Server, you can point your browser to https://my.exchange.server.com/ews/Exchange.asmx and get a WSDL for the server. Wow. That is pretty cool.

Standards are standards, and complying to a standard doesn't guarantee interop. If you are designing a web services interface, you can get better interop by keeping your WSDL fairly simple, and constraining your interface definitions to avoid some of the more esoteric nooks and crannies of the XML Schema spec. This is advice we've all been following for years. It works.

The problem with that advice is that it does not work if the information systems at the end of the wire are not simple, or if the requirements are not simple. For example, if you want WS-Addressing or delegated security, things can get complicated. And, similarly, Exchange Server is not simple. It stores lots of different types of things (email messages, contacts, tasks, notes, calendar entries, requests, resent items, recurrences, mime content type, cancellations, suggestions for new times, and so on). And then it compounds that complexity by adding in an unstructured aspect - document attachments and other unstructured data. Then there is stuff like impersonation, delegation (your admin assistant can update your calendar), out-of-office messages, and so on. It's pretty rich.

With all that capability, necessarily, the interface to Exchange is not going to be simple. Look at the user interface in Outlook - it is pretty rich. And the same is true with the network interface on the wire, described in the WSDL.

As a result, the WSDL for Exchange is complicated. It's big. It's got lots of nested structures. There are unions and restrictions and headers and bodies. It breaks all the rules for designing schema with interop in mind.

Anyway, who needs web services, right? We can use WebDAV!

Interop and the Exchange Server web services interface

First decision: which web services library to use? My first thought was to use the tried-and-true Apache AXIS 1.4 library that I had been depending on since 2006. It wasn't THAT old, and it worked for all the other interop tasks I had tried it on. It connected to .NET systems using ASMX or WCF. It was simple. I knew the tools.

My first step was to grab the Exchange.wsdl, the types.xsd and the message.xsd. I knew that I needed to add a service definition to the end of the Exchange.wsdl file, because most WSDL compilers want it:

    <wsdl:service name="ExchangeServices">
     <wsdl:port name="ExchangeServicePort" binding="tns:ExchangeServiceBinding">
       <soap:address location="https://myexchange.company.com/EWS/Exchange.asmx"/>
     </wsdl:port>
   </wsdl:service>

</wsdl:definitions>

But very early the work with AXIS 1.4 was hitting roadblocks. That library generated source code that did not compile. It choked on quite a few things. Other people have walked this path but custom modifications to the generated code did not sound like fun to me. I quickly decided that I'd need to move to something else.

My next idea was to move to AXIS2, which, coincidentally, is also at version 1.4. Now why they chose to call it "AXIS2 1.4" instead of "AXIS 2.4" is beyond me. There is a significant breakage in back-compatability with the move to AXIS to AXIS2, but still the version numbers are very confusing.

AXIS2 introduces a number of changes over AXIS (1). The AXIS guys improved the XML schema support, for one. Also, they made the XML serialization engine pluggable, so you can use the built-in simple library (called ADB), or you can switch to use JiBX or XMLBeans.

As with AXIS1, I ran the Exchange WSDL through the AXIS2 tools to generate the client-side proxies. This time, the code actually compiled. I was pleased to see that there was just one giant Java file generated for the entire Exchange interface. In AXIS 1.4, there was a whole series of files, 447 of em. Too many, thought I. But AXIS2 v1.4 generated just one file. Cool!

And it was a big one. Opening the file in emacs - emacs asked me, "are you sure you want to open this file? it's really big!" Seriously. And compiling that single file with javac, I actually got a heap overflow. Wow! That's a big file. I just used the -X heap magic incantations on that file and I was able to compile it.

I wrote up a simple proof of concept Java class that used the generated AXIS stub classes to connect to Exchange Server. When I ran the code, I got a failure: "required attribute localTraversal is null" I had no idea where that was leading me. I searched in the WSDL and schema for Exchange but couldn't find any elements or attributes by that name. I considered ripping open the AXIS2 source code to try to troubleshoot it. Wasn't real excited about that prospect. I had no idea where that was leading me.

My next step was to try the XMLbeans serialization option in AXIS 2. With this, I was back to generating ~ 400 Java files. But I didn't care, as long as the interop worked. And I had to update my client application code, because switching from ADB to XMLBeans necessitates a change in the programming model. I didn't care, though. Interop! (The things I do for interop!)

The AXIS2 + XMLbeans combo was looking promising, but also failed, this time with

 java.lang.ClassCastException: org.apache.xmlbeans.impl.values.XmlComplexContentImpl cannot be cast to com.microsoft.schemas.exchange.services._2006.types.Folder
IdType

This is apparently a common error, if there is a duplication in classes. I don't think I had that. I keep a clean workspace. Some of the XMLBeans people suggested on forums to upgrade to the later version of XMLBeans. There was no better suggestion I found. The AXIS2 library bundles XMLBeans 2.3.0, but XMLBeans standalone was at v2.4.0. So I downloaded 2.4.0, and tried to manually compile the schema with the scomp tool.

 c:\apache\xmlbeans-2.3.0\bin\scomp -out GeneratedTypes.jar contract\ExchangeServices.wsdl  -compiler c:\sunjdk\bin\javac.exe

But, then, another runtime exception:

 Exception in thread "main" java.lang.NoClassDefFoundError: com/microsoft/schemas/exchange/services/_2006/types/FolderIdType
        at ExchangeClient.Run(ExchangeClient.java:161)
        at ExchangeClient.main(ExchangeClient.java:217)
Caused by: java.lang.ClassNotFoundException: com.microsoft.schemas.exchange.services._2006.types.FolderIdType
        at java.net.URLClassLoader$1.run(Unknown Source)

It was one problem after another. Also, I had found the documentation and examples to be completely lacking for AXIS2. I couldn't find anyplace that gave me any hints. Why was it so hard to find information on this thing? And why did I have to use Class.Factory.Create(); instead of just calling new Class(). How dumb is this? And finally, there were 59 JAR files I needed to reference for AXIS2. That's what the doc told me. Commons-logging, xmlbeans, axis-api, axis-rt, corba, neethi, axiom, anogen, and on and on. I don't even know what those things are. The whole thing felt very... messy. I've already experienced JAR Hell and I don't want to go back there. I'm sure each one of those jars is required, but AXIS2 *could* be a little more seamless. As it is, it feels like a science project.

Ok, time to switch horses. The AXIS Stuff was not giving me joy. My next attempt would be to use JAXWS.

With JAXWS, there is one runtime JAR. One. The code-gen produces... a JAR. How refreshing! I don't need 400 class files. I can just reference a JAR. (Yes, I know there are 400 classes inside the jar. I don't care. It's a single jar). Hmm, someone ought to tell the AXIS project about this upstart JAXWS. The stub classes get generated and compile nicely. (small victory, cross fingers and hope for more). I am starting to like JAXWS.

The programming model exposed by JAXWS-generated client-side classes is a little different than what I was used to. Rather than using return values from a stub method for the response, JAXWS has this idea of using the javax.xml.ws.Holder class for return values. Kinda odd, but... whatever. Maybe there is a way to change the client-side programming model, but I wasn't going to worry about that right then.

There is some stuff in the Exchange interface (WSDL) having to do with impersonation and SerializedSecurityContext in each request on the EWS Service port, which is not useful. I'm using neither of those things, yet they are included in every Web services stub method. Passing null for these parameters does not satisfy, I get an error saying that I cannot set that to xsi:nil unless the schema has nillable='true'. I tried fiddling with the WSDL.. to set it to nillable in the right spots... That didn't work, so I just removed that stuff from the WSDL completely, from every wsdl:message in the WSDL file:

         <wsdl:message name="GetFolderSoapIn">
                <wsdl:part name="request" element="tns:GetFolder" />
<!-- dinoch - change for JAXWS
                <wsdl:part name="Impersonation" element="t:ExchangeImpersonation"/>
                <wsdl:part name="S2SAuth" element="t:SerializedSecurityContext"/>
                <wsdl:part name="MailboxCulture" element="t:MailboxCulture"/>
-->
                <wsdl:part name="RequestVersion" element="t:RequestServerVersion"/>
        </wsdl:message>

and also from every soap:operation:

                 <wsdl:operation name="GetItem">
                        <soap:operation soapAction="https://schemas.microsoft.com/exchange/services/2006/messages/GetItem" />
                        <wsdl:input>
<!--
                                <soap:header message="tns:GetItemSoapIn" part="Impersonation" use="literal"/>
                                <soap:header message="tns:GetItemSoapIn" part="S2SAuth" use="literal"/>
                                <soap:header message="tns:GetItemSoapIn" part="MailboxCulture" use="literal"/>
-->
                                <soap:header message="tns:GetItemSoapIn" part="RequestVersion" use="literal"/>
                                <soap:body parts="request" use="literal" />
                        </wsdl:input>
                        <wsdl:output>
                                <soap:body parts="GetItemResult" use="literal" />
                                <soap:header message="tns:GetItemSoapOut" part="ServerVersion" use="literal"/>
                        </wsdl:output>
                </wsdl:operation>
                <wsdl:operation name="CreateItem">

Ok, I was getting somewhere. With this change I was able to send a FindItem message over the wire to Exchange, and get message IDs back. Very cool. The approach I was taking was to use findItem to get the item IDs, and then use GetItem on the Ids to get the full content. The FindItem() was working flawlessly, but the GetItem() was not. I was not getting the results I expected. No exceptions, but my data was not coming back.

To troubleshoot this, I used Fiddler2 to see the HTTP messages back and forth between Java and Exchange. Because the communication was using SSL for encryption, Fiddler2 does not work. Some people have used Charles, a similar web debugging proxy. But I already had Fiddler, so... With the fiddler tool I could see exactly why the GetItem message was not giving me joy. (By the way, if you use Fiddler to decrypt SSL traffic, it's necessary to specify to the Java app that it should accept any SSL traffic - basically disable cert validation in the application. Only do this for testing purposes!!)

Correct flow from a .NET app to Exchange:

 <soap:Envelope xmlns:soap="https://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <GetItem xmlns="https://schemas.microsoft.com/exchange/services/2006/messages">
      <ItemShape>
        <BaseShape xmlns="https://schemas.microsoft.com/exchange/services/2006/types">Default</BaseShape>
        <AdditionalProperties xmlns="https://schemas.microsoft.com/exchange/services/2006/types">
          <FieldURI FieldURI="item:Subject" />
          <FieldURI FieldURI="item:Body" />
          <FieldURI FieldURI="item:Categories" />
          <ExtendedFieldURI PropertySetId="0006200E-0000-0000-C000-000000000046" PropertyId="35584" PropertyType="Integer" />
          <ExtendedFieldURI PropertyTag="0x10F3" PropertyType="String" />
          <ExtendedFieldURI PropertyTag="0x3008" PropertyType="SystemTime" />
        </AdditionalProperties>
      </ItemShape>
      <ItemIds>
        <ItemId Id="AAAZAERpbm8uQ2hpZXNhQG1pY3Jvc29mdC5jb20ARgAAAAAALBHixAbt0hGt6gCAX5/eGgcAd75VstDi0RGt3QCAX5/eGgAAAnopngAAEKx2OMNuVUuHEMvofVOxpwANFT4iewAA" ChangeKey="AAAAABQAAAAdMskaJAAzTpK1s6mN4vCnAAhyCw==" xmlns="https://schemas.microsoft.com/exchange/services/2006/types" />

Broken flow from a JAXWS app to Exchange:

 <S:Envelope xmlns:S="https://schemas.xmlsoap.org/soap/envelope/">
  <S:Header>
    <ns2:RequestServerVersion xmlns="https://schemas.microsoft.com/exchange/services/2006/messages" xmlns:ns2="https://schemas.microsoft.com/exchange/services/2006/types" Version="Exchange2007_SP1" />
  </S:Header>
  <S:Body>
    <GetItem xmlns="https://schemas.microsoft.com/exchange/services/2006/messages" xmlns:ns2="https://schemas.microsoft.com/exchange/services/2006/types">
      <ItemShape>
        <ns2:BaseShape>Default</ns2:BaseShape>
        <ns2:AdditionalProperties>
          <ns2:Path xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:PathToUnindexedFieldType" FieldURI="item:Subject" />
          <ns2:Path xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:PathToUnindexedFieldType" FieldURI="item:Body" />
          <ns2:Path xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:PathToUnindexedFieldType" FieldURI="item:Categories" />
        </ns2:AdditionalProperties>
      </ItemShape>
      <ItemIds>
        <ns2:ItemId ChangeKey="AAAAABQAAAAdMskaJAAzTpK1s6mN4vCnAAhyCw==" Id="AAMkAGUwNmJhYWVjLTI1ZTgtMTFkMy05MDc5LTAwODA1ZjMxZjgyNgBGAAAAAAAsEeLEBu3SEa3qAIBfn94aBwB3vlWy0OLREa3dAIBfn94aAAACeimeAAAQrHY4w25VS4cQy+h9U7GnAA0VPiJ7AAA=" />

Exchange wasn't sending me the data I expected, because the JAXWS request message was not what Exchange expected it to be. With this understanding, I looked into the contract. For the element in question, the schema used a ref="type" structure, and this was the only use of the ref in the entire contract. Clearly causing a problem; JAXWS was formatting it strangely. I modified it to eliminate the ref, just as a test.

   <!-- Arrays of paths -->
  <xs:complexType name="NonEmptyArrayOfPathsToElementType">
    <!-- using choice here so that proxy generator will not flatten the array and thus lose the
                item element names -->
    <xs:choice maxOccurs ="unbounded">
<!-- dinoch - change for jaxws -->
  <xs:element name="Path" type="t:BasePathToElementType"/>
  <xs:element name="FieldURI" type="t:PathToUnindexedFieldType" />
  <xs:element name="IndexedFieldURI" type="t:PathToIndexedFieldType" />
  <xs:element name="ExtendedFieldURI" type="t:PathToExtendedFieldType"/>

<!--
      <xs:element ref="t:Path"/>
-->

    </xs:choice>
  </xs:complexType>

After re-generating the client-side stubs, I was able to get the GetItem message to work nicely. Here's the message from JAXWS after the change in schema:

 <S:Envelope xmlns:S="https://schemas.xmlsoap.org/soap/envelope/">
  <S:Header>
    <ns2:RequestServerVersion xmlns="https://schemas.microsoft.com/exchange/services/2006/messages" xmlns:ns2="https://schemas.microsoft.com/exchange/services/2006/types" Version="Exchange2007_SP1" />
  </S:Header>
  <S:Body>
    <GetItem xmlns="https://schemas.microsoft.com/exchange/services/2006/messages" xmlns:ns2="https://schemas.microsoft.com/exchange/services/2006/types">
      <ItemShape>
        <ns2:BaseShape>Default</ns2:BaseShape>
        <ns2:AdditionalProperties>
          <ns2:FieldURI FieldURI="item:Subject" />
          <ns2:FieldURI FieldURI="item:Body" />
          <ns2:FieldURI FieldURI="item:Categories" />
        </ns2:AdditionalProperties>
      </ItemShape>
      <ItemIds>

After that, it was a matter of writing up the code to use the jaxws-generated stubs. It all works nicely now. It's pretty darn fast, too!

Let me summarize what I did to get Java to talk to Exchange:

Modified the exchange.wsdl to include a service definition, and to remove all occurrences of impersonation, S2SAuth, and MailboxCulture from each wsdl:message.
Modified the types.xsd to eliminate the ref="t:Path", and replace it with actual elements
Use Fiddler2 to trace and debug the SOAP messages.

The bottom line is that web services interop is possible even with complicated schema and interfaces, if you are willing to get your hands dirty on the schema, and if you choose the right tools.

In part 2 of this story, I'll show you some of the application code that uses the JAXWS-generated client classes, and I'll pack up the source code for everything too.

Great resources:

Cheers,
-Dino

Comments

Anonymous
July 31, 2008
I'm switching my preferred web services stack for Java, from Apache AXIS to JAX-WS.
Anonymous
November 19, 2008
Really Nice Post... EWS Rocks!!!
Anonymous
March 11, 2009
Just found this, as I was tasked to getting connectivity to EWS from Java for a project here. Article is nice, but I have some questions...

What happened to part 2?
How do you get past the NTLM auth security problems that JAXWS RI leaves you with? I know you can use an instance of 'Authenticator' to supply the credentials, but that is gloabl to the VM and won't work for a server type application that needs multiple, different connections to EWS. To solve 2 for myself, I actually went back to Axis2 (because you can use the Apache Commons HttpClient code), but instead of using one of the regular XML backends, I used JAXB. The end result is code that is almost identical to what is generated by JAXWS RI (all the types, classes, etc. are identical, just the SOAP wrappings are different). Doing that, I didn't have to muck with the underlying schemas, other than removing 1 refernce to an 'xml:lang' type on an attribute in types.xsd. Still looking forward to your part 2 :)

Anonymous
May 18, 2009
I'm having many of the issues others are having both here and elsewhere on the net. I've very, very close to getting JAX-WS working with EWS but with some issues. Is there any way that you could post the source code for the authentication and the example findItem call? This would be a life saver for many out there. Thanks, Jason
Anonymous
June 11, 2009
Such a good technical explaning with all details on mind...! Thanks.. I'll also try ews - java integration, will write you the result.

Udostępnij za pośrednictwem