Daily Wacky Environment Solution: One-Way trust AND ADRMS using ADFS

Consider this scenario.
Customer has two forests.

Resource Forest (RFCOM) - Houses resources like RMS and MOSS.
Account Forest (AFCOM) - House all of the user accounts.

Situation: Customers wants to have *one* RMS cluster housed in the resource forest. Has a one way trust where the resource forest trusts the account forest (for some other applications including MOSS auth (although I think MOSS will work with ADFS as well)). Wants to use ADFS to do SSO to ADRMS server to get content served by MOSS server in the resource forest.

Problem: In order to use ADFS with ADRMS there typically needs to be *no* trust between forests. The way that the RMS client works, is that it will try to obtain a RAC via the standard Windows Auth pipeline (https://rms.rfcom.com/_wmcs/certification/certification.asmx). If this fails, it will use the ADFS token pipeline (https://rms.rfcom.com/_wmcs/certificationexternal/certification.asmx), and pass the ADFS server specified in the FederationHomeRealm registry key. The RMS server *has to* reside in the same forest with MOSS since there is no ADFS pipeline for service accounts.

If there is a one way trust then the user will legitimately be able to get to the Windows Auth Pipeline through IIS. This is a problem, because the return code is 200 (which means OK), although the RMS server will reject the request from a user in another forest to this pipeline, because it expects to find the user in its own forests AD. This puts the RMS client in a bad state. It simply thinks the RMS server has rejected the request, and cries about it with an error message.

So we have a few options. 

Solution 1:  Remove the one-way trust. If you remove the one-way trust, then IIS will not authenticate the user to the Windows Auth certification pipeline, and the RMS client will automatically roll over to the ADFS tokenized pipeline.

Solution 2: (The one we used). We went to the properties of C:\Inetpub\wwwroot\_wmcs\certification\certification.asmx, and added an ACE to the security tab for the 'AFCOM\Domain Users' group, and gave it explicit *deny* permissions. This forces all clients in the AFCOM forest to rollover to the ADFS certification pipeline.

Solution 3: Setup a RMS cluster in the account forest, and then export the SLC from the AFCOM forests RMS server, into the RFCOM forests RMS server as a TUD (Trusted User Domain).

Anyways, option 2 worked for this customer. Any forests that he now adds to his forest that have a one way trust, he just needs to add a deny ace to the certification.asmx file for the 'domain users' group of that forest. We also added an SCP to the account forest with the cluster URL of the resource forest, so that all of the clients in the account forest would auto-discover the ADRMS service without needing registry overrides.

ADRMS is a pretty flexible product. Even though every single possible environment situation isn't documented, doesn't mean that with a little playing around you can't get just about any situation to work (within the support boundaries...and sometimes outside of them. <-- I didn't just say that. :)).

-Jason