Authenticated or encrypted MSMQ messages are rejected because of incorrect CSP name
There is a new hotfix released for MSMQ 3.0 on Windows XP to correct a problem where the certificate service provider name was not being written correctly to the message:
The updated files are version 5.1.0.1111 and dated 20th November, 2008.
Obviously, if you don't use external certificates for message authentication or encryption then there's no great rush to apply the hotfix.
I'd ignore references to "WCF" and "workgroup mode" in the symptoms - these seem to be just flavour text.