AD FS Relying Party Trust Error ID4037

Question

I am building a relying party trust between two separate ADFS servers (I know is not the best, but have some company to company SSO I need) and getting an error when trying to do an idp initiated signon using the Relying Party trust. The error message I get is below for the different error messages I get. I have imported the certificates into the trusted root / machine settings so know the trust is established. Also see the key data in the SAML exchange in the assertion so not sure what additional claims rules / settings need to be added.

Cannot find certificate to validate message/token signature obtained from claims provider. 
Claims provider: http://<my hosst>/adfs/services/trust 

This request failed. 

User Action 
Check that Claim Provider Trust configuration is up to date.

The Federation Service encountered an error while processing the SAML authentication request. 

Additional Data 
Exception details: 
Microsoft.IdentityModel.Protocols.XmlSignature.SignatureVerificationFailedException: ID4037: The key needed to verify the signature could not be resolved from the following security key identifier 'SecurityKeyIdentifier
    (
    IsReadOnly = False,
    Count = 1,
    Clause[0] = Microsoft.IdentityServer.Tokens.MSISSecurityKeyIdentifierClause
    )
'. Ensure that the SecurityTokenResolver is populated with the required key.
   at Microsoft.IdentityModel.Protocols.XmlSignature.EnvelopedSignatureReader.ResolveSigningCredentials()
   at Microsoft.IdentityModel.Protocols.XmlSignature.EnvelopedSignatureReader.OnEndOfRootElement()
   at Microsoft.IdentityModel.Protocols.XmlSignature.EnvelopedSignatureReader.Read()
   at System.Xml.XmlReader.ReadEndElement()
   at Microsoft.IdentityModel.Tokens.Saml2.Saml2SecurityTokenHandler.ReadAssertion(XmlReader reader)
   at Microsoft.IdentityModel.Tokens.Saml2.Saml2SecurityTokenHandler.ReadToken(XmlReader reader)
   at Microsoft.IdentityModel.Tokens.SecurityTokenHandlerCollection.ReadToken(XmlReader reader)
   at Microsoft.IdentityModel.Tokens.SecurityTokenElement.ReadSecurityToken(XmlElement securityTokenXml, SecurityTokenHandlerCollection securityTokenHandlers)
   at Microsoft.IdentityServer.Service.Tokens.SamlMessageSecurityTokenHandler.ReadToken(XmlReader reader)
   at Microsoft.IdentityModel.Tokens.SecurityTokenHandlerCollection.ReadToken(XmlReader reader)
   at Microsoft.IdentityModel.Tokens.SecurityTokenElement.ReadSecurityToken(XmlElement securityTokenXml, SecurityTokenHandlerCollection securityTokenHandlers)
   at Microsoft.IdentityServer.Web.Protocols.Saml.SamlProtocolManager.Issue(HttpSamlRequestMessage httpSamlRequestMessage, SecurityTokenElement onBehalfOf, String sessionState, String relayState, String& newSamlSession, String& samlpAuthenticationProvider, Boolean isUrlTranslationNeeded, WrappedHttpListenerContext context, Boolean isKmsiRequested)

Encountered error during federation passive request. 

Additional Data 

Protocol Name: 
Saml 

Relying Party: 
 

Exception details: 
Microsoft.IdentityModel.Protocols.XmlSignature.SignatureVerificationFailedException: ID4037: The key needed to verify the signature could not be resolved from the following security key identifier 'SecurityKeyIdentifier
    (
    IsReadOnly = False,
    Count = 1,
    Clause[0] = Microsoft.IdentityServer.Tokens.MSISSecurityKeyIdentifierClause
    )
'. Ensure that the SecurityTokenResolver is populated with the required key.
   at Microsoft.IdentityModel.Protocols.XmlSignature.EnvelopedSignatureReader.ResolveSigningCredentials()
   at Microsoft.IdentityModel.Protocols.XmlSignature.EnvelopedSignatureReader.OnEndOfRootElement()
   at Microsoft.IdentityModel.Protocols.XmlSignature.EnvelopedSignatureReader.Read()
   at System.Xml.XmlReader.ReadEndElement()
   at Microsoft.IdentityModel.Tokens.Saml2.Saml2SecurityTokenHandler.ReadAssertion(XmlReader reader)
   at Microsoft.IdentityModel.Tokens.Saml2.Saml2SecurityTokenHandler.ReadToken(XmlReader reader)
   at Microsoft.IdentityModel.Tokens.SecurityTokenHandlerCollection.ReadToken(XmlReader reader)
   at Microsoft.IdentityModel.Tokens.SecurityTokenElement.ReadSecurityToken(XmlElement securityTokenXml, SecurityTokenHandlerCollection securityTokenHandlers)
   at Microsoft.IdentityServer.Service.Tokens.SamlMessageSecurityTokenHandler.ReadToken(XmlReader reader)
   at Microsoft.IdentityModel.Tokens.SecurityTokenHandlerCollection.ReadToken(XmlReader reader)
   at Microsoft.IdentityModel.Tokens.SecurityTokenElement.ReadSecurityToken(XmlElement securityTokenXml, SecurityTokenHandlerCollection securityTokenHandlers)
   at Microsoft.IdentityServer.Web.Protocols.Saml.SamlProtocolManager.Issue(HttpSamlRequestMessage httpSamlRequestMessage, SecurityTokenElement onBehalfOf, String sessionState, String relayState, String& newSamlSession, String& samlpAuthenticationProvider, Boolean isUrlTranslationNeeded, WrappedHttpListenerContext context, Boolean isKmsiRequested)
   at Microsoft.IdentityServer.Web.Protocols.Saml.SamlProtocolHandler.RequestBearerToken(WrappedHttpListenerContext context, HttpSamlRequestMessage httpSamlRequest, SecurityTokenElement onBehalfOf, String relyingPartyIdentifier, Boolean isKmsiRequested, Boolean isApplicationProxyTokenRequired, String& samlpSessionState, String& samlpAuthenticationProvider)
   at Microsoft.IdentityServer.Web.Protocols.Saml.SamlProtocolHandler.BuildSignInResponseCoreWithSerializedToken(HttpSamlRequestMessage httpSamlRequest, WrappedHttpListenerContext context, String relyingPartyIdentifier, SecurityTokenElement signOnTokenElement, Boolean isKmsiRequested, Boolean isApplicationProxyTokenRequired)
   at Microsoft.IdentityServer.Web.Protocols.Saml.SamlProtocolHandler.SendSignInResponseForSecurityToken(GenericProtocolRequest originalRequest, SecurityTokenElement requestedTokenElement, ProtocolContext context)
   at Microsoft.IdentityServer.Web.PassiveProtocolListener.ProcessProtocolRequest(ProtocolContext protocolContext, PassiveProtocolHandler protocolHandler)
   at Microsoft.IdentityServer.Web.PassiveProtocolListener.OnGetContext(WrappedHttpListenerContext context)