@Niklas Åsberg Thanks for reaching here! Could you make sure your route rules don't conflict with your selected authentication providers. Blocking a provider with a route rule prevents users from accepting invitations.
For more information on how to configure Microsoft Entra ID, see the App Service Authentication/Authorization documentation on using an existing registration.
To configure which accounts can sign in, see Modify the accounts supported by an application and Restrict your Microsoft Entra app to a set of users in a Microsoft Entra tenant.
Also suggest you to refer to the authentication and authorization section.