Hi, @Alex Rough
From your description, can I understand that when you invite an external user in Azure, there are multiple identity records in the Exchange Admin Center? When you delete the user in Azure, there are still duplicate records in the Exchange Admin Center that cannot be deleted, and you also get the screenshot below?
Here are some steps you can try:
1.What is the error when you use the command Remove-MailUser? You can provide screenshots for me to reproduce the issue.
2.Try to remove the residual user from the M365 admin centre. Go to M365 admin portal->Guest users, select Residual Users, and delete them.
3.Double-check if there are duplicate proxy addresses and UPN.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".