Hello Avinash
Greetings!
I understand that you are facing issues with Azure Front Door (AFD) configuration.
You have an AFD, and the backend IIS is hosted in the VM.
Please check the front door origin group. In the Hostname field, add the VM public IP, and in the host header field, configure the backend host header name.
To enable HTTPS in the front door, configure a custom domain and add the certificate properly. Also, configure the same hostname and bundle the same certificate in the backend.
If you want to use a different hostname in the backend, add the backend host header in the AFD origin group properly and remove the certificate namecheck in the front door.
Check the below screen shot for more understanding:
Check the public document for more understanding:
And also check the below public document how to add custom domain in Front door:
You can also collect the tracking reference ID in the access logs when encountering the issue. Use this reference ID to check logs in your Azure portal. Please refer to the following document for collecting the logs:
https://learn.microsoft.com/en-us/azure/frontdoor/refstring?source=recommendations&tabs=edge
Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
If you have any other questions or are still running into more issues, let me know in the "comments" and I would be happy to help you.