@Jason C , You can get your custom written application working on legacy auth protocols like Kerberos and NTLM to work with Azure AD DS. Azure AD DS also supports LDAP and LDAPS. Ideally all the users who are present in the Azure AD, would be synced to the Azure AD DS's directory.
[Note: In case of hybrid scenario, the Password Hash Sync has to be enabled, so that password hash from on-prem can get synced to cloud and from there get synced to Azure AD DS]
Once you enable the Azure AD DS service in your tenant, a dedicated VNET gets created that is connected to the Domain Controllers that gets created in the backend. Now you can go ahead and create a VM and connect that VM to the same VNET and get that VM domain joined to the Azure AD DS domain. Once done with the domain join you can install the RSAT tools on that Windows VM and manage the functionalities of the Domain Controller. Also, you can domain join Webserver hosting your custom app on IIS and get them authenticated with Azure AD DS Domain Controllers.
You can get more information on Azure AD DS, by referring to the links below:
How Azure AD DS works?
Configure Secure LDAP on Azure AD DS?
Hope this helps.
If any more queries around this, please feel free to share the same so that we can help you further. If not and if the response has helped in answering your query, please do not forget to accept this response as answer which will help other members of the community who have similar queries .