So it is sad to say... but through MS Support, it appears that this is not possible: HTTP/2 will not go till the end until either:
- The proxy in front of app services implements an pure http/2 clientship or an https-clientship (this is the proxy as the one in front of docker, nodejs or any other, this has the setting that brtrach-MSFT mentioned)
- NGinx implements, for http, where http2 is mostly not working, an upgrade from http11 to http2 (this trac ticket follows this; very old)
With the MS support, which were really friendly and helpful!, we then discussed the alternatives and it appears that the only way to operate a docker-with-nginx or any other server where http2 will go till the files and till the app-server, is to use VMs. I've indeed started to go to Linux VMs and can say that I see my nginx happily do http2 (and also certbot to be working well, while it becoming more difficult for app-services).
As usual: I lost the bells and whistles of all the tuning and security protections in favour of something I control and where I can attest of the performance ;-).
I'll now provide this answer as an answer. Ideally I should mark it as a non-solution!
Paul