Cannot connect to a https backend while a load test is running

Question

I wrote a load test that in 300 threads calls a simple web service.
The situation is
test client at onprem using java 1.8 jax-ws -> public Internet -> Azure Standard Load Balancer -> backend Windows machine -> Apache Server listening on port 443 -> Apache mod AJP proxy -> Apache Tomcat.

The ws uses azure sql and the whole scenario simulates a big insert / update load to a table with size of more than 300 GB.

While the test is sucessfully running I cannot see a lot of ERORRS like 'Cannot connect' etc.
The jax-ws client uses Connection:Keep-Alive
At the LB Monitoring I can see approx 100 SYN count - so some keep alive connections were closed and new one were estabilished.
At the Vm side I can see 300 ESTABILISHED connections using netstat -ano. That correlates to number of load client test.

But If I want to download even a static page /index.html from onprem side, the connection is stucked and nothing is returned, unless I stopped the load test client.
I tried a browser or a curl command.

Why?
Where should I continue in a diagnostics?

The machine was alone, dedicated for the test - so no one else called it...

Answer 1

Hello Jan Vávra

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this!

Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

Issue: Cannot connect to a https backend while a load test is running.

Solution: Default Apache Server configuration bottleneck - winnt_mpn module has default setup of ThreadsPerChild as 64(changing value).

If you have any other questions or are still running into more issues, please let me know.

Thank you again for your time and patience throughout this issue.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

Answer 2

I turned out it was a default Apache Server configuration bottleneck - winnt_mpn module has default setup of ThreadsPerChild as 64. So Apache can handle at most 64 concurent connections.

After rising this value everything went well.