Bizarre apparent throttling of the computer vision image analysis API.

Question

I've been observing very strange behavior from the Computer Vision API related to the latency of my requests. I'm using the standard paid pricing plan and just setup a basic resource for OCR purposes. I'm going nowhere near the 10 calls per second for the API and I'm not getting any 429 responses.

Every request is successful, however some requests receive an additional 7.5 seconds delay to complete. This happens even for unsuccessful requests, as if some API Gateway is silently throttling my requests without notifying me. There is nothing related to throttling or retrying in the response headers. The delayed ones look identical to the fast ones. The normal, fast requests all happen for the same image in about 0.3 seconds, but every single time there's a delay it is precisely 7.5 seconds slower.

The most frustrating part of all of this, is that the Azure Vision Demo that is provided for testing purposes, hosted by azure on the Vision Studio, does not have any such delays. I even opened chrome dev tools to copy the exact request being sent and used it to send an exact cURL request, but still, the delays persist. And they are random - no pattern can be observed. As if there's a 20% chance at any given time for a request to be throttled/delayed, by precisely 7.5 seconds every time.

I'm attaching a screenshot of observed response times:

You can see how 2 of those 8 requests are delayed. The response is a successful API call, with perfect OCR json data, but I need the latency to be constant and low for my time-sensitive application.

Please someone help me with this issue. I am currently using GCP for OCR, but I want to use Azure's product. If this issue is resolved, I'm switching instantly.

Answer 1

Hi @stringie,

I'm glad to hear that you were able to resolve the issue by using a requests.Session() with keep-alive enabled. I apologize for the lack of this information in the official documentation; we appreciate your feedback and will make sure to pass it along for future improvements.

Thanks for sharing the information, which might be beneficial to other community members reading this thread as solution. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", so I'll reiterate your response to an answer in case you'd like to accept the answer. This will help other users who may have a similar query find the solution more easily.

Query: Bizarre apparent throttling of the computer vision image analysis API.

Solution: I solved my own problem. Turns out that you need to use an HTTP Session with keep-alive enabled in order to not experience any delays. This is nowhere to be found in the Azure docs.

I was using Python's requests library directly to call the API, but what I had to do instead was build a requests.Session() object, which keeps the TCP connection alive for some time, while you send the subsequent requests using the session. Again, there is no mention of this in the docs for Image Analysis. They only show how to make a curl request, but they don't explain why your curl request my sometimes be 8 seconds long and sometimes 0.3 seconds long. The issue is the keep-alive connection.

Hopefully this will be useful to anyone who finds it, experiencing the same problem as me.

If you have any further questions or concerns, please don't hesitate to ask. We're always here to help.

---

Do click Accept Answer and Yes for was this answer helpful.