Best Practices for Routing Requests within Inference Clusters

Question

Hi,

I have a Kubernetes Service attached as an inference cluster to an azure machine learning workspace. I have deployed multiple models to that the AKS service, each with their own endpoints. I plan to configure this such that I just need to send the request to one main endpoint, which after applying some conditions, will redirect the request to one of the endpoints (e.g. redirect the request to the appropriate model). Are there any best practices to approach this problem?

There seems to be an Azure ML router using azureml-fe that does something similar, but I cannot find any documentation about it.

Thanks,
Lawrence

Answer 1

Hello @Lawrence Wong ,
We do have a solution for this in private preview (called Many Models solution accelerator).
Please send your email id to AzCommunity[at]microsoft[dot]com). Include title and link to this thread in the email (and reply here once you do for faster response) and we can take the conversation from there.