Azure GPT-4 model overloaded requests error

[blrchen]

When invoking Azure GPT 4 models, you may encounter an error message stating That model is currently overloaded with other requests. You can retry your request, or contact us through an Azure support request. In such cases, the proxy should automatically retry.

[blrchen]

The proposed solution involves parsing event data and verifying whether an overloaded model event data is received. If such data is received, an internal retry will be initiated on the proxy side.

Here's an example of event data returned by the Azure OpenAI GPT model when the model is overloaded:
{
"error": {
"message": "That model is currently overloaded with other requests. You can retry your request, or contact us through an Azure support request at: https://go.microsoft.com/fwlink/?linkid=2213926 if the error persists. (Please include the request ID 2dcb57a0-803c-4ac1-986d-e3aee5fedd6e in your message.)",
"type": "server_error",
"param": null,
"code": null
}
}