Auto Scaling

Auto Scaling ensures that your endpoint can handle varying loads efficiently, adjusting the number of replicas between the min/max number set for the Endpoint. Auto Scaling is triggered based on the number of concurrent requests your endpoint is handling. As concurrency increases, additional replicas are provisioned (up to your configured maximum); as it decreases, replicas are removed (down to your configured minimum). Note that GPU utilisation is not currently used as a scaling triggerer.

info

Auto Scaling can take some time to provision/de-provision replicas.

Zero to Scale Policy

The Scale to Zero policy allows the Endpoint to go to zero replicas after a certain period with no incoming requests.