Skip to main content

Get Started

Step 1: Create an endpoint​

  1. Select a Model: Choose from our list of popular, pre-trained models.

  2. Configure Compute Resources:

    • GPU Selection: Select the GPU type based on your performance and cost requirements. RAIC might provide a recommended GPU that ensures the optimal cost/performance for the selected model.
    • Location: Choose the geographical location where your endpoint will be hosted.

  3. Autoscaling (Optional):

    • Replicas: Define the minimum and maximum number of replicas for your endpoint to scale between.
    • Zero to Scale Policy: If the minimum replicas are set to Zero, your endpoint will scale down to zero replicas based on the defined policy.

Step 2: Accessing your Endpoints​

After creating, you are provided with an Authorization Token. When accessing inference endpoints, an authorization token is required to ensure secure access through the Authorization Header.

caution

The Authorization Token is only shown once upon creation — copy it somewhere safe. You can request a replacement via the Request Access Token option on the endpoint, but be aware that this action revokes the existing token and issues a new one; it does not add a second token alongside it, so calling an endpoint using the previous token will fail.

You'll be able to see a sample cURL command under the endpoint details.

This is an example for a Llama 3.1 8B model:

export ACCESS_TOKEN='<authorization-token>'
export ACCESS_URL='<endpoint-url>'

curl -H "Content-Type: application/json" \
-H "Authorization: Bearer $ACCESS_TOKEN" \
-d '{"model": "model","messages":[{"role": "system", "content": "You are an assistant that speaks in pirate speak."},{"role": "user", "content": "Write a poem about colors"}],"max_tokens": 50,"stream": false}' \
"$ACCESS_URL/openai/v1/chat/completions"

Contents