Audio models

Step 1: Deploy the audio model (self-hosted)

Deploy the audio model inside your cluster (e.g., Voxtral Small via vLLM) and expose it via a Service.

1. Deploy Voxtral Small using the official vLLM OpenAI-compatible image.

   kubectl apply -f- <<EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: voxtral-vllm
     namespace: agentgateway-system
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: voxtral-vllm
     template:
       metadata:
         labels:
           app: voxtral-vllm
       spec:
         containers:
         - name: voxtral-vllm
           image: vllm/vllm-openai:latest
           args:
           - "vllm"
           - "serve"
           - "mistralai/Voxtral-Small-24B-2507"
           - "--tokenizer_mode"
           - "mistral"
           - "--config_format"
           - "mistral"
           - "--load_format"
           - "mistral"
           - "--tool-call-parser"
           - "mistral"
           - "--enable-auto-tool-choice"
           ports:
           - containerPort: 8000
             name: http
           resources:
             limits:
               nvidia.com/gpu: "1"
             requests:
               nvidia.com/gpu: "1"
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: voxtral-service
     namespace: agentgateway-system
   spec:
     selector:
       app: voxtral-vllm
     ports:
     - port: 80
       targetPort: 8000
       protocol: TCP
   EOF

   The deployment uses:

   • vllm/vllm-openai:latest — the official vLLM OpenAI-compatible server image
   • vllm serve mistralai/Voxtral-Small-24B-2507 with Mistral-specific configuration
   • Port 8000 internally (vLLM default) exposed as port 80 on the Service
   • NVIDIA GPU reservation (1 GPU)

2. Wait for the pod to be ready.

   kubectl wait --for=condition=ready pod \
     -l app=voxtral-vllm \
     -n agentgateway-system \
     --timeout=300s

Step 2: Verify the model is responding

Verify that the model is responding by listing the models that it serves. The voxtral-service Service is a ClusterIP Service, so send the request from a pod inside the cluster.

kubectl run curl-audio --rm -i --restart=Never \
  -n agentgateway-system \
  --image=curlimages/curl -- \
  curl -s http://voxtral-service.agentgateway-system.svc.cluster.local:80/v1/models

In the output, you see a JSON response listing the model.

Step 3: Enable LLM serving on a listener

Add the AgentgatewayModel route kind to the http listener on the agentgateway-proxy Gateway, and verify that the listener supports it. Follow the steps in Enable LLM serving on a listener. The gateway then serves both LLM models and ordinary HTTPRoute traffic on the same port. The model in the next step references the http listener name in parentRefs.sectionName.

Step 4: Create the AgentgatewayModel

Create an AgentgatewayModel that serves the Voxtral model on the http listener.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: voxtral-small
  namespace: agentgateway-system
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  match:
    model: mistralai/Voxtral-Small-24B-2507
  provider: OpenAI
  baseURL: http://voxtral-service.agentgateway-system.svc.cluster.local/v1
EOF

Example output:

agentgatewaymodel.agentgateway.dev/voxtral-small created

Review the following table to understand this configuration. For more information, see the API docs.

Step 5: Send audio transcription requests

1. Create a sample WAV file to transcribe. If you already have an audio file, use its path in the following requests instead.

   mkdir -p testdata
   python3 -c "
   import wave
   w = wave.open('testdata/sample-audio.wav', 'wb')
   w.setnchannels(1); w.setsampwidth(2); w.setframerate(16000)
   w.writeframes(b'\x00\x00' * 16000)
   w.close()
   "

2. Save the gateway address in an environment variable.

   Cloud Provider LoadBalancerPort-forward for local testing

   export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
   echo $INGRESS_GW_ADDRESS

   In one terminal, start a port-forward to the gateway.

   kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:80

   In a second terminal, point the requests in this guide at the forwarded port.

   export INGRESS_GW_ADDRESS=localhost:8080

3. Send a transcription request through the gateway.

   curl http://$INGRESS_GW_ADDRESS/v1/audio/transcriptions \
     -F model=mistralai/Voxtral-Small-24B-2507 \
     -F file=@./testdata/sample-audio.wav | jq

   Example output. The sample file is one second of silence, so the text is empty. vLLM reports the audio duration in usage.seconds.

   {
     "text": "",
     "usage": {
       "type": "duration",
       "seconds": 1
     }
   }

4. Check the proxy access log. The entry records the transcription operation and the audio duration. The entry includes a cost only if your model catalog has a perAudioMinute rate for the model, as described in Usage and cost.

   kubectl logs deploy/agentgateway-proxy -n agentgateway-system | grep audio/transcriptions | tail -1

   Example output:

   info	request gateway=agentgateway-system/agentgateway-proxy listener=http ... http.method=POST http.path=/v1/audio/transcriptions http.status=200 protocol=llm gen_ai.operation.name=transcription gen_ai.provider.name=openai gen_ai.request.model=mistralai/Voxtral-Small-24B-2507 gen_ai.usage.audio_seconds=1 ...