Langfuse

Verified : Code examples on this page have been automatically tested and verified.

Integrate agentgateway with Langfuse for LLM analytics and prompt management

Langfuse is an open-source LLM observability platform that provides prompt management, analytics, and evaluation.

Features

Prompt tracing - Log all prompts and responses.
Cost tracking - Monitor token usage and costs.
Latency analytics - Track response times.
Prompt management - Version and deploy prompts.
Evaluation - Score and evaluate outputs.
User tracking - Attribute usage to users.

Before you begin

Install agentgateway in your Kubernetes cluster.
Set up the OpenAI LLM provider and its /openai route. You can use another provider, but the example request in this guide uses the OpenAI route.
Set up the OpenTelemetry stack. This guide updates the opentelemetry-collector-traces collector from that guide. The tracing AgentgatewayPolicy from that guide sends the proxy’s traces to this collector.
Set up a Langfuse project.
- Langfuse Cloud: Sign up at langfuse.com and create a project.
- Self-hosted Langfuse: Langfuse needs PostgreSQL, ClickHouse, Redis, and S3-compatible storage. To deploy it in your cluster, follow the Langfuse Kubernetes (Helm) guide. Then, create a project in the Langfuse UI.

Get your Langfuse credentials

In your Langfuse project settings, create an API key pair. Copy the public key and the secret key.

Encode the key pair for HTTP Basic authentication.

export LANGFUSE_AUTH=$(echo -n "<public-key>:<secret-key>" | base64)

Set the Langfuse OTLP endpoint.
- Langfuse Cloud (EU): https://cloud.langfuse.com/api/public/otel
- Langfuse Cloud (US): https://us.cloud.langfuse.com/api/public/otel
- Self-hosted Langfuse: http://<langfuse-web-service>.<namespace>.svc.cluster.local:3000/api/public/otel
```
export LANGFUSE_OTEL_ENDPOINT=https://cloud.langfuse.com/api/public/otel
```

Configure the collector

Update the traces collector to also export traces to Langfuse. The configuration keeps the Tempo exporter from the OpenTelemetry stack guide, so traces still reach Tempo. The helm upgrade command replaces the collector’s whole configuration, so keep any other exporters that you added.

helm upgrade --install opentelemetry-collector-traces opentelemetry-collector \
  --repo https://open-telemetry.github.io/opentelemetry-helm-charts \
  --version 0.127.2 \
  --set mode=deployment \
  --set image.repository="otel/opentelemetry-collector-contrib" \
  --set command.name="otelcol-contrib" \
  --namespace=telemetry \
  --create-namespace \
  -f -<<EOF
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  exporters:
    otlp/tempo:
      endpoint: http://tempo.telemetry.svc.cluster.local:4317
      tls:
        insecure: true
    otlphttp/langfuse:
      endpoint: ${LANGFUSE_OTEL_ENDPOINT}
      headers:
        Authorization: "Basic ${LANGFUSE_AUTH}"
    debug:
      verbosity: detailed
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch]
        exporters: [debug, otlp/tempo, otlphttp/langfuse]
EOF

Wait for the collector to restart with the new configuration.

kubectl rollout status deployment/opentelemetry-collector-traces -n telemetry

Verify the integration

Send a request through agentgateway to the OpenAI route.

curl "$INGRESS_GW_ADDRESS/openai" -H content-type:application/json -d '{
  "model": "gpt-4o-mini",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

Check the collector logs for the agentgateway span. The debug exporter prints each span that the collector receives.
```
kubectl logs deploy/opentelemetry-collector-traces -n telemetry | grep -i "gen_ai"
```
In your Langfuse project, open Tracing and verify that the trace appears with the following information.
- Full prompt and response.
- Token counts (input and output).
- Model information.
- Latency metrics.

Learn more

Axiom LangSmith

Was this page helpful?