Deploying Opik

Introduction

Opik is an open-source observability platform for Large Language Model (LLM) applications, developed by Comet ML. It provides comprehensive tracing, evaluation, and monitoring capabilities for AI applications, helping teams understand how their LLM-powered features perform in production.

As LLM applications become more complex with chains, agents, and multi-step workflows, understanding what happens inside these systems becomes crucial. Opik captures traces of LLM calls, tracks token usage, measures latency, and enables evaluation of response quality, providing the visibility needed to debug, optimize, and improve AI applications.

Key highlights of Opik:

• LLM Tracing: Capture full traces of LLM interactions including prompts, completions, and metadata
• Multi-Framework Support: Integrations with LangChain, LlamaIndex, OpenAI, and more
• Cost Tracking: Monitor token usage and estimate costs across providers
• Latency Analysis: Track response times and identify performance bottlenecks
• Evaluation Tools: Score and evaluate LLM outputs programmatically
• Prompt Management: Version and compare prompt templates
• Search and Filter: Query traces by content, metadata, or performance metrics
• Team Collaboration: Share insights and annotate traces with team members
• API Access: Programmatic access for custom integrations
• Self-Hosted: Complete control over your LLM observability data

This guide walks through deploying Opik on Klutch.sh using Docker, integrating with your LLM applications, and setting up monitoring for production use.

Why Deploy Opik on Klutch.sh

Deploying Opik on Klutch.sh provides several advantages:

Simplified Deployment: Klutch.sh automatically builds and deploys your LLM observability platform. Push to GitHub, and your service deploys automatically.

Persistent Storage: Attach persistent volumes for traces and database. Your LLM interaction history survives container restarts.

HTTPS by Default: Klutch.sh provides automatic SSL certificates for secure API access.

Data Privacy: Keep all LLM traces and prompts on infrastructure you control, essential for sensitive or proprietary data.

GitHub Integration: Store configuration in Git for version-controlled infrastructure.

Scalable Resources: Allocate resources based on trace volume and query complexity.

Custom Domains: Use your organization’s domain for professional observability URLs.

Prerequisites

Before deploying Opik on Klutch.sh, ensure you have:

• A Klutch.sh account
• A GitHub account with a repository for your Opik configuration
• Basic familiarity with Docker and containerization concepts
• LLM applications to instrument (OpenAI, LangChain, etc.)

Understanding Opik Architecture

Opik consists of several components:

API Server: Handles trace ingestion, querying, and evaluation operations.

Web Interface: Dashboard for viewing traces, analyzing performance, and managing projects.

Database: Stores traces, evaluations, and project configuration. Uses ClickHouse for efficient trace storage.

Object Storage: Stores large payloads and artifacts that exceed database limits.

Preparing Your Repository

Create a GitHub repository containing your Dockerfile and configuration.

Repository Structure

opik-deploy/
├── Dockerfile
├── docker-compose.yml
├── .dockerignore
└── README.md

Creating the Dockerfile

Create a Dockerfile for Opik:

FROM python:3.11-slim

WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    git \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Clone Opik repository
RUN git clone https://github.com/comet-ml/opik.git .

# Install Python dependencies
RUN pip install -r requirements.txt

# Environment configuration
ENV OPIK_PORT=5000
ENV OPIK_HOST=0.0.0.0

# Expose the application port
EXPOSE 5000

# Start the application
CMD ["python", "-m", "opik.server"]

Docker Compose Alternative

For a complete setup with dependencies:

version: '3.8'
services:
  opik:
    build: .
    ports:
      - "5000:5000"
    environment:
      - DATABASE_URL=postgresql://opik:opik@postgres:5432/opik
      - CLICKHOUSE_HOST=clickhouse
    depends_on:
      - postgres
      - clickhouse
    volumes:
      - opik-data:/app/data

  postgres:
    image: postgres:15
    environment:
      - POSTGRES_USER=opik
      - POSTGRES_PASSWORD=opik
      - POSTGRES_DB=opik
    volumes:
      - postgres-data:/var/lib/postgresql/data

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    volumes:
      - clickhouse-data:/var/lib/clickhouse

volumes:
  opik-data:
  postgres-data:
  clickhouse-data:

Environment Variables Reference

Variable	Required	Description
DATABASE_URL	Yes	PostgreSQL connection string
CLICKHOUSE_HOST	Yes	ClickHouse server hostname
CLICKHOUSE_PORT	No	ClickHouse port (default: 8123)
OPIK_PORT	No	API server port (default: 5000)
OPIK_HOST	No	API server bind address
SECRET_KEY	Yes	Application secret for sessions
OPIK_URL	No	Public URL for the application

Deploying Opik on Klutch.sh

Follow these steps to deploy your LLM observability platform:

Deploy Required Services

Opik requires backend services:

1. PostgreSQL: For user accounts and project metadata
2. ClickHouse: For efficient trace storage and querying

Deploy these services first and note connection details.

Push Your Repository to GitHub

Initialize and push your repository:

git init
git add Dockerfile docker-compose.yml .dockerignore README.md
git commit -m "Initial Opik configuration"
git remote add origin https://github.com/yourusername/opik-deploy.git
git push -u origin main

Create a New Project on Klutch.sh

Navigate to the Klutch.sh dashboard and create a new project named “opik” or “llm-observability”.

Create a New App

Within your project, create a new app. Connect your GitHub account and select the repository containing your Opik Dockerfile.

Configure HTTP Traffic

In the deployment settings:

• Select HTTP as the traffic type
• Set the internal port to 5000

Set Environment Variables

Configure your Opik instance:

Variable	Value
DATABASE_URL	Your PostgreSQL connection string
CLICKHOUSE_HOST	Your ClickHouse hostname
SECRET_KEY	Your generated secret key
OPIK_URL	https://your-app-name.klutch.sh

Attach Persistent Volumes

Add persistent storage:

Mount Path	Recommended Size	Purpose
/app/data	50 GB	Application data and cache

Deploy Your Application

Click Deploy to start the build process.

Access Opik

Once deployment completes, access Opik at https://your-app-name.klutch.sh. Create your account and first project.

Integrating with LLM Applications

Python SDK Installation

Install the Opik Python SDK:

pip install opik

OpenAI Integration

Track OpenAI API calls:

import opik
from opik.integrations.openai import track_openai
import openai

# Initialize Opik
opik.configure(
    api_key="your-opik-api-key",
    workspace="your-workspace",
    host="https://your-app-name.klutch.sh"
)

# Wrap OpenAI client
client = track_openai(openai.OpenAI())

# All calls are now traced
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello, world!"}]
)

LangChain Integration

Track LangChain applications:

import opik
from opik.integrations.langchain import OpikTracer

# Initialize tracer
tracer = OpikTracer(
    project_name="my-langchain-app"
)

# Use with LangChain
from langchain.chains import LLMChain
from langchain.llms import OpenAI

chain = LLMChain(llm=OpenAI(), callbacks=[tracer])
result = chain.run("What is the meaning of life?")

LlamaIndex Integration

Track LlamaIndex queries:

import opik
from opik.integrations.llama_index import OpikHandler

# Initialize handler
handler = OpikHandler(project_name="my-llamaindex-app")

# Configure LlamaIndex to use handler
from llama_index.core import Settings
Settings.callback_manager.add_handler(handler)

Manual Tracing

For custom implementations:

import opik

opik.configure(host="https://your-app-name.klutch.sh")

@opik.track
def my_llm_function(prompt):
    # Your LLM logic here
    response = call_llm(prompt)
    return response

# Function calls are automatically traced
result = my_llm_function("Hello!")

Using the Dashboard

Viewing Traces

The Opik dashboard provides:

1. Trace List: Browse all traced interactions
2. Trace Details: View full request/response pairs
3. Span Tree: Visualize multi-step workflows
4. Metadata: Access token counts, latency, and costs

Filtering Traces

Search and filter traces:

• By prompt content
• By model name
• By latency range
• By token count
• By custom metadata
• By time range

Evaluating Responses

Score and annotate traces:

1. Select traces to evaluate
2. Apply scoring criteria
3. Add annotations and feedback
4. Track quality over time

Evaluation and Testing

Programmatic Evaluation

Evaluate LLM outputs:

from opik import evaluate

# Define evaluation criteria
def relevance_score(trace):
    # Custom scoring logic
    return score

# Run evaluation
results = evaluate(
    project="my-project",
    evaluator=relevance_score,
    filter={"model": "gpt-4"}
)

Dataset Management

Create test datasets:

1. Collect representative examples
2. Define expected outputs
3. Run automated evaluations
4. Track performance over time

A/B Testing

Compare prompt versions:

1. Create prompt variants
2. Route traffic to variants
3. Compare performance metrics
4. Select winning variant

Performance Monitoring

Latency Tracking

Monitor response times:

• P50, P95, P99 latency percentiles
• Breakdown by model and provider
• Identify slow queries
• Track trends over time

Cost Analysis

Track LLM spending:

• Token usage by model
• Cost estimation by provider
• Budget alerts
• Cost optimization recommendations

Error Tracking

Monitor failures:

• API error rates
• Timeout frequency
• Rate limit hits
• Error categorization

Best Practices

Trace Organization

• Use meaningful project names
• Add relevant metadata to traces
• Tag traces for easy filtering
• Archive old projects

Prompt Versioning

• Version all prompt templates
• Document changes between versions
• Track performance by version
• Maintain prompt history

Privacy Considerations

• Redact PII from traces when necessary
• Control access to sensitive projects
• Configure data retention policies
• Review stored content regularly

Troubleshooting Common Issues

Traces Not Appearing

Symptoms: Instrumented calls not showing in dashboard.

Solutions:

• Verify API key and host configuration
• Check network connectivity to Opik server
• Review SDK logs for errors
• Confirm project name matches

High Latency

Symptoms: Dashboard queries are slow.

Solutions:

• Increase ClickHouse resources
• Optimize query filters
• Archive old traces
• Add database indexes

Missing Metadata

Symptoms: Traces lack expected information.

Solutions:

• Update SDK to latest version
• Configure additional metadata in SDK
• Check integration configuration
• Review trace logging settings

Additional Resources

• Opik Official Website
• Opik GitHub Repository
• Opik Documentation
• Klutch.sh Persistent Volumes
• Klutch.sh Deployments

Conclusion

Deploying Opik on Klutch.sh provides comprehensive observability for your LLM applications. With tracing, evaluation, and performance monitoring, Opik helps you understand and optimize your AI-powered features.

The combination of persistent storage for traces, reliable uptime, and HTTPS security makes Klutch.sh well-suited for hosting Opik. Whether debugging prompt issues, optimizing costs, or evaluating model performance, your self-hosted observability platform provides the insights you need.

Start by instrumenting your primary LLM applications, then expand coverage as you gain insights into your AI systems. With Opik on Klutch.sh, you own your LLM observability data and infrastructure.