Skip to content
Klutch.sh Docs

Deploying a Kafka App

Introduction

Apache Kafka is a distributed streaming platform for building real-time data pipelines and event streaming apps. Deploying Kafka with a Dockerfile on Klutch.sh gives you reproducible brokers, managed secrets, and durable storage for logs—all configured from klutch.sh/app. This guide covers installation, repository prep, a production-ready Dockerfile, deployment steps, Nixpacks overrides, and best practices for reliable messaging.

Prerequisites

  • A Klutch.sh account (create one)
  • A GitHub repository containing your Kafka configs (GitHub is the only supported git source)
  • Docker familiarity and basic Kafka administration
  • Storage for broker logs and snapshots

For setup basics, review the Quick Start.

Architecture and ports

  • Kafka brokers speak TCP on 9092; set the internal container port to 9092 and choose TCP traffic in Klutch.sh.
  • Use KRaft mode (no ZooKeeper) for simplicity; if you add ZooKeeper, deploy it as a separate TCP app on port 2181 (external 8000).
  • Persistent storage is required for /var/lib/kafka/data.

Repository layout

kafka/
├── config/                 # server.properties and Kraft configs
├── Dockerfile              # Must be at repo root for auto-detection
├── scripts/                # Helper scripts (optional)
└── README.md

Keep secrets (SASL credentials, TLS keys) in Klutch.sh environment variables, not in Git.

Installation (local) and starter commands

Quick local test with KRaft:

Terminal window
docker run --rm -p 9092:9092 \
  -e KAFKA_CFG_PROCESS_ROLES=broker,controller \
  -e KAFKA_CFG_NODE_ID=1 \
  -e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \
  -e KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 \
  -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \
  -e KAFKA_CFG_LOG_DIRS=/var/lib/kafka/data \
  -v $(pwd)/data:/var/lib/kafka/data \
  confluentinc/cp-kafka:7.5.3

Optional helper start.sh for portability and Nixpacks fallback:

#!/usr/bin/env bash
set -euo pipefail
exec /etc/confluent/docker/run

Make it executable with chmod +x start.sh.

Dockerfile for Kafka (production-ready, KRaft mode)

Place this Dockerfile at the repo root; Klutch.sh auto-detects it (no Docker selection in the UI):

FROM confluentinc/cp-kafka:7.5.3


WORKDIR /app


# Copy optional custom configs
COPY config /etc/kafka


EXPOSE 9092
CMD ["/etc/confluent/docker/run"]

Notes:

  • Pin the Kafka image tag for reproducible behavior.
  • If you use TLS/SASL, mount certs and set the corresponding env vars.

Environment variables (Klutch.sh)

Set these in the Klutch.sh app settings (Secrets tab) before deploying:

  • KAFKA_CFG_PROCESS_ROLES=broker,controller
  • KAFKA_CFG_NODE_ID=1
  • KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@0.0.0.0:9093
  • KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
  • KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://example-app.klutch.sh:8000
  • KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
  • KAFKA_CFG_LOG_DIRS=/var/lib/kafka/data
  • PORT=9092

If you deploy without the Dockerfile and need Nixpacks overrides:

  • NIXPACKS_BUILD_CMD="echo Kafka uses prebuilt image"
  • NIXPACKS_START_CMD=/etc/confluent/docker/run
  • NIXPACKS_JDK_VERSION=17

These keep Kafka compatible with Nixpacks defaults when a Dockerfile is absent.

Attach persistent volumes

In Klutch.sh storage settings, add mount paths and sizes (no names required):

  • /var/lib/kafka/data — required for broker logs.
  • /etc/kafka — optional if you edit configs at runtime.

Ensure these paths are writable inside the container.

Deploy Kafka on Klutch.sh (Dockerfile workflow)

  1. Push your repository (with the Dockerfile at the root) to GitHub.
  2. Open klutch.sh/app, create a project, and add an app.
  1. Connect the GitHub repository; Klutch.sh automatically detects the Dockerfile.
  2. Choose TCP traffic for Kafka.
  3. Set the internal port to 9092.
  4. Add the environment variables above (listeners, advertised listeners, log dirs, and any NIXPACKS_* overrides if you temporarily deploy without the Dockerfile).
  5. Attach persistent volumes for /var/lib/kafka/data (and /etc/kafka if used), selecting sizes that match your retention needs.
  6. Deploy. Your broker will be reachable at example-app.klutch.sh:8000 over TCP for clients.

Sample producer/consumer code

Producer (Node.js):

import { Kafka } from "kafkajs";


const kafka = new Kafka({ clientId: "demo-producer", brokers: ["example-app.klutch.sh:8000"] });
const producer = kafka.producer();
await producer.connect();
await producer.send({ topic: "demo-topic", messages: [{ value: "hello from Klutch.sh" }] });
await producer.disconnect();

Consumer (Python):

from kafka import KafkaConsumer


consumer = KafkaConsumer(
    "demo-topic",
    bootstrap_servers=["example-app.klutch.sh:8000"],
    auto_offset_reset="earliest",
    enable_auto_commit=True,
)
for msg in consumer:
    print(msg.value.decode("utf-8"))

Health checks and production tips

  • Monitor broker health via JMX or external monitors; scrape metrics with a sidecar if needed.
  • Enforce TLS/SASL for production; store creds as Klutch.sh secrets.
  • Keep image tags pinned and upgrade intentionally.
  • Monitor disk usage on /var/lib/kafka/data and resize before reaching retention limits.
  • If you need multi-broker clusters, repeat the deployment for each broker with unique NODE_ID and quorum settings.

Kafka on Klutch.sh combines reproducible Docker builds with managed secrets, persistent volumes for log storage, and flexible TCP routing. With the Dockerfile at the repo root and ports set to 9092 internally (8000 externally for clients), you can stream data reliably without extra YAML or workflow overhead.