Deploying Hop

Introduction

Hop is a powerful open-source event streaming platform designed for building real-time, event-driven applications. It provides a lightweight, high-performance messaging system that enables developers to handle real-time data streams, process events, and build scalable microservices architectures. Hop supports multiple messaging patterns including publish/subscribe, request/reply, and queue-based messaging, making it ideal for modern cloud-native applications.

With Hop, you can:

Build event-driven microservices with real-time communication
Process and stream data efficiently across distributed systems
Implement reliable message queuing and event sourcing patterns
Scale horizontally to handle high-throughput workloads
Maintain data consistency across your application architecture

This comprehensive guide will walk you through deploying Hop on Klutch.sh, including detailed installation instructions, a production-ready Dockerfile, persistent storage configuration, and best practices for running Hop in production.

Prerequisites

Before you begin, ensure you have the following:

A Klutch.sh account
A GitHub account with a repository for your Hop deployment
Basic understanding of Docker and containerization concepts
Familiarity with event streaming and messaging patterns
Git installed on your local machine

Getting Started: Installing Hop

Create a new directory for your Hop deployment:
Terminal window
```
mkdir my-hop-deployment
cd my-hop-deployment
git init
```

Create a basic Hop configuration file (hop.yaml):

server:
  host: 0.0.0.0
  port: 4222

logging:
  level: info

persistence:
  enabled: true
  dir: /data/hop

cluster:
  name: hop-cluster

Create a Dockerfile for your Hop deployment (see the detailed Dockerfile example in the next section).

Initialize your Git repository and push to GitHub:

git add .
git commit -m "Initial Hop deployment setup"
git remote add origin https://github.com/yourusername/my-hop-deployment.git
git push -u origin main

Creating a Production-Ready Dockerfile

Klutch.sh automatically detects a Dockerfile in your repository’s root directory. Here’s a comprehensive, production-ready Dockerfile for deploying Hop:

# Use Alpine Linux as base for minimal image size
FROM alpine:latest

# Install necessary dependencies
RUN apk add --no-cache ca-certificates tzdata

# Set working directory
WORKDIR /app

# Create directory for persistent data
RUN mkdir -p /data/hop

# Copy Hop binary (replace with actual Hop binary from your build process)
COPY hop /app/hop

# Copy configuration files
COPY hop.yaml /app/hop.yaml

# Set environment variables for configuration
ENV HOP_CONFIG=/app/hop.yaml
ENV HOP_DATA_DIR=/data/hop
ENV HOP_PORT=4222
ENV HOP_HTTP_PORT=8222
ENV HOP_LOG_LEVEL=info

# Expose client connection port (4222)
EXPOSE 4222

# Expose HTTP monitoring port (8222)
EXPOSE 8222

# Health check to ensure the server is running
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
  CMD nc -z localhost 4222 || exit 1

# Start Hop server with configuration
CMD ["/app/hop", "serve", "-c", "/app/hop.yaml"]

For a minimal Dockerfile without custom configuration:

FROM alpine:latest

# Install dependencies
RUN apk add --no-cache ca-certificates

# Create persistent data directory
RUN mkdir -p /data/hop

WORKDIR /app

# Copy Hop binary
COPY hop /app/hop

# Set environment variables
ENV HOP_PORT=4222
ENV HOP_HTTP_PORT=8222

# Expose ports
EXPOSE 4222
EXPOSE 8222

# Start server
CMD ["/app/hop", "serve", "--port", "4222"]

Dockerfile Best Practices:

Use Alpine-based images for smaller container sizes
Set explicit environment variables for configuration
Include health checks to monitor container status
Create necessary directories for persistent data
Expose all required ports for the application

Deploying Hop on Klutch.sh

Push your Hop deployment to GitHub:

Ensure your Dockerfile, configuration files, and any other necessary files are committed and pushed to your GitHub repository.
Terminal window
```
git add .
git commit -m "Add Hop Dockerfile and configuration"
git push origin main
```
Log in to Klutch.sh:

Navigate to klutch.sh/app and sign in to your account.
Create a new project:
- Go to Create Project
- Enter a descriptive name for your project (e.g., “Hop Event Platform”)
- Click “Create Project”
Create a new app:
- Navigate to Create App
- Select your GitHub repository containing the Hop deployment
- Choose the branch you want to deploy (e.g., main)
- Klutch.sh will automatically detect your Dockerfile
Configure your app settings:
- Traffic Type: Select TCP (Hop uses TCP protocol for messaging)
- Internal Port: Set to 4222 (Hop’s default client connection port)
- Region: Choose your preferred deployment region
- Compute Resources: Select appropriate CPU and memory based on your workload
- Instances: Set the number of instances (start with 1, scale as needed)
Configure environment variables (optional):

Add any necessary environment variables for your Hop deployment:
- HOP_LOG_LEVEL: Set logging level (e.g., info, debug, warn)
- HOP_CLUSTER_NAME: Set cluster name for multi-instance deployments
- HOP_MAX_PAYLOAD: Maximum message payload size
- HOP_MAX_CONNECTIONS: Maximum concurrent connections
Attach persistent storage:

Hop requires persistent storage for data durability and message persistence.
- In the app creation form, find the “Persistent Volumes” section
- Click “Add Volume”
- Mount Path: Enter /data/hop (where Hop stores persistent data)
- Size: Choose an appropriate size (start with 10 GB, scale as needed)
- Click “Add Volume” to confirm
Deploy your application:
- Review all your settings
- Click “Create” to start the deployment
- Klutch.sh will automatically build your Docker image from the Dockerfile
- Monitor the build logs to ensure successful deployment

Access your Hop instance:

Once deployed, you can connect to your Hop instance on port 8000. Your application will be accessible at a URL like:

example-app.klutch.sh:8000

For TCP connections in your application code:

// Node.js example
const { connect } = require('nats');

async function main() {
  const nc = await connect({
    servers: 'example-app.klutch.sh:8000'
  });
  console.log('Connected to Hop!');
}

Sample Hop Client Application

Here’s a complete example of how to connect to and use your deployed Hop instance:

Node.js Client

// Using a generic WebSocket or TCP client to connect to Hop
const net = require('net');

const client = net.createConnection({
  host: 'example-app.klutch.sh',
  port: 8000
}, () => {
  console.log('Connected to Hop on Klutch.sh!');

  // Send a message to Hop
  const message = JSON.stringify({
    topic: 'events.user.created',
    data: {
      userId: '12345',
      username: 'john_doe',
      email: 'john@example.com',
      timestamp: new Date().toISOString()
    }
  });

  client.write(message + '\n');
  console.log('Message sent!');
});

client.on('data', (data) => {
  console.log('Received:', data.toString());
});

client.on('end', () => {
  console.log('Disconnected from Hop');
});

client.on('error', (err) => {
  console.error('Connection error:', err);
});

Python Client

import socket
import json
import time

def main():
    # Connect to Hop on Klutch.sh
    client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    client.connect(('example-app.klutch.sh', 8000))

    print("Connected to Hop on Klutch.sh!")

    # Publish a message
    message = {
        "topic": "events.user.created",
        "data": {
            "userId": "12345",
            "username": "john_doe",
            "email": "john@example.com",
            "timestamp": "2024-01-01T00:00:00Z"
        }
    }

    client.send((json.dumps(message) + '\n').encode())
    print("Message sent!")

    # Receive response
    response = client.recv(1024)
    print(f"Received: {response.decode()}")

    time.sleep(60)
    client.close()

if __name__ == '__main__':
    main()

Go Client

package main

import (
    "bufio"
    "encoding/json"
    "fmt"
    "net"
    "time"
)

type Message struct {
    Topic string      `json:"topic"`
    Data  interface{} `json:"data"`
}

type UserEvent struct {
    UserID    string `json:"userId"`
    Username  string `json:"username"`
    Email     string `json:"email"`
    Timestamp string `json:"timestamp"`
}

func main() {
    // Connect to Hop on Klutch.sh
    conn, err := net.Dial("tcp", "example-app.klutch.sh:8000")
    if err != nil {
        panic(err)
    }
    defer conn.Close()

    fmt.Println("Connected to Hop on Klutch.sh!")

    // Create message
    event := UserEvent{
        UserID:    "12345",
        Username:  "john_doe",
        Email:     "john@example.com",
        Timestamp: time.Now().Format(time.RFC3339),
    }

    msg := Message{
        Topic: "events.user.created",
        Data:  event,
    }

    // Send message
    data, _ := json.Marshal(msg)
    fmt.Fprintf(conn, "%s\n", data)
    fmt.Println("Message sent!")

    // Read response
    reader := bufio.NewReader(conn)
    response, _ := reader.ReadString('\n')
    fmt.Printf("Received: %s\n", response)

    time.Sleep(60 * time.Second)
}

Customizing Your Hop Deployment

Environment Variables for Nixpacks

If you need to customize the build or start commands when not using a Dockerfile, you can set Nixpacks environment variables:

START_COMMAND: Custom start command for your application
```
START_COMMAND=/app/hop serve -c /app/custom-config.yaml
```
BUILD_COMMAND: Custom build command
```
BUILD_COMMAND=./scripts/setup.sh
```

Advanced Configuration

Create an advanced hop.yaml configuration file:

server:
  host: 0.0.0.0
  port: 4222

http:
  port: 8222

logging:
  level: info
  file: /data/hop/logs/hop.log

limits:
  max_connections: 10000
  max_payload: 1048576  # 1MB
  max_pending: 67108864  # 64MB

persistence:
  enabled: true
  dir: /data/hop/store

cluster:
  name: hop-production
  routes:
    - hop://hop-node-1:6222
    - hop://hop-node-2:6222

auth:
  token: ${HOP_AUTH_TOKEN}

tls:
  enabled: false

Update your Dockerfile to use this configuration:

FROM alpine:latest

RUN apk add --no-cache ca-certificates

WORKDIR /app

# Copy Hop binary and advanced configuration
COPY hop /app/hop
COPY hop.yaml /app/hop.yaml

# Create necessary directories
RUN mkdir -p /data/hop/logs /data/hop/store

ENV HOP_CONFIG=/app/hop.yaml

EXPOSE 4222 8222 6222

CMD ["/app/hop", "serve", "-c", "/app/hop.yaml"]

Monitoring and Health Checks

HTTP Monitoring Endpoint

Hop provides an HTTP monitoring endpoint on port 8222 (by default). You can access various metrics and health information:

Server Info: http://example-app.klutch.sh:8222/varz
Connection Info: http://example-app.klutch.sh:8222/connz
Subscription Info: http://example-app.klutch.sh:8222/subsz
Route Info: http://example-app.klutch.sh:8222/routez

Application Health Check Script

Create a health check script to monitor your Hop instance:

#!/bin/bash

HOP_URL="http://example-app.klutch.sh:8222"

# Check server health
curl -s "${HOP_URL}/varz" | jq '.uptime' > /dev/null

if [ $? -eq 0 ]; then
    echo "Hop is healthy"
    exit 0
else
    echo "Hop is unhealthy"
    exit 1
fi

Scaling Your Hop Deployment

Horizontal Scaling

To handle increased load, you can scale your Hop deployment horizontally:

Update your Hop configuration to enable clustering:
```
cluster:
  name: hop-production
  port: 6222
```
In your Klutch.sh app settings, increase the number of instances.
Configure your clients to connect to multiple Hop instances for high availability:

Configure your application to support multiple Hop instances for failover and load distribution.

Vertical Scaling

If you need more resources per instance:

Go to your app settings in Klutch.sh
Update the compute resources (CPU and memory)
Redeploy your application

Persistent Storage Best Practices

When configuring persistent volumes for Hop:

Data Directory: Mount at /data/hop/store for message persistence
Log Directory: Mount at /data/hop/logs for application logs
Initial Size: Start with 10 GB and monitor usage
Scaling: Increase volume size as your message throughput grows
Backups: Regularly backup your persistent volumes through Klutch.sh

Example volume configuration in your deployment:

Mount Path: /data/hop
Size: 20 GB
Purpose: Store persistent messages, logs, and state

Security Best Practices

Authentication and Authorization

Enable authentication in your Hop configuration:

auth:
  token: ${HOP_AUTH_TOKEN}
  users:
    - user: admin
      password: ${ADMIN_PASSWORD}
      permissions:
        publish: [">"]
        subscribe: [">"]
    - user: service
      password: ${SERVICE_PASSWORD}
      permissions:
        publish: ["events.>"]
        subscribe: ["events.>"]

Add these environment variables in your Klutch.sh app settings:

HOP_AUTH_TOKEN: A secure authentication token
ADMIN_PASSWORD: Admin user password
SERVICE_PASSWORD: Service account password

TLS/SSL Configuration

For production deployments, enable TLS:

tls:
  cert_file: /app/certs/server-cert.pem
  key_file: /app/certs/server-key.pem
  ca_file: /app/certs/ca.pem
  verify: true

Store your certificates in environment variables or mount them via persistent volumes.

Network Security

Restrict access to your Hop instance using Klutch.sh network policies
Use strong authentication tokens and passwords
Regularly rotate credentials
Monitor connection logs for suspicious activity

Troubleshooting Common Issues

Connection Issues

If clients cannot connect to Hop:

Verify the internal port is set to 4222
Ensure traffic type is set to TCP
Check that your application is using port 8000 for external connections
Review the deployment logs in Klutch.sh

Performance Issues

If you experience slow message processing:

Monitor CPU and memory usage in Klutch.sh
Check message queue sizes via the HTTP monitoring endpoint
Consider scaling horizontally (more instances)
Increase compute resources (vertical scaling)
Review and optimize your message payload sizes

Persistent Storage Issues

If data is not persisting across restarts:

Verify persistent volume is correctly mounted at /data/hop
Check volume permissions in the container
Ensure persistence is enabled in your configuration
Review storage logs for write errors

Production Deployment Checklist

Before going to production with Hop on Klutch.sh:

Resources

Deploying Hop on Klutch.sh provides you with a powerful, scalable event streaming platform that can handle real-time messaging for modern cloud-native applications. With automatic Dockerfile detection, persistent storage support, and horizontal scaling capabilities, Klutch.sh makes it easy to run production-grade Hop deployments with minimal operational overhead. Whether you’re building microservices, processing real-time events, or implementing event-driven architectures, Hop on Klutch.sh offers the reliability and performance your applications need.