Deploying SeaweedFS

Introduction

SeaweedFS is a fast, distributed file system optimized for billions of small files. Originally inspired by Facebook’s Haystack paper, SeaweedFS provides a simple and highly scalable key-to-file mapping system that can be used as a blob store, object store with S3 API, or even a distributed POSIX-compatible file system.

Built entirely in Go, SeaweedFS achieves O(1) disk seeks for file reads, making it extremely efficient for serving large numbers of files. The architecture separates the file metadata (handled by master servers) from the actual file storage (handled by volume servers), enabling horizontal scaling of both components independently.

Key highlights of SeaweedFS:

O(1) Disk Seek: Direct file-to-disk mapping for maximum read performance
S3 API Compatible: Drop-in replacement for S3-based applications
FUSE Mount: Mount as a POSIX file system on Linux
Erasure Coding: Reduce storage overhead while maintaining redundancy
Tiered Storage: Automatically move cold data to cheaper storage
Cross-DC Replication: Replicate data across data centers
File Versioning: Optional versioning for file changes
TTL Support: Automatic file expiration
Encryption: Server-side and client-side encryption options
100% Open Source: Licensed under Apache-2.0

This guide walks through deploying SeaweedFS on Klutch.sh using Docker, configuring the master and volume servers, and accessing storage via the S3 API.

Why Deploy SeaweedFS on Klutch.sh

Deploying SeaweedFS on Klutch.sh provides several advantages:

Simplified Deployment: Klutch.sh automatically builds your SeaweedFS configuration without complex cluster management.

Persistent Storage: Attach persistent volumes for reliable blob storage that survives restarts.

HTTPS by Default: Klutch.sh provides automatic SSL certificates for secure API access.

Scalable Resources: Allocate CPU and memory based on your storage workload.

S3 Compatibility: Use existing S3 tools and libraries with your self-hosted storage.

Prerequisites

Before deploying SeaweedFS on Klutch.sh, ensure you have:

A Klutch.sh account
A GitHub account with a repository for your SeaweedFS configuration
Basic familiarity with Docker and distributed storage concepts

Understanding SeaweedFS Architecture

SeaweedFS consists of several components:

Master Server: Manages volume allocation and cluster topology
Volume Server: Stores the actual file data
Filer: Optional metadata layer for directory structure
S3 API Gateway: S3-compatible interface

For a single-node deployment, all components can run in a single container.

Preparing Your Repository

Create a GitHub repository with your SeaweedFS configuration.

Repository Structure

seaweedfs-deploy/
├── Dockerfile
└── .dockerignore

Creating the Dockerfile

Create a Dockerfile for a combined master/volume/filer/S3 deployment:

FROM chrislusf/seaweedfs:latest

# Environment variables
ENV WEED_MASTER_VOLUME_GROWTH_COPY_1=1
ENV WEED_MASTER_VOLUME_GROWTH_COPY_2=0
ENV WEED_MASTER_VOLUME_GROWTH_COPY_3=0

# Create data directories
RUN mkdir -p /data/master /data/volume /data/filer

# Expose ports
# Master
EXPOSE 9333
# Volume
EXPOSE 8080
# Filer
EXPOSE 8888
# S3
EXPOSE 8333
# WebDAV
EXPOSE 7333

# Start all services
ENTRYPOINT ["weed"]
CMD ["server", "-master.port=9333", "-volume.port=8080", "-filer", "-filer.port=8888", "-s3", "-s3.port=8333", "-dir=/data/volume", "-master.dir=/data/master", "-volume.max=100"]

Environment Variables Reference

Variable	Required	Default	Description
`WEED_MASTER_VOLUME_GROWTH_COPY_1`	No	1	Number of volume copies with replication level 1
`WEED_S3_AUTH_ENABLED`	No	false	Enable S3 authentication
`WEED_S3_ACCESS_KEY`	No	-	S3 access key
`WEED_S3_SECRET_KEY`	No	-	S3 secret key

Deploying SeaweedFS on Klutch.sh

Push Your Repository to GitHub

Initialize and push your repository:

git init
git add Dockerfile .dockerignore
git commit -m "Initial SeaweedFS deployment configuration"
git remote add origin https://github.com/yourusername/seaweedfs-deploy.git
git push -u origin main

Create a New Project on Klutch.sh

Navigate to the Klutch.sh dashboard and create a new project with a descriptive name like “seaweedfs” or “object-storage”.

Create a New App

Within your project, create a new app. Connect your GitHub account and select your SeaweedFS repository.

Configure HTTP Traffic

In the deployment settings:

Select HTTP as the traffic type
Set the internal port to 8333 (S3 API) or 8888 (Filer API)

Set Environment Variables

For S3 authentication, add:

Variable	Value
`WEED_S3_AUTH_ENABLED`	`true`
`WEED_S3_ACCESS_KEY`	Your access key
`WEED_S3_SECRET_KEY`	Your secret key

Attach Persistent Volumes

Add the following volumes:

Mount Path	Recommended Size	Purpose
`/data`	100+ GB	Master metadata, volume data, and filer database

Deploy Your Application

Click Deploy to start the build process.

Access SeaweedFS

Once deployment completes, access the S3 API at https://your-app-name.klutch.sh.

Using SeaweedFS

S3 API Access

Configure your S3 client with:

Endpoint: https://your-app-name.klutch.sh
Access Key: Your configured access key
Secret Key: Your configured secret key
Region: Any (not enforced)

Creating Buckets

Using AWS CLI:

aws --endpoint-url=https://your-app-name.klutch.sh s3 mb s3://my-bucket

Uploading Files

aws --endpoint-url=https://your-app-name.klutch.sh s3 cp file.txt s3://my-bucket/

Filer API

The filer provides a REST API for file operations:

POST /path/to/file - Upload a file
GET /path/to/file - Download a file
DELETE /path/to/file - Delete a file
GET /path/to/directory/?pretty=y - List directory

Troubleshooting

S3 Connection Refused

Symptoms: S3 clients cannot connect.

Solutions:

Verify the S3 port (8333) is configured correctly
Check that S3 authentication credentials match
Ensure the service is running

Files Not Persisting

Symptoms: Files disappear after restart.

Solutions:

Verify persistent volume is mounted at /data
Check volume has sufficient space
Review container logs for storage errors

Additional Resources

Conclusion

Deploying SeaweedFS on Klutch.sh gives you a high-performance, S3-compatible object storage system. Whether you need blob storage for application files or a scalable backend for millions of small objects, SeaweedFS provides the performance and flexibility to handle demanding storage workloads.