Deploying Piler

Introduction

Piler is an enterprise-grade email archiving solution designed for organizations that need to store, search, and retain email communications for compliance, legal, or business purposes. The application captures emails from mail servers, indexes them for fast searching, and provides a web interface for users to access their archived messages.

Built with performance and scalability in mind, Piler uses Sphinx for full-text search, MariaDB for metadata storage, and a custom storage format for efficient email archiving. The application supports multiple email sources including IMAP import, SMTP relay, and journal capture.

Key highlights of Piler:

• Enterprise Archiving: Archive emails for compliance and retention
• Full-Text Search: Fast search across millions of emails using Sphinx
• Multiple Import Methods: IMAP, SMTP relay, journal capture
• Deduplication: Store duplicate emails only once
• Attachment Handling: Index and search attachments
• Retention Policies: Configure automated retention rules
• Audit Trail: Track access and searches for compliance
• LDAP/AD Integration: Sync users from directory services
• Multi-Tenant: Support multiple organizations
• Legal Hold: Prevent deletion for legal requirements
• Export Capabilities: Export emails in various formats
• 100% Open Source: GPL licensed

This guide walks through deploying Piler on Klutch.sh using Docker, configuring the database, and setting up the application for production use.

Why Deploy Piler on Klutch.sh

Deploying Piler on Klutch.sh provides several advantages for email archiving:

Simplified Deployment: Klutch.sh automatically detects your Dockerfile and builds Piler without complex orchestration. Push to GitHub and your archive deploys automatically.

Persistent Storage: Attach persistent volumes for your email archive and database. Your archived emails survive container restarts.

HTTPS by Default: Klutch.sh provides automatic SSL certificates, ensuring secure access to archived emails.

GitHub Integration: Connect your configuration repository directly from GitHub. Updates trigger automatic redeployments.

Scalable Resources: Allocate CPU and memory based on archive size and search requirements.

Environment Variable Management: Securely store database credentials through Klutch.sh’s environment variable system.

Custom Domains: Assign a custom domain to your Piler instance.

Always-On Availability: Your email archive remains accessible 24/7 without managing infrastructure.

Prerequisites

Before deploying Piler on Klutch.sh, ensure you have:

• A Klutch.sh account
• A GitHub account with a repository for your Piler configuration
• A MariaDB or MySQL database
• SMTP access for email import/capture
• Basic familiarity with Docker and email systems
• (Optional) LDAP/AD for user authentication
• (Optional) A custom domain for your Piler instance

Understanding Piler Architecture

Piler consists of multiple integrated components:

Piler Core: The main archiving engine handling email processing and storage.

Sphinx Search: Full-text search engine for fast email queries.

MariaDB/MySQL: Database for metadata, users, and configuration.

Web Interface: PHP-based interface for searching and retrieving emails.

Piler SMTP: SMTP server component for receiving journal emails.

Preparing Your Repository

To deploy Piler on Klutch.sh, create a GitHub repository containing your Dockerfile.

Repository Structure

piler-deploy/
├── Dockerfile
├── .dockerignore
└── README.md

Creating the Dockerfile

Create a Dockerfile in the root of your repository:

FROM sutojo/piler:latest

# Environment variables
ENV PILER_HOSTNAME=${PILER_HOSTNAME}
ENV MYSQL_HOSTNAME=${MYSQL_HOSTNAME}
ENV MYSQL_DATABASE=${MYSQL_DATABASE:-piler}
ENV MYSQL_USER=${MYSQL_USER}
ENV MYSQL_PASSWORD=${MYSQL_PASSWORD}
ENV MEMCACHED_HOST=${MEMCACHED_HOST}
ENV ADMIN_USER=${ADMIN_USER:-admin@local}
ENV ADMIN_PASSWORD=${ADMIN_PASSWORD}

# Create data directories
RUN mkdir -p /var/piler/store /var/piler/stat /var/piler/tmp

EXPOSE 80 25

# Health check
HEALTHCHECK --interval=60s --timeout=10s --start-period=120s --retries=3 \
  CMD curl -f http://localhost/ || exit 1

Creating the .dockerignore File

Create a .dockerignore file:

.git
.github
*.md
LICENSE
.gitignore
*.log
.DS_Store

Environment Variables Reference

Variable	Required	Default	Description
PILER_HOSTNAME	Yes	-	Hostname for the Piler instance
MYSQL_HOSTNAME	Yes	-	MySQL/MariaDB server hostname
MYSQL_DATABASE	No	piler	Database name
MYSQL_USER	Yes	-	Database username
MYSQL_PASSWORD	Yes	-	Database password
ADMIN_USER	No	admin@local	Admin email address
ADMIN_PASSWORD	Yes	-	Admin password
MEMCACHED_HOST	No	-	Memcached server (optional)

Deploying Piler on Klutch.sh

Once your repository is prepared, follow these steps to deploy Piler:

Provision a MariaDB Database

Set up a MariaDB or MySQL database for Piler. Create a database and user with full access permissions.

Push Your Repository to GitHub

Initialize and push your repository:

git init
git add Dockerfile .dockerignore
git commit -m "Initial Piler deployment configuration"
git remote add origin https://github.com/yourusername/piler-deploy.git
git push -u origin main

Create a New Project on Klutch.sh

Navigate to the Klutch.sh dashboard and create a new project. Give it a descriptive name like “piler” or “email-archive”.

Create a New App

Within your project, create a new app. Connect your GitHub account if you haven’t already, then select the repository containing your Piler Dockerfile.

Configure Traffic Settings

Piler requires HTTP for the web interface and potentially SMTP:

• Select HTTP as the traffic type
• Set the internal port to 80

Set Environment Variables

Add the following environment variables:

Variable	Value
PILER_HOSTNAME	your-app-name.klutch.sh
MYSQL_HOSTNAME	Your MariaDB host
MYSQL_DATABASE	piler
MYSQL_USER	Database username
MYSQL_PASSWORD	Database password
ADMIN_PASSWORD	Your admin password

Attach Persistent Volumes

Add the following volumes for data persistence:

Mount Path	Recommended Size	Purpose
/var/piler/store	500+ GB	Email archive storage
/var/piler/stat	5 GB	Statistics data
/var/piler/sphinx	50 GB	Search index

Deploy Your Application

Click Deploy to start the build process. Klutch.sh will:

• Detect your Dockerfile automatically
• Build the container image
• Attach the persistent volumes
• Start the Piler container
• Provision an HTTPS certificate

Access Piler

Once deployment completes, access your Piler instance at https://your-app-name.klutch.sh.

Initial Setup and Configuration

Admin Login

Access the admin interface:

1. Navigate to your Piler URL
2. Click “Login”
3. Enter admin credentials
4. Access the administration panel

Configuring Email Import

Set up email sources:

IMAP Import:

1. Configure IMAP connection settings
2. Set folder to archive
3. Schedule import frequency

SMTP Journal:

1. Configure journal address
2. Set up mail server to forward journals
3. Emails archive automatically

User Management

Configure user access:

1. Navigate to User Management
2. Configure LDAP/AD integration or create local users
3. Set permissions per user or group
4. Define archive access rules

Retention Policies

Configure retention rules:

1. Go to Retention settings
2. Define retention periods
3. Set up automated cleanup
4. Configure legal hold exceptions

Production Best Practices

Security Recommendations

• Strong Authentication: Enable LDAP/AD integration
• Access Control: Define granular permissions
• Audit Logging: Enable comprehensive audit trails
• Encryption: Secure database connections
• Regular Updates: Keep Piler updated

Performance Optimization

• Sphinx Tuning: Optimize search index settings
• Database Tuning: Configure MariaDB for performance
• Memcached: Enable caching for faster searches
• Storage: Use fast storage for archive

Backup Strategy

Protect your email archive:

1. Database Backups: Regular MariaDB backups
2. Archive Backups: Back up email storage
3. Search Index: Back up Sphinx indexes
4. Configuration: Save configuration files

Troubleshooting Common Issues

Database Connection Errors

Symptoms: Application fails to start.

Solutions:

• Verify database credentials
• Check database host accessibility
• Confirm database exists
• Review connection settings

Search Not Working

Symptoms: Searches return no results.

Solutions:

• Check Sphinx service status
• Rebuild search index
• Verify disk space for indexes
• Review Sphinx logs

Email Import Failing

Symptoms: Emails not being archived.

Solutions:

• Verify IMAP/SMTP settings
• Check authentication credentials
• Review import logs
• Confirm network connectivity

Additional Resources

• Piler Official Website
• Piler Documentation
• Piler Source Code
• Klutch.sh Persistent Volumes
• Klutch.sh Deployments

Conclusion

Deploying Piler on Klutch.sh gives you an enterprise-grade email archiving solution with automatic builds, persistent storage, and secure HTTPS access. The combination of fast search, compliance features, and flexible import options makes Piler suitable for organizations with serious email retention requirements.

Whether you need to meet regulatory compliance, support legal discovery, or simply maintain email history, Piler on Klutch.sh provides the foundation for a reliable, searchable email archive.