Deploying CI-CD Target

Introduction

CI-CD Target is a powerful open-source continuous integration and deployment platform designed to automate your software development workflows. It provides a robust infrastructure for building, testing, and deploying applications with flexible pipeline configurations, artifact management, and real-time monitoring capabilities.

CI-CD Target stands out for its:

Flexible Pipeline Configuration: Define complex CI/CD workflows with ease using declarative syntax
Artifact Management: Built-in storage for build artifacts, docker images, and deployment packages
Real-Time Monitoring: Live pipeline execution tracking with detailed logs and metrics
Scalable Architecture: Designed to handle projects of any size, from small teams to enterprise deployments
Extensible Plugin System: Rich ecosystem of plugins for integrations with popular tools and services
Secure by Default: Built-in security features including secrets management, role-based access control, and audit logging
Self-Hosted Control: Complete ownership and control of your CI/CD infrastructure

Deploying CI-CD Target on Klutch.sh provides you with a managed, scalable platform for running your CI/CD pipelines with support for persistent storage, secure environment variables, and high availability. This comprehensive guide walks you through deploying CI-CD Target using Docker, including detailed installation steps, sample Dockerfile configurations, persistent storage setup, and production-ready best practices.

Prerequisites

Before you begin, ensure you have the following:

A Klutch.sh account
A GitHub account with a repository for your CI-CD Target project
Docker installed locally for testing (optional but recommended)
Basic understanding of Docker, CI/CD concepts, and continuous integration workflows
A database (PostgreSQL or MySQL) for storing pipeline configurations and execution data (can be deployed on Klutch.sh)

Installation and Setup

Create Your Project Directory

First, create a new directory for your CI-CD Target deployment project:
Terminal window
```
mkdir cicd-target-klutch
cd cicd-target-klutch
git init
```

Create the Dockerfile

Create a Dockerfile in your project root directory. This will define your CI-CD Target container configuration. Klutch.sh automatically detects the Dockerfile in the root directory and uses it for deployment:

FROM node:18-bullseye

# Set working directory
WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    git \
    curl \
    wget \
    build-essential \
    python3 \
    && rm -rf /var/lib/apt/lists/*

# Install CI-CD Target
RUN npm install -g cicd-target

# Create necessary directories
RUN mkdir -p /data/pipelines /data/artifacts /data/logs /data/config

# Copy custom configuration if any
COPY . .

# Expose the default CI-CD Target port
EXPOSE 8080

# Set default environment variables
ENV NODE_ENV=production
ENV CICD_PORT=8080
ENV CICD_DATA_DIR=/data

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

# Start CI-CD Target
CMD ["cicd-target", "start", "--host", "0.0.0.0", "--port", "8080"]

Note: This Dockerfile uses Node.js 18 on Debian Bullseye for a stable production deployment. The application binds to 0.0.0.0 to ensure Klutch.sh can route traffic properly.

Advanced Dockerfile with Custom Build Steps

For a more customized setup with additional build tools and pipeline runners:

FROM node:18-bullseye

# Install system dependencies and build tools
RUN apt-get update && apt-get install -y \
    git \
    curl \
    wget \
    build-essential \
    python3 \
    python3-pip \
    docker.io \
    openjdk-11-jdk \
    maven \
    && rm -rf /var/lib/apt/lists/*

# Set working directory
WORKDIR /app

# Install CI-CD Target and additional tools
RUN npm install -g cicd-target cicd-target-cli

# Install common pipeline dependencies
RUN pip3 install pytest pylint black

# Create directory structure
RUN mkdir -p /data/pipelines \
             /data/artifacts \
             /data/logs \
             /data/config \
             /data/cache \
             /data/workspace

# Copy application files
COPY package*.json ./
RUN npm ci --production
COPY . .

# Set up proper permissions
RUN chown -R node:node /data /app

# Switch to non-root user
USER node

# Expose port
EXPOSE 8080

# Environment variables
ENV NODE_ENV=production
ENV CICD_PORT=8080
ENV CICD_DATA_DIR=/data
ENV CICD_LOG_LEVEL=info

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

# Start the application
CMD ["node", "server.js"]

Create a Custom Start Script (Optional)

Create a server.js file for custom startup logic:

const { CICDTarget } = require('cicd-target');
const path = require('path');

// Configuration
const config = {
  port: process.env.CICD_PORT || 8080,
  host: '0.0.0.0',
  dataDir: process.env.CICD_DATA_DIR || '/data',
  database: {
    type: process.env.DB_TYPE || 'postgres',
    host: process.env.DB_HOST || 'localhost',
    port: process.env.DB_PORT || 5432,
    database: process.env.DB_NAME || 'cicd_target',
    username: process.env.DB_USER || 'cicd',
    password: process.env.DB_PASSWORD || 'changeme',
    ssl: process.env.DB_SSL === 'true'
  },
  storage: {
    artifacts: path.join(process.env.CICD_DATA_DIR || '/data', 'artifacts'),
    logs: path.join(process.env.CICD_DATA_DIR || '/data', 'logs'),
    cache: path.join(process.env.CICD_DATA_DIR || '/data', 'cache')
  },
  security: {
    jwtSecret: process.env.JWT_SECRET || 'change-this-secret',
    sessionSecret: process.env.SESSION_SECRET || 'change-this-secret'
  }
};

// Initialize and start CI-CD Target
const app = new CICDTarget(config);

app.start()
  .then(() => {
    console.log(`CI-CD Target running on http://${config.host}:${config.port}`);
  })
  .catch((error) => {
    console.error('Failed to start CI-CD Target:', error);
    process.exit(1);
  });

// Graceful shutdown
process.on('SIGTERM', async () => {
  console.log('SIGTERM signal received: closing HTTP server');
  await app.stop();
  process.exit(0);
});

Create Configuration File

Create a config.yml file for pipeline and system configurations:

# CI-CD Target Configuration

server:
  host: 0.0.0.0
  port: 8080

pipelines:
  max_concurrent: 10
  default_timeout: 3600
  artifact_retention_days: 30

storage:
  type: filesystem
  base_path: /data

logging:
  level: info
  format: json
  output: /data/logs/cicd-target.log

security:
  enable_auth: true
  allow_signup: false
  require_2fa: false

integrations:
  github:
    enabled: true
  slack:
    enabled: false
  email:
    enabled: false

Create .gitignore File

Create a .gitignore file to exclude sensitive and build files:

# Dependencies
node_modules/

# Environment variables
.env
.env.local

# Data directories (should be volumes)
/data/

# Logs
*.log

# Build artifacts
dist/
build/

# IDE
.vscode/
.idea/

Initialize Git and Push to GitHub

Initialize your repository and push it to GitHub:

git add .
git commit -m "Initial CI-CD Target setup for Klutch.sh"
git branch -M main
git remote add origin https://github.com/yourusername/cicd-target-klutch.git
git push -u origin main

Deploying on Klutch.sh

Connect Your GitHub Repository
- Navigate to klutch.sh/app
- Click “New Project” or select an existing project
- Click “New App” and select “Import from GitHub”
- Authorize Klutch.sh to access your GitHub account if you haven’t already
- Select your CI-CD Target repository from the list
Configure Build Settings

Since Klutch.sh automatically detects the Dockerfile in your repository root, the build configuration is handled automatically. The platform uses Nixpacks as the build system, but when a Dockerfile is present, it takes precedence.
Configure the Internal Port

CI-CD Target needs to expose its web interface on port 8080:
- In the app configuration, set the Internal Port to 8080
- Select HTTP as the traffic type
- This ensures that Klutch.sh routes incoming HTTP traffic to the correct port inside your container
Set Up Persistent Storage

CI-CD Target requires persistent storage for pipeline data, artifacts, logs, and configuration:

In your app settings, add the following volume mounts:
- Mount Path: /data/pipelines | Size: 5 GB (for pipeline configurations)
- Mount Path: /data/artifacts | Size: 20 GB (for build artifacts)
- Mount Path: /data/logs | Size: 5 GB (for application and pipeline logs)
- Mount Path: /data/config | Size: 1 GB (for configuration files)
Note: Adjust the volume sizes based on your expected usage. Projects with many pipelines or large artifacts may require more storage.

Configure Environment Variables

Add the following environment variables in your Klutch.sh app settings. Mark sensitive values as secret:

Database Configuration:

DB_TYPE=postgres
DB_HOST=your-postgres-host.klutch.sh
DB_PORT=5432
DB_NAME=cicd_target
DB_USER=cicd_admin
DB_PASSWORD=<mark-as-secret>
DB_SSL=true

Application Configuration:

NODE_ENV=production
CICD_PORT=8080
CICD_DATA_DIR=/data
CICD_LOG_LEVEL=info
CICD_BASE_URL=https://example-app.klutch.sh

Security Configuration:

JWT_SECRET=<mark-as-secret>
SESSION_SECRET=<mark-as-secret>
ADMIN_EMAIL=admin@yourdomain.com
ADMIN_PASSWORD=<mark-as-secret>

Optional - GitHub Integration:

GITHUB_CLIENT_ID=your-github-client-id
GITHUB_CLIENT_SECRET=<mark-as-secret>
GITHUB_WEBHOOK_SECRET=<mark-as-secret>

Optional - Notification Services:

SLACK_WEBHOOK_URL=<mark-as-secret>
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=notifications@yourdomain.com
SMTP_PASSWORD=<mark-as-secret>

Deploy the Application
- Review all configurations
- Click “Deploy” to start the deployment
- Klutch.sh will build the Docker image from your Dockerfile and deploy it
- Monitor the build logs to ensure everything completes successfully
Verify the Deployment

Once the deployment is complete:
- Access your CI-CD Target instance at https://example-app.klutch.sh
- Log in with the admin credentials you configured
- Verify that you can access the dashboard and create a test pipeline
- Check that persistent storage is working by creating a pipeline and ensuring it persists after a restart

Database Setup

CI-CD Target requires a database for storing pipeline configurations, execution history, and user data. You can deploy a PostgreSQL database on Klutch.sh or use an external managed database service.

Option 1: Deploy PostgreSQL on Klutch.sh

Create a new app for PostgreSQL
Use the official PostgreSQL Docker image
Set up persistent storage for /var/lib/postgresql/data
Configure environment variables for database credentials
Note the internal hostname and port for connecting from your CI-CD Target app

Option 2: Use External Database

You can use external managed database services like:

AWS RDS PostgreSQL
Google Cloud SQL
Azure Database for PostgreSQL
DigitalOcean Managed Databases

Ensure your database is accessible from Klutch.sh and configure the connection details in your environment variables.

Getting Started with CI-CD Target

Initial Setup

Access the Web Interface: Navigate to https://example-app.klutch.sh
Complete Initial Setup: On first launch, you’ll be prompted to:
- Create an admin account (if not using environment variables)
- Configure system settings
- Set up integrations

Create Your First Pipeline:

name: Build and Test

triggers:
  - type: push
    branches:
      - main
      - develop

stages:
  - name: Build
    steps:
      - name: Install Dependencies
        run: npm install

      - name: Build Application
        run: npm run build

  - name: Test
    steps:
      - name: Run Unit Tests
        run: npm test

      - name: Run Integration Tests
        run: npm run test:integration

  - name: Deploy
    condition: branch == 'main'
    steps:
      - name: Deploy to Production
        run: ./deploy.sh

Connect GitHub Repository:
- Go to Settings → Integrations
- Add your GitHub repository
- Configure webhook for automatic pipeline triggers

Example: Simple Build Pipeline

Here’s a complete example of a build pipeline for a Node.js application:

name: Node.js CI Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  NODE_VERSION: '18'

stages:
  - name: setup
    steps:
      - name: Checkout Code
        uses: checkout@v2

      - name: Setup Node.js
        uses: setup-node@v2
        with:
          node-version: ${{ env.NODE_VERSION }}

      - name: Cache Dependencies
        uses: cache@v2
        with:
          path: node_modules
          key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}

  - name: build
    steps:
      - name: Install Dependencies
        run: npm ci

      - name: Lint Code
        run: npm run lint

      - name: Build Application
        run: npm run build

      - name: Upload Artifacts
        uses: upload-artifact@v2
        with:
          name: build-output
          path: dist/

  - name: test
    steps:
      - name: Run Unit Tests
        run: npm run test:unit

      - name: Run Integration Tests
        run: npm run test:integration

      - name: Generate Coverage Report
        run: npm run test:coverage

      - name: Upload Coverage
        uses: upload-artifact@v2
        with:
          name: coverage-report
          path: coverage/

  - name: deploy
    condition: branch == 'main' && success()
    steps:
      - name: Download Build Artifacts
        uses: download-artifact@v2
        with:
          name: build-output

      - name: Deploy to Klutch.sh
        run: ./scripts/deploy.sh
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

Environment Variables Reference

Required Environment Variables

Variable	Description	Example
`DB_TYPE`	Database type (postgres, mysql)	`postgres`
`DB_HOST`	Database host address	`postgres.klutch.sh`
`DB_PORT`	Database port	`5432`
`DB_NAME`	Database name	`cicd_target`
`DB_USER`	Database username	`cicd_admin`
`DB_PASSWORD`	Database password	`securepassword`
`JWT_SECRET`	Secret for JWT tokens	`random-secure-string`
`SESSION_SECRET`	Secret for sessions	`random-secure-string`

Optional Environment Variables

Variable	Description	Default
`CICD_PORT`	Application port	`8080`
`CICD_LOG_LEVEL`	Logging level (debug, info, warn, error)	`info`
`CICD_BASE_URL`	Public URL of the application	`http://localhost:8080`
`MAX_CONCURRENT_PIPELINES`	Maximum concurrent pipeline executions	`10`
`ARTIFACT_RETENTION_DAYS`	Days to keep build artifacts	`30`
`ENABLE_METRICS`	Enable Prometheus metrics	`true`
`METRICS_PORT`	Metrics endpoint port	`9090`

Nixpacks Environment Variables

If you need to customize the build or start commands, you can use these Nixpacks environment variables:

Variable	Description	Example
`NIXPACKS_BUILD_CMD`	Custom build command	`npm run build:production`
`NIXPACKS_START_CMD`	Custom start command	`node server.js`
`NIXPACKS_INSTALL_CMD`	Custom install command	`npm ci --production`

Note: Since Klutch.sh automatically detects and uses the Dockerfile, these Nixpacks variables are only relevant if you’re deploying without a Dockerfile.

Persistent Storage Best Practices

Storage Layout

Organize your persistent storage for optimal performance and management:

/data/
├── pipelines/          # Pipeline configurations
├── artifacts/          # Build artifacts and outputs
├── logs/              # Application and pipeline logs
├── config/            # Configuration files
├── cache/             # Build cache for faster executions
└── workspace/         # Temporary workspace for pipeline executions

Volume Size Recommendations

Based on project scale:

Small Projects (1-5 pipelines):

Pipelines: 1 GB
Artifacts: 5 GB
Logs: 2 GB

Medium Projects (5-20 pipelines):

Pipelines: 5 GB
Artifacts: 20 GB
Logs: 5 GB

Large Projects (20+ pipelines):

Pipelines: 10 GB
Artifacts: 50 GB+
Logs: 10 GB

Artifact Cleanup

Implement regular cleanup to manage storage:

// Example cleanup script
const RETENTION_DAYS = 30;
const ARTIFACT_PATH = '/data/artifacts';

async function cleanupOldArtifacts() {
  const cutoffDate = new Date();
  cutoffDate.setDate(cutoffDate.getDate() - RETENTION_DAYS);

  // Implementation to remove old artifacts
  // This should be run as a scheduled job
}

Production Best Practices

Security

Use Strong Secrets: Generate cryptographically secure random strings for JWT and session secrets
Terminal window
```
openssl rand -hex 32
```
Enable Database SSL: Always use SSL/TLS for database connections in production
Implement Rate Limiting: Configure rate limits to prevent abuse
Regular Updates: Keep CI-CD Target and dependencies up to date

High Availability

Database Backups: Schedule regular database backups
- Store backups in secure, off-site locations
- Test restoration procedures regularly
Health Monitoring: Use the built-in health check endpoint
Terminal window
```
curl https://example-app.klutch.sh/health
```
Log Aggregation: Configure external log aggregation for better monitoring

Performance Optimization

Cache Dependencies: Use build caching to speed up pipeline executions
Concurrent Execution Limits: Set appropriate limits based on your resource availability
Artifact Retention: Configure automatic cleanup of old artifacts
Database Indexing: Ensure proper database indexes for pipeline queries

Monitoring and Alerts

Set up monitoring for:

Application uptime and response times
Pipeline execution success rates
Storage usage and growth trends
Database performance metrics
Error rates and types

Troubleshooting

Common Issues

Issue: Application fails to start

Solution: Check database connectivity and ensure all required environment variables are set
Debug: View application logs in Klutch.sh dashboard

Issue: Pipelines not triggering

Solution: Verify GitHub webhook configuration and check webhook delivery logs
Debug: Test webhook manually using the webhook test feature

Issue: Out of storage space

Solution: Implement artifact cleanup or increase volume size
Debug: Check storage usage: df -h /data

Issue: Slow pipeline executions

Solution: Review concurrent execution limits and consider caching strategies
Debug: Check system resources and database performance

Debug Mode

Enable debug logging for troubleshooting:

CICD_LOG_LEVEL=debug
NODE_ENV=development

Database Connection Issues

Test database connectivity:

# Inside container
psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -c "SELECT 1"

Advanced Configuration

Custom Pipeline Runners

Configure custom runners for specific pipeline types:

runners:
  - name: docker-runner
    type: docker
    config:
      image: docker:latest
      privileged: true

  - name: kubernetes-runner
    type: kubernetes
    config:
      namespace: cicd-pipelines
      service_account: pipeline-runner

Integration with External Services

GitHub Integration

Create a GitHub OAuth App
Add credentials to environment variables
Configure webhook endpoints

Slack Notifications

notifications:
  slack:
    enabled: true
    webhook_url: ${SLACK_WEBHOOK_URL}
    channels:
      success: '#deployments'
      failure: '#alerts'

Scaling Considerations

For high-load scenarios:

Horizontal Scaling: Deploy multiple CI-CD Target instances behind a load balancer
Worker Nodes: Add dedicated worker nodes for pipeline execution
Distributed Caching: Use Redis for distributed caching
Queue Management: Implement message queue for pipeline scheduling

Docker Compose for Local Development

For local testing before deploying to Klutch.sh, use this Docker Compose configuration:

version: '3.8'

services:
  cicd-target:
    build: .
    ports:
      - "8080:8080"
    environment:
      - NODE_ENV=development
      - DB_TYPE=postgres
      - DB_HOST=postgres
      - DB_PORT=5432
      - DB_NAME=cicd_target
      - DB_USER=cicd
      - DB_PASSWORD=devpassword
      - JWT_SECRET=dev-jwt-secret
      - SESSION_SECRET=dev-session-secret
    volumes:
      - ./data/pipelines:/data/pipelines
      - ./data/artifacts:/data/artifacts
      - ./data/logs:/data/logs
      - ./data/config:/data/config
    depends_on:
      - postgres

  postgres:
    image: postgres:15-alpine
    environment:
      - POSTGRES_DB=cicd_target
      - POSTGRES_USER=cicd
      - POSTGRES_PASSWORD=devpassword
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data

volumes:
  postgres-data:

Run locally with:

docker-compose up -d

Note: Docker Compose is only for local development. Klutch.sh does not support Docker Compose for deployments.

Backup and Recovery

Database Backup

Create regular backups of your PostgreSQL database:

# Backup script
#!/bin/bash
BACKUP_DIR=/data/backups
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="cicd_target_backup_${DATE}.sql"

pg_dump -h $DB_HOST -U $DB_USER -d $DB_NAME > "${BACKUP_DIR}/${BACKUP_FILE}"
gzip "${BACKUP_DIR}/${BACKUP_FILE}"

# Keep only last 7 days of backups
find $BACKUP_DIR -name "*.sql.gz" -mtime +7 -delete

Volume Backup

Backup persistent volumes regularly:

# Backup volumes
tar -czf artifacts_backup_$(date +%Y%m%d).tar.gz /data/artifacts
tar -czf pipelines_backup_$(date +%Y%m%d).tar.gz /data/pipelines

Restore Procedure

Stop the application
Restore database from backup
Restore volume data
Restart the application
Verify functionality