Deploying DebOps

DebOps is a comprehensive Ansible-based framework that provides a collection of over 100 integrated Ansible roles for automating the deployment and management of Debian and Ubuntu-based infrastructure. Unlike traditional configuration management tools that require extensive custom scripting, DebOps offers pre-built, production-tested roles that handle everything from basic system configuration and security hardening to complex application deployments and database management. The framework follows infrastructure-as-code principles, allowing you to define your entire infrastructure in declarative YAML configurations that Ansible executes consistently across environments.

What makes DebOps powerful is its holistic approach to infrastructure automation. Rather than treating each component in isolation, DebOps roles are designed to work together seamlessly, handling dependencies, security policies, and best practices automatically. The framework includes roles for web servers (nginx, Apache), databases (PostgreSQL, MySQL, MariaDB), programming language environments (Python, Ruby, Node.js, PHP), monitoring systems (Prometheus, Grafana), and much more. Whether you’re managing a single server or orchestrating a complex multi-tier application across dozens of hosts, DebOps provides the building blocks to automate infrastructure consistently and maintainably.

Why Deploy DebOps on Klutch.sh?

Deploying DebOps on Klutch.sh offers several advantages for hosting your infrastructure automation platform:

Automatic Docker Detection: Klutch.sh recognizes your Dockerfile and handles containerization without manual configuration
Persistent Storage: Built-in volume management ensures your playbooks, inventories, and SSH keys persist across deployments
HTTPS by Default: Secure access to your DebOps control server with automatic SSL certificates
Environment Management: Securely configure SSH keys, vault passwords, and Ansible variables through environment variables
Always-On Control Server: Run scheduled automation tasks and on-demand playbook executions 24/7
Version Control Integration: Direct integration with GitHub for infrastructure-as-code workflows
Isolated Execution Environment: Containerized DebOps installation prevents conflicts with managed hosts

Prerequisites

Before deploying DebOps to Klutch.sh, ensure you have:

A Klutch.sh account (sign up here)
A GitHub account with a repository for your DebOps project
Basic understanding of Docker and containerization
Target servers (Debian/Ubuntu) for DebOps to manage
SSH access to target servers
Basic knowledge of Ansible and YAML
Git installed on your local development machine
Understanding of infrastructure automation concepts

Understanding DebOps Architecture

DebOps follows a controller-based architecture for infrastructure automation:

Core Components

Ansible Control Node

DebOps runs on an Ansible control node, which is the central server from which all automation tasks are executed. The control node contains:

Ansible engine for executing playbooks
DebOps collection of roles and playbooks
Python environment with required dependencies
SSH client for connecting to managed hosts
Inventory files defining managed infrastructure
Configuration files and variables

The control node doesn’t need to be powerful since it only orchestrates tasks; actual work happens on managed hosts.

DebOps Roles Collection

DebOps provides over 100 Ansible roles organized by function:

System Roles: Base system configuration including users, groups, sudo access, SSH hardening, firewall rules (ferm), fail2ban, unattended upgrades, NTP time synchronization, and sysctl tuning.

Application Roles: Web servers (nginx, Apache), databases (PostgreSQL, MySQL, MariaDB, Redis), programming environments (Python virtualenvs, Ruby rbenv, Node.js nvm, PHP-FPM), application servers (Gunicorn, uWSGI, Passenger), and containerization (Docker, LXC).

Service Roles: Monitoring (Prometheus, Grafana, Elasticsearch, Kibana), logging (rsyslog, journald), backup systems (borg, restic), DNS (bind, dnsmasq), mail services (Postfix, Dovecot), and reverse proxies (HAProxy).

Security Roles: SSL/TLS certificate management (PKI infrastructure, Let’s Encrypt integration), secret management (ansible-vault, pass), security hardening (AppArmor, SELinux), intrusion detection, and compliance enforcement.

Each role is self-contained but integrates with others through standardized interfaces, allowing complex multi-role configurations to work together seamlessly.

Inventory System

DebOps uses Ansible inventory to define infrastructure:

[webservers]
web1.example.com ansible_host=192.168.1.10
web2.example.com ansible_host=192.168.1.11

[databases]
db1.example.com ansible_host=192.168.1.20

[monitoring]
monitor.example.com ansible_host=192.168.1.30

Inventory can be static (YAML/INI files) or dynamic (scripts that query cloud APIs). Host variables and group variables define role-specific configuration.

Playbook System

DebOps playbooks orchestrate role application:

---
- name: Configure web servers
  hosts: webservers
  roles:
    - role: debops.apt
    - role: debops.nginx
    - role: debops.php
    - role: debops.postgresql

Playbooks define which roles apply to which hosts, execution order, and conditional logic.

Secret Management

DebOps integrates with multiple secret management approaches:

Ansible Vault: Encrypt sensitive data files with AES256. Passwords stored separately, decrypted at runtime.

Pass (password-store): GPG-based password manager integration. Secrets stored in git-compatible format.

Environment Variables: Runtime injection of secrets without storing in version control.

The framework handles secret distribution securely, ensuring credentials never appear in logs or version control.

Directory Structure

A typical DebOps project follows this structure:

debops-project/
├── ansible/
│   ├── inventory/
│   │   ├── hosts
│   │   ├── host_vars/
│   │   └── group_vars/
│   ├── playbooks/
│   │   ├── site.yml
│   │   └── custom_playbooks/
│   └── roles/
│       └── custom_roles/
├── .debops.cfg
└── ansible.cfg

Configuration files control behavior, inventory defines infrastructure, and playbooks orchestrate automation.

Execution Flow

Administrator initiates playbook execution from control node
Ansible reads inventory to determine target hosts
DebOps roles are loaded with their default variables
Host-specific and group-specific variables override defaults
Ansible establishes SSH connections to managed hosts
Tasks execute on remote hosts in defined order
Facts gathered from hosts inform conditional logic
Handlers trigger service restarts when configuration changes
Results reported back to control node
Logs and reports generated for audit trail

Role Dependencies

DebOps roles declare dependencies automatically:

debops.nginx depends on debops.apt (package management)
debops.postgresql depends on debops.python (psycopg2 library)
debops.letsencrypt depends on debops.nginx (ACME challenge serving)

Ansible resolves and executes dependencies in correct order without manual intervention.

Data Flow and State Management

DebOps is idempotent—running the same playbook multiple times produces the same result without unintended changes. State is stored on managed hosts, not the control node. The control node maintains:

Inventory (what hosts exist)
Variables (desired configuration state)
Secrets (encrypted credentials)
Playbooks (automation instructions)

Managed hosts maintain their actual state, and Ansible converges actual state toward desired state defined in configuration.

Storage Requirements

DebOps control node requires persistent storage for:

Project Files: Playbooks, roles, inventory (100MB-1GB)
SSH Keys: Authentication credentials for managed hosts (1KB per key)
Secrets: Ansible vault files and password stores (1-10MB)
Logs: Playbook execution logs and audit trails (grows over time)
Facts Cache: Gathered host information for faster runs (1-100MB)

A typical deployment needs 2-5GB initially, with modest growth based on inventory size and execution history.

Installation and Setup

Let’s walk through setting up DebOps for deployment on Klutch.sh.

Step 1: Create the Project Structure

First, create a new directory for your DebOps deployment:

mkdir debops-deployment
cd debops-deployment
git init

Step 2: Create the Dockerfile

Create a Dockerfile in the root directory:

FROM python:3.11-slim

# Set environment variables
ENV DEBIAN_FRONTEND=noninteractive \
    PYTHONUNBUFFERED=1 \
    ANSIBLE_HOST_KEY_CHECKING=False \
    ANSIBLE_FORCE_COLOR=1

# Install system dependencies
RUN apt-get update && apt-get install -y \
    git \
    openssh-client \
    sshpass \
    gnupg2 \
    pass \
    vim \
    curl \
    rsync \
    && rm -rf /var/lib/apt/lists/*

# Create working directory
WORKDIR /opt/debops

# Create debops user
RUN useradd -m -s /bin/bash debops && \
    mkdir -p /home/debops/.ssh && \
    chown -R debops:debops /home/debops

# Install DebOps and dependencies
RUN pip install --no-cache-dir \
    debops==3.1.0 \
    ansible==8.5.0 \
    ansible-core==2.15.5 \
    netaddr \
    dnspython \
    passlib \
    jmespath

# Initialize DebOps project
RUN debops project init debops-project && \
    chown -R debops:debops debops-project

# Set working directory to project
WORKDIR /opt/debops/debops-project

# Copy project files
COPY --chown=debops:debops ansible/ ./ansible/
COPY --chown=debops:debops .debops.cfg ./
COPY --chown=debops:debops ansible.cfg ./

# Create necessary directories
RUN mkdir -p ansible/secret logs && \
    chown -R debops:debops ansible/secret logs

# Switch to debops user
USER debops

# Expose port for web interface (if using AWX/Semaphore)
EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=20s --retries=3 \
  CMD ansible --version || exit 1

# Default command
CMD ["/bin/bash"]

Step 3: Create DebOps Configuration

Create .debops.cfg:

[ansible defaults]
host_key_checking = False
retry_files_enabled = False
inventory = ansible/inventory/hosts
roles_path = ansible/roles
collections_paths = ~/.ansible/collections:/usr/share/ansible/collections
vault_password_file = .vault-password
log_path = logs/ansible.log
stdout_callback = yaml
callback_whitelist = profile_tasks, timer

[ssh_connection]
pipelining = True
control_path = ~/.ansible/cp/%%h-%%p-%%r

Step 4: Create Ansible Configuration

Create ansible.cfg:

[defaults]
inventory = ansible/inventory/hosts
roles_path = ansible/roles:~/.ansible/roles:/usr/share/ansible/roles
collections_paths = ~/.ansible/collections:/usr/share/ansible/collections
host_key_checking = False
retry_files_enabled = False
stdout_callback = yaml
callback_whitelist = profile_tasks, timer
forks = 20
timeout = 30
vault_password_file = .vault-password

[inventory]
enable_plugins = host_list, yaml, ini, script

[privilege_escalation]
become = True
become_method = sudo
become_user = root
become_ask_pass = False

[ssh_connection]
pipelining = True
control_path = ~/.ansible/cp/ansible-ssh-%%h-%%p-%%r

Step 5: Create Initial Inventory

Create ansible/inventory/hosts:

# DebOps Inventory File

[debops_all_hosts]
# Define all your managed hosts here

# Example web servers
[webservers]
web1.example.com ansible_host=192.168.1.10 ansible_user=root
web2.example.com ansible_host=192.168.1.11 ansible_user=root

# Example database servers
[databases]
db1.example.com ansible_host=192.168.1.20 ansible_user=root

# Example monitoring servers
[monitoring]
monitor.example.com ansible_host=192.168.1.30 ansible_user=root

# Group all web and database servers
[application:children]
webservers
databases

# Variables for all hosts
[debops_all_hosts:vars]
ansible_python_interpreter=/usr/bin/python3

Step 6: Create Site Playbook

Create ansible/playbooks/site.yml:

---
# Main DebOps site playbook

- name: Common configuration for all hosts
  hosts: debops_all_hosts
  become: True
  roles:
    - role: debops.apt
      tags: ['role::apt']

    - role: debops.core
      tags: ['role::core']

    - role: debops.ferm
      tags: ['role::ferm']

    - role: debops.users
      tags: ['role::users']

    - role: debops.sshd
      tags: ['role::sshd']

    - role: debops.unattended_upgrades
      tags: ['role::unattended_upgrades']

- name: Configure web servers
  hosts: webservers
  become: True
  roles:
    - role: debops.nginx
      tags: ['role::nginx']

    - role: debops.php
      tags: ['role::php']
      when: install_php | default(True)

    - role: debops.postgresql
      tags: ['role::postgresql']
      when: install_postgresql | default(False)

- name: Configure database servers
  hosts: databases
  become: True
  roles:
    - role: debops.postgresql
      tags: ['role::postgresql']

    - role: debops.redis_server
      tags: ['role::redis_server']
      when: install_redis | default(False)

- name: Configure monitoring servers
  hosts: monitoring
  become: True
  roles:
    - role: debops.prometheus
      tags: ['role::prometheus']
      when: install_prometheus | default(True)

Step 7: Create Group Variables

Create ansible/inventory/group_vars/all/main.yml:

---
# Global variables for all hosts

# Domain configuration
domain: 'example.com'

# Admin email for certificates and notifications
admin_email: 'admin@example.com'

# Timezone configuration
timezone: 'UTC'

# Enable automatic security updates
unattended_upgrades__enabled: True
unattended_upgrades__automatic_reboot: False

# SSH configuration
sshd__permit_root_login: 'prohibit-password'
sshd__password_authentication: False
sshd__port: [22]

# Firewall configuration
ferm__enabled: True

# User management
users__list:
  - name: 'deploy'
    group: 'deploy'
    groups: ['sudo']
    shell: '/bin/bash'
    comment: 'Deployment user'

Step 8: Create Webserver Variables

Create ansible/inventory/group_vars/webservers/nginx.yml:

---
# Nginx configuration for web servers

nginx__enabled: True
nginx__deploy_state: 'present'

nginx__servers:
  - name: 'default'
    filename: 'default'
    default_server: True
    root: '/var/www/html'
    index: 'index.html index.htm'

  - name: 'app.example.com'
    filename: 'app-example-com'
    root: '/var/www/app'
    index: 'index.php index.html'

    location_list:
      - pattern: '~ \.php$'
        options: |
          fastcgi_pass unix:/run/php/php-fpm.sock;
          fastcgi_index index.php;
          include fastcgi_params;
          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

nginx__upstreams:
  - name: 'backend_app'
    server: 'localhost:3000'
    state: 'present'

Step 9: Create Database Variables

Create ansible/inventory/group_vars/databases/postgresql.yml:

---
# PostgreSQL configuration for database servers

postgresql__enabled: True
postgresql__version: '15'

postgresql__databases:
  - name: 'production_db'
    owner: 'app_user'
    encoding: 'UTF-8'
    lc_collate: 'en_US.UTF-8'
    lc_ctype: 'en_US.UTF-8'

postgresql__users:
  - name: 'app_user'
    password: "{{ lookup('env', 'DB_PASSWORD') | default('changeme') }}"
    role_attr_flags: 'CREATEDB'

postgresql__pg_hba_entries:
  - type: 'host'
    database: 'production_db'
    user: 'app_user'
    address: '192.168.1.0/24'
    method: 'md5'

Step 10: Create Requirements File

Create requirements.txt:

debops==3.1.0
ansible==8.5.0
ansible-core==2.15.5
netaddr>=0.8.0
dnspython>=2.3.0
passlib>=1.7.4
jmespath>=1.0.1

Step 11: Create Entrypoint Script

Create entrypoint.sh:

#!/bin/bash
set -e

# Setup SSH keys if provided
if [ -n "$SSH_PRIVATE_KEY" ]; then
    echo "Setting up SSH key..."
    mkdir -p ~/.ssh
    echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
    chmod 600 ~/.ssh/id_rsa

    # Add known hosts
    if [ -n "$SSH_KNOWN_HOSTS" ]; then
        echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
    fi
fi

# Setup vault password if provided
if [ -n "$ANSIBLE_VAULT_PASSWORD" ]; then
    echo "$ANSIBLE_VAULT_PASSWORD" > .vault-password
    chmod 600 .vault-password
fi

# Run DebOps command if provided
if [ $# -gt 0 ]; then
    exec "$@"
else
    # Start interactive shell
    exec /bin/bash
fi

Make it executable:

chmod +x entrypoint.sh

Step 12: Create .dockerignore

Create .dockerignore:

.git
.gitignore
*.md
README.md
.DS_Store
Thumbs.db
logs/
*.log
.vault-password
.vault-password.txt
ansible/secret/
.ssh/
.ansible/
*.retry
__pycache__/
*.pyc
.vscode
.idea

Step 13: Create Documentation

Create README.md:

# DebOps Infrastructure Automation

This repository contains a DebOps project for automating infrastructure deployment and management.

## Features

- 100+ pre-built Ansible roles
- Security hardening by default
- Multi-tier application support
- Database management
- Web server configuration
- Monitoring and logging
- SSL/TLS certificate management

## Project Structure

- `ansible/inventory/` - Host inventory and variables
- `ansible/playbooks/` - Custom playbooks
- `ansible/roles/` - Custom roles
- `.debops.cfg` - DebOps configuration
- `ansible.cfg` - Ansible configuration

## Running Playbooks

```bash
# Run full site playbook
debops run site

# Run specific role
debops run site -t role::nginx

# Run against specific hosts
debops run site -l webservers

# Check mode (dry run)
debops run site --check

Deployment

This project is configured to deploy on Klutch.sh with automatic Docker detection.

### Step 14: Initialize Git Repository

```bash
git add .
git commit -m "Initial DebOps setup for Klutch.sh deployment"
git branch -M master
git remote add origin https://github.com/yourusername/debops-deployment.git
git push -u origin master

Deploying to Klutch.sh

Now that your DebOps control node is configured, let’s deploy it to Klutch.sh.

Log in to Klutch.sh
Navigate to klutch.sh/app and sign in with your GitHub account.
Create a New Project
Click “New Project” and select “Import from GitHub”. Choose the repository containing your DebOps deployment.
Configure Build Settings
Klutch.sh will automatically detect the Dockerfile in your repository. The platform will use this for building your container.
Configure Traffic Settings
Select “HTTP” as the traffic type. While DebOps primarily runs via SSH to managed hosts, you may want HTTP access for web-based management interfaces like AWX or Semaphore on port 8080.
Set Environment Variables
In the project settings, add the following environment variables:
- ANSIBLE_HOST_KEY_CHECKING: False
- ANSIBLE_FORCE_COLOR: 1
SSH configuration (required):
- SSH_PRIVATE_KEY: Your SSH private key for accessing managed hosts (paste full key including headers)
- SSH_KNOWN_HOSTS: Known hosts entries for your servers (optional but recommended)
Vault password (for encrypted secrets):
- ANSIBLE_VAULT_PASSWORD: Password for decrypting ansible-vault encrypted files
Database passwords (if using PostgreSQL role):
- DB_PASSWORD: Password for database users
Optional configuration:
- ADMIN_EMAIL: admin@example.com
- DOMAIN: example.com
- TIMEZONE: UTC
Configure Persistent Storage
DebOps requires persistent storage for project files and logs:
- Project Volume:
  - Mount path: /opt/debops/debops-project
  - Size: 5GB
- SSH Keys Volume:
  - Mount path: /home/debops/.ssh
  - Size: 100MB
- Logs Volume:
  - Mount path: /opt/debops/debops-project/logs
  - Size: 2GB
These volumes ensure your inventory, playbooks, and execution logs persist across deployments.
Deploy the Application
Click “Deploy” to start the build process. Klutch.sh will:
- Clone your repository
- Build the Docker image using your Dockerfile
- Install Python dependencies and DebOps
- Initialize the DebOps project structure
- Deploy the container
- Mount persistent volumes
The build process typically takes 3-5 minutes.
Verify Deployment
Once deployment completes, you can access your DebOps control node via SSH or web interface (if configured).
Run Initial Playbook
Execute your first playbook to configure managed hosts. You can use the Klutch.sh terminal or run commands through a scheduled task.

Getting Started with DebOps

Once your DebOps control node is deployed, here’s how to automate infrastructure:

Running Your First Playbook

Access the Container

From Klutch.sh terminal or local SSH:

# Navigate to project directory
cd /opt/debops/debops-project

# Verify inventory
ansible-inventory --list -y

# Check connectivity to managed hosts
ansible all -m ping

Execute Site Playbook

# Run full site playbook (all roles)
debops run site

# Run with verbose output
debops run site -vv

# Check mode (dry run without changes)
debops run site --check

# Run against specific host group
debops run site -l webservers

Run Specific Roles

# Run only nginx role
debops run site -t role::nginx

# Run multiple roles
debops run site -t role::nginx,role::php

# Skip specific roles
debops run site --skip-tags role::postgresql

Configuring Web Servers

Basic Nginx Setup

Edit ansible/inventory/group_vars/webservers/nginx.yml:

---
nginx__enabled: True

nginx__servers:
  - name: 'myapp.example.com'
    filename: 'myapp'
    root: '/var/www/myapp/public'
    index: 'index.html'

    location_list:
      - pattern: '/'
        options: |
          try_files $uri $uri/ /index.html;

      - pattern: '/api'
        options: |
          proxy_pass http://localhost:3000;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;

Apply configuration:

debops run site -l webservers -t role::nginx

SSL/TLS with Let’s Encrypt

Configure automatic certificate management:

---
pki__enabled: True
pki__acme: True
pki__acme_ca: 'le-live-v2'
pki__acme_contacts: ['mailto:admin@example.com']

nginx__servers:
  - name: 'myapp.example.com'
    filename: 'myapp'
    ssl: True
    ssl_redirect: True
    root: '/var/www/myapp/public'

Run playbook:

debops run site -l webservers -t role::pki,role::nginx

Database Management

PostgreSQL Setup

Configure PostgreSQL with database and users:

---
postgresql__enabled: True
postgresql__version: '15'

postgresql__databases:
  - name: 'app_production'
    owner: 'app_user'
    encoding: 'UTF-8'

  - name: 'app_staging'
    owner: 'app_user'
    encoding: 'UTF-8'

postgresql__users:
  - name: 'app_user'
    password: "{{ vault_app_db_password }}"
    role_attr_flags: 'CREATEDB'

  - name: 'readonly_user'
    password: "{{ vault_readonly_password }}"
    role_attr_flags: 'LOGIN'

postgresql__pg_hba_entries:
  - type: 'host'
    database: 'app_production'
    user: 'app_user'
    address: '192.168.1.0/24'
    method: 'md5'

  - type: 'host'
    database: 'app_production'
    user: 'readonly_user'
    address: '192.168.1.0/24'
    method: 'md5'

Apply database configuration:

debops run site -l databases -t role::postgresql

MySQL/MariaDB Setup

---
mariadb__enabled: True
mariadb__version: '10.11'

mariadb__databases:
  - name: 'wordpress_db'

  - name: 'drupal_db'

mariadb__users:
  - name: 'wordpress_user'
    password: "{{ vault_wordpress_password }}"
    priv: 'wordpress_db.*:ALL'
    host: '192.168.1.%'

  - name: 'drupal_user'
    password: "{{ vault_drupal_password }}"
    priv: 'drupal_db.*:ALL'
    host: '192.168.1.%'

Run playbook:

debops run site -l databases -t role::mariadb

User and Access Management

Creating System Users

Define users in ansible/inventory/group_vars/all/users.yml:

---
users__list:
  - name: 'john'
    group: 'developers'
    groups: ['sudo', 'docker']
    shell: '/bin/bash'
    comment: 'John Doe'
    ssh_authorized_keys:
      - 'ssh-rsa AAAAB3NzaC1yc2EA... john@workstation'

  - name: 'jane'
    group: 'developers'
    groups: ['sudo']
    shell: '/bin/bash'
    comment: 'Jane Smith'
    ssh_authorized_keys:
      - 'ssh-rsa AAAAB3NzaC1yc2EA... jane@laptop'

  - name: 'deploy'
    group: 'deploy'
    groups: ['www-data']
    shell: '/bin/bash'
    comment: 'Deployment user'
    ssh_authorized_keys:
      - 'ssh-rsa AAAAB3NzaC1yc2EA... deploy@ci-server'

users__groups:
  - name: 'developers'
    gid: 3000
    system: False

Apply user configuration:

debops run site -t role::users

Security Hardening

Firewall Configuration

DebOps uses ferm for firewall management:

---
ferm__enabled: True

# Allow specific ports
ferm__input_list:
  - type: 'dport_accept'
    dport: [22, 80, 443]
    protocol: 'tcp'
    saddr: ['0.0.0.0/0']
    comment: 'Allow SSH and HTTP/HTTPS'

  - type: 'dport_accept'
    dport: [5432]
    protocol: 'tcp'
    saddr: ['192.168.1.0/24']
    comment: 'Allow PostgreSQL from internal network'

Apply firewall rules:

debops run site -t role::ferm

SSH Hardening

Configure secure SSH settings:

---
sshd__permit_root_login: 'no'
sshd__password_authentication: False
sshd__challenge_response_authentication: False
sshd__x11_forwarding: False
sshd__max_auth_tries: 3
sshd__max_sessions: 3
sshd__port: [22]
sshd__listen: ['0.0.0.0', '::']

# Allowed users
sshd__allow_users: ['john', 'jane', 'deploy']

# Allowed groups
sshd__allow_groups: ['sudo', 'developers']

Apply SSH configuration:

debops run site -t role::sshd

Managing Secrets

Using Ansible Vault

Create encrypted variables file:

# Create new encrypted file
ansible-vault create ansible/inventory/group_vars/all/vault.yml

Add secrets:

---
vault_app_db_password: 'super-secret-password'
vault_readonly_password: 'readonly-password'
vault_api_key: 'api-key-12345'

Edit encrypted file:

ansible-vault edit ansible/inventory/group_vars/all/vault.yml

Reference in playbooks:

postgresql__users:
  - name: 'app_user'
    password: "{{ vault_app_db_password }}"

Encrypting Existing Files

# Encrypt file
ansible-vault encrypt ansible/inventory/group_vars/databases/secrets.yml

# Decrypt file
ansible-vault decrypt ansible/inventory/group_vars/databases/secrets.yml

# View encrypted file without editing
ansible-vault view ansible/inventory/group_vars/databases/secrets.yml

Application Deployment

Deploy Node.js Application

Configure Node.js environment:

---
nodejs__enabled: True
nodejs__npm_version: 'latest'
nodejs__upstream_release: 'node_18.x'

# Application configuration
nodejs__applications:
  - name: 'myapp'
    user: 'deploy'
    home: '/var/www/myapp'
    repository: 'https://github.com/username/myapp.git'
    version: 'main'
    npm_install: True
    npm_production: True

    environment:
      NODE_ENV: 'production'
      PORT: '3000'
      DATABASE_URL: "postgresql://app_user:{{ vault_app_db_password }}@db1.example.com:5432/app_production"

Deploy application:

debops run site -l webservers -t role::nodejs

Deploy Python Application

---
python__enabled: True
python__v3: True

# Virtual environments
python__virtualenvs:
  - name: 'myapp'
    path: '/var/www/myapp/venv'
    python: 'python3.11'

# Application configuration
gunicorn__applications:
  - name: 'myapp'
    user: 'deploy'
    group: 'www-data'
    working_dir: '/var/www/myapp'
    wsgi_module: 'app:application'
    virtualenv: '/var/www/myapp/venv'
    bind: '127.0.0.1:8000'
    workers: 4

Deploy:

debops run site -l webservers -t role::python,role::gunicorn

Monitoring Setup

Prometheus Configuration

---
prometheus__enabled: True
prometheus__version: '2.45.0'

prometheus__scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets:
          - 'web1.example.com:9100'
          - 'web2.example.com:9100'
          - 'db1.example.com:9100'

  - job_name: 'nginx'
    static_configs:
      - targets:
          - 'web1.example.com:9113'
          - 'web2.example.com:9113'

  - job_name: 'postgresql'
    static_configs:
      - targets:
          - 'db1.example.com:9187'

prometheus__alerting_rules:
  - name: 'instance_down'
    rules:
      - alert: 'InstanceDown'
        expr: 'up == 0'
        for: '5m'
        labels:
          severity: 'critical'
        annotations:
          summary: 'Instance {{ $labels.instance }} down'

Deploy monitoring:

debops run site -l monitoring -t role::prometheus

Advanced Configuration

Custom Playbooks

Create Custom Playbook

Create ansible/playbooks/deploy_app.yml:

---
- name: Deploy application
  hosts: webservers
  become: True
  vars:
    app_name: 'myapp'
    app_repo: 'https://github.com/username/myapp.git'
    app_version: 'main'
    app_path: '/var/www/{{ app_name }}'

  tasks:
    - name: Ensure app directory exists
      file:
        path: "{{ app_path }}"
        state: directory
        owner: deploy
        group: www-data
        mode: '0755'

    - name: Clone application repository
      git:
        repo: "{{ app_repo }}"
        dest: "{{ app_path }}"
        version: "{{ app_version }}"
        force: yes
      become_user: deploy
      notify: restart app

    - name: Install dependencies
      command: npm ci --production
      args:
        chdir: "{{ app_path }}"
      become_user: deploy
      when: ansible_facts['stat']['exists']

    - name: Copy environment file
      template:
        src: env.j2
        dest: "{{ app_path }}/.env"
        owner: deploy
        group: www-data
        mode: '0600'
      notify: restart app

  handlers:
    - name: restart app
      systemd:
        name: "{{ app_name }}"
        state: restarted

Run custom playbook:

ansible-playbook ansible/playbooks/deploy_app.yml

Custom Roles

Create Custom Role

mkdir -p ansible/roles/myapp/{tasks,templates,handlers,files,defaults,vars}

Define tasks in ansible/roles/myapp/tasks/main.yml:

---
- name: Install application dependencies
  apt:
    name:
      - git
      - build-essential
    state: present
    update_cache: yes

- name: Create application user
  user:
    name: "{{ myapp_user }}"
    home: "{{ myapp_home }}"
    shell: /bin/bash
    state: present

- name: Deploy application
  git:
    repo: "{{ myapp_repository }}"
    dest: "{{ myapp_path }}"
    version: "{{ myapp_version }}"
  become_user: "{{ myapp_user }}"
  notify: restart myapp

- name: Configure application
  template:
    src: config.yml.j2
    dest: "{{ myapp_path }}/config.yml"
    owner: "{{ myapp_user }}"
    mode: '0644'
  notify: restart myapp

Define defaults in ansible/roles/myapp/defaults/main.yml:

---
myapp_user: 'myapp'
myapp_home: '/opt/myapp'
myapp_path: '{{ myapp_home }}/app'
myapp_repository: 'https://github.com/username/myapp.git'
myapp_version: 'main'

Use custom role in playbook:

---
- name: Deploy custom application
  hosts: webservers
  roles:
    - role: myapp

Dynamic Inventory

AWS EC2 Dynamic Inventory

Create ansible/inventory/aws_ec2.yml:

---
plugin: aws_ec2
regions:
  - us-east-1
  - us-west-2

filters:
  tag:Environment: production
  instance-state-name: running

keyed_groups:
  - key: tags.Role
    prefix: role
  - key: tags.Environment
    prefix: env

hostnames:
  - ip-address
  - dns-name
  - tag:Name

Test dynamic inventory:

ansible-inventory -i ansible/inventory/aws_ec2.yml --list

Multi-Environment Configuration

Production Environment

Create ansible/inventory/production/hosts:

[webservers]
web1.prod.example.com
web2.prod.example.com

[databases]
db1.prod.example.com

Create ansible/inventory/production/group_vars/all/main.yml:

---
environment: 'production'
domain: 'example.com'
enable_monitoring: True

Staging Environment

Create ansible/inventory/staging/hosts:

[webservers]
web1.staging.example.com

[databases]
db1.staging.example.com

Create ansible/inventory/staging/group_vars/all/main.yml:

---
environment: 'staging'
domain: 'staging.example.com'
enable_monitoring: False

Run against specific environment:

# Production
debops run site -i ansible/inventory/production/hosts

# Staging
debops run site -i ansible/inventory/staging/hosts

Scheduled Playbook Execution

Cron-based Automation

Create scheduled task to run playbooks:

# Edit crontab
crontab -e

# Run security updates nightly at 2 AM
0 2 * * * cd /opt/debops/debops-project && debops run site -t role::unattended_upgrades >> logs/nightly-updates.log 2>&1

# Run full site playbook weekly on Sunday at 3 AM
0 3 * * 0 cd /opt/debops/debops-project && debops run site >> logs/weekly-full-run.log 2>&1

# Check for configuration drift daily at 1 AM
0 1 * * * cd /opt/debops/debops-project && debops run site --check >> logs/daily-drift-check.log 2>&1

Rollback Strategies

Git-based Rollback

Track configuration changes in git:

# Before making changes
git add ansible/
git commit -m "Update nginx configuration"

# If changes cause issues, revert
git revert HEAD
debops run site -t role::nginx

Ansible Snapshots

Take snapshots before major changes:

- name: Create snapshot before updates
  hosts: all
  become: True
  tasks:
    - name: Create filesystem snapshot
      command: lvcreate -L 10G -s -n backup-{{ ansible_date_time.date }} /dev/vg0/root
      when: ansible_lvm.vgs.vg0 is defined

Production Best Practices

Follow these recommendations for running DebOps in production:

Security

SSH Key Management

Use separate SSH keys for different environments
Rotate SSH keys regularly (quarterly recommended)
Use SSH agent forwarding cautiously
Never commit private keys to version control
Use passphrase-protected keys

Ansible Vault Security

Use strong vault passwords (20+ characters)
Store vault password in secure location (password manager)
Never commit .vault-password files
Rotate vault passwords annually
Use different vault passwords per environment

Privilege Escalation

# Use become with specific users, not root when possible
become: True
become_user: 'app_user'

# Use sudo with password when needed
become_method: sudo
become_ask_pass: True

# Limit sudo access
users__sudo_entries:
  - user: 'deploy'
    commands: ['/usr/bin/systemctl restart myapp']
    nopasswd: True

Firewall Rules

Default deny all traffic
Explicitly allow only required ports
Restrict database access to application servers
Use IP whitelisting for administrative access
Log dropped packets for security monitoring

Testing

Syntax Validation

# Check playbook syntax
ansible-playbook ansible/playbooks/site.yml --syntax-check

# Validate inventory
ansible-inventory --list -y

# Check for best practices
ansible-lint ansible/playbooks/site.yml

Dry Run Testing

# Check mode (no changes made)
debops run site --check

# Diff mode (show what would change)
debops run site --check --diff

# Limit to single host for testing
debops run site -l web1.example.com --check

Staging Environment

Always test in staging before production
Maintain staging identical to production
Use same playbooks for both environments
Validate changes with monitoring

Version Control

Branching Strategy

# Feature development
git checkout -b feature/new-role

# Testing in staging
git checkout staging
git merge feature/new-role

# Production deployment
git checkout production
git merge staging
git tag -a v1.2.0 -m "Production release 1.2.0"

Commit Practices

Descriptive commit messages
Atomic commits (one logical change)
Reference issue numbers
Sign commits with GPG

Monitoring and Logging

Playbook Execution Logging

Configure comprehensive logging:

# ansible.cfg
[defaults]
log_path = logs/ansible.log
log_filter = all

# Callback plugins for detailed reporting
callback_whitelist = profile_tasks, timer, yaml

# JSON logging for parsing
stdout_callback = json

Monitoring Playbook Success

Track playbook execution metrics:

Success/failure rates
Execution duration
Changed tasks count
Failed hosts
Task timing profiles

Alerting

Set up alerts for:

Playbook execution failures
Unreachable hosts
Configuration drift detection
Security updates pending
Certificate expiration

Performance Optimization

Parallel Execution

# ansible.cfg
[defaults]
forks = 50  # Increase parallel execution

[ssh_connection]
pipelining = True  # Reduce SSH operations
control_path = ~/.ansible/cp/ansible-ssh-%%h-%%p-%%r

Facts Caching

[defaults]
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_facts
fact_caching_timeout = 86400

Task Optimization

# Use async for long-running tasks
- name: Long running task
  command: /usr/bin/long-process
  async: 3600
  poll: 0
  register: long_task

# Check on async task later
- name: Check on long task
  async_status:
    jid: "{{ long_task.ansible_job_id }}"
  register: job_result
  until: job_result.finished
  retries: 100

Backup and Recovery

Configuration Backup

- name: Backup configuration
  hosts: all
  become: True
  tasks:
    - name: Archive configuration files
      archive:
        path:
          - /etc/nginx
          - /etc/postgresql
          - /etc/mysql
        dest: "/backup/config-{{ inventory_hostname }}-{{ ansible_date_time.date }}.tar.gz"

Database Backup

- name: Backup PostgreSQL databases
  hosts: databases
  become: True
  become_user: postgres
  tasks:
    - name: Dump all databases
      shell: pg_dumpall > /backup/postgresql-{{ ansible_date_time.date }}.sql

Automated Backup Schedule

# Daily configuration backups
0 1 * * * cd /opt/debops/debops-project && ansible-playbook ansible/playbooks/backup.yml

# Weekly full backups
0 2 * * 0 cd /opt/debops/debops-project && ansible-playbook ansible/playbooks/full-backup.yml

Scaling Considerations

Large Inventory Management

For 100+ hosts:

Use dynamic inventory
Implement batching with serial
Group hosts logically
Use fact caching
Increase fork count

Batched Execution

- name: Update web servers in batches
  hosts: webservers
  serial: 5  # Update 5 at a time
  max_fail_percentage: 20
  roles:
    - role: debops.nginx

Load Balancer Coordination

- name: Rolling updates with load balancer
  hosts: webservers
  serial: 1
  tasks:
    - name: Remove from load balancer
      haproxy:
        state: disabled
        backend: web_backend
        host: "{{ inventory_hostname }}"
      delegate_to: "{{ item }}"
      loop: "{{ groups['loadbalancers'] }}"

    - name: Update application
      include_role:
        name: myapp

    - name: Add back to load balancer
      haproxy:
        state: enabled
        backend: web_backend
        host: "{{ inventory_hostname }}"
      delegate_to: "{{ item }}"
      loop: "{{ groups['loadbalancers'] }}"

Troubleshooting

Connection Issues

Problem: Cannot connect to managed hosts

Solutions:

Verify SSH key is loaded: ssh-add -l
Test SSH connection: ssh user@host
Check SSH port: ssh -p 22 user@host
Verify firewall allows SSH
Check ansible_host in inventory
Test with: ansible all -m ping -vvv

Problem: Permission denied (publickey)

Solutions:

Verify SSH key is correct
Check SSH key permissions (should be 600)
Ensure key is added to remote authorized_keys
Verify SSH user has proper permissions
Check SSH config on remote host

Playbook Execution Issues

Problem: Playbook fails with “unreachable”

Solutions:

Check network connectivity
Verify DNS resolution
Test SSH connection manually
Check firewall rules
Verify host is running
Check ansible_host value in inventory

Problem: Task fails with permission error

Solutions:

Use become: True for privileged operations
Verify sudo access: sudo -l
Check become_method setting
Verify user in sudoers file
Use become_user for specific user

Problem: Role dependencies not found

Solutions:

Install DebOps collection: ansible-galaxy collection install debops.debops
Check roles_path in ansible.cfg
Verify role names are correct
Update DebOps: pip install --upgrade debops

Variable Issues

Problem: Variables not being applied

Solutions:

Check variable precedence order
Verify variable file location
Use -e flag to override: ansible-playbook site.yml -e "var=value"
Debug variables: ansible all -m debug -a "var=my_variable"
Check for typos in variable names

Problem: Vault password incorrect

Solutions:

Verify vault password file exists
Check vault_password_file setting
Try manual password: ansible-playbook site.yml --ask-vault-pass
Verify encrypted file: ansible-vault view file.yml
Re-encrypt if needed: ansible-vault rekey file.yml

Performance Issues

Problem: Playbooks run very slowly

Solutions:

Enable SSH pipelining
Increase fork count
Use fact caching
Disable fact gathering when not needed: gather_facts: False
Use async tasks for long operations
Check network latency

Problem: Gathering facts takes long time

Solutions:

Use gather_facts: False if facts not needed
Enable smart gathering
Use fact caching
Gather subset only: gather_subset: '!all,network'

Debugging

Problem: Need to debug playbook execution

Solutions:

# Verbose output
ansible-playbook site.yml -vvv

# Very verbose (connection debugging)
ansible-playbook site.yml -vvvv

# Debug specific task
- name: Debug task
  debug:
    var: my_variable
    verbosity: 2

# Step through playbook
ansible-playbook site.yml --step

# Start at specific task
ansible-playbook site.yml --start-at-task="Install nginx"

Additional Resources

Conclusion

DebOps transforms infrastructure automation from a complex, error-prone manual process into a repeatable, tested, and maintainable system. By providing over 100 production-ready Ansible roles that work together seamlessly, DebOps eliminates the need to reinvent the wheel for common infrastructure tasks. You get security hardening, monitoring, backups, and best practices built-in, not as afterthoughts you have to implement yourself.

Deploying DebOps on Klutch.sh gives you a persistent control node that’s always available to execute playbooks, respond to infrastructure events, and maintain your servers in their desired state. Your entire infrastructure becomes code that you can version, review, test, and deploy with confidence. Whether you’re managing a handful of servers or orchestrating complex multi-tier applications across dozens of hosts, DebOps provides the automation foundation to operate infrastructure efficiently and reliably.

Start automating your infrastructure today and spend less time on repetitive tasks and more time building value for your users.