Deploying GRR

Introduction

GRR Rapid Response is an incident response framework developed by Google that enables remote live forensics on large-scale enterprise networks. Originally built to handle Google’s own security investigation needs, GRR provides security teams with the ability to conduct forensic analysis, collect artifacts, perform memory analysis, and investigate compromised systems across thousands of endpoints simultaneously.

The framework consists of two primary components: a Python-based server infrastructure that handles client communication, data processing, and provides a web-based administrative interface, and lightweight agents (called “clients”) that deploy to endpoints across Windows, macOS, and Linux systems. GRR uses a client-server architecture where agents periodically check in with the server for instructions, making it ideal for investigating systems behind firewalls or NAT.

Deploying GRR on Klutch.sh provides a managed, scalable platform for your incident response infrastructure with automatic Dockerfile detection, persistent storage for forensic data and MySQL databases, secure environment variable management, and the ability to handle large-scale agent deployments. This guide walks through deploying the GRR server components with production-ready MySQL configuration, complete with sample Dockerfiles, environment setup, agent configuration, and best practices for enterprise forensics.

Why Deploy GRR on Klutch.sh?

Klutch.sh offers several advantages for running GRR in production:

Automatic Dockerfile Detection: Push your repository with a Dockerfile and Klutch.sh handles the build and deployment automatically
Persistent Volumes: Store forensic artifacts, flow data, and database files reliably across deployments
Environment Variable Management: Securely store database credentials, encryption keys, and API tokens
Scalable Infrastructure: Handle large numbers of agent check-ins and forensic data collection
GitHub Integration: Connect directly to your GitHub repository for seamless CI/CD workflows
HTTP/TCP Traffic Support: Configure appropriate traffic routing for web UI and agent communication
Custom Domains: Use your own domain for the GRR admin interface and agent communication endpoints

Prerequisites

Before deploying GRR on Klutch.sh, ensure you have:

A Klutch.sh account
A GitHub account with a repository for your project
Docker installed locally for testing (optional but recommended)
Basic familiarity with Docker, Python, and security operations
Access to a MySQL database (deploy separately on Klutch.sh - see MySQL guide)
Understanding of incident response and forensics concepts
Network access to endpoints where GRR agents will be deployed

What is GRR?

GRR (Google Rapid Response) is a comprehensive incident response and remote live forensics platform that enables security teams to:

Remote Live Forensics: Collect forensic artifacts from endpoints without physical access
Large-Scale Investigations: Investigate thousands of machines simultaneously
Memory Analysis: Perform memory acquisition and analysis on remote systems
File Collection: Retrieve files of interest from endpoints for analysis
Timeline Analysis: Build forensic timelines from collected artifacts
Hunt Capabilities: Search for indicators of compromise across your entire fleet
Artifact Collection: Gather system information, registry data, browser history, and more
Flow Management: Execute predefined or custom forensic workflows
Agent Management: Deploy and manage lightweight agents across Windows, macOS, and Linux

GRR Architecture Overview

Understanding GRR’s architecture helps with proper deployment configuration:

┌─────────────────────────────────────────────────────────────────────┐
│                         GRR Server Stack                            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐ │
│  │   Admin UI      │    │   API Server    │    │   Frontend      │ │
│  │   (Port 8000)   │    │   (Port 8080)   │    │   (Port 8080)   │ │
│  └────────┬────────┘    └────────┬────────┘    └────────┬────────┘ │
│           │                      │                      │          │
│           └──────────────────────┼──────────────────────┘          │
│                                  │                                  │
│                    ┌─────────────▼─────────────┐                   │
│                    │      GRR Server Core      │                   │
│                    │    (Message Processing)   │                   │
│                    └─────────────┬─────────────┘                   │
│                                  │                                  │
│           ┌──────────────────────┼──────────────────────┐          │
│           │                      │                      │          │
│  ┌────────▼────────┐    ┌───────▼────────┐    ┌────────▼────────┐ │
│  │   MySQL DB      │    │   Blobstore    │    │  Message Queue  │ │
│  │   (Metadata)    │    │   (Artifacts)  │    │   (Fleetspeak)  │ │
│  └─────────────────┘    └────────────────┘    └─────────────────┘ │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  │ HTTPS (Agent Communication)
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         GRR Agents (Clients)                        │
├─────────────────────────────────────────────────────────────────────┤
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐             │
│  │   Windows   │    │    macOS    │    │    Linux    │             │
│  │   Endpoint  │    │   Endpoint  │    │   Endpoint  │    ...      │
│  └─────────────┘    └─────────────┘    └─────────────┘             │
└─────────────────────────────────────────────────────────────────────┘

Key Components:

Admin UI: Web-based interface for managing investigations, viewing results, and configuring hunts
API Server: RESTful API for programmatic access and integrations
Frontend Server: Handles agent (client) communication and message processing
MySQL Database: Stores metadata, client information, flow states, and hunt data
Blobstore: Stores collected artifacts and forensic data (can use filesystem or cloud storage)
GRR Agents: Lightweight clients deployed to endpoints for forensic data collection

Project Structure

A typical repository layout for deploying GRR on Klutch.sh:

grr-deployment/
├── Dockerfile
├── docker-entrypoint.sh
├── config/
│   ├── server.yaml
│   └── client_template.yaml
├── scripts/
│   └── init-db.sh
├── .env.example
└── README.md

For production deployments, you’ll maintain separate MySQL configuration and implement proper backup strategies for both the database and collected forensic artifacts.

Sample Dockerfile

Create a Dockerfile in the root of your repository. Klutch.sh automatically detects and uses it for deployment:

FROM ubuntu:22.04

# Prevent interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive

# Install system dependencies
RUN apt-get update && apt-get install -y \
    python3 \
    python3-pip \
    python3-venv \
    python3-dev \
    build-essential \
    libmysqlclient-dev \
    pkg-config \
    wget \
    curl \
    gnupg2 \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Create GRR user and directories
RUN useradd -r -s /bin/false grr && \
    mkdir -p /opt/grr /var/grr/blobstore /var/grr/logs /etc/grr && \
    chown -R grr:grr /opt/grr /var/grr /etc/grr

# Set up Python virtual environment
RUN python3 -m venv /opt/grr/venv
ENV PATH="/opt/grr/venv/bin:$PATH"

# Upgrade pip and install GRR
RUN pip install --upgrade pip setuptools wheel && \
    pip install grr-response-server grr-response-client-builder mysqlclient

# Copy configuration files
COPY config/ /etc/grr/
COPY docker-entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/docker-entrypoint.sh

# Create data directories
RUN mkdir -p /var/grr/blobstore /var/grr/logs && \
    chown -R grr:grr /var/grr

# Expose ports
# 8000 - Admin UI
# 8080 - Frontend (Agent Communication)
EXPOSE 8000 8080

# Set working directory
WORKDIR /opt/grr

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
    CMD curl -f http://localhost:8000/api/health || exit 1

# Run as grr user
USER grr

ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["grr_server", "--config=/etc/grr/server.yaml"]

Docker Entrypoint Script

Create a docker-entrypoint.sh file to handle initialization and configuration:

#!/bin/bash
set -e

# Wait for MySQL to be ready
wait_for_mysql() {
    echo "Waiting for MySQL to be ready..."
    max_attempts=30
    attempt=0

    while [ $attempt -lt $max_attempts ]; do
        if python3 -c "import MySQLdb; MySQLdb.connect(host='${MYSQL_HOST}', user='${MYSQL_USER}', passwd='${MYSQL_PASSWORD}', db='${MYSQL_DATABASE}')" 2>/dev/null; then
            echo "MySQL is ready!"
            return 0
        fi
        attempt=$((attempt + 1))
        echo "Attempt $attempt/$max_attempts - MySQL not ready yet..."
        sleep 5
    done

    echo "ERROR: MySQL did not become ready in time"
    exit 1
}

# Generate server configuration from environment variables
generate_config() {
    cat > /etc/grr/server.yaml << EOF
Database.implementation: MysqlDB
Mysql.host: ${MYSQL_HOST:-localhost}
Mysql.port: ${MYSQL_PORT:-3306}
Mysql.database_name: ${MYSQL_DATABASE:-grr}
Mysql.database_username: ${MYSQL_USER:-grr}
Mysql.database_password: ${MYSQL_PASSWORD:-grr}

Blobstore.implementation: FileBasedBlobStore
FileBasedBlobStore.root_path: /var/grr/blobstore

Logging.path: /var/grr/logs
Logging.filename: grr-server.log

AdminUI.url: ${GRR_ADMIN_URL:-http://localhost:8000}
AdminUI.port: 8000
AdminUI.bind: 0.0.0.0

Frontend.bind_address: 0.0.0.0
Frontend.bind_port: 8080

Server.initialized: True
EOF

    # Add optional configuration
    if [ -n "$GRR_ADMIN_PASSWORD" ]; then
        echo "AdminUI.password: $GRR_ADMIN_PASSWORD" >> /etc/grr/server.yaml
    fi

    if [ -n "$GRR_SECRET_KEY" ]; then
        echo "Server.secret_key: $GRR_SECRET_KEY" >> /etc/grr/server.yaml
    fi
}

# Initialize the database if needed
initialize_database() {
    echo "Checking database initialization..."

    if [ "${GRR_INIT_DB:-false}" = "true" ]; then
        echo "Initializing GRR database..."
        grr_config_updater initialize --config=/etc/grr/server.yaml --noprompt
        echo "Database initialization complete."
    fi
}

# Main execution
echo "Starting GRR Server..."
echo "================================"
echo "MySQL Host: ${MYSQL_HOST:-localhost}"
echo "MySQL Database: ${MYSQL_DATABASE:-grr}"
echo "Admin UI URL: ${GRR_ADMIN_URL:-http://localhost:8000}"
echo "================================"

# Wait for dependencies
wait_for_mysql

# Generate configuration
generate_config

# Initialize database on first run
initialize_database

# Execute the main command
exec "$@"

Server Configuration

Create a config/server.yaml file with baseline GRR server settings:

# GRR Server Configuration
# Environment variables override these defaults

# Database Configuration
Database.implementation: MysqlDB
Mysql.host: "%(MYSQL_HOST|localhost)"
Mysql.port: "%(MYSQL_PORT|3306)"
Mysql.database_name: "%(MYSQL_DATABASE|grr)"
Mysql.database_username: "%(MYSQL_USER|grr)"
Mysql.database_password: "%(MYSQL_PASSWORD|)"

# Connection pool settings
Mysql.max_pool_size: 10
Mysql.conn_pool_min: 1
Mysql.conn_pool_max: 10

# Blobstore Configuration
Blobstore.implementation: FileBasedBlobStore
FileBasedBlobStore.root_path: /var/grr/blobstore

# Logging Configuration
Logging.path: /var/grr/logs
Logging.filename: grr-server.log
Logging.verbose: 1

# Admin UI Configuration
AdminUI.url: "%(GRR_ADMIN_URL|http://localhost:8000)"
AdminUI.port: 8000
AdminUI.bind: 0.0.0.0
AdminUI.webauth_manager: BasicWebAuthManager
AdminUI.csrf_secret_key: "%(GRR_CSRF_SECRET|change-this-in-production)"

# Frontend Configuration (Agent Communication)
Frontend.bind_address: 0.0.0.0
Frontend.bind_port: 8080
Frontend.processes: 4

# Client Configuration
Client.server_urls:
  - "%(GRR_FRONTEND_URL|http://localhost:8080)"

# Hunt Configuration
Hunt.default_client_limit: 1000
Hunt.default_client_rate: 100

# Flow Configuration
Flow.default_timeout: 3600

# Security Settings
Server.secret_key: "%(GRR_SECRET_KEY|generate-a-secure-key)"

# Data Retention
DataRetention.cron_jobs_ttl: 604800
DataRetention.tmp_ttl: 604800

Client Configuration Template

Create a config/client_template.yaml for generating agent installers:

# GRR Client Configuration Template
# Used for building client installers

Client:
  # Server URLs for agent communication
  server_urls:
    - "%(GRR_FRONTEND_URL|https://grr.example.com:8080)"

  # Polling interval in seconds
  poll_min: 60
  poll_max: 600

  # Client name prefix
  name_prefix: "grr"

  # Platform-specific settings
  platform_specific:
    windows:
      install_path: "C:\\Program Files\\GRR"
      service_name: "GRR Monitor"
    darwin:
      install_path: "/usr/local/lib/grr"
      plist_path: "/Library/LaunchDaemons/com.google.grr.plist"
    linux:
      install_path: "/usr/lib/grr"
      service_name: "grr-client"

# Communication settings
Communication:
  # Use HTTPS for agent communication
  use_ssl: true

  # Certificate verification
  ssl_verify_server: true

# Logging
Logging:
  verbose: 0
  path: "%(CLIENT_LOG_PATH|/var/log/grr)"

Environment Variables

GRR requires several environment variables for proper operation. Configure these in the Klutch.sh dashboard:

Required Variables

Variable	Description	Example
`MYSQL_HOST`	MySQL server hostname	`mysql-app.klutch.sh`
`MYSQL_PORT`	MySQL server port	`8000`
`MYSQL_DATABASE`	Database name for GRR	`grr`
`MYSQL_USER`	MySQL username	`grr_user`
`MYSQL_PASSWORD`	MySQL password	`secure_password`
`GRR_SECRET_KEY`	Server secret key for encryption	`random-64-char-string`

Optional Variables

Variable	Description	Default
`GRR_ADMIN_URL`	Public URL for admin interface	`http://localhost:8000`
`GRR_FRONTEND_URL`	Public URL for agent communication	`http://localhost:8080`
`GRR_ADMIN_PASSWORD`	Initial admin password	Auto-generated
`GRR_CSRF_SECRET`	CSRF protection secret	Auto-generated
`GRR_INIT_DB`	Initialize database on startup	`false`
`GRR_LOG_LEVEL`	Logging verbosity (0-3)	`1`

Generating Secure Keys

Generate secure random keys for production:

# Generate GRR_SECRET_KEY
python3 -c "import secrets; print(secrets.token_hex(32))"

# Generate GRR_CSRF_SECRET
python3 -c "import secrets; print(secrets.token_hex(16))"

# Generate secure admin password
python3 -c "import secrets; print(secrets.token_urlsafe(16))"

Deploying MySQL Database

GRR requires a MySQL database for storing metadata, client information, and flow states. Deploy MySQL on Klutch.sh first:

Follow the MySQL deployment guide to set up a MySQL instance

Create a database and user for GRR:

CREATE DATABASE grr CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'grr_user'@'%' IDENTIFIED BY 'your_secure_password';
GRANT ALL PRIVILEGES ON grr.* TO 'grr_user'@'%';
FLUSH PRIVILEGES;

Note the connection details: - Host: `your-mysql-app.klutch.sh` - Port: `8000` (external TCP port) - Database: `grr` - Username: `grr_user`

Deploying GRR on Klutch.sh

Follow these steps to deploy GRR on Klutch.sh:

Prepare Your Repository

Create a new GitHub repository with the Dockerfile, configuration files, and entrypoint script from the previous sections.

# Clone your repository
git clone https://github.com/yourusername/grr-deployment.git
cd grr-deployment

# Add the files
# - Dockerfile
# - docker-entrypoint.sh
# - config/server.yaml
# - config/client_template.yaml

# Commit and push
git add .
git commit -m "Initial GRR deployment configuration"
git push origin main

Log in to Klutch.sh

Navigate to klutch.sh/app and sign in to your account.
Create a New Project

Click “New Project” and provide a name for your GRR deployment project (e.g., “grr-forensics”).
Create a New App

Within your project, click “New App” and configure:
- App Name: grr-server
- Git Repository: Connect your GitHub repository
- Branch: main (or your default branch)

Configure Environment Variables

In the app settings, add the following environment variables:

MYSQL_HOST=your-mysql-app.klutch.sh
MYSQL_PORT=8000
MYSQL_DATABASE=grr
MYSQL_USER=grr_user
MYSQL_PASSWORD=your_secure_password
GRR_SECRET_KEY=your_64_character_secret_key
GRR_ADMIN_URL=https://grr-server.klutch.sh
GRR_FRONTEND_URL=https://grr-server.klutch.sh:8080
GRR_INIT_DB=true

Mark MYSQL_PASSWORD, GRR_SECRET_KEY, and any other sensitive values as secrets.

Configure Network Settings
- Traffic Type: HTTP
- Internal Port: 8000 (for admin UI)
Attach Persistent Volume

GRR requires persistent storage for the blobstore (collected artifacts):
- Click “Add Volume” in the app settings
- Mount Path: /var/grr
- Size: Start with 50GB or more depending on expected forensic data volume
Deploy the Application

Click “Create” to start the deployment. Klutch.sh will:
- Detect the Dockerfile automatically
- Build the container image
- Deploy with your configured environment variables
- Attach the persistent volume
Verify Deployment

Once deployed, access the GRR admin interface at:
```
https://grr-server.klutch.sh
```
The initial admin credentials are either set via GRR_ADMIN_PASSWORD or can be found in the application logs.

Deploying the Frontend Server (Optional)

For production deployments with high agent traffic, deploy the frontend server separately:

Frontend Dockerfile

FROM ubuntu:22.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y \
    python3 \
    python3-pip \
    python3-venv \
    python3-dev \
    build-essential \
    libmysqlclient-dev \
    pkg-config \
    curl \
    && rm -rf /var/lib/apt/lists/*

RUN useradd -r -s /bin/false grr && \
    mkdir -p /opt/grr /var/grr /etc/grr && \
    chown -R grr:grr /opt/grr /var/grr /etc/grr

RUN python3 -m venv /opt/grr/venv
ENV PATH="/opt/grr/venv/bin:$PATH"

RUN pip install --upgrade pip setuptools wheel && \
    pip install grr-response-server mysqlclient

COPY config/ /etc/grr/
COPY frontend-entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/frontend-entrypoint.sh

EXPOSE 8080

WORKDIR /opt/grr
USER grr

HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
    CMD curl -f http://localhost:8080/server.pem || exit 1

ENTRYPOINT ["frontend-entrypoint.sh"]
CMD ["grr_frontend", "--config=/etc/grr/server.yaml"]

Frontend Entrypoint

#!/bin/bash
set -e

echo "Starting GRR Frontend Server..."
echo "Frontend URL: ${GRR_FRONTEND_URL:-http://localhost:8080}"

# Generate frontend-specific configuration
cat > /etc/grr/server.yaml << EOF
Database.implementation: MysqlDB
Mysql.host: ${MYSQL_HOST:-localhost}
Mysql.port: ${MYSQL_PORT:-3306}
Mysql.database_name: ${MYSQL_DATABASE:-grr}
Mysql.database_username: ${MYSQL_USER:-grr}
Mysql.database_password: ${MYSQL_PASSWORD:-grr}

Blobstore.implementation: FileBasedBlobStore
FileBasedBlobStore.root_path: /var/grr/blobstore

Frontend.bind_address: 0.0.0.0
Frontend.bind_port: 8080
Frontend.processes: ${GRR_FRONTEND_PROCESSES:-4}

Server.secret_key: ${GRR_SECRET_KEY}
EOF

exec "$@"

Deploy as a separate app with:

Traffic Type: HTTP
Internal Port: 8080

Building GRR Agents

Once your GRR server is deployed, build agents for deployment to endpoints:

Generate Agent Installers

Access the GRR admin UI and navigate to Configuration → Binary Management:

Click "Build new template" to create platform-specific installers
Configure the server URL to match your Klutch.sh deployment:
```
https://grr-server.klutch.sh:8080
```
Download installers for your target platforms: - Windows: `.exe` or `.msi` - macOS: `.pkg` - Linux: `.deb` or `.rpm`

Agent Deployment Script (Windows)

# Download and install GRR agent
$InstallerUrl = "https://grr-server.klutch.sh/static/installers/GRR_installer.exe"
$InstallerPath = "$env:TEMP\GRR_installer.exe"

# Download installer
Invoke-WebRequest -Uri $InstallerUrl -OutFile $InstallerPath

# Install silently
Start-Process -FilePath $InstallerPath -ArgumentList "/S" -Wait

# Verify installation
Get-Service "GRR Monitor"

Agent Deployment Script (Linux)

#!/bin/bash
# Download and install GRR agent on Linux

INSTALLER_URL="https://grr-server.klutch.sh/static/installers/grr_client.deb"
INSTALLER_PATH="/tmp/grr_client.deb"

# Download installer
wget -O "$INSTALLER_PATH" "$INSTALLER_URL"

# Install package
sudo dpkg -i "$INSTALLER_PATH"

# Start service
sudo systemctl enable grr-client
sudo systemctl start grr-client

# Verify
sudo systemctl status grr-client

Agent Deployment Script (macOS)

#!/bin/bash
# Download and install GRR agent on macOS

INSTALLER_URL="https://grr-server.klutch.sh/static/installers/grr_client.pkg"
INSTALLER_PATH="/tmp/grr_client.pkg"

# Download installer
curl -o "$INSTALLER_PATH" "$INSTALLER_URL"

# Install package
sudo installer -pkg "$INSTALLER_PATH" -target /

# Verify
sudo launchctl list | grep grr

Using the GRR Admin Interface

Searching for Clients

# API example: Search for clients
import requests

GRR_URL = "https://grr-server.klutch.sh"
API_TOKEN = "your_api_token"

headers = {
    "Authorization": f"Bearer {API_TOKEN}",
    "Content-Type": "application/json"
}

# Search for Windows clients
response = requests.get(
    f"{GRR_URL}/api/v2/clients",
    headers=headers,
    params={
        "query": "os:windows",
        "count": 100
    }
)

clients = response.json()
for client in clients.get("items", []):
    print(f"Client: {client['clientId']} - {client['hostname']}")

Starting a Hunt

# API example: Create a hunt to collect browser history
import requests
import json

GRR_URL = "https://grr-server.klutch.sh"
API_TOKEN = "your_api_token"

headers = {
    "Authorization": f"Bearer {API_TOKEN}",
    "Content-Type": "application/json"
}

hunt_config = {
    "description": "Collect browser history from all Windows clients",
    "flowName": "ArtifactCollectorFlow",
    "flowArgs": {
        "artifactList": ["ChromeHistory", "FirefoxHistory", "EdgeHistory"]
    },
    "clientRule": {
        "osWindows": True
    },
    "clientLimit": 1000,
    "clientRate": 50
}

response = requests.post(
    f"{GRR_URL}/api/v2/hunts",
    headers=headers,
    json=hunt_config
)

hunt = response.json()
print(f"Hunt created: {hunt['huntId']}")

Collecting Files

# API example: Collect specific files from a client
import requests

GRR_URL = "https://grr-server.klutch.sh"
API_TOKEN = "your_api_token"
CLIENT_ID = "C.1234567890abcdef"

headers = {
    "Authorization": f"Bearer {API_TOKEN}",
    "Content-Type": "application/json"
}

flow_config = {
    "name": "FileFinder",
    "args": {
        "paths": [
            "/etc/passwd",
            "/etc/shadow",
            "/var/log/auth.log"
        ],
        "action": {
            "actionType": "DOWNLOAD"
        }
    }
}

response = requests.post(
    f"{GRR_URL}/api/v2/clients/{CLIENT_ID}/flows",
    headers=headers,
    json=flow_config
)

flow = response.json()
print(f"Flow started: {flow['flowId']}")

Common Forensic Artifacts

GRR can collect numerous forensic artifacts. Here are commonly used ones:

Windows Artifacts

Artifact	Description
`WindowsEventLogs`	Security, System, Application event logs
`WindowsPrefetchFiles`	Application execution artifacts
`WindowsRegistryFiles`	SAM, SYSTEM, SOFTWARE hives
`ChromeHistory`	Chrome browser history and downloads
`WindowsScheduledTasks`	Scheduled task configurations
`WindowsServices`	Installed services
`WMILogicalDisks`	Disk information
`WindowsPersistence`	Common persistence mechanisms

Linux Artifacts

Artifact	Description
`LinuxAuthLogs`	Authentication logs
`LinuxCronTabs`	Scheduled cron jobs
`LinuxPasswdFile`	User account information
`LinuxSudoersFile`	Sudo configuration
`SSHAuthorizedKeys`	SSH authorized keys
`BashHistory`	Bash command history

macOS Artifacts

Artifact	Description
`MacOSLoginItems`	Login items and launch agents
`MacOSUsers`	User account information
`SafariHistory`	Safari browser history
`MacOSQuarantineEvents`	Quarantine events database
`MacOSApplications`	Installed applications

Advanced Configuration

Fleetspeak Integration

For larger deployments, consider using Fleetspeak for agent communication:

# server.yaml additions for Fleetspeak
Server.fleetspeak_enabled: true
Server.fleetspeak_server: "fleetspeak-server:9091"

# Fleetspeak message priority settings
Fleetspeak.message_priority_high: 1
Fleetspeak.message_priority_medium: 5
Fleetspeak.message_priority_low: 10

External Authentication

Configure LDAP or OAuth for enterprise authentication:

# LDAP Authentication
AdminUI.webauth_manager: LDAPWebAuthManager
LDAP.server: "ldap://your-ldap-server:389"
LDAP.bind_dn: "cn=admin,dc=example,dc=com"
LDAP.bind_password: "%(LDAP_PASSWORD|)"
LDAP.user_base_dn: "ou=users,dc=example,dc=com"
LDAP.user_search_filter: "(uid=%s)"

S3 Blobstore

For cloud-based artifact storage:

# S3 Blobstore Configuration
Blobstore.implementation: S3BlobStore
S3BlobStore.bucket: "grr-forensics-data"
S3BlobStore.region: "us-east-1"
S3BlobStore.access_key: "%(AWS_ACCESS_KEY|)"
S3BlobStore.secret_key: "%(AWS_SECRET_KEY|)"

Rate Limiting

Configure client rate limiting to prevent server overload:

# Rate limiting settings
Frontend.max_queue_size: 50000
Frontend.max_retransmission_time: 86400
Frontend.message_expiry_time: 86400

# Hunt rate limiting
Hunt.default_client_rate: 100
Hunt.max_client_rate: 500

Monitoring and Health Checks

Server Health Endpoint

GRR provides health check endpoints:

# Check server health
curl -s https://grr-server.klutch.sh/api/health | jq .

# Check frontend health
curl -s https://grr-server.klutch.sh:8080/server.pem

Monitoring with Prometheus

Export metrics for Prometheus monitoring:

# Enable Prometheus metrics
Monitoring.http_port: 9090
Monitoring.prometheus_enabled: true

Key Metrics to Monitor

Metric	Description	Alert Threshold
`grr_frontend_active_count`	Active agent connections	> 10000
`grr_flow_completion_time`	Flow execution duration	> 3600s
`grr_hunt_clients_completed`	Clients completing hunts	< expected
`grr_database_connections`	Active DB connections	> pool_size * 0.8
`grr_blobstore_size_bytes`	Total artifact storage	> 80% capacity

For comprehensive monitoring setup, see the Monitoring guide.

Backup and Disaster Recovery

Database Backup

Regular MySQL backups are critical:

#!/bin/bash
# GRR database backup script

BACKUP_DIR="/var/grr/backups"
DATE=$(date +%Y%m%d_%H%M%S)
MYSQL_HOST="your-mysql-app.klutch.sh"
MYSQL_PORT="8000"

# Create backup
mysqldump -h "$MYSQL_HOST" -P "$MYSQL_PORT" \
    -u "$MYSQL_USER" -p"$MYSQL_PASSWORD" \
    --single-transaction \
    --routines \
    --triggers \
    grr > "$BACKUP_DIR/grr_db_$DATE.sql"

# Compress
gzip "$BACKUP_DIR/grr_db_$DATE.sql"

# Retain last 30 days
find "$BACKUP_DIR" -name "grr_db_*.sql.gz" -mtime +30 -delete

echo "Backup completed: grr_db_$DATE.sql.gz"

Blobstore Backup

For artifact data stored in the blobstore:

#!/bin/bash
# Blobstore backup script

BLOBSTORE_PATH="/var/grr/blobstore"
BACKUP_PATH="/var/grr/backups/blobstore"
DATE=$(date +%Y%m%d)

# Incremental backup using rsync
rsync -av --delete \
    "$BLOBSTORE_PATH/" \
    "$BACKUP_PATH/$DATE/"

# Create weekly full archive
if [ $(date +%u) -eq 7 ]; then
    tar -czvf "$BACKUP_PATH/blobstore_full_$DATE.tar.gz" \
        -C "$BLOBSTORE_PATH" .
fi

echo "Blobstore backup completed"

Troubleshooting

Common Issues

Database Connection Errors

Error: Can't connect to MySQL server

Solution:

Verify MySQL is running and accessible
Check MYSQL_HOST and MYSQL_PORT environment variables
Ensure the MySQL user has proper permissions
Test connection: mysql -h $MYSQL_HOST -P $MYSQL_PORT -u $MYSQL_USER -p

Agent Not Checking In

Client not found in client list

Solution:

Verify the frontend URL in client configuration
Check firewall rules allow outbound HTTPS
Review agent logs:
- Windows: C:\Program Files\GRR\GRR.log
- Linux: /var/log/grr/grr.log
- macOS: /var/log/grr/grr.log

Frontend Server Overload

Too many pending messages

Solution:

Increase Frontend.processes in configuration
Deploy additional frontend servers
Reduce hunt client rate: Hunt.default_client_rate
Scale up server resources

Blobstore Disk Space

Error: No space left on device

Solution:

Increase persistent volume size
Implement data retention policies
Archive old artifacts to external storage
Configure DataRetention.blobs_ttl setting

Authentication Failures

Error: Invalid credentials

Solution:

Reset admin password using grr_config_updater
Verify LDAP configuration if using external auth
Check API token validity and permissions

Debug Mode

Enable verbose logging for troubleshooting:

# Increase logging verbosity
Logging.verbose: 3
Logging.engines: "stderr,file"

# Enable component-specific debugging
Debug.flow_execution: true
Debug.hunt_processing: true

Log Locations

Component	Log Path
Server	`/var/grr/logs/grr-server.log`
Frontend	`/var/grr/logs/grr-frontend.log`
Worker	`/var/grr/logs/grr-worker.log`
Admin UI	`/var/grr/logs/grr-admin-ui.log`

Security Best Practices

Network Security

Use HTTPS for all agent communication
Configure firewall rules to restrict access to admin UI
Deploy behind a reverse proxy with WAF capabilities
Use VPN for administrative access

Authentication Security

Enable multi-factor authentication when available
Use LDAP/OAuth integration for enterprise deployments
Implement role-based access control (RBAC)
Regularly rotate API tokens and secrets

Data Security

Encrypt sensitive artifacts at rest
Implement data retention policies
Audit access to forensic data
Use separate storage for sensitive investigations

Agent Security

Sign agent installers with code signing certificates
Use certificate pinning for server communication
Implement client-side access controls
Regular agent updates and patching

Production Checklist

Before going live with GRR in production:

Additional Resources

Conclusion

GRR provides enterprise-grade incident response and forensics capabilities that scale from small teams to organizations with hundreds of thousands of endpoints. By deploying GRR on Klutch.sh, you get a managed, scalable platform that handles the infrastructure complexity while you focus on security investigations.

The combination of GRR’s powerful forensic collection capabilities with Klutch.sh’s automatic Dockerfile detection, persistent storage, and environment management creates an ideal platform for running incident response operations. Whether you’re conducting routine security monitoring or responding to active incidents, this deployment provides the foundation for effective digital forensics at scale.