Deploying Kibitzr

Introduction

Kibitzr is a personal web assistant that monitors websites and performs automated tasks based on changes. Think of it as a programmable helper that periodically checks web pages, applies transformations to the content, and notifies you of relevant changes through various channels.

Built with Python and featuring Selenium integration for JavaScript-heavy sites, Kibitzr can handle everything from simple HTML scraping to complex browser automation scenarios. The YAML-based configuration makes it accessible while Python transforms provide unlimited flexibility for power users.

Key highlights of Kibitzr:

• Website Monitoring: Periodically check web pages for changes
• Content Transforms: Extract, filter, and process page content
• Multiple Notifiers: Send alerts via Slack, email, webhooks, and more
• Browser Automation: Handle JavaScript with Selenium integration
• Python Extensions: Write custom transforms and checks in Python
• YAML Configuration: Simple, readable check definitions
• Headless Operation: Run without a display for server deployments
• Flexible Scheduling: Configure check intervals per target
• Change Detection: Compare current content with previous state
• Lightweight: Minimal resource usage between checks

This guide walks through deploying Kibitzr on Klutch.sh using Docker, configuring web monitoring checks, and setting up notifications.

Why Deploy Kibitzr on Klutch.sh

Deploying Kibitzr on Klutch.sh provides several advantages:

Simplified Deployment: Klutch.sh automatically builds Kibitzr from your Dockerfile with all dependencies configured.

Persistent Storage: Store your configuration files and check state across container restarts.

Always-On Monitoring: Your checks run 24/7 without maintaining your own server.

Reliable Scheduling: Consistent check execution without worrying about uptime.

Environment Variable Management: Securely store API tokens and credentials for notifiers.

Prerequisites

Before deploying Kibitzr on Klutch.sh, ensure you have:

• A Klutch.sh account
• A GitHub account with a repository for your configuration
• Basic familiarity with Docker and YAML
• (Optional) Slack webhook URL or other notification service credentials

Preparing Your Repository

To deploy Kibitzr on Klutch.sh, create a GitHub repository containing your Dockerfile and configuration.

Repository Structure

kibitzr-deploy/
├── Dockerfile
├── kibitzr.yml
├── kibitzr-creds.yml
└── .dockerignore

Creating the Dockerfile

FROM peterdemin/kibitzr:latest

WORKDIR /root

# Copy configuration files
COPY kibitzr.yml /root/kibitzr.yml
COPY kibitzr-creds.yml /root/kibitzr-creds.yml

# The container will run kibitzr in the foreground
CMD ["kibitzr", "run"]

Creating kibitzr.yml

Define your monitoring checks:

checks:
  - name: Example Site Monitor
    url: https://example.com
    transform:
      - css: h1
      - text
    notify:
      - stdout
    period: 1 hour

  - name: Price Tracker
    url: https://example-store.com/product
    transform:
      - css: .price
      - text
    notify:
      - slack
    period: 30 minutes

Creating kibitzr-creds.yml

Store sensitive credentials (use environment variable substitution):

slack:
  url: ${SLACK_WEBHOOK_URL}

telegram:
  token: ${TELEGRAM_BOT_TOKEN}
  chat: ${TELEGRAM_CHAT_ID}

Environment Variables Reference

Variable	Required	Description
SLACK_WEBHOOK_URL	No	Slack incoming webhook URL
TELEGRAM_BOT_TOKEN	No	Telegram bot API token
TELEGRAM_CHAT_ID	No	Telegram chat ID for notifications
SMTP_HOST	No	Email server for mail notifications
SMTP_USER	No	Email username
SMTP_PASSWORD	No	Email password

Creating the .dockerignore File

.git
.github
*.md
LICENSE
.gitignore
*.log
.DS_Store
.env
__pycache__/

Deploying Kibitzr on Klutch.sh

Push Your Repository to GitHub

git init
git add Dockerfile kibitzr.yml kibitzr-creds.yml .dockerignore
git commit -m "Initial Kibitzr deployment configuration"
git remote add origin https://github.com/yourusername/kibitzr-deploy.git
git push -u origin main

Create a New Project on Klutch.sh

Navigate to the Klutch.sh dashboard and create a new project. Give it a descriptive name like “kibitzr” or “web-monitor”.

Create a New App

Within your project, create a new app. Connect your GitHub account and select your Kibitzr repository.

Configure TCP Traffic

Kibitzr runs as a background service without a web interface. In the deployment settings:

• Select TCP as the traffic type
• Set the external port to 8000

Note: Kibitzr doesn’t expose ports by default, but Klutch.sh requires a port configuration.

Set Environment Variables

Variable	Value
SLACK_WEBHOOK_URL	Your Slack webhook (if using Slack)
TELEGRAM_BOT_TOKEN	Your Telegram token (if using Telegram)
TELEGRAM_CHAT_ID	Your Telegram chat ID (if using Telegram)

Attach Persistent Volumes

Mount Path	Recommended Size	Purpose
/root/.kibitzr	1 GB	State files and change tracking

Deploy Your Application

Click Deploy to start the build process.

Verify Operation

Check the application logs in the Klutch.sh dashboard to confirm checks are running.

Configuration Examples

Basic HTML Monitoring

Monitor a webpage for any changes:

checks:
  - name: Blog Updates
    url: https://example-blog.com
    transform:
      - css: article.latest
      - text
    notify:
      - slack
    period: 6 hours

JavaScript-Heavy Sites

Use the browser fetcher for dynamic content:

checks:
  - name: Dynamic Content
    url: https://spa-example.com
    fetch:
      browser: true
      wait: 5  # Wait 5 seconds for JS to load
    transform:
      - css: .dynamic-content
      - text
    notify:
      - slack
    period: 1 hour

Price Monitoring

Track product prices:

checks:
  - name: Amazon Price
    url: https://amazon.com/dp/B0EXAMPLE
    fetch:
      browser: true
    transform:
      - css: span.a-price-whole
      - text
    notify:
      - slack
    period: 4 hours

Python Custom Transform

Use Python for complex transformations:

checks:
  - name: Custom Processing
    url: https://api.example.com/data
    transform:
      - jq: .items
      - python: |
          import json
          data = json.loads(content)
          content = f"Found {len(data)} items"
    notify:
      - slack
    period: 30 minutes

Multiple Notifiers

Send to multiple channels:

checks:
  - name: Critical Monitor
    url: https://important-site.com/status
    transform:
      - css: .status
      - text
    notify:
      - slack
      - email
      - telegram
    period: 5 minutes

Notification Configuration

Slack

# In kibitzr-creds.yml
slack:
  url: https://hooks.slack.com/services/XXX/YYY/ZZZ

Email

# In kibitzr-creds.yml
email:
  host: smtp.gmail.com
  port: 587
  user: your-email@gmail.com
  password: your-app-password
  to: recipient@example.com

Webhooks

# In kibitzr.yml check
notify:
  - webhook:
      url: https://your-webhook-endpoint.com/notify
      method: POST

Production Best Practices

Check Intervals

• Respect Rate Limits: Don’t check too frequently
• Consider Server Load: Space out checks for the same domain
• Use Appropriate Intervals: Match frequency to expected change rate

Error Handling

• Monitor Logs: Check for failed fetches or transforms
• Handle Outages: Sites may be temporarily unavailable
• Test Transforms: Verify CSS selectors work correctly

Troubleshooting Common Issues

Checks Not Running

Solutions:

• Verify kibitzr.yml syntax is valid YAML
• Check that URLs are accessible
• Review application logs for errors

Notifications Not Sending

Solutions:

• Verify credentials in kibitzr-creds.yml
• Test webhook URLs independently
• Check that environment variables are set correctly

Dynamic Content Not Loading

Solutions:

• Enable browser fetcher with fetch: browser: true
• Increase wait time for slow pages
• Verify CSS selectors match rendered content

Additional Resources

• Kibitzr Website
• Kibitzr Documentation
• Kibitzr GitHub Repository
• Kibitzr Docker Hub
• Klutch.sh Deployments

Conclusion

Deploying Kibitzr on Klutch.sh gives you a powerful, always-on web monitoring system without managing your own infrastructure. The combination of Kibitzr’s flexible configuration and Klutch.sh’s deployment simplicity means you can focus on what to monitor rather than how to run the monitoring system.

From simple webpage change detection to complex automation workflows with Python, Kibitzr on Klutch.sh provides the foundation for staying informed about the web content that matters to you.