thrillwiki_django_no_react/shared/docs/UNRAID_COMPLETE_AUTOMATION.md

ThrillWiki Complete Unraid Automation Guide

This guide provides complete automation for ThrillWiki deployment on Unraid, including VM creation, configuration, and CI/CD setup. Everything is automated with a single command.

🚀 One-Command Complete Setup

Run this single command to automate everything:

./scripts/unraid/setup-complete-automation.sh

This will:

1. ✅ Create and configure VM on Unraid
2. ✅ Install Ubuntu Server with all dependencies
3. ✅ Configure PostgreSQL database
4. ✅ Deploy ThrillWiki application
5. ✅ Set up systemd services
6. ✅ Configure SSH access
7. ✅ Set up webhook listener
8. ✅ Test the entire system

System Architecture

GitHub Push → Webhook → Local Listener → SSH → Unraid VM → Deploy & Restart

Prerequisites

Local Machine

• Python 3.8+
• SSH client
• Internet connection

Unraid Server

• Unraid 6.8+ with VM support enabled
• SSH access to Unraid server
• Sufficient resources (4GB RAM, 50GB disk minimum)
• Ubuntu Server 22.04 ISO in /mnt/user/isos/

Automated Components

1. VM Manager (scripts/unraid/vm-manager.py)

• Creates VM with proper specifications
• Configures networking and storage
• Manages VM lifecycle (start/stop/status)
• Retrieves VM IP addresses

2. Complete Automation (scripts/unraid/setup-complete-automation.sh)

• Orchestrates entire setup process
• Handles SSH key generation and distribution
• Configures all services automatically
• Performs end-to-end testing

3. VM Configuration

• Ubuntu Server 22.04 LTS
• PostgreSQL database
• UV package manager
• Systemd services for ThrillWiki
• Nginx (optional)

Step-by-Step Process

Phase 1: Initial Setup

The automation script will prompt for:

• Unraid server IP address
• Unraid credentials
• VM specifications (memory, CPU, disk)
• GitHub repository URL
• Webhook secret

Phase 2: SSH Key Setup

• Generates SSH keys for VM access
• Generates SSH keys for Unraid access
• Configures SSH client settings
• Tests connectivity

Phase 3: VM Creation

• Creates VM XML configuration
• Creates virtual disk (QCOW2 format)
• Defines VM in libvirt
• Starts VM with Ubuntu installation

Phase 4: VM Configuration

• Installs Ubuntu Server 22.04
• Configures user account with SSH keys
• Installs required packages:
  • Python 3.8+
  • UV package manager
  • PostgreSQL
  • Git
  • Build tools

Phase 5: ThrillWiki Deployment

• Clones repository
• Installs Python dependencies with UV
• Creates database and user
• Runs initial migrations
• Configures systemd services
• Starts ThrillWiki service

Phase 6: CI/CD Setup

• Configures webhook listener
• Tests deployment pipeline
• Verifies all services

Configuration Files Generated

***REMOVED***.unraid

UNRAID_HOST=192.168.1.100
UNRAID_USER=root
VM_NAME=thrillwiki-vm
VM_MEMORY=4096
VM_VCPUS=2
VM_DISK_SIZE=50
SSH_PUBLIC_KEY=ssh-rsa AAAAB3...

***REMOVED***.webhook

WEBHOOK_PORT=9000
WEBHOOK_SECRET=your_secret
VM_HOST=192.168.1.101
VM_USER=ubuntu
VM_KEY_PATH=/home/user/.ssh/thrillwiki_vm
VM_PROJECT_PATH=/home/ubuntu/thrillwiki
REPO_URL=https://github.com/user/repo.git
DEPLOY_BRANCH=main

SSH Configuration

Host thrillwiki-vm
    HostName 192.168.1.101
    User ubuntu
    IdentityFile ~/.ssh/thrillwiki_vm
    StrictHostKeyChecking no

Host unraid
    HostName 192.168.1.100
    User root
    IdentityFile ~/.ssh/unraid_access
    StrictHostKeyChecking no

VM Specifications

Default Configuration

• OS: Ubuntu Server 22.04 LTS
• Memory: 4GB RAM
• vCPUs: 2
• Storage: 50GB (expandable)
• Network: Bridge mode (br0)
• Boot: UEFI with OVMF

Customizable Options

All specifications can be customized during setup:

• Memory allocation
• CPU cores
• Disk size
• VM name
• Network configuration

Services Installed

On VM

• ThrillWiki Django App: Port 8000
• PostgreSQL Database: Port 5432
• SSH Server: Port 22
• Systemd Services: Auto-start on boot

On Local Machine

• Webhook Listener: Configurable port (default 9000)
• SSH Client: Configured for VM access

Management Commands

VM Management

# Check VM status
python3 scripts/unraid/vm-manager.py status

# Start VM
python3 scripts/unraid/vm-manager.py start

# Stop VM
python3 scripts/unraid/vm-manager.py stop

# Get VM IP
python3 scripts/unraid/vm-manager.py ip

# Complete VM setup
python3 scripts/unraid/vm-manager.py setup

Service Management

# Connect to VM
ssh thrillwiki-vm

# Check ThrillWiki service
sudo systemctl status thrillwiki

# Restart service
sudo systemctl restart thrillwiki

# View logs
journalctl -u thrillwiki -f

# Manual deployment
cd thrillwiki && ./scripts/vm-deploy.sh

Webhook Management

# Start webhook listener
./start-webhook.sh

# Or manually
source ***REMOVED***.webhook && python3 scripts/webhook-listener.py

# Test webhook
curl -X GET http://localhost:9000/health

Automated Testing

The setup includes comprehensive testing:

Connectivity Tests

• SSH access to Unraid server
• SSH access to VM
• Network connectivity

Service Tests

• ThrillWiki application startup
• Database connectivity
• Web server response

Deployment Tests

• Git repository access
• Deployment script execution
• Service restart verification

Security Features

SSH Security

• Dedicated SSH keys for each connection
• No password authentication
• Key-based access only

Network Security

• VM isolated in bridge network
• Firewall rules (configurable)
• SSH key rotation support

Service Security

• Non-root service execution
• Systemd security features
• Log rotation and monitoring

Troubleshooting

Common Issues

1. VM Creation Fails

   # Check Unraid VM support
   ssh unraid "virsh list --all"
   
   # Verify ISO exists
   ssh unraid "ls -la /mnt/user/isos/*.iso"
   

2. VM Won't Start

   # Check VM configuration
   python3 scripts/unraid/vm-manager.py status
   
   # Check Unraid logs
   ssh unraid "tail -f /var/log/libvirt/qemu/thrillwiki-vm.log"
   

3. Can't Connect to VM

   # Check VM IP
   python3 scripts/unraid/vm-manager.py ip
   
   # Test SSH key
   ssh -i ~/.ssh/thrillwiki_vm ubuntu@VM_IP
   

4. Service Won't Start

   # Check service logs
   ssh thrillwiki-vm "journalctl -u thrillwiki --no-pager"
   
   # Manual start
   ssh thrillwiki-vm "cd thrillwiki && ./scripts/ci-start.sh"
   

Log Locations

• Setup logs: logs/unraid-automation.log
• VM logs: SSH to VM, then journalctl -u thrillwiki
• Webhook logs: logs/webhook.log
• Deployment logs: On VM at ~/thrillwiki/logs/deploy.log

Advanced Configuration

Custom VM Specifications

Edit variables in the automation script:

VM_MEMORY=8192    # 8GB RAM
VM_VCPUS=4        # 4 CPU cores
VM_DISK_SIZE=100  # 100GB disk

Network Configuration

For static IP assignment, modify the VM XML template in vm-manager.py.

Storage Configuration

The automation uses QCOW2 format for efficient storage. For better performance, consider:

• Raw disk format
• NVMe storage on Unraid
• Dedicated SSD for VM

Performance Optimization

Recommended Settings

• Memory: 4GB minimum, 8GB recommended
• CPU: 2 cores minimum, 4 cores for production
• Storage: SSD recommended for database
• Network: 1Gbps for fast deployments

Production Considerations

• Use dedicated hardware for database
• Configure backup strategies
• Monitor resource usage
• Set up log rotation

Backup and Recovery

Automated Backups

The deployment script automatically creates backups before each deployment in ~/thrillwiki/backups/.

VM Snapshots

# Create VM snapshot
ssh unraid "virsh snapshot-create-as thrillwiki-vm snapshot-name"

# List snapshots
ssh unraid "virsh snapshot-list thrillwiki-vm"

# Restore snapshot
ssh unraid "virsh snapshot-revert thrillwiki-vm snapshot-name"

Database Backups

# Manual database backup
ssh thrillwiki-vm "pg_dump thrillwiki > backup.sql"

# Automated backup (add to cron)
ssh thrillwiki-vm "crontab -e"
# Add: 0 2 * * * pg_dump thrillwiki > /home/ubuntu/db-backup-$(date +\%Y\%m\%d).sql

Monitoring

Health Checks

The system includes built-in health checks:

• VM status monitoring
• Service health verification
• Network connectivity tests
• Application response checks

Alerts (Optional)

Configure alerts for:

• Service failures
• Resource exhaustion
• Deployment failures
• Network issues

This complete automation provides a production-ready ThrillWiki deployment with minimal manual intervention. The entire process from VM creation to application deployment is handled automatically.