thrillwiki_django_no_react/docs/UNRAID_COMPLETE_AUTOMATION.md

# ThrillWiki Complete Unraid Automation Guide

This guide provides **complete automation** for ThrillWiki deployment on Unraid, including VM creation, configuration, and CI/CD setup. Everything is automated with a single command.

## 🚀 One-Command Complete Setup

Run this single command to automate everything:

```bash
./scripts/unraid/setup-complete-automation.sh
```

This will:
1. ✅ Create and configure VM on Unraid
2. ✅ Install Ubuntu Server with all dependencies
3. ✅ Configure PostgreSQL database
4. ✅ Deploy ThrillWiki application
5. ✅ Set up systemd services
6. ✅ Configure SSH access
7. ✅ Set up webhook listener
8. ✅ Test the entire system

## System Architecture

```
GitHub Push → Webhook → Local Listener → SSH → Unraid VM → Deploy & Restart
```

## Prerequisites

### Local Machine
- Python 3.8+
- SSH client
- Internet connection

### Unraid Server
- Unraid 6.8+ with VM support enabled
- SSH access to Unraid server
- Sufficient resources (4GB RAM, 50GB disk minimum)
- Ubuntu Server 22.04 ISO in `/mnt/user/isos/`

## Automated Components

### 1. VM Manager (`scripts/unraid/vm-manager.py`)
- Creates VM with proper specifications
- Configures networking and storage
- Manages VM lifecycle (start/stop/status)
- Retrieves VM IP addresses

### 2. Complete Automation (`scripts/unraid/setup-complete-automation.sh`)
- Orchestrates entire setup process
- Handles SSH key generation and distribution
- Configures all services automatically
- Performs end-to-end testing

### 3. VM Configuration
- Ubuntu Server 22.04 LTS
- PostgreSQL database
- UV package manager
- Systemd services for ThrillWiki
- Nginx (optional)

## Step-by-Step Process

### Phase 1: Initial Setup
The automation script will prompt for:
- Unraid server IP address
- Unraid credentials
- VM specifications (memory, CPU, disk)
- GitHub repository URL
- Webhook secret

### Phase 2: SSH Key Setup
- Generates SSH keys for VM access
- Generates SSH keys for Unraid access
- Configures SSH client settings
- Tests connectivity

### Phase 3: VM Creation
- Creates VM XML configuration
- Creates virtual disk (QCOW2 format)
- Defines VM in libvirt
- Starts VM with Ubuntu installation

### Phase 4: VM Configuration
- Installs Ubuntu Server 22.04
- Configures user account with SSH keys
- Installs required packages:
  - Python 3.8+
  - UV package manager
  - PostgreSQL
  - Git
  - Build tools

### Phase 5: ThrillWiki Deployment
- Clones repository
- Installs Python dependencies with UV
- Creates database and user
- Runs initial migrations
- Configures systemd services
- Starts ThrillWiki service

### Phase 6: CI/CD Setup
- Configures webhook listener
- Tests deployment pipeline
- Verifies all services

## Configuration Files Generated

### `***REMOVED***.unraid`
```bash
UNRAID_HOST=192.168.1.100
UNRAID_USER=root
VM_NAME=thrillwiki-vm
VM_MEMORY=4096
VM_VCPUS=2
VM_DISK_SIZE=50
SSH_PUBLIC_KEY=ssh-rsa AAAAB3...
```

### `***REMOVED***.webhook`
```bash
WEBHOOK_PORT=9000
WEBHOOK_SECRET=your_secret
VM_HOST=192.168.1.101
VM_USER=ubuntu
VM_KEY_PATH=/home/user/.ssh/thrillwiki_vm
VM_PROJECT_PATH=/home/ubuntu/thrillwiki
REPO_URL=https://github.com/user/repo.git
DEPLOY_BRANCH=main
```

### SSH Configuration
```
Host thrillwiki-vm
    HostName 192.168.1.101
    User ubuntu
    IdentityFile ~/.ssh/thrillwiki_vm
    StrictHostKeyChecking no

Host unraid
    HostName 192.168.1.100
    User root
    IdentityFile ~/.ssh/unraid_access
    StrictHostKeyChecking no
```

## VM Specifications

### Default Configuration
- **OS**: Ubuntu Server 22.04 LTS
- **Memory**: 4GB RAM
- **vCPUs**: 2
- **Storage**: 50GB (expandable)
- **Network**: Bridge mode (br0)
- **Boot**: UEFI with OVMF

### Customizable Options
All specifications can be customized during setup:
- Memory allocation
- CPU cores
- Disk size
- VM name
- Network configuration

## Services Installed

### On VM
- **ThrillWiki Django App**: Port 8000
- **PostgreSQL Database**: Port 5432
- **SSH Server**: Port 22
- **Systemd Services**: Auto-start on boot

### On Local Machine
- **Webhook Listener**: Configurable port (default 9000)
- **SSH Client**: Configured for VM access

## Management Commands

### VM Management
```bash
# Check VM status
python3 scripts/unraid/vm-manager.py status

# Start VM
python3 scripts/unraid/vm-manager.py start

# Stop VM
python3 scripts/unraid/vm-manager.py stop

# Get VM IP
python3 scripts/unraid/vm-manager.py ip

# Complete VM setup
python3 scripts/unraid/vm-manager.py setup
```

### Service Management
```bash
# Connect to VM
ssh thrillwiki-vm

# Check ThrillWiki service
sudo systemctl status thrillwiki

# Restart service
sudo systemctl restart thrillwiki

# View logs
journalctl -u thrillwiki -f

# Manual deployment
cd thrillwiki && ./scripts/vm-deploy.sh
```

### Webhook Management
```bash
# Start webhook listener
./start-webhook.sh

# Or manually
source ***REMOVED***.webhook && python3 scripts/webhook-listener.py

# Test webhook
curl -X GET http://localhost:9000/health
```

## Automated Testing

The setup includes comprehensive testing:

### Connectivity Tests
- SSH access to Unraid server
- SSH access to VM
- Network connectivity

### Service Tests
- ThrillWiki application startup
- Database connectivity
- Web server response

### Deployment Tests
- Git repository access
- Deployment script execution
- Service restart verification

## Security Features

### SSH Security
- Dedicated SSH keys for each connection
- No password authentication
- Key-based access only

### Network Security
- VM isolated in bridge network
- Firewall rules (configurable)
- SSH key rotation support

### Service Security
- Non-root service execution
- Systemd security features
- Log rotation and monitoring

## Troubleshooting

### Common Issues

1. **VM Creation Fails**
   ```bash
   # Check Unraid VM support
   ssh unraid "virsh list --all"
   
   # Verify ISO exists
   ssh unraid "ls -la /mnt/user/isos/*.iso"
   ```

2. **VM Won't Start**
   ```bash
   # Check VM configuration
   python3 scripts/unraid/vm-manager.py status
   
   # Check Unraid logs
   ssh unraid "tail -f /var/log/libvirt/qemu/thrillwiki-vm.log"
   ```

3. **Can't Connect to VM**
   ```bash
   # Check VM IP
   python3 scripts/unraid/vm-manager.py ip
   
   # Test SSH key
   ssh -i ~/.ssh/thrillwiki_vm ubuntu@VM_IP
   ```

4. **Service Won't Start**
   ```bash
   # Check service logs
   ssh thrillwiki-vm "journalctl -u thrillwiki --no-pager"
   
   # Manual start
   ssh thrillwiki-vm "cd thrillwiki && ./scripts/ci-start.sh"
   ```

### Log Locations

- **Setup logs**: `logs/unraid-automation.log`
- **VM logs**: SSH to VM, then `journalctl -u thrillwiki`
- **Webhook logs**: `logs/webhook.log`
- **Deployment logs**: On VM at `~/thrillwiki/logs/deploy.log`

## Advanced Configuration

### Custom VM Specifications
Edit variables in the automation script:
```bash
VM_MEMORY=8192    # 8GB RAM
VM_VCPUS=4        # 4 CPU cores
VM_DISK_SIZE=100  # 100GB disk
```

### Network Configuration
For static IP assignment, modify the VM XML template in `vm-manager.py`.

### Storage Configuration
The automation uses QCOW2 format for efficient storage. For better performance, consider:
- Raw disk format
- NVMe storage on Unraid
- Dedicated SSD for VM

## Performance Optimization

### Recommended Settings
- **Memory**: 4GB minimum, 8GB recommended
- **CPU**: 2 cores minimum, 4 cores for production
- **Storage**: SSD recommended for database
- **Network**: 1Gbps for fast deployments

### Production Considerations
- Use dedicated hardware for database
- Configure backup strategies
- Monitor resource usage
- Set up log rotation

## Backup and Recovery

### Automated Backups
The deployment script automatically creates backups before each deployment in `~/thrillwiki/backups/`.

### VM Snapshots
```bash
# Create VM snapshot
ssh unraid "virsh snapshot-create-as thrillwiki-vm snapshot-name"

# List snapshots
ssh unraid "virsh snapshot-list thrillwiki-vm"

# Restore snapshot
ssh unraid "virsh snapshot-revert thrillwiki-vm snapshot-name"
```

### Database Backups
```bash
# Manual database backup
ssh thrillwiki-vm "pg_dump thrillwiki > backup.sql"

# Automated backup (add to cron)
ssh thrillwiki-vm "crontab -e"
# Add: 0 2 * * * pg_dump thrillwiki > /home/ubuntu/db-backup-$(date +\%Y\%m\%d).sql
```

## Monitoring

### Health Checks
The system includes built-in health checks:
- VM status monitoring
- Service health verification
- Network connectivity tests
- Application response checks

### Alerts (Optional)
Configure alerts for:
- Service failures
- Resource exhaustion
- Deployment failures
- Network issues

This complete automation provides a production-ready ThrillWiki deployment with minimal manual intervention. The entire process from VM creation to application deployment is handled automatically.