mirror of
https://github.com/pacnpal/thrillwiki_django_no_react.git
synced 2025-12-20 08:31:08 -05:00
feat: complete monorepo structure with frontend and shared resources
- Add complete backend/ directory with full Django application - Add frontend/ directory with Vite + TypeScript setup ready for Next.js - Add comprehensive shared/ directory with: - Complete documentation and memory-bank archives - Media files and avatars (letters, park/ride images) - Deployment scripts and automation tools - Shared types and utilities - Add architecture/ directory with migration guides - Configure pnpm workspace for monorepo development - Update .gitignore to exclude .django_tailwind_cli/ build artifacts - Preserve all historical documentation in shared/docs/memory-bank/ - Set up proper structure for full-stack development with shared resources
This commit is contained in:
pacnpal
387
shared/docs/UNRAID_COMPLETE_AUTOMATION.md
Normal file
387
shared/docs/UNRAID_COMPLETE_AUTOMATION.md
Normal file
@@ -0,0 +1,387 @@
|
||||
# ThrillWiki Complete Unraid Automation Guide
|
||||
|
||||
This guide provides **complete automation** for ThrillWiki deployment on Unraid, including VM creation, configuration, and CI/CD setup. Everything is automated with a single command.
|
||||
|
||||
## 🚀 One-Command Complete Setup
|
||||
|
||||
Run this single command to automate everything:
|
||||
|
||||
```bash
|
||||
./scripts/unraid/setup-complete-automation.sh
|
||||
```
|
||||
|
||||
This will:
|
||||
1. ✅ Create and configure VM on Unraid
|
||||
2. ✅ Install Ubuntu Server with all dependencies
|
||||
3. ✅ Configure PostgreSQL database
|
||||
4. ✅ Deploy ThrillWiki application
|
||||
5. ✅ Set up systemd services
|
||||
6. ✅ Configure SSH access
|
||||
7. ✅ Set up webhook listener
|
||||
8. ✅ Test the entire system
|
||||
|
||||
## System Architecture
|
||||
|
||||
```
|
||||
GitHub Push → Webhook → Local Listener → SSH → Unraid VM → Deploy & Restart
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Local Machine
|
||||
- Python 3.8+
|
||||
- SSH client
|
||||
- Internet connection
|
||||
|
||||
### Unraid Server
|
||||
- Unraid 6.8+ with VM support enabled
|
||||
- SSH access to Unraid server
|
||||
- Sufficient resources (4GB RAM, 50GB disk minimum)
|
||||
- Ubuntu Server 22.04 ISO in `/mnt/user/isos/`
|
||||
|
||||
## Automated Components
|
||||
|
||||
### 1. VM Manager (`scripts/unraid/vm-manager.py`)
|
||||
- Creates VM with proper specifications
|
||||
- Configures networking and storage
|
||||
- Manages VM lifecycle (start/stop/status)
|
||||
- Retrieves VM IP addresses
|
||||
|
||||
### 2. Complete Automation (`scripts/unraid/setup-complete-automation.sh`)
|
||||
- Orchestrates entire setup process
|
||||
- Handles SSH key generation and distribution
|
||||
- Configures all services automatically
|
||||
- Performs end-to-end testing
|
||||
|
||||
### 3. VM Configuration
|
||||
- Ubuntu Server 22.04 LTS
|
||||
- PostgreSQL database
|
||||
- UV package manager
|
||||
- Systemd services for ThrillWiki
|
||||
- Nginx (optional)
|
||||
|
||||
## Step-by-Step Process
|
||||
|
||||
### Phase 1: Initial Setup
|
||||
The automation script will prompt for:
|
||||
- Unraid server IP address
|
||||
- Unraid credentials
|
||||
- VM specifications (memory, CPU, disk)
|
||||
- GitHub repository URL
|
||||
- Webhook secret
|
||||
|
||||
### Phase 2: SSH Key Setup
|
||||
- Generates SSH keys for VM access
|
||||
- Generates SSH keys for Unraid access
|
||||
- Configures SSH client settings
|
||||
- Tests connectivity
|
||||
|
||||
### Phase 3: VM Creation
|
||||
- Creates VM XML configuration
|
||||
- Creates virtual disk (QCOW2 format)
|
||||
- Defines VM in libvirt
|
||||
- Starts VM with Ubuntu installation
|
||||
|
||||
### Phase 4: VM Configuration
|
||||
- Installs Ubuntu Server 22.04
|
||||
- Configures user account with SSH keys
|
||||
- Installs required packages:
|
||||
- Python 3.8+
|
||||
- UV package manager
|
||||
- PostgreSQL
|
||||
- Git
|
||||
- Build tools
|
||||
|
||||
### Phase 5: ThrillWiki Deployment
|
||||
- Clones repository
|
||||
- Installs Python dependencies with UV
|
||||
- Creates database and user
|
||||
- Runs initial migrations
|
||||
- Configures systemd services
|
||||
- Starts ThrillWiki service
|
||||
|
||||
### Phase 6: CI/CD Setup
|
||||
- Configures webhook listener
|
||||
- Tests deployment pipeline
|
||||
- Verifies all services
|
||||
|
||||
## Configuration Files Generated
|
||||
|
||||
### `***REMOVED***.unraid`
|
||||
```bash
|
||||
UNRAID_HOST=192.168.1.100
|
||||
UNRAID_USER=root
|
||||
VM_NAME=thrillwiki-vm
|
||||
VM_MEMORY=4096
|
||||
VM_VCPUS=2
|
||||
VM_DISK_SIZE=50
|
||||
SSH_PUBLIC_KEY=ssh-rsa AAAAB3...
|
||||
```
|
||||
|
||||
### `***REMOVED***.webhook`
|
||||
```bash
|
||||
WEBHOOK_PORT=9000
|
||||
WEBHOOK_SECRET=your_secret
|
||||
VM_HOST=192.168.1.101
|
||||
VM_USER=ubuntu
|
||||
VM_KEY_PATH=/home/user/.ssh/thrillwiki_vm
|
||||
VM_PROJECT_PATH=/home/ubuntu/thrillwiki
|
||||
REPO_URL=https://github.com/user/repo.git
|
||||
DEPLOY_BRANCH=main
|
||||
```
|
||||
|
||||
### SSH Configuration
|
||||
```
|
||||
Host thrillwiki-vm
|
||||
HostName 192.168.1.101
|
||||
User ubuntu
|
||||
IdentityFile ~/.ssh/thrillwiki_vm
|
||||
StrictHostKeyChecking no
|
||||
|
||||
Host unraid
|
||||
HostName 192.168.1.100
|
||||
User root
|
||||
IdentityFile ~/.ssh/unraid_access
|
||||
StrictHostKeyChecking no
|
||||
```
|
||||
|
||||
## VM Specifications
|
||||
|
||||
### Default Configuration
|
||||
- **OS**: Ubuntu Server 22.04 LTS
|
||||
- **Memory**: 4GB RAM
|
||||
- **vCPUs**: 2
|
||||
- **Storage**: 50GB (expandable)
|
||||
- **Network**: Bridge mode (br0)
|
||||
- **Boot**: UEFI with OVMF
|
||||
|
||||
### Customizable Options
|
||||
All specifications can be customized during setup:
|
||||
- Memory allocation
|
||||
- CPU cores
|
||||
- Disk size
|
||||
- VM name
|
||||
- Network configuration
|
||||
|
||||
## Services Installed
|
||||
|
||||
### On VM
|
||||
- **ThrillWiki Django App**: Port 8000
|
||||
- **PostgreSQL Database**: Port 5432
|
||||
- **SSH Server**: Port 22
|
||||
- **Systemd Services**: Auto-start on boot
|
||||
|
||||
### On Local Machine
|
||||
- **Webhook Listener**: Configurable port (default 9000)
|
||||
- **SSH Client**: Configured for VM access
|
||||
|
||||
## Management Commands
|
||||
|
||||
### VM Management
|
||||
```bash
|
||||
# Check VM status
|
||||
python3 scripts/unraid/vm-manager.py status
|
||||
|
||||
# Start VM
|
||||
python3 scripts/unraid/vm-manager.py start
|
||||
|
||||
# Stop VM
|
||||
python3 scripts/unraid/vm-manager.py stop
|
||||
|
||||
# Get VM IP
|
||||
python3 scripts/unraid/vm-manager.py ip
|
||||
|
||||
# Complete VM setup
|
||||
python3 scripts/unraid/vm-manager.py setup
|
||||
```
|
||||
|
||||
### Service Management
|
||||
```bash
|
||||
# Connect to VM
|
||||
ssh thrillwiki-vm
|
||||
|
||||
# Check ThrillWiki service
|
||||
sudo systemctl status thrillwiki
|
||||
|
||||
# Restart service
|
||||
sudo systemctl restart thrillwiki
|
||||
|
||||
# View logs
|
||||
journalctl -u thrillwiki -f
|
||||
|
||||
# Manual deployment
|
||||
cd thrillwiki && ./scripts/vm-deploy.sh
|
||||
```
|
||||
|
||||
### Webhook Management
|
||||
```bash
|
||||
# Start webhook listener
|
||||
./start-webhook.sh
|
||||
|
||||
# Or manually
|
||||
source ***REMOVED***.webhook && python3 scripts/webhook-listener.py
|
||||
|
||||
# Test webhook
|
||||
curl -X GET http://localhost:9000/health
|
||||
```
|
||||
|
||||
## Automated Testing
|
||||
|
||||
The setup includes comprehensive testing:
|
||||
|
||||
### Connectivity Tests
|
||||
- SSH access to Unraid server
|
||||
- SSH access to VM
|
||||
- Network connectivity
|
||||
|
||||
### Service Tests
|
||||
- ThrillWiki application startup
|
||||
- Database connectivity
|
||||
- Web server response
|
||||
|
||||
### Deployment Tests
|
||||
- Git repository access
|
||||
- Deployment script execution
|
||||
- Service restart verification
|
||||
|
||||
## Security Features
|
||||
|
||||
### SSH Security
|
||||
- Dedicated SSH keys for each connection
|
||||
- No password authentication
|
||||
- Key-based access only
|
||||
|
||||
### Network Security
|
||||
- VM isolated in bridge network
|
||||
- Firewall rules (configurable)
|
||||
- SSH key rotation support
|
||||
|
||||
### Service Security
|
||||
- Non-root service execution
|
||||
- Systemd security features
|
||||
- Log rotation and monitoring
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **VM Creation Fails**
|
||||
```bash
|
||||
# Check Unraid VM support
|
||||
ssh unraid "virsh list --all"
|
||||
|
||||
# Verify ISO exists
|
||||
ssh unraid "ls -la /mnt/user/isos/*.iso"
|
||||
```
|
||||
|
||||
2. **VM Won't Start**
|
||||
```bash
|
||||
# Check VM configuration
|
||||
python3 scripts/unraid/vm-manager.py status
|
||||
|
||||
# Check Unraid logs
|
||||
ssh unraid "tail -f /var/log/libvirt/qemu/thrillwiki-vm.log"
|
||||
```
|
||||
|
||||
3. **Can't Connect to VM**
|
||||
```bash
|
||||
# Check VM IP
|
||||
python3 scripts/unraid/vm-manager.py ip
|
||||
|
||||
# Test SSH key
|
||||
ssh -i ~/.ssh/thrillwiki_vm ubuntu@VM_IP
|
||||
```
|
||||
|
||||
4. **Service Won't Start**
|
||||
```bash
|
||||
# Check service logs
|
||||
ssh thrillwiki-vm "journalctl -u thrillwiki --no-pager"
|
||||
|
||||
# Manual start
|
||||
ssh thrillwiki-vm "cd thrillwiki && ./scripts/ci-start.sh"
|
||||
```
|
||||
|
||||
### Log Locations
|
||||
|
||||
- **Setup logs**: `logs/unraid-automation.log`
|
||||
- **VM logs**: SSH to VM, then `journalctl -u thrillwiki`
|
||||
- **Webhook logs**: `logs/webhook.log`
|
||||
- **Deployment logs**: On VM at `~/thrillwiki/logs/deploy.log`
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Custom VM Specifications
|
||||
Edit variables in the automation script:
|
||||
```bash
|
||||
VM_MEMORY=8192 # 8GB RAM
|
||||
VM_VCPUS=4 # 4 CPU cores
|
||||
VM_DISK_SIZE=100 # 100GB disk
|
||||
```
|
||||
|
||||
### Network Configuration
|
||||
For static IP assignment, modify the VM XML template in `vm-manager.py`.
|
||||
|
||||
### Storage Configuration
|
||||
The automation uses QCOW2 format for efficient storage. For better performance, consider:
|
||||
- Raw disk format
|
||||
- NVMe storage on Unraid
|
||||
- Dedicated SSD for VM
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Recommended Settings
|
||||
- **Memory**: 4GB minimum, 8GB recommended
|
||||
- **CPU**: 2 cores minimum, 4 cores for production
|
||||
- **Storage**: SSD recommended for database
|
||||
- **Network**: 1Gbps for fast deployments
|
||||
|
||||
### Production Considerations
|
||||
- Use dedicated hardware for database
|
||||
- Configure backup strategies
|
||||
- Monitor resource usage
|
||||
- Set up log rotation
|
||||
|
||||
## Backup and Recovery
|
||||
|
||||
### Automated Backups
|
||||
The deployment script automatically creates backups before each deployment in `~/thrillwiki/backups/`.
|
||||
|
||||
### VM Snapshots
|
||||
```bash
|
||||
# Create VM snapshot
|
||||
ssh unraid "virsh snapshot-create-as thrillwiki-vm snapshot-name"
|
||||
|
||||
# List snapshots
|
||||
ssh unraid "virsh snapshot-list thrillwiki-vm"
|
||||
|
||||
# Restore snapshot
|
||||
ssh unraid "virsh snapshot-revert thrillwiki-vm snapshot-name"
|
||||
```
|
||||
|
||||
### Database Backups
|
||||
```bash
|
||||
# Manual database backup
|
||||
ssh thrillwiki-vm "pg_dump thrillwiki > backup.sql"
|
||||
|
||||
# Automated backup (add to cron)
|
||||
ssh thrillwiki-vm "crontab -e"
|
||||
# Add: 0 2 * * * pg_dump thrillwiki > /home/ubuntu/db-backup-$(date +\%Y\%m\%d).sql
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
The system includes built-in health checks:
|
||||
- VM status monitoring
|
||||
- Service health verification
|
||||
- Network connectivity tests
|
||||
- Application response checks
|
||||
|
||||
### Alerts (Optional)
|
||||
Configure alerts for:
|
||||
- Service failures
|
||||
- Resource exhaustion
|
||||
- Deployment failures
|
||||
- Network issues
|
||||
|
||||
This complete automation provides a production-ready ThrillWiki deployment with minimal manual intervention. The entire process from VM creation to application deployment is handled automatically.
|
||||
Reference in New Issue
Block a user