thrillwiki_django_no_react/shared/docs/VM_DEPLOYMENT_SETUP.md

ThrillWiki VM Deployment Setup Guide

This guide explains how to set up a local CI/CD system that automatically deploys ThrillWiki to a Linux VM when commits are pushed to GitHub.

System Overview

The deployment system consists of three main components:

1. Local CI Start Script (scripts/ci-start.sh) - Starts the Django server locally
2. GitHub Webhook Listener (scripts/webhook-listener.py) - Listens for GitHub push events
3. VM Deployment Script (scripts/vm-deploy.sh) - Deploys code changes to the Linux VM

Architecture Flow

GitHub Push → Webhook → Local Listener → SSH to VM → Deploy Script → Restart Server

Prerequisites

Local Machine (Webhook Listener Host)

• Python 3.8+
• SSH access to the Linux VM
• Git repository with webhook access

Linux VM (Deployment Target)

• Ubuntu 20.04+ (recommended)
• Python 3.8+
• UV package manager
• Git
• PostgreSQL (if using database)
• SSH server running
• Sudo access for the deployment user

Step 1: Linux VM Setup

1.1 Create Deployment User

# On the Linux VM
sudo adduser ubuntu
sudo usermod -aG sudo ubuntu
su - ubuntu

1.2 Install Required Software

# Update system
sudo apt update && sudo apt upgrade -y

# Install essential packages
sudo apt install -y git curl build-essential python3-pip python3-venv postgresql postgresql-contrib nginx

# Install UV package manager
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.cargo/env

1.3 Set up SSH Keys

# Generate SSH key on local machine
ssh-keygen -t rsa -b 4096 -f ~/.ssh/thrillwiki_vm

# Copy public key to VM
ssh-copy-id -i ~/.ssh/thrillwiki_vm.pub ubuntu@VM_IP_ADDRESS

1.4 Clone Repository

# On the VM
cd /home/ubuntu
git clone https://github.com/YOUR_USERNAME/thrillwiki_django_no_react.git thrillwiki
cd thrillwiki

1.5 Install Dependencies

# Install Python dependencies
uv sync

# Create required directories
mkdir -p logs backups

Step 2: Configure Services

2.1 Install Systemd Services

# Copy service files to systemd directory
sudo cp scripts/systemd/thrillwiki.service /etc/systemd/system/
sudo cp scripts/systemd/thrillwiki-webhook.service /etc/systemd/system/

# Edit service files to match your paths
sudo nano /etc/systemd/system/thrillwiki.service
sudo nano /etc/systemd/system/thrillwiki-webhook.service

# Reload systemd and enable services
sudo systemctl daemon-reload
sudo systemctl enable thrillwiki.service
sudo systemctl enable thrillwiki-webhook.service

2.2 Configure Environment Variables

Create /home/ubuntu/thrillwiki/***REMOVED***:

# Database configuration
DATABASE_URL=[DATABASE-URL-REMOVED]

# Django settings
DJANGO_SECRET_KEY=your_secret_key_here
DJANGO_DEBUG=False
DJANGO_ALLOWED_HOSTS=your_domain.com,VM_IP_ADDRESS

# Webhook configuration
WEBHOOK_SECRET=your_github_webhook_secret
WEBHOOK_PORT=9000
VM_HOST=localhost
VM_USER=ubuntu
VM_PROJECT_PATH=/home/ubuntu/thrillwiki
REPO_URL=https://github.com/YOUR_USERNAME/thrillwiki_django_no_react.git

Step 3: Local Machine Setup

3.1 Configure Webhook Listener

Create a configuration file for the webhook listener:

# Create environment file
cat > ***REMOVED***.webhook << EOF
WEBHOOK_PORT=9000
WEBHOOK_SECRET=your_github_webhook_secret
VM_HOST=VM_IP_ADDRESS
VM_PORT=22
VM_USER=ubuntu
VM_KEY_PATH=/home/your_user/.ssh/thrillwiki_vm
VM_PROJECT_PATH=/home/ubuntu/thrillwiki
REPO_URL=https://github.com/YOUR_USERNAME/thrillwiki_django_no_react.git
DEPLOY_BRANCH=main
EOF

3.2 Set up GitHub Webhook

1. Go to your GitHub repository
2. Navigate to Settings → Webhooks
3. Click "Add webhook"
4. Configure:
   • Payload URL: http://YOUR_PUBLIC_IP:9000/webhook
   • Content type: application/json
   • Secret: Your webhook secret
   • Events: Select "Just the push event"

Step 4: Database Setup

4.1 PostgreSQL Configuration

# On the VM
sudo -u postgres psql

-- Create database and user
CREATE DATABASE thrillwiki;
CREATE USER thrillwiki_user WITH ENCRYPTED PASSWORD 'your_password';
GRANT ALL PRIVILEGES ON DATABASE thrillwiki TO thrillwiki_user;
\q

# Install PostGIS (if using geographic features)
sudo apt install -y postgresql-postgis postgresql-postgis-scripts
sudo -u postgres psql -d thrillwiki -c "CREATE EXTENSION postgis;"

4.2 Run Initial Migration

# On the VM
cd /home/ubuntu/thrillwiki
uv run manage.py migrate
uv run manage.py collectstatic --noinput
uv run manage.py createsuperuser

Step 5: Start Services

5.1 Start VM Services

# On the VM
sudo systemctl start thrillwiki
sudo systemctl start thrillwiki-webhook
sudo systemctl status thrillwiki
sudo systemctl status thrillwiki-webhook

5.2 Start Local Webhook Listener

# On local machine
source ***REMOVED***.webhook
python3 scripts/webhook-listener.py

Step 6: Testing

6.1 Test Local Server

# Start local development server
./scripts/ci-start.sh

# Check if server is running
curl http://localhost:8000/health

6.2 Test VM Deployment

# On the VM, test deployment script
./scripts/vm-deploy.sh

# Check service status
./scripts/vm-deploy.sh status

# View logs
journalctl -u thrillwiki -f

6.3 Test Webhook

# Test webhook endpoint
curl -X GET http://localhost:9000/health

# Make a test commit and push to trigger deployment
git add .
git commit -m "Test deployment"
git push origin main

Monitoring and Logs

Service Logs

# View service logs
journalctl -u thrillwiki -f
journalctl -u thrillwiki-webhook -f

# View deployment logs
tail -f /home/ubuntu/thrillwiki/logs/deploy.log
tail -f /home/ubuntu/thrillwiki/logs/webhook.log

Health Checks

# Check services status
systemctl status thrillwiki
systemctl status thrillwiki-webhook

# Manual health check
curl http://localhost:8000/health
curl http://localhost:9000/health

Troubleshooting

Common Issues

1. Permission Denied

   # Fix file permissions
   chmod +x scripts/*.sh
   chown ubuntu:ubuntu -R /home/ubuntu/thrillwiki
   

2. Service Won't Start

   # Check service logs
   journalctl -u thrillwiki --no-pager
   
   # Verify paths in service files
   sudo systemctl edit thrillwiki
   

3. Webhook Not Triggering

   # Check webhook listener logs
   tail -f logs/webhook.log
   
   # Verify GitHub webhook configuration
   # Check firewall settings for port 9000
   

4. Database Connection Issues

   # Test database connection
   uv run manage.py dbshell
   
   # Check PostgreSQL status
   sudo systemctl status postgresql
   

Rollback Procedure

If deployment fails, you can rollback:

# On the VM
./scripts/vm-deploy.sh
# The script automatically handles rollback on failure

# Manual rollback to specific commit
cd /home/ubuntu/thrillwiki
git reset --hard COMMIT_HASH
./scripts/vm-deploy.sh restart

Security Considerations

1. SSH Keys: Use dedicated SSH keys for deployment
2. Webhook Secret: Use a strong, unique webhook secret
3. Firewall: Only open necessary ports (22, 8000, 9000)
4. User Permissions: Use dedicated deployment user with minimal privileges
5. Environment Variables: Store sensitive data in environment files, not in code

Maintenance

Regular Tasks

1. Update Dependencies: Run uv sync regularly
2. Log Rotation: Set up logrotate for application logs
3. Backup Database: Schedule regular database backups
4. Monitor Disk Space: Ensure sufficient space for logs and backups

Cleanup Old Backups

# The deployment script automatically cleans old backups
# Manual cleanup if needed:
find /home/ubuntu/thrillwiki/backups -name "backup_*.commit" -mtime +30 -delete

Performance Optimization

1. Use Production WSGI Server: Consider using Gunicorn instead of development server
2. Reverse Proxy: Set up Nginx as reverse proxy
3. Database Optimization: Configure PostgreSQL for production
4. Static Files: Serve static files through Nginx

This setup provides a robust CI/CD pipeline for automatic deployment of ThrillWiki to your Linux VM whenever code is pushed to GitHub.