thrilltrack-explorer/django/MIGRATION_PLAN.md

ThrillWiki Django Backend Migration Plan

🎯 Project Overview

Objective: Migrate ThrillWiki from Supabase backend to Django REST backend while preserving 100% of functionality.

Timeline: 12-16 weeks with 2 developers
Status: Foundation Phase - In Progress
Branch: django-backend

---

📊 Architecture Overview

Current Stack (Supabase)

• Frontend: React 18.3 + TypeScript + Vite + React Query
• Backend: Supabase (PostgreSQL + Edge Functions)
• Database: PostgreSQL with 80+ tables
• Auth: Supabase Auth (OAuth + MFA)
• Storage: CloudFlare Images
• Notifications: Novu Cloud
• Real-time: Supabase Realtime

Target Stack (Django)

• Frontend: React 18.3 + TypeScript + Vite (unchanged)
• Backend: Django 4.2 + django-ninja
• Database: PostgreSQL (migrated schema)
• Auth: Django + django-allauth + django-otp
• Storage: CloudFlare Images (unchanged)
• Notifications: Novu Cloud (unchanged)
• Real-time: Django Channels + WebSockets
• Tasks: Celery + Redis
• Caching: Redis + django-cacheops

---

🏗️ Project Structure

django/
├── manage.py
├── config/                      # Project settings
│   ├── settings/
│   │   ├── __init__.py
│   │   ├── base.py             # Shared settings
│   │   ├── local.py            # Development
│   │   └── production.py       # Production
│   ├── urls.py
│   ├── wsgi.py
│   └── asgi.py                 # For Channels
│
├── apps/
│   ├── core/                   # Base models, utilities
│   │   ├── models.py           # Abstract base models
│   │   ├── permissions.py      # Reusable permissions
│   │   ├── mixins.py           # Model mixins
│   │   └── utils.py
│   │
│   ├── entities/               # Parks, Rides, Companies
│   │   ├── models/
│   │   │   ├── park.py
│   │   │   ├── ride.py
│   │   │   ├── company.py
│   │   │   └── ride_model.py
│   │   ├── api/
│   │   │   ├── views.py
│   │   │   ├── serializers.py
│   │   │   └── filters.py
│   │   ├── services.py
│   │   └── tasks.py
│   │
│   ├── moderation/             # Content moderation
│   │   ├── models.py
│   │   ├── state_machine.py    # django-fsm workflow
│   │   ├── services.py
│   │   └── api/
│   │
│   ├── versioning/             # Entity versioning
│   │   ├── models.py
│   │   ├── signals.py
│   │   └── services.py
│   │
│   ├── users/                  # User management
│   │   ├── models.py
│   │   ├── managers.py
│   │   └── api/
│   │
│   ├── media/                  # Photo management
│   │   ├── models.py
│   │   ├── storage.py
│   │   └── tasks.py
│   │
│   └── notifications/          # Notification system
│       ├── models.py
│       ├── providers/
│       │   └── novu.py
│       └── tasks.py
│
├── api/
│   └── v1/
│       ├── router.py           # Main API router
│       └── schemas.py          # Pydantic schemas
│
└── scripts/
    ├── migrate_from_supabase.py
    └── validate_data.py

---

📋 Implementation Phases

✅ Phase 0: Foundation (CURRENT - Week 1)

• Create git branch django-backend
• Set up Python virtual environment
• Install all dependencies (Django 4.2, django-ninja, celery, etc.)
• Create Django project structure
• Create app directories
• Create .env.example
• Configure Django settings (base, local, production)
• Create base models and utilities
• Set up database connection
• Create initial migrations

Phase 1: Core Models (Week 2-3)

• Create abstract base models (TimeStamped, Versioned, etc.)
• Implement entity models (Park, Ride, Company, RideModel)
• Implement location models
• Implement user models with custom User
• Implement photo/media models
• Create Django migrations
• Test model relationships

Phase 2: Authentication System (Week 3-4)

• Set up django-allauth for OAuth (Google, Discord)
• Implement JWT authentication with djangorestframework-simplejwt
• Set up django-otp for MFA (TOTP)
• Create user registration/login endpoints
• Implement permission system (django-guardian)
• Create role-based access control
• Test authentication flow

Phase 3: Moderation System (Week 5-7)

• Create ContentSubmission and SubmissionItem models
• Implement django-fsm state machine
• Create ModerationService with atomic transactions
• Implement submission creation endpoints
• Implement approval/rejection endpoints
• Implement selective approval logic
• Create moderation queue API
• Add rate limiting with django-ratelimit
• Test moderation workflow end-to-end

Phase 4: Versioning System (Week 7-8)

• Create version models for all entities
• Implement django-lifecycle hooks for auto-versioning
• Create VersioningService
• Implement version history endpoints
• Add version diff functionality
• Test versioning with submissions

Phase 5: API Layer with django-ninja (Week 8-10)

• Set up django-ninja router
• Create Pydantic schemas for all entities
• Implement CRUD endpoints for parks
• Implement CRUD endpoints for rides
• Implement CRUD endpoints for companies
• Add filtering with django-filter
• Add search functionality
• Implement pagination
• Add API documentation (auto-generated)
• Test all endpoints

Phase 6: Celery Tasks (Week 10-11)

• Set up Celery with Redis
• Set up django-celery-beat for periodic tasks
• Migrate edge functions to Celery tasks:
  • cleanup_old_page_views
  • update_entity_view_counts
  • process_submission_notifications
  • generate_daily_stats
• Create notification tasks for Novu
• Set up Flower for monitoring
• Test async task execution

Phase 7: Real-time Features (Week 11-12)

• Set up Django Channels with Redis
• Create WebSocket consumers
• Implement moderation queue real-time updates
• Implement notification real-time delivery
• Test WebSocket connections
• OR: Implement Server-Sent Events as alternative

Phase 8: Caching & Performance (Week 12-13)

• Set up django-redis for caching
• Configure django-cacheops for automatic ORM caching
• Add cache invalidation logic
• Optimize database queries (select_related, prefetch_related)
• Add database indexes
• Profile with django-silk
• Load testing

Phase 9: Data Migration (Week 13-14)

• Export all data from Supabase
• Create migration script for entities
• Migrate user data (preserve UUIDs)
• Migrate submissions (pending only)
• Migrate version history
• Migrate photos/media references
• Validate data integrity
• Test with migrated data

Phase 10: Frontend Integration (Week 14-15)

• Create new API client (replace Supabase client)
• Update authentication logic
• Update all API calls to point to Django
• Update real-time subscriptions to WebSockets
• Test all user flows
• Fix any integration issues

Phase 11: Testing & QA (Week 15-16)

• Write unit tests for all models
• Write unit tests for all services
• Write API integration tests
• Write end-to-end tests
• Security audit
• Performance testing
• Load testing
• Bug fixes

Phase 12: Deployment (Week 16-17)

• Set up production environment
• Configure PostgreSQL
• Configure Redis
• Set up Celery workers
• Configure Gunicorn/Daphne
• Set up Docker containers
• Configure CI/CD
• Deploy to staging
• Final testing
• Deploy to production
• Monitor for issues

---

🔑 Key Technical Decisions

1. django-ninja vs Django REST Framework

Choice: django-ninja

• FastAPI-style syntax (modern, intuitive)
• Better performance
• Automatic OpenAPI documentation
• Pydantic integration for validation

2. State Machine for Moderation

Choice: django-fsm

• Declarative state transitions
• Built-in guards and conditions
• Prevents invalid state changes
• Easy to visualize workflow

3. Auto-versioning Strategy

Choice: django-lifecycle hooks

• Automatic version creation on model changes
• No manual intervention needed
• Tracks what changed
• Preserves full history

4. Real-time Communication

Primary: Django Channels (WebSockets) Fallback: Server-Sent Events (SSE)

• WebSockets for bidirectional communication
• SSE as simpler alternative
• Redis channel layer for scaling

5. Caching Strategy

Tool: django-cacheops

• Automatic ORM query caching
• Transparent invalidation
• Minimal code changes
• Redis backend for consistency

---

🚀 Critical Features to Preserve

1. Moderation System

• ✅ Atomic transactions for approvals
• ✅ Selective approval (approve individual items)
• ✅ State machine workflow (pending → reviewing → approved/rejected)
• ✅ Lock mechanism (15-minute lock on review)
• ✅ Automatic unlock on timeout
• ✅ Batch operations

2. Versioning System

• ✅ Full version history for all entities
• ✅ Track who made changes
• ✅ Track what changed
• ✅ Link versions to submissions
• ✅ Version diffs
• ✅ Rollback capability

3. Authentication

• ✅ Password-based login
• ✅ Google OAuth
• ✅ Discord OAuth
• ✅ Two-factor authentication (TOTP)
• ✅ Session management
• ✅ JWT tokens for API

4. Permissions & Security

• ✅ Role-based access control (user, moderator, admin, superuser)
• ✅ Object-level permissions
• ✅ Rate limiting
• ✅ CORS configuration
• ✅ Brute force protection

5. Image Management

• ✅ CloudFlare direct upload
• ✅ Image validation
• ✅ Image metadata storage
• ✅ Multiple image variants (thumbnails, etc.)

6. Notifications

• ✅ Email notifications via Novu
• ✅ In-app notifications
• ✅ Notification templates
• ✅ User preferences

7. Search & Filtering

• ✅ Full-text search
• ✅ Advanced filtering
• ✅ Sorting options
• ✅ Pagination

---

📊 Database Schema Preservation

Core Entity Tables (Must Migrate)

✅ parks (80+ fields including dates, locations, operators)
✅ rides (100+ fields including ride_models, parks, manufacturers)
✅ companies (manufacturers, operators, designers)
✅ ride_models (coaster models, flat ride models)
✅ locations (countries, subdivisions, localities)
✅ profiles (user profiles linked to auth.users)
✅ user_roles (role assignments)
✅ content_submissions (moderation queue)
✅ submission_items (individual changes in submissions)
✅ park_versions, ride_versions, etc. (version history)
✅ photos (image metadata)
✅ photo_submissions (photo approval queue)
✅ reviews (user reviews)
✅ reports (user reports)
✅ entity_timeline_events (history timeline)
✅ notification_logs
✅ notification_templates

Computed Fields Strategy

Some Supabase tables have computed fields. Options:

1. Cache in model (recommended for frequently accessed)
2. Property method (for rarely accessed)
3. Cached query (using django-cacheops)

Example:

class Park(models.Model):
    # Cached computed fields
    ride_count = models.IntegerField(default=0)
    coaster_count = models.IntegerField(default=0)
    
    def update_counts(self):
        """Update cached counts"""
        self.ride_count = self.rides.count()
        self.coaster_count = self.rides.filter(
            is_coaster=True
        ).count()
        self.save()

---

🔧 Development Setup

Prerequisites

# System requirements
Python 3.11+
PostgreSQL 15+
Redis 7+
Node.js 18+ (for frontend)

Initial Setup

# 1. Clone and checkout branch
git checkout django-backend

# 2. Set up Python environment
cd django
python3 -m venv venv
source venv/bin/activate

# 3. Install dependencies
pip install -r requirements/local.txt

# 4. Set up environment
cp .env.example .env
# Edit .env with your credentials

# 5. Run migrations
python manage.py migrate

# 6. Create superuser
python manage.py createsuperuser

# 7. Run development server
python manage.py runserver

# 8. Run Celery worker (separate terminal)
celery -A config worker -l info

# 9. Run Celery beat (separate terminal)
celery -A config beat -l info

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=apps --cov-report=html

# Run specific test file
pytest apps/moderation/tests/test_services.py

---

📝 Edge Functions to Migrate

Supabase Edge Functions → Django/Celery

Edge Function	Django Implementation	Priority
process-submission	ModerationService.submit()	P0
process-selective-approval	ModerationService.approve()	P0
reject-submission	ModerationService.reject()	P0
unlock-submission	Celery periodic task	P0
cleanup_old_page_views	Celery periodic task	P1
update_entity_view_counts	Celery periodic task	P1
send-notification	NotificationService.send()	P0
process-photo-submission	MediaService.submit_photo()	P1
generate-daily-stats	Celery periodic task	P2

---

🎯 Success Criteria

Must Have (P0)

• ✅ All 80+ database tables migrated
• ✅ All user data preserved (with UUIDs)
• ✅ Authentication working (password + OAuth + MFA)
• ✅ Moderation workflow functional
• ✅ Versioning system working
• ✅ All API endpoints functional
• ✅ Frontend fully integrated
• ✅ No data loss during migration
• ✅ Performance equivalent or better

Should Have (P1)

• ✅ Real-time updates working
• ✅ All Celery tasks running
• ✅ Caching operational
• ✅ Image uploads working
• ✅ Notifications working
• ✅ Search functional
• ✅ Comprehensive test coverage (>80%)

Nice to Have (P2)

• Admin dashboard improvements
• Enhanced monitoring/observability
• API rate limiting per user
• Advanced analytics
• GraphQL endpoint (optional)

---

🚨 Risk Mitigation

Risk 1: Data Loss During Migration

Mitigation:

• Comprehensive backup before migration
• Dry-run migration multiple times
• Validation scripts to check data integrity
• Rollback plan

Risk 2: Downtime During Cutover

Mitigation:

• Blue-green deployment strategy
• Run both systems in parallel briefly
• Feature flags to toggle between backends
• Quick rollback capability

Risk 3: Performance Degradation

Mitigation:

• Load testing before production
• Database query optimization
• Aggressive caching strategy
• Monitoring and alerting

Risk 4: Missing Edge Cases

Mitigation:

• Comprehensive test suite
• Manual QA testing
• Beta testing period
• Staged rollout

---

📞 Support & Resources

Documentation

• Django: https://docs.djangoproject.com/
• django-ninja: https://django-ninja.rest-framework.com/
• Celery: https://docs.celeryq.dev/
• Django Channels: https://channels.readthedocs.io/

Key Files to Reference

• Original database schema: supabase/migrations/
• Current API endpoints: src/lib/supabaseClient.ts
• Moderation logic: src/components/moderation/
• Existing docs: docs/moderation/, docs/versioning/

---

🎉 Next Steps

1. Immediate (This Week):

   • Configure Django settings
   • Create base models
   • Set up database connection

2. Short-term (Next 2 Weeks):

   • Implement entity models
   • Set up authentication
   • Create basic API endpoints

3. Medium-term (Next 4-8 Weeks):

   • Build moderation system
   • Implement versioning
   • Migrate edge functions

4. Long-term (8-16 Weeks):

   • Complete API layer
   • Frontend integration
   • Testing and deployment

---

Last Updated: November 8, 2025
Status: Foundation Phase - Dependencies Installed, Structure Created
Next: Configure Django settings and create base models