thrilltrack-explorer/django-backend/MIGRATION_PLAN.md

# ThrillWiki Django Backend Migration Plan

## 🎯 Project Overview

**Objective**: Migrate ThrillWiki from Supabase backend to Django REST backend while preserving 100% of functionality.

**Timeline**: 12-16 weeks with 2 developers  
**Status**: Foundation Phase - In Progress  
**Branch**: `django-backend`

---

## 📊 Architecture Overview

### Current Stack (Supabase)
- **Frontend**: React 18.3 + TypeScript + Vite + React Query
- **Backend**: Supabase (PostgreSQL + Edge Functions)
- **Database**: PostgreSQL with 80+ tables
- **Auth**: Supabase Auth (OAuth + MFA)
- **Storage**: CloudFlare Images
- **Notifications**: Novu Cloud
- **Real-time**: Supabase Realtime

### Target Stack (Django)
- **Frontend**: React 18.3 + TypeScript + Vite (unchanged)
- **Backend**: Django 4.2 + django-ninja
- **Database**: PostgreSQL (migrated schema)
- **Auth**: Django + django-allauth + django-otp
- **Storage**: CloudFlare Images (unchanged)
- **Notifications**: Novu Cloud (unchanged)
- **Real-time**: Django Channels + WebSockets
- **Tasks**: Celery + Redis
- **Caching**: Redis + django-cacheops

---

## 🏗️ Project Structure

```
django/
├── manage.py
├── config/                      # Project settings
│   ├── settings/
│   │   ├── __init__.py
│   │   ├── base.py             # Shared settings
│   │   ├── local.py            # Development
│   │   └── production.py       # Production
│   ├── urls.py
│   ├── wsgi.py
│   └── asgi.py                 # For Channels
│
├── apps/
│   ├── core/                   # Base models, utilities
│   │   ├── models.py           # Abstract base models
│   │   ├── permissions.py      # Reusable permissions
│   │   ├── mixins.py           # Model mixins
│   │   └── utils.py
│   │
│   ├── entities/               # Parks, Rides, Companies
│   │   ├── models/
│   │   │   ├── park.py
│   │   │   ├── ride.py
│   │   │   ├── company.py
│   │   │   └── ride_model.py
│   │   ├── api/
│   │   │   ├── views.py
│   │   │   ├── serializers.py
│   │   │   └── filters.py
│   │   ├── services.py
│   │   └── tasks.py
│   │
│   ├── moderation/             # Content moderation
│   │   ├── models.py
│   │   ├── state_machine.py    # django-fsm workflow
│   │   ├── services.py
│   │   └── api/
│   │
│   ├── versioning/             # Entity versioning
│   │   ├── models.py
│   │   ├── signals.py
│   │   └── services.py
│   │
│   ├── users/                  # User management
│   │   ├── models.py
│   │   ├── managers.py
│   │   └── api/
│   │
│   ├── media/                  # Photo management
│   │   ├── models.py
│   │   ├── storage.py
│   │   └── tasks.py
│   │
│   └── notifications/          # Notification system
│       ├── models.py
│       ├── providers/
│       │   └── novu.py
│       └── tasks.py
│
├── api/
│   └── v1/
│       ├── router.py           # Main API router
│       └── schemas.py          # Pydantic schemas
│
└── scripts/
    ├── migrate_from_supabase.py
    └── validate_data.py
```

---

## 📋 Implementation Phases

### ✅ Phase 0: Foundation (CURRENT - Week 1)
- [x] Create git branch `django-backend`
- [x] Set up Python virtual environment
- [x] Install all dependencies (Django 4.2, django-ninja, celery, etc.)
- [x] Create Django project structure
- [x] Create app directories
- [x] Create .env.example
- [ ] Configure Django settings (base, local, production)
- [ ] Create base models and utilities
- [ ] Set up database connection
- [ ] Create initial migrations

### Phase 1: Core Models (Week 2-3)
- [ ] Create abstract base models (TimeStamped, Versioned, etc.)
- [ ] Implement entity models (Park, Ride, Company, RideModel)
- [ ] Implement location models
- [ ] Implement user models with custom User
- [ ] Implement photo/media models
- [ ] Create Django migrations
- [ ] Test model relationships

### Phase 2: Authentication System (Week 3-4)
- [ ] Set up django-allauth for OAuth (Google, Discord)
- [ ] Implement JWT authentication with djangorestframework-simplejwt
- [ ] Set up django-otp for MFA (TOTP)
- [ ] Create user registration/login endpoints
- [ ] Implement permission system (django-guardian)
- [ ] Create role-based access control
- [ ] Test authentication flow

### Phase 3: Moderation System (Week 5-7)
- [ ] Create ContentSubmission and SubmissionItem models
- [ ] Implement django-fsm state machine
- [ ] Create ModerationService with atomic transactions
- [ ] Implement submission creation endpoints
- [ ] Implement approval/rejection endpoints
- [ ] Implement selective approval logic
- [ ] Create moderation queue API
- [ ] Add rate limiting with django-ratelimit
- [ ] Test moderation workflow end-to-end

### Phase 4: Versioning System (Week 7-8)
- [ ] Create version models for all entities
- [ ] Implement django-lifecycle hooks for auto-versioning
- [ ] Create VersioningService
- [ ] Implement version history endpoints
- [ ] Add version diff functionality
- [ ] Test versioning with submissions

### Phase 5: API Layer with django-ninja (Week 8-10)
- [ ] Set up django-ninja router
- [ ] Create Pydantic schemas for all entities
- [ ] Implement CRUD endpoints for parks
- [ ] Implement CRUD endpoints for rides
- [ ] Implement CRUD endpoints for companies
- [ ] Add filtering with django-filter
- [ ] Add search functionality
- [ ] Implement pagination
- [ ] Add API documentation (auto-generated)
- [ ] Test all endpoints

### Phase 6: Celery Tasks (Week 10-11)
- [ ] Set up Celery with Redis
- [ ] Set up django-celery-beat for periodic tasks
- [ ] Migrate edge functions to Celery tasks:
  - [ ] cleanup_old_page_views
  - [ ] update_entity_view_counts
  - [ ] process_submission_notifications
  - [ ] generate_daily_stats
- [ ] Create notification tasks for Novu
- [ ] Set up Flower for monitoring
- [ ] Test async task execution

### Phase 7: Real-time Features (Week 11-12)
- [ ] Set up Django Channels with Redis
- [ ] Create WebSocket consumers
- [ ] Implement moderation queue real-time updates
- [ ] Implement notification real-time delivery
- [ ] Test WebSocket connections
- [ ] OR: Implement Server-Sent Events as alternative

### Phase 8: Caching & Performance (Week 12-13)
- [ ] Set up django-redis for caching
- [ ] Configure django-cacheops for automatic ORM caching
- [ ] Add cache invalidation logic
- [ ] Optimize database queries (select_related, prefetch_related)
- [ ] Add database indexes
- [ ] Profile with django-silk
- [ ] Load testing

### Phase 9: Data Migration (Week 13-14)
- [ ] Export all data from Supabase
- [ ] Create migration script for entities
- [ ] Migrate user data (preserve UUIDs)
- [ ] Migrate submissions (pending only)
- [ ] Migrate version history
- [ ] Migrate photos/media references
- [ ] Validate data integrity
- [ ] Test with migrated data

### Phase 10: Frontend Integration (Week 14-15)
- [ ] Create new API client (replace Supabase client)
- [ ] Update authentication logic
- [ ] Update all API calls to point to Django
- [ ] Update real-time subscriptions to WebSockets
- [ ] Test all user flows
- [ ] Fix any integration issues

### Phase 11: Testing & QA (Week 15-16)
- [ ] Write unit tests for all models
- [ ] Write unit tests for all services
- [ ] Write API integration tests
- [ ] Write end-to-end tests
- [ ] Security audit
- [ ] Performance testing
- [ ] Load testing
- [ ] Bug fixes

### Phase 12: Deployment (Week 16-17)
- [ ] Set up production environment
- [ ] Configure PostgreSQL
- [ ] Configure Redis
- [ ] Set up Celery workers
- [ ] Configure Gunicorn/Daphne
- [ ] Set up Docker containers
- [ ] Configure CI/CD
- [ ] Deploy to staging
- [ ] Final testing
- [ ] Deploy to production
- [ ] Monitor for issues

---

## 🔑 Key Technical Decisions

### 1. **django-ninja vs Django REST Framework**
**Choice**: django-ninja
- FastAPI-style syntax (modern, intuitive)
- Better performance
- Automatic OpenAPI documentation
- Pydantic integration for validation

### 2. **State Machine for Moderation**
**Choice**: django-fsm
- Declarative state transitions
- Built-in guards and conditions
- Prevents invalid state changes
- Easy to visualize workflow

### 3. **Auto-versioning Strategy**
**Choice**: django-lifecycle hooks
- Automatic version creation on model changes
- No manual intervention needed
- Tracks what changed
- Preserves full history

### 4. **Real-time Communication**
**Primary**: Django Channels (WebSockets)
**Fallback**: Server-Sent Events (SSE)
- WebSockets for bidirectional communication
- SSE as simpler alternative
- Redis channel layer for scaling

### 5. **Caching Strategy**
**Tool**: django-cacheops
- Automatic ORM query caching
- Transparent invalidation
- Minimal code changes
- Redis backend for consistency

---

## 🚀 Critical Features to Preserve

### 1. **Moderation System**
- ✅ Atomic transactions for approvals
- ✅ Selective approval (approve individual items)
- ✅ State machine workflow (pending → reviewing → approved/rejected)
- ✅ Lock mechanism (15-minute lock on review)
- ✅ Automatic unlock on timeout
- ✅ Batch operations

### 2. **Versioning System**
- ✅ Full version history for all entities
- ✅ Track who made changes
- ✅ Track what changed
- ✅ Link versions to submissions
- ✅ Version diffs
- ✅ Rollback capability

### 3. **Authentication**
- ✅ Password-based login
- ✅ Google OAuth
- ✅ Discord OAuth
- ✅ Two-factor authentication (TOTP)
- ✅ Session management
- ✅ JWT tokens for API

### 4. **Permissions & Security**
- ✅ Role-based access control (user, moderator, admin, superuser)
- ✅ Object-level permissions
- ✅ Rate limiting
- ✅ CORS configuration
- ✅ Brute force protection

### 5. **Image Management**
- ✅ CloudFlare direct upload
- ✅ Image validation
- ✅ Image metadata storage
- ✅ Multiple image variants (thumbnails, etc.)

### 6. **Notifications**
- ✅ Email notifications via Novu
- ✅ In-app notifications
- ✅ Notification templates
- ✅ User preferences

### 7. **Search & Filtering**
- ✅ Full-text search
- ✅ Advanced filtering
- ✅ Sorting options
- ✅ Pagination

---

## 📊 Database Schema Preservation

### Core Entity Tables (Must Migrate)
```
✅ parks (80+ fields including dates, locations, operators)
✅ rides (100+ fields including ride_models, parks, manufacturers)
✅ companies (manufacturers, operators, designers)
✅ ride_models (coaster models, flat ride models)
✅ locations (countries, subdivisions, localities)
✅ profiles (user profiles linked to auth.users)
✅ user_roles (role assignments)
✅ content_submissions (moderation queue)
✅ submission_items (individual changes in submissions)
✅ park_versions, ride_versions, etc. (version history)
✅ photos (image metadata)
✅ photo_submissions (photo approval queue)
✅ reviews (user reviews)
✅ reports (user reports)
✅ entity_timeline_events (history timeline)
✅ notification_logs
✅ notification_templates
```

### Computed Fields Strategy
Some Supabase tables have computed fields. Options:
1. **Cache in model** (recommended for frequently accessed)
2. **Property method** (for rarely accessed)
3. **Cached query** (using django-cacheops)

Example:
```python
class Park(models.Model):
    # Cached computed fields
    ride_count = models.IntegerField(default=0)
    coaster_count = models.IntegerField(default=0)
    
    def update_counts(self):
        """Update cached counts"""
        self.ride_count = self.rides.count()
        self.coaster_count = self.rides.filter(
            is_coaster=True
        ).count()
        self.save()
```

---

## 🔧 Development Setup

### Prerequisites
```bash
# System requirements
Python 3.11+
PostgreSQL 15+
Redis 7+
Node.js 18+ (for frontend)
```

### Initial Setup
```bash
# 1. Clone and checkout branch
git checkout django-backend

# 2. Set up Python environment
cd django
python3 -m venv venv
source venv/bin/activate

# 3. Install dependencies
pip install -r requirements/local.txt

# 4. Set up environment
cp .env.example .env
# Edit .env with your credentials

# 5. Run migrations
python manage.py migrate

# 6. Create superuser
python manage.py createsuperuser

# 7. Run development server
python manage.py runserver

# 8. Run Celery worker (separate terminal)
celery -A config worker -l info

# 9. Run Celery beat (separate terminal)
celery -A config beat -l info
```

### Running Tests
```bash
# Run all tests
pytest

# Run with coverage
pytest --cov=apps --cov-report=html

# Run specific test file
pytest apps/moderation/tests/test_services.py
```

---

## 📝 Edge Functions to Migrate

### Supabase Edge Functions → Django/Celery

| Edge Function | Django Implementation | Priority |
|---------------|----------------------|----------|
| `process-submission` | `ModerationService.submit()` | P0 |
| `process-selective-approval` | `ModerationService.approve()` | P0 |
| `reject-submission` | `ModerationService.reject()` | P0 |
| `unlock-submission` | Celery periodic task | P0 |
| `cleanup_old_page_views` | Celery periodic task | P1 |
| `update_entity_view_counts` | Celery periodic task | P1 |
| `send-notification` | `NotificationService.send()` | P0 |
| `process-photo-submission` | `MediaService.submit_photo()` | P1 |
| `generate-daily-stats` | Celery periodic task | P2 |

---

## 🎯 Success Criteria

### Must Have (P0)
- ✅ All 80+ database tables migrated
- ✅ All user data preserved (with UUIDs)
- ✅ Authentication working (password + OAuth + MFA)
- ✅ Moderation workflow functional
- ✅ Versioning system working
- ✅ All API endpoints functional
- ✅ Frontend fully integrated
- ✅ No data loss during migration
- ✅ Performance equivalent or better

### Should Have (P1)
- ✅ Real-time updates working
- ✅ All Celery tasks running
- ✅ Caching operational
- ✅ Image uploads working
- ✅ Notifications working
- ✅ Search functional
- ✅ Comprehensive test coverage (>80%)

### Nice to Have (P2)
- Admin dashboard improvements
- Enhanced monitoring/observability
- API rate limiting per user
- Advanced analytics
- GraphQL endpoint (optional)

---

## 🚨 Risk Mitigation

### Risk 1: Data Loss During Migration
**Mitigation**: 
- Comprehensive backup before migration
- Dry-run migration multiple times
- Validation scripts to check data integrity
- Rollback plan

### Risk 2: Downtime During Cutover
**Mitigation**:
- Blue-green deployment strategy
- Run both systems in parallel briefly
- Feature flags to toggle between backends
- Quick rollback capability

### Risk 3: Performance Degradation
**Mitigation**:
- Load testing before production
- Database query optimization
- Aggressive caching strategy
- Monitoring and alerting

### Risk 4: Missing Edge Cases
**Mitigation**:
- Comprehensive test suite
- Manual QA testing
- Beta testing period
- Staged rollout

---

## 📞 Support & Resources

### Documentation
- Django: https://docs.djangoproject.com/
- django-ninja: https://django-ninja.rest-framework.com/
- Celery: https://docs.celeryq.dev/
- Django Channels: https://channels.readthedocs.io/

### Key Files to Reference
- Original database schema: `supabase/migrations/`
- Current API endpoints: `src/lib/supabaseClient.ts`
- Moderation logic: `src/components/moderation/`
- Existing docs: `docs/moderation/`, `docs/versioning/`

---

## 🎉 Next Steps

1. **Immediate** (This Week):
   - Configure Django settings
   - Create base models
   - Set up database connection
   
2. **Short-term** (Next 2 Weeks):
   - Implement entity models
   - Set up authentication
   - Create basic API endpoints
   
3. **Medium-term** (Next 4-8 Weeks):
   - Build moderation system
   - Implement versioning
   - Migrate edge functions
   
4. **Long-term** (8-16 Weeks):
   - Complete API layer
   - Frontend integration
   - Testing and deployment

---

**Last Updated**: November 8, 2025  
**Status**: Foundation Phase - Dependencies Installed, Structure Created  
**Next**: Configure Django settings and create base models