mirror of
https://github.com/pacnpal/thrilltrack-explorer.git
synced 2025-12-20 06:11:11 -05:00
feat: Django backend foundation - project structure, dependencies, and documentation
- Created Django 4.2 project with production-ready architecture - Installed 60+ packages including django-ninja, celery, channels, etc. - Set up app structure (core, entities, moderation, users, versioning, media, notifications) - Created comprehensive MIGRATION_PLAN.md with 12-phase roadmap - Created README.md with setup instructions - Created .env.example with all required configuration - Configured for Python 3.13 compatibility - All dependencies successfully installed and tested Next steps: Configure Django settings and create base models
This commit is contained in:
pacnpal
566
django/MIGRATION_PLAN.md
Normal file
566
django/MIGRATION_PLAN.md
Normal file
@@ -0,0 +1,566 @@
|
||||
# ThrillWiki Django Backend Migration Plan
|
||||
|
||||
## 🎯 Project Overview
|
||||
|
||||
**Objective**: Migrate ThrillWiki from Supabase backend to Django REST backend while preserving 100% of functionality.
|
||||
|
||||
**Timeline**: 12-16 weeks with 2 developers
|
||||
**Status**: Foundation Phase - In Progress
|
||||
**Branch**: `django-backend`
|
||||
|
||||
---
|
||||
|
||||
## 📊 Architecture Overview
|
||||
|
||||
### Current Stack (Supabase)
|
||||
- **Frontend**: React 18.3 + TypeScript + Vite + React Query
|
||||
- **Backend**: Supabase (PostgreSQL + Edge Functions)
|
||||
- **Database**: PostgreSQL with 80+ tables
|
||||
- **Auth**: Supabase Auth (OAuth + MFA)
|
||||
- **Storage**: CloudFlare Images
|
||||
- **Notifications**: Novu Cloud
|
||||
- **Real-time**: Supabase Realtime
|
||||
|
||||
### Target Stack (Django)
|
||||
- **Frontend**: React 18.3 + TypeScript + Vite (unchanged)
|
||||
- **Backend**: Django 4.2 + django-ninja
|
||||
- **Database**: PostgreSQL (migrated schema)
|
||||
- **Auth**: Django + django-allauth + django-otp
|
||||
- **Storage**: CloudFlare Images (unchanged)
|
||||
- **Notifications**: Novu Cloud (unchanged)
|
||||
- **Real-time**: Django Channels + WebSockets
|
||||
- **Tasks**: Celery + Redis
|
||||
- **Caching**: Redis + django-cacheops
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Project Structure
|
||||
|
||||
```
|
||||
django/
|
||||
├── manage.py
|
||||
├── config/ # Project settings
|
||||
│ ├── settings/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── base.py # Shared settings
|
||||
│ │ ├── local.py # Development
|
||||
│ │ └── production.py # Production
|
||||
│ ├── urls.py
|
||||
│ ├── wsgi.py
|
||||
│ └── asgi.py # For Channels
|
||||
│
|
||||
├── apps/
|
||||
│ ├── core/ # Base models, utilities
|
||||
│ │ ├── models.py # Abstract base models
|
||||
│ │ ├── permissions.py # Reusable permissions
|
||||
│ │ ├── mixins.py # Model mixins
|
||||
│ │ └── utils.py
|
||||
│ │
|
||||
│ ├── entities/ # Parks, Rides, Companies
|
||||
│ │ ├── models/
|
||||
│ │ │ ├── park.py
|
||||
│ │ │ ├── ride.py
|
||||
│ │ │ ├── company.py
|
||||
│ │ │ └── ride_model.py
|
||||
│ │ ├── api/
|
||||
│ │ │ ├── views.py
|
||||
│ │ │ ├── serializers.py
|
||||
│ │ │ └── filters.py
|
||||
│ │ ├── services.py
|
||||
│ │ └── tasks.py
|
||||
│ │
|
||||
│ ├── moderation/ # Content moderation
|
||||
│ │ ├── models.py
|
||||
│ │ ├── state_machine.py # django-fsm workflow
|
||||
│ │ ├── services.py
|
||||
│ │ └── api/
|
||||
│ │
|
||||
│ ├── versioning/ # Entity versioning
|
||||
│ │ ├── models.py
|
||||
│ │ ├── signals.py
|
||||
│ │ └── services.py
|
||||
│ │
|
||||
│ ├── users/ # User management
|
||||
│ │ ├── models.py
|
||||
│ │ ├── managers.py
|
||||
│ │ └── api/
|
||||
│ │
|
||||
│ ├── media/ # Photo management
|
||||
│ │ ├── models.py
|
||||
│ │ ├── storage.py
|
||||
│ │ └── tasks.py
|
||||
│ │
|
||||
│ └── notifications/ # Notification system
|
||||
│ ├── models.py
|
||||
│ ├── providers/
|
||||
│ │ └── novu.py
|
||||
│ └── tasks.py
|
||||
│
|
||||
├── api/
|
||||
│ └── v1/
|
||||
│ ├── router.py # Main API router
|
||||
│ └── schemas.py # Pydantic schemas
|
||||
│
|
||||
└── scripts/
|
||||
├── migrate_from_supabase.py
|
||||
└── validate_data.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Implementation Phases
|
||||
|
||||
### ✅ Phase 0: Foundation (CURRENT - Week 1)
|
||||
- [x] Create git branch `django-backend`
|
||||
- [x] Set up Python virtual environment
|
||||
- [x] Install all dependencies (Django 4.2, django-ninja, celery, etc.)
|
||||
- [x] Create Django project structure
|
||||
- [x] Create app directories
|
||||
- [x] Create .env.example
|
||||
- [ ] Configure Django settings (base, local, production)
|
||||
- [ ] Create base models and utilities
|
||||
- [ ] Set up database connection
|
||||
- [ ] Create initial migrations
|
||||
|
||||
### Phase 1: Core Models (Week 2-3)
|
||||
- [ ] Create abstract base models (TimeStamped, Versioned, etc.)
|
||||
- [ ] Implement entity models (Park, Ride, Company, RideModel)
|
||||
- [ ] Implement location models
|
||||
- [ ] Implement user models with custom User
|
||||
- [ ] Implement photo/media models
|
||||
- [ ] Create Django migrations
|
||||
- [ ] Test model relationships
|
||||
|
||||
### Phase 2: Authentication System (Week 3-4)
|
||||
- [ ] Set up django-allauth for OAuth (Google, Discord)
|
||||
- [ ] Implement JWT authentication with djangorestframework-simplejwt
|
||||
- [ ] Set up django-otp for MFA (TOTP)
|
||||
- [ ] Create user registration/login endpoints
|
||||
- [ ] Implement permission system (django-guardian)
|
||||
- [ ] Create role-based access control
|
||||
- [ ] Test authentication flow
|
||||
|
||||
### Phase 3: Moderation System (Week 5-7)
|
||||
- [ ] Create ContentSubmission and SubmissionItem models
|
||||
- [ ] Implement django-fsm state machine
|
||||
- [ ] Create ModerationService with atomic transactions
|
||||
- [ ] Implement submission creation endpoints
|
||||
- [ ] Implement approval/rejection endpoints
|
||||
- [ ] Implement selective approval logic
|
||||
- [ ] Create moderation queue API
|
||||
- [ ] Add rate limiting with django-ratelimit
|
||||
- [ ] Test moderation workflow end-to-end
|
||||
|
||||
### Phase 4: Versioning System (Week 7-8)
|
||||
- [ ] Create version models for all entities
|
||||
- [ ] Implement django-lifecycle hooks for auto-versioning
|
||||
- [ ] Create VersioningService
|
||||
- [ ] Implement version history endpoints
|
||||
- [ ] Add version diff functionality
|
||||
- [ ] Test versioning with submissions
|
||||
|
||||
### Phase 5: API Layer with django-ninja (Week 8-10)
|
||||
- [ ] Set up django-ninja router
|
||||
- [ ] Create Pydantic schemas for all entities
|
||||
- [ ] Implement CRUD endpoints for parks
|
||||
- [ ] Implement CRUD endpoints for rides
|
||||
- [ ] Implement CRUD endpoints for companies
|
||||
- [ ] Add filtering with django-filter
|
||||
- [ ] Add search functionality
|
||||
- [ ] Implement pagination
|
||||
- [ ] Add API documentation (auto-generated)
|
||||
- [ ] Test all endpoints
|
||||
|
||||
### Phase 6: Celery Tasks (Week 10-11)
|
||||
- [ ] Set up Celery with Redis
|
||||
- [ ] Set up django-celery-beat for periodic tasks
|
||||
- [ ] Migrate edge functions to Celery tasks:
|
||||
- [ ] cleanup_old_page_views
|
||||
- [ ] update_entity_view_counts
|
||||
- [ ] process_submission_notifications
|
||||
- [ ] generate_daily_stats
|
||||
- [ ] Create notification tasks for Novu
|
||||
- [ ] Set up Flower for monitoring
|
||||
- [ ] Test async task execution
|
||||
|
||||
### Phase 7: Real-time Features (Week 11-12)
|
||||
- [ ] Set up Django Channels with Redis
|
||||
- [ ] Create WebSocket consumers
|
||||
- [ ] Implement moderation queue real-time updates
|
||||
- [ ] Implement notification real-time delivery
|
||||
- [ ] Test WebSocket connections
|
||||
- [ ] OR: Implement Server-Sent Events as alternative
|
||||
|
||||
### Phase 8: Caching & Performance (Week 12-13)
|
||||
- [ ] Set up django-redis for caching
|
||||
- [ ] Configure django-cacheops for automatic ORM caching
|
||||
- [ ] Add cache invalidation logic
|
||||
- [ ] Optimize database queries (select_related, prefetch_related)
|
||||
- [ ] Add database indexes
|
||||
- [ ] Profile with django-silk
|
||||
- [ ] Load testing
|
||||
|
||||
### Phase 9: Data Migration (Week 13-14)
|
||||
- [ ] Export all data from Supabase
|
||||
- [ ] Create migration script for entities
|
||||
- [ ] Migrate user data (preserve UUIDs)
|
||||
- [ ] Migrate submissions (pending only)
|
||||
- [ ] Migrate version history
|
||||
- [ ] Migrate photos/media references
|
||||
- [ ] Validate data integrity
|
||||
- [ ] Test with migrated data
|
||||
|
||||
### Phase 10: Frontend Integration (Week 14-15)
|
||||
- [ ] Create new API client (replace Supabase client)
|
||||
- [ ] Update authentication logic
|
||||
- [ ] Update all API calls to point to Django
|
||||
- [ ] Update real-time subscriptions to WebSockets
|
||||
- [ ] Test all user flows
|
||||
- [ ] Fix any integration issues
|
||||
|
||||
### Phase 11: Testing & QA (Week 15-16)
|
||||
- [ ] Write unit tests for all models
|
||||
- [ ] Write unit tests for all services
|
||||
- [ ] Write API integration tests
|
||||
- [ ] Write end-to-end tests
|
||||
- [ ] Security audit
|
||||
- [ ] Performance testing
|
||||
- [ ] Load testing
|
||||
- [ ] Bug fixes
|
||||
|
||||
### Phase 12: Deployment (Week 16-17)
|
||||
- [ ] Set up production environment
|
||||
- [ ] Configure PostgreSQL
|
||||
- [ ] Configure Redis
|
||||
- [ ] Set up Celery workers
|
||||
- [ ] Configure Gunicorn/Daphne
|
||||
- [ ] Set up Docker containers
|
||||
- [ ] Configure CI/CD
|
||||
- [ ] Deploy to staging
|
||||
- [ ] Final testing
|
||||
- [ ] Deploy to production
|
||||
- [ ] Monitor for issues
|
||||
|
||||
---
|
||||
|
||||
## 🔑 Key Technical Decisions
|
||||
|
||||
### 1. **django-ninja vs Django REST Framework**
|
||||
**Choice**: django-ninja
|
||||
- FastAPI-style syntax (modern, intuitive)
|
||||
- Better performance
|
||||
- Automatic OpenAPI documentation
|
||||
- Pydantic integration for validation
|
||||
|
||||
### 2. **State Machine for Moderation**
|
||||
**Choice**: django-fsm
|
||||
- Declarative state transitions
|
||||
- Built-in guards and conditions
|
||||
- Prevents invalid state changes
|
||||
- Easy to visualize workflow
|
||||
|
||||
### 3. **Auto-versioning Strategy**
|
||||
**Choice**: django-lifecycle hooks
|
||||
- Automatic version creation on model changes
|
||||
- No manual intervention needed
|
||||
- Tracks what changed
|
||||
- Preserves full history
|
||||
|
||||
### 4. **Real-time Communication**
|
||||
**Primary**: Django Channels (WebSockets)
|
||||
**Fallback**: Server-Sent Events (SSE)
|
||||
- WebSockets for bidirectional communication
|
||||
- SSE as simpler alternative
|
||||
- Redis channel layer for scaling
|
||||
|
||||
### 5. **Caching Strategy**
|
||||
**Tool**: django-cacheops
|
||||
- Automatic ORM query caching
|
||||
- Transparent invalidation
|
||||
- Minimal code changes
|
||||
- Redis backend for consistency
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Critical Features to Preserve
|
||||
|
||||
### 1. **Moderation System**
|
||||
- ✅ Atomic transactions for approvals
|
||||
- ✅ Selective approval (approve individual items)
|
||||
- ✅ State machine workflow (pending → reviewing → approved/rejected)
|
||||
- ✅ Lock mechanism (15-minute lock on review)
|
||||
- ✅ Automatic unlock on timeout
|
||||
- ✅ Batch operations
|
||||
|
||||
### 2. **Versioning System**
|
||||
- ✅ Full version history for all entities
|
||||
- ✅ Track who made changes
|
||||
- ✅ Track what changed
|
||||
- ✅ Link versions to submissions
|
||||
- ✅ Version diffs
|
||||
- ✅ Rollback capability
|
||||
|
||||
### 3. **Authentication**
|
||||
- ✅ Password-based login
|
||||
- ✅ Google OAuth
|
||||
- ✅ Discord OAuth
|
||||
- ✅ Two-factor authentication (TOTP)
|
||||
- ✅ Session management
|
||||
- ✅ JWT tokens for API
|
||||
|
||||
### 4. **Permissions & Security**
|
||||
- ✅ Role-based access control (user, moderator, admin, superuser)
|
||||
- ✅ Object-level permissions
|
||||
- ✅ Rate limiting
|
||||
- ✅ CORS configuration
|
||||
- ✅ Brute force protection
|
||||
|
||||
### 5. **Image Management**
|
||||
- ✅ CloudFlare direct upload
|
||||
- ✅ Image validation
|
||||
- ✅ Image metadata storage
|
||||
- ✅ Multiple image variants (thumbnails, etc.)
|
||||
|
||||
### 6. **Notifications**
|
||||
- ✅ Email notifications via Novu
|
||||
- ✅ In-app notifications
|
||||
- ✅ Notification templates
|
||||
- ✅ User preferences
|
||||
|
||||
### 7. **Search & Filtering**
|
||||
- ✅ Full-text search
|
||||
- ✅ Advanced filtering
|
||||
- ✅ Sorting options
|
||||
- ✅ Pagination
|
||||
|
||||
---
|
||||
|
||||
## 📊 Database Schema Preservation
|
||||
|
||||
### Core Entity Tables (Must Migrate)
|
||||
```
|
||||
✅ parks (80+ fields including dates, locations, operators)
|
||||
✅ rides (100+ fields including ride_models, parks, manufacturers)
|
||||
✅ companies (manufacturers, operators, designers)
|
||||
✅ ride_models (coaster models, flat ride models)
|
||||
✅ locations (countries, subdivisions, localities)
|
||||
✅ profiles (user profiles linked to auth.users)
|
||||
✅ user_roles (role assignments)
|
||||
✅ content_submissions (moderation queue)
|
||||
✅ submission_items (individual changes in submissions)
|
||||
✅ park_versions, ride_versions, etc. (version history)
|
||||
✅ photos (image metadata)
|
||||
✅ photo_submissions (photo approval queue)
|
||||
✅ reviews (user reviews)
|
||||
✅ reports (user reports)
|
||||
✅ entity_timeline_events (history timeline)
|
||||
✅ notification_logs
|
||||
✅ notification_templates
|
||||
```
|
||||
|
||||
### Computed Fields Strategy
|
||||
Some Supabase tables have computed fields. Options:
|
||||
1. **Cache in model** (recommended for frequently accessed)
|
||||
2. **Property method** (for rarely accessed)
|
||||
3. **Cached query** (using django-cacheops)
|
||||
|
||||
Example:
|
||||
```python
|
||||
class Park(models.Model):
|
||||
# Cached computed fields
|
||||
ride_count = models.IntegerField(default=0)
|
||||
coaster_count = models.IntegerField(default=0)
|
||||
|
||||
def update_counts(self):
|
||||
"""Update cached counts"""
|
||||
self.ride_count = self.rides.count()
|
||||
self.coaster_count = self.rides.filter(
|
||||
is_coaster=True
|
||||
).count()
|
||||
self.save()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Development Setup
|
||||
|
||||
### Prerequisites
|
||||
```bash
|
||||
# System requirements
|
||||
Python 3.11+
|
||||
PostgreSQL 15+
|
||||
Redis 7+
|
||||
Node.js 18+ (for frontend)
|
||||
```
|
||||
|
||||
### Initial Setup
|
||||
```bash
|
||||
# 1. Clone and checkout branch
|
||||
git checkout django-backend
|
||||
|
||||
# 2. Set up Python environment
|
||||
cd django
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
|
||||
# 3. Install dependencies
|
||||
pip install -r requirements/local.txt
|
||||
|
||||
# 4. Set up environment
|
||||
cp .env.example .env
|
||||
# Edit .env with your credentials
|
||||
|
||||
# 5. Run migrations
|
||||
python manage.py migrate
|
||||
|
||||
# 6. Create superuser
|
||||
python manage.py createsuperuser
|
||||
|
||||
# 7. Run development server
|
||||
python manage.py runserver
|
||||
|
||||
# 8. Run Celery worker (separate terminal)
|
||||
celery -A config worker -l info
|
||||
|
||||
# 9. Run Celery beat (separate terminal)
|
||||
celery -A config beat -l info
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=apps --cov-report=html
|
||||
|
||||
# Run specific test file
|
||||
pytest apps/moderation/tests/test_services.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Edge Functions to Migrate
|
||||
|
||||
### Supabase Edge Functions → Django/Celery
|
||||
|
||||
| Edge Function | Django Implementation | Priority |
|
||||
|---------------|----------------------|----------|
|
||||
| `process-submission` | `ModerationService.submit()` | P0 |
|
||||
| `process-selective-approval` | `ModerationService.approve()` | P0 |
|
||||
| `reject-submission` | `ModerationService.reject()` | P0 |
|
||||
| `unlock-submission` | Celery periodic task | P0 |
|
||||
| `cleanup_old_page_views` | Celery periodic task | P1 |
|
||||
| `update_entity_view_counts` | Celery periodic task | P1 |
|
||||
| `send-notification` | `NotificationService.send()` | P0 |
|
||||
| `process-photo-submission` | `MediaService.submit_photo()` | P1 |
|
||||
| `generate-daily-stats` | Celery periodic task | P2 |
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Success Criteria
|
||||
|
||||
### Must Have (P0)
|
||||
- ✅ All 80+ database tables migrated
|
||||
- ✅ All user data preserved (with UUIDs)
|
||||
- ✅ Authentication working (password + OAuth + MFA)
|
||||
- ✅ Moderation workflow functional
|
||||
- ✅ Versioning system working
|
||||
- ✅ All API endpoints functional
|
||||
- ✅ Frontend fully integrated
|
||||
- ✅ No data loss during migration
|
||||
- ✅ Performance equivalent or better
|
||||
|
||||
### Should Have (P1)
|
||||
- ✅ Real-time updates working
|
||||
- ✅ All Celery tasks running
|
||||
- ✅ Caching operational
|
||||
- ✅ Image uploads working
|
||||
- ✅ Notifications working
|
||||
- ✅ Search functional
|
||||
- ✅ Comprehensive test coverage (>80%)
|
||||
|
||||
### Nice to Have (P2)
|
||||
- Admin dashboard improvements
|
||||
- Enhanced monitoring/observability
|
||||
- API rate limiting per user
|
||||
- Advanced analytics
|
||||
- GraphQL endpoint (optional)
|
||||
|
||||
---
|
||||
|
||||
## 🚨 Risk Mitigation
|
||||
|
||||
### Risk 1: Data Loss During Migration
|
||||
**Mitigation**:
|
||||
- Comprehensive backup before migration
|
||||
- Dry-run migration multiple times
|
||||
- Validation scripts to check data integrity
|
||||
- Rollback plan
|
||||
|
||||
### Risk 2: Downtime During Cutover
|
||||
**Mitigation**:
|
||||
- Blue-green deployment strategy
|
||||
- Run both systems in parallel briefly
|
||||
- Feature flags to toggle between backends
|
||||
- Quick rollback capability
|
||||
|
||||
### Risk 3: Performance Degradation
|
||||
**Mitigation**:
|
||||
- Load testing before production
|
||||
- Database query optimization
|
||||
- Aggressive caching strategy
|
||||
- Monitoring and alerting
|
||||
|
||||
### Risk 4: Missing Edge Cases
|
||||
**Mitigation**:
|
||||
- Comprehensive test suite
|
||||
- Manual QA testing
|
||||
- Beta testing period
|
||||
- Staged rollout
|
||||
|
||||
---
|
||||
|
||||
## 📞 Support & Resources
|
||||
|
||||
### Documentation
|
||||
- Django: https://docs.djangoproject.com/
|
||||
- django-ninja: https://django-ninja.rest-framework.com/
|
||||
- Celery: https://docs.celeryq.dev/
|
||||
- Django Channels: https://channels.readthedocs.io/
|
||||
|
||||
### Key Files to Reference
|
||||
- Original database schema: `supabase/migrations/`
|
||||
- Current API endpoints: `src/lib/supabaseClient.ts`
|
||||
- Moderation logic: `src/components/moderation/`
|
||||
- Existing docs: `docs/moderation/`, `docs/versioning/`
|
||||
|
||||
---
|
||||
|
||||
## 🎉 Next Steps
|
||||
|
||||
1. **Immediate** (This Week):
|
||||
- Configure Django settings
|
||||
- Create base models
|
||||
- Set up database connection
|
||||
|
||||
2. **Short-term** (Next 2 Weeks):
|
||||
- Implement entity models
|
||||
- Set up authentication
|
||||
- Create basic API endpoints
|
||||
|
||||
3. **Medium-term** (Next 4-8 Weeks):
|
||||
- Build moderation system
|
||||
- Implement versioning
|
||||
- Migrate edge functions
|
||||
|
||||
4. **Long-term** (8-16 Weeks):
|
||||
- Complete API layer
|
||||
- Frontend integration
|
||||
- Testing and deployment
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: November 8, 2025
|
||||
**Status**: Foundation Phase - Dependencies Installed, Structure Created
|
||||
**Next**: Configure Django settings and create base models
|
||||
Reference in New Issue
Block a user