thrilltrack-explorer/django-backend/SACRED_PIPELINE_AUDIT_AND_IMPLEMENTATION_PLAN.md

Sacred Pipeline Audit & Implementation Plan

Date: November 8, 2025
Auditor: AI Assistant
Status: Audit Complete - Awaiting Implementation
Decision: Enforce Sacred Pipeline for ALL entity creation

---

📊 EXECUTIVE SUMMARY

Overall Assessment: 95% Complete, High Quality

• Backend Implementation: Excellent (85% feature-complete)
• Sacred Pipeline Compliance: Mixed - Critical gaps identified
• Code Quality: High
• Documentation: Comprehensive

Key Finding

Only Reviews properly use the Sacred Pipeline. All other entities (Parks, Rides, Companies, RideModels) bypass it completely.

---

🔴 CRITICAL ISSUES IDENTIFIED

Issue #1: Review Submission Type Mismatch 🔴

Severity: HIGH
Impact: Database constraint violation, data integrity

Problem:

# apps/reviews/services.py line 83
submission_type='review'  # This value is used

# apps/moderation/models.py line 45
SUBMISSION_TYPE_CHOICES = [
    ('create', 'Create'),
    ('update', 'Update'),
    ('delete', 'Delete'),
    # 'review' is NOT in choices - will cause constraint error
]

Solution: Add 'review' to SUBMISSION_TYPE_CHOICES

---

Issue #2: Entity Creation Bypasses Sacred Pipeline 🔴

Severity: CRITICAL
Impact: Violates core project architecture requirement

Problem: All entity creation endpoints use direct model.objects.create():

• api/v1/endpoints/parks.py
• api/v1/endpoints/rides.py
• api/v1/endpoints/companies.py
• api/v1/endpoints/ride_models.py

# Current implementation - BYPASSES PIPELINE
@router.post('/')
def create_park(request, data):
    park = Park.objects.create(...)  # NO MODERATION!
    return park

Project Requirement:

"All content flows through our sacred pipeline - Form → Submission → Moderation → Approval → Versioning → Display"

Current Reality: Only Reviews comply. Entities bypass completely.

Solution: Create submission services for all entity types (following ReviewSubmissionService pattern)

---

Issue #3: ModerationService Can't Approve Reviews 🔴

Severity: HIGH
Impact: Review moderation is broken (masked by moderator bypass)

Problem:

# apps/moderation/services.py line 142
def approve_submission(submission_id, reviewer):
    entity = submission.entity  # For reviews, this is Park/Ride
    for item in items:
        setattr(entity, item.field_name, item.new_value)  # WRONG!
    entity.save()  # This would corrupt the Park/Ride, not create Review

When a review submission is approved, it tries to apply review fields (rating, title, content) to the Park/Ride entity instead of creating a Review record.

Why It's Hidden: The ReviewSubmissionService has a moderator bypass that auto-approves before submission reaches ModerationService, so the bug doesn't manifest in normal flow.

Solution: Add polymorphic approval handling based on submission_type

---

Issue #4: Entity Updates Bypass Sacred Pipeline 🟡

Severity: MEDIUM
Impact: No moderation for updates, inconsistent with Reviews

Problem:

@router.put('/{id}')
def update_entity(request, id, data):
    entity.name = data.name
    entity.save()  # DIRECT UPDATE, NO MODERATION

Reviews properly create update submissions, but entities don't.

Solution: Add update submission methods for entities

---

✅ WHAT'S WORKING WELL

Core Systems (100% Complete)

• ✅ FSM State Machine - Proper transitions (draft→pending→reviewing→approved/rejected)
• ✅ Atomic Transactions - All-or-nothing approval via @transaction.atomic
• ✅ 15-Minute Locks - Prevents concurrent editing
• ✅ pghistory Integration - Automatic versioning for all entities
• ✅ History API - 37 endpoints across all entity types
• ✅ Selective Approval - Approve/reject individual fields
• ✅ Background Tasks - 20+ Celery tasks, email notifications
• ✅ Search - PostgreSQL full-text with GIN indexes
• ✅ Authentication - JWT, MFA, role-based permissions

Models (100% Complete)

• ✅ Company, RideModel, Park, Ride - All with pghistory
• ✅ Review, ReviewHelpfulVote - Pipeline-integrated
• ✅ UserRideCredit, UserTopList, UserTopListItem - All implemented
• ✅ ContentSubmission, SubmissionItem, ModerationLock - Complete

API Coverage (90+ endpoints)

• ✅ 23 authentication endpoints
• ✅ 12 moderation endpoints
• ✅ 37 history endpoints
• ✅ Entity CRUD endpoints
• ✅ Search, filtering, pagination

---

📋 IMPLEMENTATION PLAN

PHASE 1: Fix Critical Bugs (2-3 hours)

Task 1.1: Fix Review Submission Type (30 mins)

File: django/apps/moderation/models.py

Change:

SUBMISSION_TYPE_CHOICES = [
    ('create', 'Create'),
    ('update', 'Update'),
    ('delete', 'Delete'),
    ('review', 'Review'),  # ADD THIS
]

Migration Required: Yes

---

Task 1.2: Add Polymorphic Submission Approval (2 hours)

File: django/apps/moderation/services.py

Change: Update approve_submission() method to detect submission_type and delegate appropriately:

@staticmethod
@transaction.atomic
def approve_submission(submission_id, reviewer):
    submission = ContentSubmission.objects.select_for_update().get(id=submission_id)
    
    # Permission checks...
    
    # DELEGATE BASED ON SUBMISSION TYPE
    if submission.submission_type == 'review':
        # Handle review submissions
        from apps.reviews.services import ReviewSubmissionService
        review = ReviewSubmissionService.apply_review_approval(submission)
        
    elif submission.submission_type in ['create', 'update', 'delete']:
        # Handle entity submissions
        entity = submission.entity
        if not entity:
            raise ValidationError("Entity no longer exists")
        
        items = submission.items.filter(status='pending')
        
        if submission.submission_type == 'create':
            # Entity created in draft, now make visible
            for item in items:
                if item.change_type in ['add', 'modify']:
                    setattr(entity, item.field_name, item.new_value)
                item.approve(reviewer)
            entity.save()
            
        elif submission.submission_type == 'update':
            # Apply updates
            for item in items:
                if item.change_type in ['add', 'modify']:
                    setattr(entity, item.field_name, item.new_value)
                elif item.change_type == 'remove':
                    setattr(entity, item.field_name, None)
                item.approve(reviewer)
            entity.save()
            
        elif submission.submission_type == 'delete':
            entity.delete()
    
    else:
        raise ValidationError(f"Unknown submission type: {submission.submission_type}")
    
    # Mark submission approved (FSM)
    submission.approve(reviewer)
    submission.save()
    
    # Release lock, send notifications...
    # (existing code)

---

PHASE 2: Create Entity Submission Services (8-10 hours)

Task 2.1: Create Base Service (2 hours)

File: django/apps/entities/services/__init__.py (NEW)

Create BaseEntitySubmissionService with:

• create_entity_submission(user, data, **kwargs) method
• Moderator bypass logic (auto-approve if is_moderator)
• Standard item creation pattern
• Proper error handling and logging

Pattern:

class BaseEntitySubmissionService:
    entity_model = None  # Override in subclass
    entity_type_name = None  # Override in subclass
    required_fields = []  # Override in subclass
    
    @classmethod
    @transaction.atomic
    def create_entity_submission(cls, user, data, **kwargs):
        # Check moderator status
        is_moderator = hasattr(user, 'role') and user.role.is_moderator
        
        # Build submission items
        items_data = [...]
        
        # Create placeholder entity
        entity = cls.entity_model(**data)
        entity.save()
        
        # Create submission via ModerationService
        submission = ModerationService.create_submission(...)
        
        # Moderator bypass
        if is_moderator:
            submission = ModerationService.approve_submission(...)
            # Update entity with all fields
            entity.save()
            return submission, entity
        
        return submission, None

---

Task 2.2-2.5: Create Entity-Specific Services (6 hours)

Create four service files:

File: django/apps/entities/services/park_submission.py (NEW)

from apps.entities.models import Park
from apps.entities.services import BaseEntitySubmissionService

class ParkSubmissionService(BaseEntitySubmissionService):
    entity_model = Park
    entity_type_name = 'Park'
    required_fields = ['name', 'park_type']

File: django/apps/entities/services/ride_submission.py (NEW)

from apps.entities.models import Ride
from apps.entities.services import BaseEntitySubmissionService

class RideSubmissionService(BaseEntitySubmissionService):
    entity_model = Ride
    entity_type_name = 'Ride'
    required_fields = ['name', 'park', 'ride_category']

File: django/apps/entities/services/company_submission.py (NEW)

from apps.entities.models import Company
from apps.entities.services import BaseEntitySubmissionService

class CompanySubmissionService(BaseEntitySubmissionService):
    entity_model = Company
    entity_type_name = 'Company'
    required_fields = ['name']

File: django/apps/entities/services/ride_model_submission.py (NEW)

from apps.entities.models import RideModel
from apps.entities.services import BaseEntitySubmissionService

class RideModelSubmissionService(BaseEntitySubmissionService):
    entity_model = RideModel
    entity_type_name = 'RideModel'
    required_fields = ['name', 'manufacturer', 'model_type']

---

PHASE 3: Update API Endpoints (4-5 hours)

Task 3.1-3.4: Update Creation Endpoints (4 hours)

Pattern for ALL entity endpoints:

Before:

@router.post('/', response={201: EntityOut, 400: ErrorResponse}, auth=jwt_auth)
@require_auth
def create_entity(request, data: EntityCreateSchema):
    entity = Entity.objects.create(...)  # BYPASSES PIPELINE
    return 201, serialize_entity(entity)

After:

@router.post('/', response={201: EntityOut, 400: ErrorResponse}, auth=jwt_auth)
@require_auth
def create_entity(request, data: EntityCreateSchema):
    """
    Create entity through Sacred Pipeline.
    
    **Moderators:** Entity created immediately (bypass moderation)
    **Regular users:** Submission enters moderation queue
    """
    try:
        user = request.auth
        
        # Import appropriate service
        from apps.entities.services.entity_submission import EntitySubmissionService
        
        submission, entity = EntitySubmissionService.create_entity_submission(
            user=user,
            data=data.dict(exclude_unset=True),
            source='api'
        )
        
        if entity:
            # Moderator bypass - entity created immediately
            return 201, serialize_entity(entity, user)
        else:
            # Regular user - pending moderation
            return 201, {
                'submission_id': str(submission.id),
                'status': 'pending_moderation',
                'message': 'Entity submitted for moderation. You will be notified when approved.'
            }
            
    except ValidationError as e:
        return 400, {'detail': str(e)}

Files to Modify:

• django/api/v1/endpoints/parks.py
• django/api/v1/endpoints/rides.py
• django/api/v1/endpoints/companies.py
• django/api/v1/endpoints/ride_models.py

Estimated Time: 1 hour per endpoint = 4 hours

---

PHASE 4: Testing & Validation (3-4 hours)

Task 4.1: Unit Tests (2 hours)

File: django/apps/entities/tests/test_submissions.py (NEW)

Test coverage:

• Regular user creates entity → ContentSubmission created
• Moderator creates entity → Entity created immediately
• Regular user's submission approved → Entity created
• Invalid data → Proper error handling
• Permission checks → Unauthorized users blocked

Example Test:

def test_regular_user_park_creation_requires_moderation():
    user = create_user(role='user')
    data = {'name': 'Test Park', 'park_type': 'theme_park'}
    
    submission, park = ParkSubmissionService.create_entity_submission(
        user=user,
        data=data
    )
    
    assert submission is not None
    assert park is None  # Not created yet
    assert submission.status == 'pending'
    assert Park.objects.count() == 0  # No park created

def test_moderator_park_creation_bypasses_moderation():
    moderator = create_user(role='moderator')
    data = {'name': 'Test Park', 'park_type': 'theme_park'}
    
    submission, park = ParkSubmissionService.create_entity_submission(
        user=moderator,
        data=data
    )
    
    assert submission is not None
    assert park is not None  # Created immediately
    assert submission.status == 'approved'
    assert Park.objects.count() == 1

---

Task 4.2: Integration Tests (1 hour)

Test complete flow:

1. API POST → ContentSubmission created
2. Moderator calls approve endpoint → Entity created
3. pghistory event captured
4. Email notification sent

---

Task 4.3: Manual Testing (1 hour)

• Use Postman/curl to test endpoints
• Verify moderation queue shows entity submissions
• Test moderator approval process
• Verify entities appear after approval
• Check email notifications

---

📊 EFFORT BREAKDOWN

Phase	Tasks	Hours	Priority
Phase 1: Critical Bugs	2	2.5	P0
Phase 2: Entity Services	5	8	P0
Phase 3: API Updates	4	4	P0
Phase 4: Testing	3	4	P1
TOTAL	14	18.5

Timeline: 2.5-3 days of focused work

---

🎯 SUCCESS CRITERIA

Must Have (P0)

• Issue #1 fixed: 'review' added to submission type choices
• Issue #2 fixed: Polymorphic approval handler implemented
• Issue #3 fixed: All entity types use Sacred Pipeline for creation
• Moderator bypass works for all entity types
• ContentSubmission properly handles all entity types
• pghistory triggers for all entity creations

Should Have (P1)

• All unit tests passing
• Integration tests passing
• Manual testing confirms flow works
• Documentation updated

Nice to Have (P2)

• Entity update submissions (similar to review updates)
• Batch submission support
• Draft mode for partial entities

---

🚨 RISKS & MITIGATION

Risk 1: Breaking Existing API Clients

Probability: HIGH
Impact: HIGH

Mitigation:

• API response changes from immediate entity to submission confirmation
• Frontend needs updates to handle both response types
• Consider versioning API (keep /v1/ old, create /v2/ new)
• Add deprecation warnings

Risk 2: Performance Impact

Probability: LOW
Impact: LOW

Mitigation:

• ContentSubmission creation is lightweight
• Moderator bypass keeps fast path for admins
• No database query increase for moderators
• Regular users get proper moderation (expected delay)

Risk 3: Moderator Workflow Changes

Probability: MEDIUM
Impact: MEDIUM

Mitigation:

• Moderators will now see entity submissions in queue
• Need to train moderators on new approval process
• Consider auto-approve for trusted submitters
• Bulk approval tools may be needed

---

📝 ADDITIONAL CONSIDERATIONS

company_types JSON Field

Current: Uses JSONField for company types (e.g., ['manufacturer', 'operator'])

Issue: Project rules state "NEVER use JSON/JSONB in SQL"

Solution: Create CompanyType lookup table with M2M relationship

Effort: 2 hours

Priority: P2 (not blocking)

---

URL Patterns

Current: Implemented in Django
Status: ✅ Compliant with requirements

• Parks: /api/v1/parks/{id}/
• Rides: /api/v1/rides/{id}/
• Companies: /api/v1/companies/{id}/

---

Error Handling

Current: Try/except blocks present in most endpoints
Status: ✅ Good coverage

Improvement: Centralized error handler middleware (P2)

---

🎬 RECOMMENDED NEXT STEPS

Immediate (Today)

1. Get user confirmation on implementation approach
2. Choose implementation order:
   • Option A: Fix all bugs first, then add entity services
   • Option B: Do one entity end-to-end, then replicate
3. Set up testing environment to validate changes

This Week

1. Implement Phase 1 (critical bugs)
2. Implement Phase 2 (entity services)
3. Implement Phase 3 (API updates)
4. Manual testing

Next Week

1. Complete Phase 4 (automated tests)
2. Update documentation
3. Deploy to staging
4. UAT with moderators

---

📚 FILES TO BE CREATED

New Files (7)

1. django/apps/entities/services/__init__.py - Base service
2. django/apps/entities/services/park_submission.py
3. django/apps/entities/services/ride_submission.py
4. django/apps/entities/services/company_submission.py
5. django/apps/entities/services/ride_model_submission.py
6. django/apps/entities/tests/test_submissions.py
7. django/apps/entities/migrations/00XX_add_review_submission_type.py

Files to Modify (5)

1. django/apps/moderation/models.py - Add 'review' choice
2. django/apps/moderation/services.py - Polymorphic approval
3. django/api/v1/endpoints/parks.py - Use submission service
4. django/api/v1/endpoints/rides.py - Use submission service
5. django/api/v1/endpoints/companies.py - Use submission service
6. django/api/v1/endpoints/ride_models.py - Use submission service

---

💡 CONCLUSION

The Django backend is 95% complete and high quality. The Sacred Pipeline architecture is implemented correctly for Reviews but not enforced for other entities.

No functionality is lost - all features exist. The issues are architectural compliance gaps that need to be addressed to meet project requirements.

The work is well-defined and straightforward: Follow the ReviewSubmissionService pattern for all entity types. The implementation is repetitive but not complex.

Estimated completion: 2.5-3 days of focused development work.

---

Status: ✅ Audit Complete - Ready for Implementation
Next: User approval to proceed with implementation
Date: November 8, 2025