thrilltrack-explorer/django/PHASE_7_CELERY_COMPLETE.md

# Phase 7: Background Tasks with Celery - COMPLETE ✅

**Completion Date:** November 8, 2025  
**Status:** Successfully Implemented

## Overview

Phase 7 implements a comprehensive background task processing system using Celery with Redis as the message broker. This phase adds asynchronous processing capabilities for long-running operations, scheduled tasks, and email notifications.

## What Was Implemented

### 1. Celery Infrastructure ✅
- **Celery App Configuration** (`config/celery.py`)
  - Auto-discovery of tasks from all apps
  - Signal handlers for task failure/success logging
  - Integration with Sentry for error tracking
  
- **Django Integration** (`config/__init__.py`)
  - Celery app loaded on Django startup
  - Shared task decorators available throughout the project

### 2. Email System ✅
- **Email Templates** (`templates/emails/`)
  - `base.html` - Base template with ThrillWiki branding
  - `welcome.html` - Welcome email for new users
  - `password_reset.html` - Password reset instructions
  - `moderation_approved.html` - Submission approved notification
  - `moderation_rejected.html` - Submission rejection notification

- **Email Configuration**
  - Development: Console backend (emails print to console)
  - Production: SMTP/SendGrid (configurable via environment variables)

### 3. Background Tasks ✅

#### Media Tasks (`apps/media/tasks.py`)
- `process_uploaded_image(photo_id)` - Post-upload image processing
- `cleanup_rejected_photos(days_old=30)` - Remove old rejected photos
- `generate_photo_thumbnails(photo_id)` - On-demand thumbnail generation
- `cleanup_orphaned_cloudflare_images()` - Remove orphaned images
- `update_photo_statistics()` - Update photo-related statistics

#### Moderation Tasks (`apps/moderation/tasks.py`)
- `send_moderation_notification(submission_id, status)` - Email notifications
- `cleanup_expired_locks()` - Remove stale moderation locks
- `send_batch_moderation_summary(moderator_id)` - Daily moderator summaries
- `update_moderation_statistics()` - Update moderation statistics
- `auto_unlock_stale_reviews(hours=1)` - Auto-unlock stale submissions
- `notify_moderators_of_queue_size()` - Alert on queue threshold

#### User Tasks (`apps/users/tasks.py`)
- `send_welcome_email(user_id)` - Welcome new users
- `send_password_reset_email(user_id, token, reset_url)` - Password resets
- `cleanup_expired_tokens()` - Remove expired JWT tokens
- `send_account_notification(user_id, type, data)` - Generic notifications
- `cleanup_inactive_users(days_inactive=365)` - Flag inactive accounts
- `update_user_statistics()` - Update user statistics
- `send_bulk_notification(user_ids, subject, message)` - Bulk emails
- `send_email_verification_reminder(user_id)` - Verification reminders

#### Entity Tasks (`apps/entities/tasks.py`)
- `update_entity_statistics(entity_type, entity_id)` - Update entity stats
- `update_all_statistics()` - Bulk statistics update
- `generate_entity_report(entity_type, entity_id)` - Generate reports
- `cleanup_duplicate_entities()` - Detect duplicates
- `calculate_global_statistics()` - Global statistics
- `validate_entity_data(entity_type, entity_id)` - Data validation

### 4. Scheduled Tasks (Celery Beat) ✅

Configured in `config/settings/base.py`:

| Task | Schedule | Purpose |
|------|----------|---------|
| `cleanup-expired-locks` | Every 5 minutes | Remove expired moderation locks |
| `cleanup-expired-tokens` | Daily at 2 AM | Clean up expired JWT tokens |
| `update-all-statistics` | Every 6 hours | Update entity statistics |
| `cleanup-rejected-photos` | Weekly Mon 3 AM | Remove old rejected photos |
| `auto-unlock-stale-reviews` | Every 30 minutes | Auto-unlock stale reviews |
| `check-moderation-queue` | Every hour | Check queue size threshold |
| `update-photo-statistics` | Daily at 1 AM | Update photo statistics |
| `update-moderation-statistics` | Daily at 1:30 AM | Update moderation statistics |
| `update-user-statistics` | Daily at 4 AM | Update user statistics |
| `calculate-global-statistics` | Every 12 hours | Calculate global statistics |

### 5. Service Integration ✅
- **PhotoService** - Triggers `process_uploaded_image` on photo creation
- **ModerationService** - Sends email notifications on approval/rejection
- Error handling ensures service operations don't fail if tasks fail to queue

### 6. Monitoring ✅
- **Flower** - Web-based Celery monitoring (production only)
- **Task Logging** - Success/failure logging for all tasks
- **Sentry Integration** - Error tracking for failed tasks

## Setup Instructions

### Development Setup

1. **Install Redis** (if not using eager mode):
   ```bash
   # macOS with Homebrew
   brew install redis
   brew services start redis
   
   # Or using Docker
   docker run -d -p 6379:6379 redis:latest
   ```

2. **Configure Environment** (`.env`):
   ```env
   # Redis Configuration
   REDIS_URL=redis://localhost:6379/0
   CELERY_BROKER_URL=redis://localhost:6379/0
   CELERY_RESULT_BACKEND=redis://localhost:6379/1
   
   # Email Configuration (Development)
   EMAIL_BACKEND=django.core.mail.backends.console.EmailBackend
   DEFAULT_FROM_EMAIL=noreply@thrillwiki.com
   SITE_URL=http://localhost:8000
   ```

3. **Run Celery Worker** (in separate terminal):
   ```bash
   cd django
   celery -A config worker --loglevel=info
   ```

4. **Run Celery Beat** (in separate terminal):
   ```bash
   cd django
   celery -A config beat --loglevel=info
   ```

5. **Development Mode** (No Redis Required):
   - Tasks run synchronously when `CELERY_TASK_ALWAYS_EAGER = True` (default in `local.py`)
   - Useful for debugging and testing without Redis

### Production Setup

1. **Configure Environment**:
   ```env
   # Redis Configuration
   REDIS_URL=redis://your-redis-host:6379/0
   CELERY_BROKER_URL=redis://your-redis-host:6379/0
   CELERY_RESULT_BACKEND=redis://your-redis-host:6379/1
   
   # Email Configuration (Production)
   EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
   EMAIL_HOST=smtp.sendgrid.net
   EMAIL_PORT=587
   EMAIL_USE_TLS=True
   EMAIL_HOST_USER=apikey
   EMAIL_HOST_PASSWORD=your-sendgrid-api-key
   DEFAULT_FROM_EMAIL=noreply@thrillwiki.com
   SITE_URL=https://thrillwiki.com
   
   # Flower Monitoring (Optional)
   FLOWER_ENABLED=True
   FLOWER_BASIC_AUTH=username:password
   ```

2. **Run Celery Worker** (systemd service):
   ```ini
   [Unit]
   Description=ThrillWiki Celery Worker
   After=network.target redis.target
   
   [Service]
   Type=forking
   User=www-data
   Group=www-data
   WorkingDirectory=/var/www/thrillwiki/django
   Environment="PATH=/var/www/thrillwiki/venv/bin"
   ExecStart=/var/www/thrillwiki/venv/bin/celery -A config worker \
             --loglevel=info \
             --logfile=/var/log/celery/worker.log \
             --pidfile=/var/run/celery/worker.pid
   
   [Install]
   WantedBy=multi-user.target
   ```

3. **Run Celery Beat** (systemd service):
   ```ini
   [Unit]
   Description=ThrillWiki Celery Beat
   After=network.target redis.target
   
   [Service]
   Type=forking
   User=www-data
   Group=www-data
   WorkingDirectory=/var/www/thrillwiki/django
   Environment="PATH=/var/www/thrillwiki/venv/bin"
   ExecStart=/var/www/thrillwiki/venv/bin/celery -A config beat \
             --loglevel=info \
             --logfile=/var/log/celery/beat.log \
             --pidfile=/var/run/celery/beat.pid \
             --schedule=/var/run/celery/celerybeat-schedule
   
   [Install]
   WantedBy=multi-user.target
   ```

4. **Run Flower** (optional):
   ```bash
   celery -A config flower --port=5555 --basic_auth=$FLOWER_BASIC_AUTH
   ```
   Access at: `https://your-domain.com/flower/`

## Testing

### Manual Testing

1. **Test Photo Upload Task**:
   ```python
   from apps.media.tasks import process_uploaded_image
   result = process_uploaded_image.delay(photo_id)
   print(result.get())  # Wait for result
   ```

2. **Test Email Notification**:
   ```python
   from apps.moderation.tasks import send_moderation_notification
   result = send_moderation_notification.delay(str(submission_id), 'approved')
   # Check console output for email
   ```

3. **Test Scheduled Task**:
   ```python
   from apps.moderation.tasks import cleanup_expired_locks
   result = cleanup_expired_locks.delay()
   print(result.get())
   ```

### Integration Testing

Test that services properly queue tasks:

```python
# Test PhotoService integration
from apps.media.services import PhotoService
service = PhotoService()
photo = service.create_photo(file, user)
# Task should be queued automatically

# Test ModerationService integration
from apps.moderation.services import ModerationService
ModerationService.approve_submission(submission_id, reviewer)
# Email notification should be queued
```

## Task Catalog

### Task Retry Configuration

All tasks implement retry logic:
- **Max Retries:** 2-3 (task-dependent)
- **Retry Delay:** 60 seconds base (exponential backoff)
- **Failure Handling:** Logged to Sentry and application logs

### Task Priority

Tasks are executed in the order they're queued. For priority queuing, configure Celery with multiple queues:

```python
# config/celery.py (future enhancement)
CELERY_TASK_ROUTES = {
    'apps.media.tasks.process_uploaded_image': {'queue': 'media'},
    'apps.moderation.tasks.send_moderation_notification': {'queue': 'notifications'},
}
```

## Monitoring & Debugging

### View Task Status

```python
from celery.result import AsyncResult

result = AsyncResult('task-id-here')
print(result.state)  # PENDING, STARTED, SUCCESS, FAILURE
print(result.info)   # Result or error details
```

### Flower Dashboard

Access Flower at `/flower/` (production only) to:
- View active tasks
- Monitor worker status
- View task history
- Inspect failed tasks
- Retry failed tasks

### Logs

```bash
# View worker logs
tail -f /var/log/celery/worker.log

# View beat logs
tail -f /var/log/celery/beat.log

# View Django logs (includes task execution)
tail -f django/logs/django.log
```

## Troubleshooting

### Common Issues

1. **Tasks not executing**
   - Check Redis connection: `redis-cli ping`
   - Verify Celery worker is running: `ps aux | grep celery`
   - Check for errors in worker logs

2. **Emails not sending**
   - Verify EMAIL_BACKEND configuration
   - Check SMTP credentials
   - Review email logs in console (development)

3. **Scheduled tasks not running**
   - Ensure Celery Beat is running
   - Check Beat logs for scheduling errors
   - Verify CELERY_BEAT_SCHEDULE configuration

4. **Task failures**
   - Check Sentry for error reports
   - Review worker logs
   - Test task in Django shell

### Performance Tuning

```python
# Increase worker concurrency
celery -A config worker --concurrency=4

# Use different pool implementation
celery -A config worker --pool=gevent

# Set task time limits
CELERY_TASK_TIME_LIMIT = 30 * 60  # 30 minutes (already configured)
```

## Configuration Options

### Environment Variables

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `REDIS_URL` | Yes* | `redis://localhost:6379/0` | Redis connection URL |
| `CELERY_BROKER_URL` | Yes* | Same as REDIS_URL | Celery message broker |
| `CELERY_RESULT_BACKEND` | Yes* | `redis://localhost:6379/1` | Task result storage |
| `EMAIL_BACKEND` | No | Console (dev) / SMTP (prod) | Email backend |
| `EMAIL_HOST` | Yes** | - | SMTP host |
| `EMAIL_PORT` | Yes** | 587 | SMTP port |
| `EMAIL_HOST_USER` | Yes** | - | SMTP username |
| `EMAIL_HOST_PASSWORD` | Yes** | - | SMTP password |
| `DEFAULT_FROM_EMAIL` | Yes | `noreply@thrillwiki.com` | From email address |
| `SITE_URL` | Yes | `http://localhost:8000` | Site URL for emails |
| `FLOWER_ENABLED` | No | False | Enable Flower monitoring |
| `FLOWER_BASIC_AUTH` | No** | - | Flower authentication |

\* Not required if using eager mode in development  
\*\* Required for production email sending

## Next Steps

### Future Enhancements

1. **Task Prioritization**
   - Implement multiple queues for different priority levels
   - Critical tasks (password reset) in high-priority queue
   - Bulk operations in low-priority queue

2. **Advanced Monitoring**
   - Set up Prometheus metrics
   - Configure Grafana dashboards
   - Add task duration tracking

3. **Email Improvements**
   - Add plain text email versions
   - Implement email templates for all notification types
   - Add email preference management

4. **Scalability**
   - Configure multiple Celery workers
   - Implement auto-scaling based on queue size
   - Add Redis Sentinel for high availability

5. **Additional Tasks**
   - Backup generation tasks
   - Data export tasks
   - Analytics report generation

## Success Criteria ✅

All success criteria for Phase 7 have been met:

- ✅ Celery workers running successfully
- ✅ Tasks executing asynchronously
- ✅ Email notifications working (console backend configured)
- ✅ Scheduled tasks configured and ready
- ✅ Flower monitoring configured for production
- ✅ Error handling and retries implemented
- ✅ Integration with existing services complete
- ✅ Comprehensive documentation created

## Files Created

- `config/celery.py` - Celery app configuration
- `config/__init__.py` - Updated to load Celery
- `templates/emails/base.html` - Base email template
- `templates/emails/welcome.html` - Welcome email
- `templates/emails/password_reset.html` - Password reset email
- `templates/emails/moderation_approved.html` - Approval notification
- `templates/emails/moderation_rejected.html` - Rejection notification
- `apps/media/tasks.py` - Media processing tasks
- `apps/moderation/tasks.py` - Moderation workflow tasks
- `apps/users/tasks.py` - User management tasks
- `apps/entities/tasks.py` - Entity statistics tasks
- `PHASE_7_CELERY_COMPLETE.md` - This documentation

## Files Modified

- `config/settings/base.py` - Added Celery Beat schedule, SITE_URL, DEFAULT_FROM_EMAIL
- `config/urls.py` - Added Flower URL routing
- `apps/media/services.py` - Integrated photo processing task
- `apps/moderation/services.py` - Integrated email notification tasks

## Dependencies

All dependencies were already included in `requirements/base.txt`:
- `celery[redis]==5.3.4`
- `django-celery-beat==2.5.0`
- `django-celery-results==2.5.1`
- `flower==2.0.1`

## Summary

Phase 7 successfully implements a complete background task processing system with Celery. The system handles:
- Asynchronous image processing
- Email notifications for moderation workflow
- Scheduled maintenance tasks
- Statistics updates
- Token cleanup

The implementation is production-ready with proper error handling, retry logic, monitoring, and documentation.

**Phase 7: COMPLETE** ✅