thrilltrack-explorer/django/PHASE_7_CELERY_COMPLETE.md

Phase 7: Background Tasks with Celery - COMPLETE ✅

Completion Date: November 8, 2025
Status: Successfully Implemented

Overview

Phase 7 implements a comprehensive background task processing system using Celery with Redis as the message broker. This phase adds asynchronous processing capabilities for long-running operations, scheduled tasks, and email notifications.

What Was Implemented

1. Celery Infrastructure ✅

• Celery App Configuration (config/celery.py)

  • Auto-discovery of tasks from all apps
  • Signal handlers for task failure/success logging
  • Integration with Sentry for error tracking

• Django Integration (config/__init__.py)

  • Celery app loaded on Django startup
  • Shared task decorators available throughout the project

2. Email System ✅

• Email Templates (templates/emails/)

  • base.html - Base template with ThrillWiki branding
  • welcome.html - Welcome email for new users
  • password_reset.html - Password reset instructions
  • moderation_approved.html - Submission approved notification
  • moderation_rejected.html - Submission rejection notification

• Email Configuration

  • Development: Console backend (emails print to console)
  • Production: SMTP/SendGrid (configurable via environment variables)

3. Background Tasks ✅

Media Tasks (apps/media/tasks.py)

• process_uploaded_image(photo_id) - Post-upload image processing
• cleanup_rejected_photos(days_old=30) - Remove old rejected photos
• generate_photo_thumbnails(photo_id) - On-demand thumbnail generation
• cleanup_orphaned_cloudflare_images() - Remove orphaned images
• update_photo_statistics() - Update photo-related statistics

Moderation Tasks (apps/moderation/tasks.py)

• send_moderation_notification(submission_id, status) - Email notifications
• cleanup_expired_locks() - Remove stale moderation locks
• send_batch_moderation_summary(moderator_id) - Daily moderator summaries
• update_moderation_statistics() - Update moderation statistics
• auto_unlock_stale_reviews(hours=1) - Auto-unlock stale submissions
• notify_moderators_of_queue_size() - Alert on queue threshold

User Tasks (apps/users/tasks.py)

• send_welcome_email(user_id) - Welcome new users
• send_password_reset_email(user_id, token, reset_url) - Password resets
• cleanup_expired_tokens() - Remove expired JWT tokens
• send_account_notification(user_id, type, data) - Generic notifications
• cleanup_inactive_users(days_inactive=365) - Flag inactive accounts
• update_user_statistics() - Update user statistics
• send_bulk_notification(user_ids, subject, message) - Bulk emails
• send_email_verification_reminder(user_id) - Verification reminders

Entity Tasks (apps/entities/tasks.py)

• update_entity_statistics(entity_type, entity_id) - Update entity stats
• update_all_statistics() - Bulk statistics update
• generate_entity_report(entity_type, entity_id) - Generate reports
• cleanup_duplicate_entities() - Detect duplicates
• calculate_global_statistics() - Global statistics
• validate_entity_data(entity_type, entity_id) - Data validation

4. Scheduled Tasks (Celery Beat) ✅

Configured in config/settings/base.py:

Task	Schedule	Purpose
cleanup-expired-locks	Every 5 minutes	Remove expired moderation locks
cleanup-expired-tokens	Daily at 2 AM	Clean up expired JWT tokens
update-all-statistics	Every 6 hours	Update entity statistics
cleanup-rejected-photos	Weekly Mon 3 AM	Remove old rejected photos
auto-unlock-stale-reviews	Every 30 minutes	Auto-unlock stale reviews
check-moderation-queue	Every hour	Check queue size threshold
update-photo-statistics	Daily at 1 AM	Update photo statistics
update-moderation-statistics	Daily at 1:30 AM	Update moderation statistics
update-user-statistics	Daily at 4 AM	Update user statistics
calculate-global-statistics	Every 12 hours	Calculate global statistics

5. Service Integration ✅

• PhotoService - Triggers process_uploaded_image on photo creation
• ModerationService - Sends email notifications on approval/rejection
• Error handling ensures service operations don't fail if tasks fail to queue

6. Monitoring ✅

• Flower - Web-based Celery monitoring (production only)
• Task Logging - Success/failure logging for all tasks
• Sentry Integration - Error tracking for failed tasks

Setup Instructions

Development Setup

1. Install Redis (if not using eager mode):

   # macOS with Homebrew
   brew install redis
   brew services start redis
   
   # Or using Docker
   docker run -d -p 6379:6379 redis:latest
   

2. Configure Environment (.env):

   # Redis Configuration
   REDIS_URL=redis://localhost:6379/0
   CELERY_BROKER_URL=redis://localhost:6379/0
   CELERY_RESULT_BACKEND=redis://localhost:6379/1
   
   # Email Configuration (Development)
   EMAIL_BACKEND=django.core.mail.backends.console.EmailBackend
   DEFAULT_FROM_EMAIL=noreply@thrillwiki.com
   SITE_URL=http://localhost:8000
   

3. Run Celery Worker (in separate terminal):

   cd django
   celery -A config worker --loglevel=info
   

4. Run Celery Beat (in separate terminal):

   cd django
   celery -A config beat --loglevel=info
   

5. Development Mode (No Redis Required):

   • Tasks run synchronously when CELERY_TASK_ALWAYS_EAGER = True (default in local.py)
   • Useful for debugging and testing without Redis

Production Setup

1. Configure Environment:

   # Redis Configuration
   REDIS_URL=redis://your-redis-host:6379/0
   CELERY_BROKER_URL=redis://your-redis-host:6379/0
   CELERY_RESULT_BACKEND=redis://your-redis-host:6379/1
   
   # Email Configuration (Production)
   EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
   EMAIL_HOST=smtp.sendgrid.net
   EMAIL_PORT=587
   EMAIL_USE_TLS=True
   EMAIL_HOST_USER=apikey
   EMAIL_HOST_PASSWORD=your-sendgrid-api-key
   DEFAULT_FROM_EMAIL=noreply@thrillwiki.com
   SITE_URL=https://thrillwiki.com
   
   # Flower Monitoring (Optional)
   FLOWER_ENABLED=True
   FLOWER_BASIC_AUTH=username:password
   

2. Run Celery Worker (systemd service):

   [Unit]
   Description=ThrillWiki Celery Worker
   After=network.target redis.target
   
   [Service]
   Type=forking
   User=www-data
   Group=www-data
   WorkingDirectory=/var/www/thrillwiki/django
   Environment="PATH=/var/www/thrillwiki/venv/bin"
   ExecStart=/var/www/thrillwiki/venv/bin/celery -A config worker \
             --loglevel=info \
             --logfile=/var/log/celery/worker.log \
             --pidfile=/var/run/celery/worker.pid
   
   [Install]
   WantedBy=multi-user.target
   

3. Run Celery Beat (systemd service):

   [Unit]
   Description=ThrillWiki Celery Beat
   After=network.target redis.target
   
   [Service]
   Type=forking
   User=www-data
   Group=www-data
   WorkingDirectory=/var/www/thrillwiki/django
   Environment="PATH=/var/www/thrillwiki/venv/bin"
   ExecStart=/var/www/thrillwiki/venv/bin/celery -A config beat \
             --loglevel=info \
             --logfile=/var/log/celery/beat.log \
             --pidfile=/var/run/celery/beat.pid \
             --schedule=/var/run/celery/celerybeat-schedule
   
   [Install]
   WantedBy=multi-user.target
   

4. Run Flower (optional):

   celery -A config flower --port=5555 --basic_auth=$FLOWER_BASIC_AUTH
   

   Access at: https://your-domain.com/flower/

Testing

Manual Testing

1. Test Photo Upload Task:

   from apps.media.tasks import process_uploaded_image
   result = process_uploaded_image.delay(photo_id)
   print(result.get())  # Wait for result
   

2. Test Email Notification:

   from apps.moderation.tasks import send_moderation_notification
   result = send_moderation_notification.delay(str(submission_id), 'approved')
   # Check console output for email
   

3. Test Scheduled Task:

   from apps.moderation.tasks import cleanup_expired_locks
   result = cleanup_expired_locks.delay()
   print(result.get())
   

Integration Testing

Test that services properly queue tasks:

# Test PhotoService integration
from apps.media.services import PhotoService
service = PhotoService()
photo = service.create_photo(file, user)
# Task should be queued automatically

# Test ModerationService integration
from apps.moderation.services import ModerationService
ModerationService.approve_submission(submission_id, reviewer)
# Email notification should be queued

Task Catalog

Task Retry Configuration

All tasks implement retry logic:

• Max Retries: 2-3 (task-dependent)
• Retry Delay: 60 seconds base (exponential backoff)
• Failure Handling: Logged to Sentry and application logs

Task Priority

Tasks are executed in the order they're queued. For priority queuing, configure Celery with multiple queues:

# config/celery.py (future enhancement)
CELERY_TASK_ROUTES = {
    'apps.media.tasks.process_uploaded_image': {'queue': 'media'},
    'apps.moderation.tasks.send_moderation_notification': {'queue': 'notifications'},
}

Monitoring & Debugging

View Task Status

from celery.result import AsyncResult

result = AsyncResult('task-id-here')
print(result.state)  # PENDING, STARTED, SUCCESS, FAILURE
print(result.info)   # Result or error details

Flower Dashboard

Access Flower at /flower/ (production only) to:

• View active tasks
• Monitor worker status
• View task history
• Inspect failed tasks
• Retry failed tasks

Logs

# View worker logs
tail -f /var/log/celery/worker.log

# View beat logs
tail -f /var/log/celery/beat.log

# View Django logs (includes task execution)
tail -f django/logs/django.log

Troubleshooting

Common Issues

1. Tasks not executing

   • Check Redis connection: redis-cli ping
   • Verify Celery worker is running: ps aux | grep celery
   • Check for errors in worker logs

2. Emails not sending

   • Verify EMAIL_BACKEND configuration
   • Check SMTP credentials
   • Review email logs in console (development)

3. Scheduled tasks not running

   • Ensure Celery Beat is running
   • Check Beat logs for scheduling errors
   • Verify CELERY_BEAT_SCHEDULE configuration

4. Task failures

   • Check Sentry for error reports
   • Review worker logs
   • Test task in Django shell

Performance Tuning

# Increase worker concurrency
celery -A config worker --concurrency=4

# Use different pool implementation
celery -A config worker --pool=gevent

# Set task time limits
CELERY_TASK_TIME_LIMIT = 30 * 60  # 30 minutes (already configured)

Configuration Options

Environment Variables

Variable	Required	Default	Description
REDIS_URL	Yes*	redis://localhost:6379/0	Redis connection URL
CELERY_BROKER_URL	Yes*	Same as REDIS_URL	Celery message broker
CELERY_RESULT_BACKEND	Yes*	redis://localhost:6379/1	Task result storage
EMAIL_BACKEND	No	Console (dev) / SMTP (prod)	Email backend
EMAIL_HOST	Yes**	-	SMTP host
EMAIL_PORT	Yes**	587	SMTP port
EMAIL_HOST_USER	Yes**	-	SMTP username
EMAIL_HOST_PASSWORD	Yes**	-	SMTP password
DEFAULT_FROM_EMAIL	Yes	noreply@thrillwiki.com	From email address
SITE_URL	Yes	http://localhost:8000	Site URL for emails
FLOWER_ENABLED	No	False	Enable Flower monitoring
FLOWER_BASIC_AUTH	No**	-	Flower authentication

* Not required if using eager mode in development
** Required for production email sending

Next Steps

Future Enhancements

1. Task Prioritization

   • Implement multiple queues for different priority levels
   • Critical tasks (password reset) in high-priority queue
   • Bulk operations in low-priority queue

2. Advanced Monitoring

   • Set up Prometheus metrics
   • Configure Grafana dashboards
   • Add task duration tracking

3. Email Improvements

   • Add plain text email versions
   • Implement email templates for all notification types
   • Add email preference management

4. Scalability

   • Configure multiple Celery workers
   • Implement auto-scaling based on queue size
   • Add Redis Sentinel for high availability

5. Additional Tasks

   • Backup generation tasks
   • Data export tasks
   • Analytics report generation

Success Criteria ✅

All success criteria for Phase 7 have been met:

• ✅ Celery workers running successfully
• ✅ Tasks executing asynchronously
• ✅ Email notifications working (console backend configured)
• ✅ Scheduled tasks configured and ready
• ✅ Flower monitoring configured for production
• ✅ Error handling and retries implemented
• ✅ Integration with existing services complete
• ✅ Comprehensive documentation created

Files Created

• config/celery.py - Celery app configuration
• config/__init__.py - Updated to load Celery
• templates/emails/base.html - Base email template
• templates/emails/welcome.html - Welcome email
• templates/emails/password_reset.html - Password reset email
• templates/emails/moderation_approved.html - Approval notification
• templates/emails/moderation_rejected.html - Rejection notification
• apps/media/tasks.py - Media processing tasks
• apps/moderation/tasks.py - Moderation workflow tasks
• apps/users/tasks.py - User management tasks
• apps/entities/tasks.py - Entity statistics tasks
• PHASE_7_CELERY_COMPLETE.md - This documentation

Files Modified

• config/settings/base.py - Added Celery Beat schedule, SITE_URL, DEFAULT_FROM_EMAIL
• config/urls.py - Added Flower URL routing
• apps/media/services.py - Integrated photo processing task
• apps/moderation/services.py - Integrated email notification tasks

Dependencies

All dependencies were already included in requirements/base.txt:

• celery[redis]==5.3.4
• django-celery-beat==2.5.0
• django-celery-results==2.5.1
• flower==2.0.1

Summary

Phase 7 successfully implements a complete background task processing system with Celery. The system handles:

• Asynchronous image processing
• Email notifications for moderation workflow
• Scheduled maintenance tasks
• Statistics updates
• Token cleanup

The implementation is production-ready with proper error handling, retry logic, monitoring, and documentation.

Phase 7: COMPLETE ✅