Properly exposing QueueProcessor in processor/init.py
Updating queue/init.py to import and expose all necessary types (QueuePriority, ProcessingMetrics)
Ensuring consistent import patterns across the codebase
The changes maintain the cog's architecture while fixing the import issues:
Module reloading system now includes all dependencies
Package hierarchy properly exposes all needed types
Import structure is consistent across components
All necessary types and models are accessible
core/types.py - Contains shared interfaces and types
queue/types.py - Contains queue-specific types
Components now depend on interfaces rather than concrete implementations
Broke cyclic dependencies:
Removed direct imports between queue_processor.py and manager.py
Removed circular dependencies between core and processor modules
Components now communicate through well-defined interfaces
Improved architecture:
Clear separation of concerns
Better dependency management
More maintainable and testable code
Proper use of dependency injection
from videoarchiver.utils import progress_tracker
Updated the main init.py to properly handle module reloading:
Added 'videoarchiver.processor' and 'videoarchiver.processor.core' to modules_to_reload
Added explicit import and reload of the processor module
These changes ensure that:
The progress_tracker is properly imported from the utils package
All necessary modules are reloaded during initialization
Circular imports are avoided
The package structure maintains proper dependency flow
Updated all files to import REACTIONS from constants.py instead of reactions.py:
queue_processor.py
message_handler.py
reactions.py (now only contains reaction handling functions)
events.py (now imports REACTIONS from constants.py and handle_archived_reaction from reactions.py)
core.py
queue_handler.py
processor/init.py
Component-based architecture with lifecycle management
Enhanced error handling and recovery mechanisms
Comprehensive state management and tracking
Event-driven architecture with monitoring
Queue Management:
Multiple processing strategies for different scenarios
Advanced state management with recovery
Comprehensive metrics and health monitoring
Sophisticated cleanup system with multiple strategies
Processing Pipeline:
Enhanced message handling with validation
Improved URL extraction and processing
Better queue management and monitoring
Advanced cleanup mechanisms
Overall Benefits:
Better code organization and maintainability
Improved error handling and recovery
Enhanced monitoring and reporting
More robust and reliable system
Method delegates to queue_handler.process_video
Maintains proper separation of concerns between components
Preserves the existing video processing logic
Adding proper processing state tracking and timing
Implementing more aggressive monitoring (1-minute intervals)
Adding activity tracking to detect and recover from hung states
Improving error handling and logging throughout the system
Reducing timeouts and deadlock thresholds for faster recovery
Properly storing and handling the db instance
Ensuring the db is passed to the queue_handler when available
The QueueHandler class was already properly set up to handle the db parameter, so no changes were needed there. The database integration should now work correctly for:
Checking if videos are already archived
Adding newly archived videos to the database
Providing archived video information to users
Core processing logic in VideoProcessor class
Dedicated message handling in MessageHandler
Queue operations in QueueHandler
Progress tracking in ProgressTracker
Reaction management in reactions.py
Improved Maintainability:
Each component has a single responsibility
Easier to test individual components
Better code organization and readability
Reduced file sizes for better version control
Better Resource Management:
Centralized progress tracking
Isolated queue operations
Cleaner cleanup processes
Optimized Imports:
Components can be imported individually as needed
Main processor.py provides backward compatibility
Clear module interface through init.py
