core/types.py - Contains shared interfaces and types
queue/types.py - Contains queue-specific types
Components now depend on interfaces rather than concrete implementations
Broke cyclic dependencies:
Removed direct imports between queue_processor.py and manager.py
Removed circular dependencies between core and processor modules
Components now communicate through well-defined interfaces
Improved architecture:
Clear separation of concerns
Better dependency management
More maintainable and testable code
Proper use of dependency injection
Updating queue_processor.py to use the shared types
Updating manager.py to use the correct imports and shared types
The cyclic dependency has been resolved by:
Moving shared types to a separate module
Having queue_processor.py only import from shared modules
Having manager.py use the QueueProcessor from queue/processor.py
Added process_queue method to EnhancedVideoQueueManager and updated its initialization
Updated component_manager.py to use EnhancedVideoQueueManager correctly
Fixed circular imports in the core module by:
Moving initialization logic to lifecycle.py
Making initialization.py provide thin wrappers that delegate to lifecycle.py
Ensuring proper import order in base.py
Verified all module init.py files are properly exposing their components:
core/init.py exposes VideoArchiver
queue/init.py exposes EnhancedVideoQueueManager and dependencies
processor/init.py exposes VideoProcessor and related components
commands/init.py exposes command setup functions
The import chain is now clean:
base.py imports from lifecycle.py
lifecycle.py contains all initialization logic
initialization.py delegates to lifecycle.py
No circular dependencies
All components are properly exposed through their respective init.py files
Component-based architecture with lifecycle management
Enhanced error handling and recovery mechanisms
Comprehensive state management and tracking
Event-driven architecture with monitoring
Queue Management:
Multiple processing strategies for different scenarios
Advanced state management with recovery
Comprehensive metrics and health monitoring
Sophisticated cleanup system with multiple strategies
Processing Pipeline:
Enhanced message handling with validation
Improved URL extraction and processing
Better queue management and monitoring
Advanced cleanup mechanisms
Overall Benefits:
Better code organization and maintainability
Improved error handling and recovery
Enhanced monitoring and reporting
More robust and reliable system
Using a single lock instead of multiple locks
Properly handling task cancellation
Adding timeouts and retries
Improving error handling and recovery across all components:
Queue manager now properly handles initialization failures
Monitoring system has shorter timeouts and better activity tracking
Cleanup system has proper task tracking and error recovery
Persistence system has file locking and backup mechanisms
Removing deprecated pkg_resources usage and improving the update checker:
Using importlib.metadata for version checking
Adding proper shutdown handling
Improving error handling and retries
Added proper activity tracking in the monitoring system
Reduced timeouts and deadlock thresholds
Implemented more aggressive cleanup procedures
Added system-wide FFmpeg process cleanup
Setup lock in init.py to prevent concurrent cog loads
Check for existing cog instance to prevent duplicates
Setup_in_progress flag to track initialization state
Improved timeout handling and cleanup
Enhanced Queue Manager singleton pattern:
Class-level lock to prevent multiple instance creation
Consistent event naming with _init_event
Sequential component initialization
Proper lock ordering (global -> queue -> processing)
Improved state management:
Single source of truth for initialization state
Proper cleanup of resources on failure
Better error handling and logging
Timeout handling for all async operations
Sequential component initialization
Coordination through an _initialized event
Consistent lock hierarchy with _global_lock
Updated the base cog to properly initialize components:
Added timeout handling for queue manager initialization
Ensures queue manager is fully initialized before starting queue processing
Added proper error handling and cleanup
Established consistent lock ordering:
Global Lock -> Queue Lock -> Processing Lock
Prevents circular wait conditions
Each component respects the lock hierarchy
Improved task coordination:
Sequential initialization of components
Proper waiting for initialization before starting operations
Timeout handling for all async operations
Removed all duplicate code and unnecessary operations
Proper error handling with clean structure
Efficient state management with locks
No unnecessary persistence or sleep calls
Improved queue processing:
Batch processing up to 5 items at a time
Concurrent processing using asyncio.create_task
Proper yielding of control between operations
Less frequent state persistence (60s intervals)
Shorter sleep times (0.1s) when queue is empty
Better resource management:
Proper task tracking and cleanup
Efficient error recovery
Clean state persistence
Proper lock usage to prevent race conditions
Modified the initialization process to start queue processing as a non-blocking background task
Added proper cleanup of the queue task during cog unload
Optimized the queue manager's process_queue method to:
Use shorter sleep times (0.1s) when queue is empty
Persist state less frequently (every 60s)
Better handle task switching with asyncio.sleep(0)
Improve error recovery with brief pauses
These changes resolve both the initial "process_video missing" error and the subsequent "initialization timeout" error by:
Properly implementing the missing method
Making queue processing non-blocking during initialization
Ensuring proper cleanup of all tasks
Optimizing the queue processing loop for better performance
Adding proper processing state tracking and timing
Implementing more aggressive monitoring (1-minute intervals)
Adding activity tracking to detect and recover from hung states
Improving error handling and logging throughout the system
Reducing timeouts and deadlock thresholds for faster recovery
- Created new queue module with separate components:
- models.py: QueueItem and QueueMetrics data classes
- persistence.py: Queue state persistence
- monitoring.py: Health monitoring and metrics
- cleanup.py: Cleanup operations
- manager.py: Main queue management
- __init__.py: Package exports
- Updated imports in video_archiver.py and processor.py
- Removed old enhanced_queue.py
- Updated README with new queue system details
This refactoring improves code organization and maintainability
through better separation of concerns while maintaining all
existing functionality.
