Added process_queue method to EnhancedVideoQueueManager and updated its initialization
Updated component_manager.py to use EnhancedVideoQueueManager correctly
Fixed circular imports in the core module by:
Moving initialization logic to lifecycle.py
Making initialization.py provide thin wrappers that delegate to lifecycle.py
Ensuring proper import order in base.py
Verified all module init.py files are properly exposing their components:
core/init.py exposes VideoArchiver
queue/init.py exposes EnhancedVideoQueueManager and dependencies
processor/init.py exposes VideoProcessor and related components
commands/init.py exposes command setup functions
The import chain is now clean:
base.py imports from lifecycle.py
lifecycle.py contains all initialization logic
initialization.py delegates to lifecycle.py
No circular dependencies
All components are properly exposed through their respective init.py files
Database components with proper schema management
FFmpeg components with process management
Queue system with state management
Processor components with proper handlers
Utility components with shared instances
Configuration components with validation
Initialization sequence is now properly ordered:
Config Manager initialization
Path setup
Database initialization
FFmpeg setup
Queue Manager initialization
Video Processor setup
Guild Components initialization
Update Checker startup
Queue Processing start
Proper cleanup handling is in place:
Component cleanup in reverse order
Resource cleanup with timeouts
Force cleanup for hung processes
System-wide FFmpeg process cleanup
Health monitoring is implemented for all components:
Database connection monitoring
Queue health checks
Processor status tracking
Component state validation
guild.py correctly imports cleanup_downloads directly from file_ops.py
initialization.py now correctly imports cleanup_downloads from file_ops.py
cleanup.py now correctly imports cleanup_downloads from file_ops.py
utils/init.py correctly re-exports both functions
The circular import has been resolved by:
Having all files import directly from file_ops.py
Removing the re-export of cleanup_downloads from cleanup.py
Ensuring proper async/await usage throughout
Component-based architecture with lifecycle management
Enhanced error handling and recovery mechanisms
Comprehensive state management and tracking
Event-driven architecture with monitoring
Queue Management:
Multiple processing strategies for different scenarios
Advanced state management with recovery
Comprehensive metrics and health monitoring
Sophisticated cleanup system with multiple strategies
Processing Pipeline:
Enhanced message handling with validation
Improved URL extraction and processing
Better queue management and monitoring
Advanced cleanup mechanisms
Overall Benefits:
Better code organization and maintainability
Improved error handling and recovery
Enhanced monitoring and reporting
More robust and reliable system
Improved the interaction response handling system to better manage both deferred and non-deferred responses
Enhanced error handling around interaction responses
Added proper fallback mechanisms for response handling
Fixed redbot import issues by importing specific exception classes
Ensured database setting is properly synchronized between base.py and config_manager.py
Added proper database initialization and reference handling
New database functionality and commands
Additional video format support (MKV)
Message duration now in hours (0-168)
Enhanced error handling and logging
Queue persistence and management improvements
Hardware acceleration updates
More detailed command descriptions
Improved performance & limitations documentation
Added missing asyncio import in events.py
Ensured all long-running operations are handled in background tasks
Fixed task initialization and management in base.py
Commands will now work correctly during initialization
Video processing features still wait for full initialization
Properly handle both single string and list prefixes
Only video processing features are blocked until initialization completes
The on_message event handler now checks if a message is a command first and lets it process normally
Using a single lock instead of multiple locks
Properly handling task cancellation
Adding timeouts and retries
Improving error handling and recovery across all components:
Queue manager now properly handles initialization failures
Monitoring system has shorter timeouts and better activity tracking
Cleanup system has proper task tracking and error recovery
Persistence system has file locking and backup mechanisms
Removing deprecated pkg_resources usage and improving the update checker:
Using importlib.metadata for version checking
Adding proper shutdown handling
Improving error handling and retries
Added proper activity tracking in the monitoring system
Reduced timeouts and deadlock thresholds
Implemented more aggressive cleanup procedures
Added system-wide FFmpeg process cleanup
Setup lock in init.py to prevent concurrent cog loads
Check for existing cog instance to prevent duplicates
Setup_in_progress flag to track initialization state
Improved timeout handling and cleanup
Enhanced Queue Manager singleton pattern:
Class-level lock to prevent multiple instance creation
Consistent event naming with _init_event
Sequential component initialization
Proper lock ordering (global -> queue -> processing)
Improved state management:
Single source of truth for initialization state
Proper cleanup of resources on failure
Better error handling and logging
Timeout handling for all async operations
Sequential component initialization
Coordination through an _initialized event
Consistent lock hierarchy with _global_lock
Updated the base cog to properly initialize components:
Added timeout handling for queue manager initialization
Ensures queue manager is fully initialized before starting queue processing
Added proper error handling and cleanup
Established consistent lock ordering:
Global Lock -> Queue Lock -> Processing Lock
Prevents circular wait conditions
Each component respects the lock hierarchy
Improved task coordination:
Sequential initialization of components
Proper waiting for initialization before starting operations
Timeout handling for all async operations
Removed all duplicate code and unnecessary operations
Proper error handling with clean structure
Efficient state management with locks
No unnecessary persistence or sleep calls
Improved queue processing:
Batch processing up to 5 items at a time
Concurrent processing using asyncio.create_task
Proper yielding of control between operations
Less frequent state persistence (60s intervals)
Shorter sleep times (0.1s) when queue is empty
Better resource management:
Proper task tracking and cleanup
Efficient error recovery
Clean state persistence
Proper lock usage to prevent race conditions
Modified the initialization process to start queue processing as a non-blocking background task
Added proper cleanup of the queue task during cog unload
Optimized the queue manager's process_queue method to:
Use shorter sleep times (0.1s) when queue is empty
Persist state less frequently (every 60s)
Better handle task switching with asyncio.sleep(0)
Improve error recovery with brief pauses
These changes resolve both the initial "process_video missing" error and the subsequent "initialization timeout" error by:
Properly implementing the missing method
Making queue processing non-blocking during initialization
Ensuring proper cleanup of all tasks
Optimizing the queue processing loop for better performance
Method delegates to queue_handler.process_video
Maintains proper separation of concerns between components
Preserves the existing video processing logic
Adding proper processing state tracking and timing
Implementing more aggressive monitoring (1-minute intervals)
Adding activity tracking to detect and recover from hung states
Improving error handling and logging throughout the system
Reducing timeouts and deadlock thresholds for faster recovery
Removed the separate commands cog loading from init.py since all commands are now part of the main VideoArchiver class
Verified the core/init.py is correctly exporting the VideoArchiver class
Removed the redundant video_archiver.py file
These changes should resolve both the "No module named 'videoarchiver.commands'" error and the asyncio-related issues by:
Eliminating circular imports that could cause import-time issues
Simplifying the cog loading process to only load the main VideoArchiver class
Ensuring proper module structure and exports
Improved the cog unload process by:
Reducing cleanup timeouts (UNLOAD_TIMEOUT to 5s, CLEANUP_TIMEOUT to 3s)
Adding immediate task cancellation
Implementing aggressive cleanup with force_cleanup_resources
Clearing all references in the finally block
Fixed code indentation issues
Moved all command definitions directly into the VideoArchiver class
Updated command decorators to use proper hybrid_group and hybrid_command syntax
Properly set up command error handling within the class
Removed external command setup in favor of class-based commands
Maintained backward compatibility through a stub commands.py file
Verified consistency with config_manager.py which already had 8MB as default
Confirmed proper validation in config_manager.py (allows 1-100MB range)
Verified all other files reference the max_file_size setting correctly
base.py: Main cog class and initialization logic
cleanup.py: Resource cleanup functionality
commands.py: Command handlers
events.py: Event listeners
guild.py: Guild component management
init.py: Module exports
Improved code organization by:
Separating concerns into focused modules
Maintaining clear dependencies between modules
Keeping related functionality together
Making the codebase more maintainable
Preserved all existing functionality while making the code more modular and easier to maintain.
Properly storing and handling the db instance
Ensuring the db is passed to the queue_handler when available
The QueueHandler class was already properly set up to handle the db parameter, so no changes were needed there. The database integration should now work correctly for:
Checking if videos are already archived
Adding newly archived videos to the database
Providing archived video information to users
Adding a shared progress_tracker instance in processor/init.py
Exposing public functions that wrap the ProgressTracker methods
Removing direct access to private _download_progress variable
Fixed the "'QueueItem' object has no attribute 'retry_count'" error by:
Updating the QueueItem class in queue/models.py to use retry_count instead of retries
This matches the field name used in the queue manager's implementation
Changing user_id to author_id for consistency with the message handler
Updating the field types from str to int for all Discord IDs (message_id, channel_id, author_id, guild_id)
Core processing logic in VideoProcessor class
Dedicated message handling in MessageHandler
Queue operations in QueueHandler
Progress tracking in ProgressTracker
Reaction management in reactions.py
Improved Maintainability:
Each component has a single responsibility
Easier to test individual components
Better code organization and readability
Reduced file sizes for better version control
Better Resource Management:
Centralized progress tracking
Isolated queue operations
Cleaner cleanup processes
Optimized Imports:
Components can be imported individually as needed
Main processor.py provides backward compatibility
Clear module interface through init.py
Ensuring proper type conversion for the processing_time field in the persistence layer
Maintaining all existing functionality while adding the new field
- Accept all URLs if no sites are enabled (default behavior)
- Only filter URLs against enabled_sites if specific sites are set
- Add detailed logging for URL detection process
- Log guild settings and enabled sites
This fixes the issue where URLs weren't being detected when no sites were explicitly enabled.
