Sequential component initialization
Coordination through an _initialized event
Consistent lock hierarchy with _global_lock
Updated the base cog to properly initialize components:
Added timeout handling for queue manager initialization
Ensures queue manager is fully initialized before starting queue processing
Added proper error handling and cleanup
Established consistent lock ordering:
Global Lock -> Queue Lock -> Processing Lock
Prevents circular wait conditions
Each component respects the lock hierarchy
Improved task coordination:
Sequential initialization of components
Proper waiting for initialization before starting operations
Timeout handling for all async operations
Removed all duplicate code and unnecessary operations
Proper error handling with clean structure
Efficient state management with locks
No unnecessary persistence or sleep calls
Improved queue processing:
Batch processing up to 5 items at a time
Concurrent processing using asyncio.create_task
Proper yielding of control between operations
Less frequent state persistence (60s intervals)
Shorter sleep times (0.1s) when queue is empty
Better resource management:
Proper task tracking and cleanup
Efficient error recovery
Clean state persistence
Proper lock usage to prevent race conditions
Modified the initialization process to start queue processing as a non-blocking background task
Added proper cleanup of the queue task during cog unload
Optimized the queue manager's process_queue method to:
Use shorter sleep times (0.1s) when queue is empty
Persist state less frequently (every 60s)
Better handle task switching with asyncio.sleep(0)
Improve error recovery with brief pauses
These changes resolve both the initial "process_video missing" error and the subsequent "initialization timeout" error by:
Properly implementing the missing method
Making queue processing non-blocking during initialization
Ensuring proper cleanup of all tasks
Optimizing the queue processing loop for better performance
Method delegates to queue_handler.process_video
Maintains proper separation of concerns between components
Preserves the existing video processing logic
Adding proper processing state tracking and timing
Implementing more aggressive monitoring (1-minute intervals)
Adding activity tracking to detect and recover from hung states
Improving error handling and logging throughout the system
Reducing timeouts and deadlock thresholds for faster recovery
Removed the separate commands cog loading from init.py since all commands are now part of the main VideoArchiver class
Verified the core/init.py is correctly exporting the VideoArchiver class
Removed the redundant video_archiver.py file
These changes should resolve both the "No module named 'videoarchiver.commands'" error and the asyncio-related issues by:
Eliminating circular imports that could cause import-time issues
Simplifying the cog loading process to only load the main VideoArchiver class
Ensuring proper module structure and exports
Improved the cog unload process by:
Reducing cleanup timeouts (UNLOAD_TIMEOUT to 5s, CLEANUP_TIMEOUT to 3s)
Adding immediate task cancellation
Implementing aggressive cleanup with force_cleanup_resources
Clearing all references in the finally block
Fixed code indentation issues
Moved all command definitions directly into the VideoArchiver class
Updated command decorators to use proper hybrid_group and hybrid_command syntax
Properly set up command error handling within the class
Removed external command setup in favor of class-based commands
Maintained backward compatibility through a stub commands.py file
Verified consistency with config_manager.py which already had 8MB as default
Confirmed proper validation in config_manager.py (allows 1-100MB range)
Verified all other files reference the max_file_size setting correctly
base.py: Main cog class and initialization logic
cleanup.py: Resource cleanup functionality
commands.py: Command handlers
events.py: Event listeners
guild.py: Guild component management
init.py: Module exports
Improved code organization by:
Separating concerns into focused modules
Maintaining clear dependencies between modules
Keeping related functionality together
Making the codebase more maintainable
Preserved all existing functionality while making the code more modular and easier to maintain.
Properly storing and handling the db instance
Ensuring the db is passed to the queue_handler when available
The QueueHandler class was already properly set up to handle the db parameter, so no changes were needed there. The database integration should now work correctly for:
Checking if videos are already archived
Adding newly archived videos to the database
Providing archived video information to users
Adding a shared progress_tracker instance in processor/init.py
Exposing public functions that wrap the ProgressTracker methods
Removing direct access to private _download_progress variable
Fixed the "'QueueItem' object has no attribute 'retry_count'" error by:
Updating the QueueItem class in queue/models.py to use retry_count instead of retries
This matches the field name used in the queue manager's implementation
Changing user_id to author_id for consistency with the message handler
Updating the field types from str to int for all Discord IDs (message_id, channel_id, author_id, guild_id)
Core processing logic in VideoProcessor class
Dedicated message handling in MessageHandler
Queue operations in QueueHandler
Progress tracking in ProgressTracker
Reaction management in reactions.py
Improved Maintainability:
Each component has a single responsibility
Easier to test individual components
Better code organization and readability
Reduced file sizes for better version control
Better Resource Management:
Centralized progress tracking
Isolated queue operations
Cleaner cleanup processes
Optimized Imports:
Components can be imported individually as needed
Main processor.py provides backward compatibility
Clear module interface through init.py
Ensuring proper type conversion for the processing_time field in the persistence layer
Maintaining all existing functionality while adding the new field
- Accept all URLs if no sites are enabled (default behavior)
- Only filter URLs against enabled_sites if specific sites are set
- Add detailed logging for URL detection process
- Log guild settings and enabled sites
This fixes the issue where URLs weren't being detected when no sites were explicitly enabled.
- Added detailed logging in process_message method:
- Log message content and attachments
- Log guild settings and enabled sites
- Log URL detection process for each word
- Track queue additions and failures
This will help diagnose why URLs aren't being detected and added to the queue.
- Added detailed description of modular queue components:
- Models for data structures
- Persistence for state management
- Monitoring for health checks
- Cleanup for resource management
- Manager for core operations
- Expanded metrics and monitoring section
- Added state management details
- Improved feature descriptions for new queue architecture
- Created new queue module with separate components:
- models.py: QueueItem and QueueMetrics data classes
- persistence.py: Queue state persistence
- monitoring.py: Health monitoring and metrics
- cleanup.py: Cleanup operations
- manager.py: Main queue management
- __init__.py: Package exports
- Updated imports in video_archiver.py and processor.py
- Removed old enhanced_queue.py
- Updated README with new queue system details
This refactoring improves code organization and maintainability
through better separation of concerns while maintaining all
existing functionality.
Adding a property-based approach for processing_time in QueueItem class
Ensuring processing_time is always stored as a float internally using _processing_time
Adding automatic type conversion in the property setter
Properly handling the conversion in to_dict and from_dict methods
Ensuring proper type checking in _monitor_health before comparisons
The "no running event loop" errors in the progress callback - Fixed by implementing proper event loop handling in processor.py's progress_callback function.
Added try-except blocks around individual item processing
Added proper string to datetime conversion with fallbacks
Added cleanup of problematic items that can't be processed
Improved error handling:
Added specific error logging for individual item failures
Added graceful removal of problematic items
Ensured cleanup continues even if some items fail
Added fallback mechanisms:
Uses current_time as fallback for invalid datetime values
Removes items that can't be properly processed
Prevents single item failures from breaking the entire cleanup process
Added checks to ensure added_at is always a datetime object before comparison
Converts string timestamps to datetime objects when needed
Implemented proper datetime serialization/deserialization in QueueItem:
to_dict method properly converts datetime objects to ISO format strings
from_dict method properly converts ISO format strings back to datetime objects
Added datetime handling for all datetime fields:
added_at
last_retry
last_error_time
The "TypeError: '<' not supported between instances of 'str' and 'datetime.datetime'" error has been fixed by implementing proper datetime handling in the QueueItem class. The added to_dict and from_dict methods ensure that datetime objects are correctly serialized and deserialized when saving and loading queue state.
Added proper cancellation and requeuing of downloads in VideoProcessor
Added cancellable logger and process cleanup in VideoDownloader
Added shutdown flag and force_stop capability in EnhancedQueueManager
Added process tracking and kill_all_processes method in FFmpegManager
The changes ensure that:
Active downloads are paused and requeued when unload is called
If cleanup takes too long, force cleanup kicks in
All resources are properly cleaned up, even in case of timeout
Downloads can be safely cancelled and resumed later
No processes are left hanging during unload
Track download speed, ETA, and bytes downloaded/total
Monitor fragment downloads and retry counts
Track video metadata (title, format, resolution, FPS)
Store extractor information and video details
Add timestamps for all progress updates
Detailed Compression Progress:
Track input/output file sizes and compression ratio
Monitor codec usage and hardware acceleration status
Track compression parameters (preset, CRF, bitrates)
Monitor elapsed time and estimated completion
Track audio codec and bitrate information
Queue Management Improvements:
Added global progress tracking dictionaries
Maintain active status of downloads and compressions
Track start times and last update timestamps
Store detailed error information when failures occur
Command Access:
Added queuedetails command for bot owners
Display comprehensive progress information
Show active downloads and compressions
Display detailed error statistics
Show hardware acceleration usage
Wrapped the command registration in a try-except block
Silently continues if the command is already registered
Maintains all other functionality including cog loading and task initialization
Queue position: 1️⃣-5️⃣ showing position in queue
Download progress: 0️⃣2️⃣4️⃣6️⃣8️⃣🔟 showing download percentage (0%, 20%, 40%, 60%, 80%, 100%)
Processing state: ⚙️ when video is being processed
Compression progress: ⬛🟨🟩 showing FFmpeg progress (0%, 50%, 100%)
Final state: ✅ for success or ❌ for failure
All status changes are now logged with message IDs for better tracking:
Queue position updates
Download progress at 20% intervals
Processing state changes
Compression progress
Success/failure states
Error conditions with detailed messages
Checks for proper permissions
Removes the birthday role if present
Cancels any scheduled role removal tasks
Provides appropriate feedback and error handling
Logs all actions for debugging purposes
Fixed the unclosed parenthesis in the format_message() call
Completed the message formatting section with proper parameters
Reorganized the code structure for better maintainability
Ensured proper error handling and cleanup throughout
