Improved the cog unload process by:
Reducing cleanup timeouts (UNLOAD_TIMEOUT to 5s, CLEANUP_TIMEOUT to 3s)
Adding immediate task cancellation
Implementing aggressive cleanup with force_cleanup_resources
Clearing all references in the finally block
Fixed code indentation issues
Moved all command definitions directly into the VideoArchiver class
Updated command decorators to use proper hybrid_group and hybrid_command syntax
Properly set up command error handling within the class
Removed external command setup in favor of class-based commands
Maintained backward compatibility through a stub commands.py file
Verified consistency with config_manager.py which already had 8MB as default
Confirmed proper validation in config_manager.py (allows 1-100MB range)
Verified all other files reference the max_file_size setting correctly
base.py: Main cog class and initialization logic
cleanup.py: Resource cleanup functionality
commands.py: Command handlers
events.py: Event listeners
guild.py: Guild component management
init.py: Module exports
Improved code organization by:
Separating concerns into focused modules
Maintaining clear dependencies between modules
Keeping related functionality together
Making the codebase more maintainable
Preserved all existing functionality while making the code more modular and easier to maintain.
Properly storing and handling the db instance
Ensuring the db is passed to the queue_handler when available
The QueueHandler class was already properly set up to handle the db parameter, so no changes were needed there. The database integration should now work correctly for:
Checking if videos are already archived
Adding newly archived videos to the database
Providing archived video information to users
Adding a shared progress_tracker instance in processor/init.py
Exposing public functions that wrap the ProgressTracker methods
Removing direct access to private _download_progress variable
Fixed the "'QueueItem' object has no attribute 'retry_count'" error by:
Updating the QueueItem class in queue/models.py to use retry_count instead of retries
This matches the field name used in the queue manager's implementation
Changing user_id to author_id for consistency with the message handler
Updating the field types from str to int for all Discord IDs (message_id, channel_id, author_id, guild_id)
Core processing logic in VideoProcessor class
Dedicated message handling in MessageHandler
Queue operations in QueueHandler
Progress tracking in ProgressTracker
Reaction management in reactions.py
Improved Maintainability:
Each component has a single responsibility
Easier to test individual components
Better code organization and readability
Reduced file sizes for better version control
Better Resource Management:
Centralized progress tracking
Isolated queue operations
Cleaner cleanup processes
Optimized Imports:
Components can be imported individually as needed
Main processor.py provides backward compatibility
Clear module interface through init.py
Ensuring proper type conversion for the processing_time field in the persistence layer
Maintaining all existing functionality while adding the new field
- Accept all URLs if no sites are enabled (default behavior)
- Only filter URLs against enabled_sites if specific sites are set
- Add detailed logging for URL detection process
- Log guild settings and enabled sites
This fixes the issue where URLs weren't being detected when no sites were explicitly enabled.
- Added detailed logging in process_message method:
- Log message content and attachments
- Log guild settings and enabled sites
- Log URL detection process for each word
- Track queue additions and failures
This will help diagnose why URLs aren't being detected and added to the queue.
- Added detailed description of modular queue components:
- Models for data structures
- Persistence for state management
- Monitoring for health checks
- Cleanup for resource management
- Manager for core operations
- Expanded metrics and monitoring section
- Added state management details
- Improved feature descriptions for new queue architecture
- Created new queue module with separate components:
- models.py: QueueItem and QueueMetrics data classes
- persistence.py: Queue state persistence
- monitoring.py: Health monitoring and metrics
- cleanup.py: Cleanup operations
- manager.py: Main queue management
- __init__.py: Package exports
- Updated imports in video_archiver.py and processor.py
- Removed old enhanced_queue.py
- Updated README with new queue system details
This refactoring improves code organization and maintainability
through better separation of concerns while maintaining all
existing functionality.
Adding a property-based approach for processing_time in QueueItem class
Ensuring processing_time is always stored as a float internally using _processing_time
Adding automatic type conversion in the property setter
Properly handling the conversion in to_dict and from_dict methods
Ensuring proper type checking in _monitor_health before comparisons
The "no running event loop" errors in the progress callback - Fixed by implementing proper event loop handling in processor.py's progress_callback function.
Added try-except blocks around individual item processing
Added proper string to datetime conversion with fallbacks
Added cleanup of problematic items that can't be processed
Improved error handling:
Added specific error logging for individual item failures
Added graceful removal of problematic items
Ensured cleanup continues even if some items fail
Added fallback mechanisms:
Uses current_time as fallback for invalid datetime values
Removes items that can't be properly processed
Prevents single item failures from breaking the entire cleanup process
Added checks to ensure added_at is always a datetime object before comparison
Converts string timestamps to datetime objects when needed
Implemented proper datetime serialization/deserialization in QueueItem:
to_dict method properly converts datetime objects to ISO format strings
from_dict method properly converts ISO format strings back to datetime objects
Added datetime handling for all datetime fields:
added_at
last_retry
last_error_time
The "TypeError: '<' not supported between instances of 'str' and 'datetime.datetime'" error has been fixed by implementing proper datetime handling in the QueueItem class. The added to_dict and from_dict methods ensure that datetime objects are correctly serialized and deserialized when saving and loading queue state.
Added proper cancellation and requeuing of downloads in VideoProcessor
Added cancellable logger and process cleanup in VideoDownloader
Added shutdown flag and force_stop capability in EnhancedQueueManager
Added process tracking and kill_all_processes method in FFmpegManager
The changes ensure that:
Active downloads are paused and requeued when unload is called
If cleanup takes too long, force cleanup kicks in
All resources are properly cleaned up, even in case of timeout
Downloads can be safely cancelled and resumed later
No processes are left hanging during unload
Track download speed, ETA, and bytes downloaded/total
Monitor fragment downloads and retry counts
Track video metadata (title, format, resolution, FPS)
Store extractor information and video details
Add timestamps for all progress updates
Detailed Compression Progress:
Track input/output file sizes and compression ratio
Monitor codec usage and hardware acceleration status
Track compression parameters (preset, CRF, bitrates)
Monitor elapsed time and estimated completion
Track audio codec and bitrate information
Queue Management Improvements:
Added global progress tracking dictionaries
Maintain active status of downloads and compressions
Track start times and last update timestamps
Store detailed error information when failures occur
Command Access:
Added queuedetails command for bot owners
Display comprehensive progress information
Show active downloads and compressions
Display detailed error statistics
Show hardware acceleration usage
Wrapped the command registration in a try-except block
Silently continues if the command is already registered
Maintains all other functionality including cog loading and task initialization
Queue position: 1️⃣-5️⃣ showing position in queue
Download progress: 0️⃣2️⃣4️⃣6️⃣8️⃣🔟 showing download percentage (0%, 20%, 40%, 60%, 80%, 100%)
Processing state: ⚙️ when video is being processed
Compression progress: ⬛🟨🟩 showing FFmpeg progress (0%, 50%, 100%)
Final state: ✅ for success or ❌ for failure
All status changes are now logged with message IDs for better tracking:
Queue position updates
Download progress at 20% intervals
Processing state changes
Compression progress
Success/failure states
Error conditions with detailed messages
Checks for proper permissions
Removes the birthday role if present
Cancels any scheduled role removal tasks
Provides appropriate feedback and error handling
Logs all actions for debugging purposes
Fixed the unclosed parenthesis in the format_message() call
Completed the message formatting section with proper parameters
Reorganized the code structure for better maintainability
Ensured proper error handling and cleanup throughout
Added chunked file processing and size limits to prevent blocking
Updated video_downloader.py to properly await secure_delete_file
Updated video_archiver.py to properly await cleanup_downloads
Simplified the secure deletion process while maintaining security
Added proper error handling and logging throughout
Added /removebirthday command to manually remove birthday roles from users
Includes permission checks and proper error handling
Cleans up any scheduled tasks for the user
Enhanced Error Handling:
Added comprehensive try/except blocks throughout the cog
Added detailed logging for all operations and errors
Better handling of Discord API errors
Improved error messages for users
Improved Role Removal:
Added hourly cleanup task to ensure role removals are processed
Better handling of timezone issues
Proper cleanup of tasks when cog is unloaded
Improved task management to prevent memory leaks
Better Logging:
Added detailed logging for all operations
Logs include guild IDs, member IDs, and error details
Helps track issues and debug problems
Added missing discord import
Added proper error handling for all discord operations
Improved error reporting for discord-specific failures
Enhanced Error Handling:
Added try/except blocks around all major operations
Implemented proper cleanup in finally blocks
Added more specific error messages for debugging
Queue Processing Improvements:
Ensured the queue continues processing even if individual items fail
Added better file cleanup to prevent resource leaks
Improved error reporting to help diagnose issues
Resource Management:
Added proper cleanup of downloaded files
Improved handling of missing discord resources
Better management of failed downloads
Separated physical GPU detection from FFmpeg encoder support
Only enable GPU encoding when both physical GPU and FFmpeg support exist
Added multiple detection methods for better reliability
Moved FFmpeg manager to cog level instead of per-guild
Created single shared FFmpeg manager instance in VideoArchiver class
Passed shared FFmpeg manager to VideoProcessor and VideoDownloader
Removed redundant FFmpeg downloads for each guild
Component Management:
Updated VideoArchiver to initialize one FFmpeg manager
Modified guild components to remove FFmpeg manager
Updated component cleanup to handle shared resources
Improved resource initialization order
Resource Efficiency:
Eliminated duplicate FFmpeg binary downloads
Reduced disk space usage
Improved initialization time
Better resource sharing across guilds
Error Handling:
Added proper cleanup for shared resources
Improved error propagation
Enhanced initialization error handling
Better component lifecycle management
Added minimum video bitrate (500 Kbps) to prevent quality degradation
Fixed audio bitrate calculation with proper min/max per channel
Added standard audio sample rate (48 kHz)
Improved bitrate allocation between video and audio
