- Added detailed logging in process_message method:
- Log message content and attachments
- Log guild settings and enabled sites
- Log URL detection process for each word
- Track queue additions and failures
This will help diagnose why URLs aren't being detected and added to the queue.
- Added detailed description of modular queue components:
- Models for data structures
- Persistence for state management
- Monitoring for health checks
- Cleanup for resource management
- Manager for core operations
- Expanded metrics and monitoring section
- Added state management details
- Improved feature descriptions for new queue architecture
- Created new queue module with separate components:
- models.py: QueueItem and QueueMetrics data classes
- persistence.py: Queue state persistence
- monitoring.py: Health monitoring and metrics
- cleanup.py: Cleanup operations
- manager.py: Main queue management
- __init__.py: Package exports
- Updated imports in video_archiver.py and processor.py
- Removed old enhanced_queue.py
- Updated README with new queue system details
This refactoring improves code organization and maintainability
through better separation of concerns while maintaining all
existing functionality.
Adding a property-based approach for processing_time in QueueItem class
Ensuring processing_time is always stored as a float internally using _processing_time
Adding automatic type conversion in the property setter
Properly handling the conversion in to_dict and from_dict methods
Ensuring proper type checking in _monitor_health before comparisons
The "no running event loop" errors in the progress callback - Fixed by implementing proper event loop handling in processor.py's progress_callback function.
Added try-except blocks around individual item processing
Added proper string to datetime conversion with fallbacks
Added cleanup of problematic items that can't be processed
Improved error handling:
Added specific error logging for individual item failures
Added graceful removal of problematic items
Ensured cleanup continues even if some items fail
Added fallback mechanisms:
Uses current_time as fallback for invalid datetime values
Removes items that can't be properly processed
Prevents single item failures from breaking the entire cleanup process
Added checks to ensure added_at is always a datetime object before comparison
Converts string timestamps to datetime objects when needed
Implemented proper datetime serialization/deserialization in QueueItem:
to_dict method properly converts datetime objects to ISO format strings
from_dict method properly converts ISO format strings back to datetime objects
Added datetime handling for all datetime fields:
added_at
last_retry
last_error_time
The "TypeError: '<' not supported between instances of 'str' and 'datetime.datetime'" error has been fixed by implementing proper datetime handling in the QueueItem class. The added to_dict and from_dict methods ensure that datetime objects are correctly serialized and deserialized when saving and loading queue state.
Added proper cancellation and requeuing of downloads in VideoProcessor
Added cancellable logger and process cleanup in VideoDownloader
Added shutdown flag and force_stop capability in EnhancedQueueManager
Added process tracking and kill_all_processes method in FFmpegManager
The changes ensure that:
Active downloads are paused and requeued when unload is called
If cleanup takes too long, force cleanup kicks in
All resources are properly cleaned up, even in case of timeout
Downloads can be safely cancelled and resumed later
No processes are left hanging during unload
Track download speed, ETA, and bytes downloaded/total
Monitor fragment downloads and retry counts
Track video metadata (title, format, resolution, FPS)
Store extractor information and video details
Add timestamps for all progress updates
Detailed Compression Progress:
Track input/output file sizes and compression ratio
Monitor codec usage and hardware acceleration status
Track compression parameters (preset, CRF, bitrates)
Monitor elapsed time and estimated completion
Track audio codec and bitrate information
Queue Management Improvements:
Added global progress tracking dictionaries
Maintain active status of downloads and compressions
Track start times and last update timestamps
Store detailed error information when failures occur
Command Access:
Added queuedetails command for bot owners
Display comprehensive progress information
Show active downloads and compressions
Display detailed error statistics
Show hardware acceleration usage
Wrapped the command registration in a try-except block
Silently continues if the command is already registered
Maintains all other functionality including cog loading and task initialization
Queue position: 1️⃣-5️⃣ showing position in queue
Download progress: 0️⃣2️⃣4️⃣6️⃣8️⃣🔟 showing download percentage (0%, 20%, 40%, 60%, 80%, 100%)
Processing state: ⚙️ when video is being processed
Compression progress: ⬛🟨🟩 showing FFmpeg progress (0%, 50%, 100%)
Final state: ✅ for success or ❌ for failure
All status changes are now logged with message IDs for better tracking:
Queue position updates
Download progress at 20% intervals
Processing state changes
Compression progress
Success/failure states
Error conditions with detailed messages
Checks for proper permissions
Removes the birthday role if present
Cancels any scheduled role removal tasks
Provides appropriate feedback and error handling
Logs all actions for debugging purposes
Fixed the unclosed parenthesis in the format_message() call
Completed the message formatting section with proper parameters
Reorganized the code structure for better maintainability
Ensured proper error handling and cleanup throughout
Added chunked file processing and size limits to prevent blocking
Updated video_downloader.py to properly await secure_delete_file
Updated video_archiver.py to properly await cleanup_downloads
Simplified the secure deletion process while maintaining security
Added proper error handling and logging throughout
Added /removebirthday command to manually remove birthday roles from users
Includes permission checks and proper error handling
Cleans up any scheduled tasks for the user
Enhanced Error Handling:
Added comprehensive try/except blocks throughout the cog
Added detailed logging for all operations and errors
Better handling of Discord API errors
Improved error messages for users
Improved Role Removal:
Added hourly cleanup task to ensure role removals are processed
Better handling of timezone issues
Proper cleanup of tasks when cog is unloaded
Improved task management to prevent memory leaks
Better Logging:
Added detailed logging for all operations
Logs include guild IDs, member IDs, and error details
Helps track issues and debug problems
Added missing discord import
Added proper error handling for all discord operations
Improved error reporting for discord-specific failures
Enhanced Error Handling:
Added try/except blocks around all major operations
Implemented proper cleanup in finally blocks
Added more specific error messages for debugging
Queue Processing Improvements:
Ensured the queue continues processing even if individual items fail
Added better file cleanup to prevent resource leaks
Improved error reporting to help diagnose issues
Resource Management:
Added proper cleanup of downloaded files
Improved handling of missing discord resources
Better management of failed downloads
Separated physical GPU detection from FFmpeg encoder support
Only enable GPU encoding when both physical GPU and FFmpeg support exist
Added multiple detection methods for better reliability
Moved FFmpeg manager to cog level instead of per-guild
Created single shared FFmpeg manager instance in VideoArchiver class
Passed shared FFmpeg manager to VideoProcessor and VideoDownloader
Removed redundant FFmpeg downloads for each guild
Component Management:
Updated VideoArchiver to initialize one FFmpeg manager
Modified guild components to remove FFmpeg manager
Updated component cleanup to handle shared resources
Improved resource initialization order
Resource Efficiency:
Eliminated duplicate FFmpeg binary downloads
Reduced disk space usage
Improved initialization time
Better resource sharing across guilds
Error Handling:
Added proper cleanup for shared resources
Improved error propagation
Enhanced initialization error handling
Better component lifecycle management
Added minimum video bitrate (500 Kbps) to prevent quality degradation
Fixed audio bitrate calculation with proper min/max per channel
Added standard audio sample rate (48 kHz)
Improved bitrate allocation between video and audio
Added URL pre-filtering to avoid unnecessary yt-dlp checks
Added common video platform patterns for quick filtering
Reduced error logging noise from non-URL words
Improved URL validation efficiency
FFmpeg Management:
Enhanced FFmpeg binary verification
Added robust error handling for subprocess calls
Improved cleanup of failed operations
Added detailed logging for binary operations
Error Handling:
Fixed exception hierarchy in utils/exceptions.py
Added proper error types for different failure scenarios
Enhanced error messages with more context
Improved error propagation through the system
Process Flow:
Added proper timeout handling for subprocess calls
Enhanced environment variable handling
Better cleanup after failures
Added retry mechanisms for failed operations
Created proper exception hierarchy in utils/exceptions.py
Fixed circular imports in utils/init.py
Ensured consistent exception imports across all files
Properly aliased FFmpeg exceptions to avoid naming conflicts
Import Fixes:
Updated processor.py to use VerificationError from ffmpeg.exceptions
Fixed video_downloader.py to use correct exception imports
Created root exceptions.py for better organization
Ensured consistent exception usage across the codebase
Error Handling:
Improved error propagation through the system
Better error messages for different failure types
Enhanced cleanup on errors
More detailed logging of error conditions
FFmpeg Integration:
Fixed FFmpeg binary verification
Better error handling for FFmpeg operations
Improved cleanup of failed operations
Enhanced logging of FFmpeg-related errors
Added FileCleanupError to utils/exceptions.py
Created root exceptions.py for better organization
Fixed circular imports in utils/init.py
Updated imports in video_archiver.py and update_checker.py
Fixed FFmpeg Management:
Updated extraction logic for BtbN's new archive structure
Added fallback for backward compatibility
Better binary verification and permissions handling
Improved Error Handling:
Proper exception hierarchy
Better error propagation
More detailed error messages
Enhanced cleanup on errors
Updated extraction logic to handle BtbN's new archive structure with binaries in bin directory
Added fallback to old structure for backward compatibility
Improved binary verification and permissions handling
Enhanced error handling during extraction and verification
Fixed Exception Handling:
Added missing FileCleanupError exception
Updated file operations to use proper exceptions
Improved error propagation and logging
Better cleanup error handling
Improved Resource Management:
Better cleanup of failed downloads
Proper handling of temporary files
Enhanced queue management
Improved file deletion with retries
Enhanced Error Reporting:
More detailed error messages
Better logging of cleanup operations
Proper error propagation through the system
Improved user feedback
The system now:
Properly handles FFmpeg's new archive structure
Better manages file operations and cleanup
Provides more detailed error messages
Cleans up resources properly
Shows appropriate error messages for different failure types
Updated extraction logic to handle BtbN's new archive structure with binaries in the bin directory
Added fallback to old structure for backward compatibility
Improved binary verification and permissions handling
Enhanced error handling during extraction and verification
Error Handling:
Added specific exception types for different errors
Better error propagation through the system
Improved error logging and user feedback
Enhanced cleanup on errors
Resource Management:
Better cleanup of failed downloads
Proper handling of temporary files
Enhanced queue management
Improved file deletion with retries
Verification:
FFmpeg binary verification with version checks
Video file integrity verification
Compression result verification
Archive structure verification
The system now:
Properly handles FFmpeg's new archive structure
Better manages binary downloads and verification
Provides more detailed error messages
Cleans up resources properly
Shows appropriate reactions for different error types
FFmpeg is now properly managed:
Binaries are downloaded and verified on startup
Permissions are properly set
Hardware acceleration is detected and used when available
Resources are cleaned up properly
Error handling has been improved:
Specific exception types for different errors
Better error messages and logging
Appropriate reaction indicators
Enhanced component error handling
Resource management has been enhanced:
Failed downloads are tracked and cleaned up
Temporary files are handled properly
Queue management is more robust
Concurrent downloads are better managed
Verification has been strengthened:
FFmpeg binaries are verified
Video files are validated
Compression results are checked
Component initialization is verified
Added proper ffprobe binary download alongside FFmpeg
Added proper path handling for both binaries
Added verification of both binaries
Fixed FFmpeg compression by using subprocess directly instead of ffmpeg-python
Improved URL Detection:
Switched from regex patterns to yt-dlp simulation for URL detection
Added better error handling for URL checking
Added detailed logging of URL detection results
Reduced Logging Noise:
Only log when a message contains a video URL
Only log URL check errors for actual URLs (containing http://, https://, or www.)
Removed unnecessary debug logging
