- Added detailed logging in process_message method:
- Log message content and attachments
- Log guild settings and enabled sites
- Log URL detection process for each word
- Track queue additions and failures
This will help diagnose why URLs aren't being detected and added to the queue.
- Created new queue module with separate components:
- models.py: QueueItem and QueueMetrics data classes
- persistence.py: Queue state persistence
- monitoring.py: Health monitoring and metrics
- cleanup.py: Cleanup operations
- manager.py: Main queue management
- __init__.py: Package exports
- Updated imports in video_archiver.py and processor.py
- Removed old enhanced_queue.py
- Updated README with new queue system details
This refactoring improves code organization and maintainability
through better separation of concerns while maintaining all
existing functionality.
Added checks to ensure added_at is always a datetime object before comparison
Converts string timestamps to datetime objects when needed
Implemented proper datetime serialization/deserialization in QueueItem:
to_dict method properly converts datetime objects to ISO format strings
from_dict method properly converts ISO format strings back to datetime objects
Added datetime handling for all datetime fields:
added_at
last_retry
last_error_time
The "TypeError: '<' not supported between instances of 'str' and 'datetime.datetime'" error has been fixed by implementing proper datetime handling in the QueueItem class. The added to_dict and from_dict methods ensure that datetime objects are correctly serialized and deserialized when saving and loading queue state.
Added proper cancellation and requeuing of downloads in VideoProcessor
Added cancellable logger and process cleanup in VideoDownloader
Added shutdown flag and force_stop capability in EnhancedQueueManager
Added process tracking and kill_all_processes method in FFmpegManager
The changes ensure that:
Active downloads are paused and requeued when unload is called
If cleanup takes too long, force cleanup kicks in
All resources are properly cleaned up, even in case of timeout
Downloads can be safely cancelled and resumed later
No processes are left hanging during unload
Track download speed, ETA, and bytes downloaded/total
Monitor fragment downloads and retry counts
Track video metadata (title, format, resolution, FPS)
Store extractor information and video details
Add timestamps for all progress updates
Detailed Compression Progress:
Track input/output file sizes and compression ratio
Monitor codec usage and hardware acceleration status
Track compression parameters (preset, CRF, bitrates)
Monitor elapsed time and estimated completion
Track audio codec and bitrate information
Queue Management Improvements:
Added global progress tracking dictionaries
Maintain active status of downloads and compressions
Track start times and last update timestamps
Store detailed error information when failures occur
Command Access:
Added queuedetails command for bot owners
Display comprehensive progress information
Show active downloads and compressions
Display detailed error statistics
Show hardware acceleration usage
Queue position: 1️⃣-5️⃣ showing position in queue
Download progress: 0️⃣2️⃣4️⃣6️⃣8️⃣🔟 showing download percentage (0%, 20%, 40%, 60%, 80%, 100%)
Processing state: ⚙️ when video is being processed
Compression progress: ⬛🟨🟩 showing FFmpeg progress (0%, 50%, 100%)
Final state: ✅ for success or ❌ for failure
All status changes are now logged with message IDs for better tracking:
Queue position updates
Download progress at 20% intervals
Processing state changes
Compression progress
Success/failure states
Error conditions with detailed messages
Fixed the unclosed parenthesis in the format_message() call
Completed the message formatting section with proper parameters
Reorganized the code structure for better maintainability
Ensured proper error handling and cleanup throughout
Added missing discord import
Added proper error handling for all discord operations
Improved error reporting for discord-specific failures
Enhanced Error Handling:
Added try/except blocks around all major operations
Implemented proper cleanup in finally blocks
Added more specific error messages for debugging
Queue Processing Improvements:
Ensured the queue continues processing even if individual items fail
Added better file cleanup to prevent resource leaks
Improved error reporting to help diagnose issues
Resource Management:
Added proper cleanup of downloaded files
Improved handling of missing discord resources
Better management of failed downloads
Moved FFmpeg manager to cog level instead of per-guild
Created single shared FFmpeg manager instance in VideoArchiver class
Passed shared FFmpeg manager to VideoProcessor and VideoDownloader
Removed redundant FFmpeg downloads for each guild
Component Management:
Updated VideoArchiver to initialize one FFmpeg manager
Modified guild components to remove FFmpeg manager
Updated component cleanup to handle shared resources
Improved resource initialization order
Resource Efficiency:
Eliminated duplicate FFmpeg binary downloads
Reduced disk space usage
Improved initialization time
Better resource sharing across guilds
Error Handling:
Added proper cleanup for shared resources
Improved error propagation
Enhanced initialization error handling
Better component lifecycle management
Added URL pre-filtering to avoid unnecessary yt-dlp checks
Added common video platform patterns for quick filtering
Reduced error logging noise from non-URL words
Improved URL validation efficiency
FFmpeg Management:
Enhanced FFmpeg binary verification
Added robust error handling for subprocess calls
Improved cleanup of failed operations
Added detailed logging for binary operations
Error Handling:
Fixed exception hierarchy in utils/exceptions.py
Added proper error types for different failure scenarios
Enhanced error messages with more context
Improved error propagation through the system
Process Flow:
Added proper timeout handling for subprocess calls
Enhanced environment variable handling
Better cleanup after failures
Added retry mechanisms for failed operations
Created proper exception hierarchy in utils/exceptions.py
Fixed circular imports in utils/init.py
Ensured consistent exception imports across all files
Properly aliased FFmpeg exceptions to avoid naming conflicts
Import Fixes:
Updated processor.py to use VerificationError from ffmpeg.exceptions
Fixed video_downloader.py to use correct exception imports
Created root exceptions.py for better organization
Ensured consistent exception usage across the codebase
Error Handling:
Improved error propagation through the system
Better error messages for different failure types
Enhanced cleanup on errors
More detailed logging of error conditions
FFmpeg Integration:
Fixed FFmpeg binary verification
Better error handling for FFmpeg operations
Improved cleanup of failed operations
Enhanced logging of FFmpeg-related errors
FFmpeg is now properly managed:
Binaries are downloaded and verified on startup
Permissions are properly set
Hardware acceleration is detected and used when available
Resources are cleaned up properly
Error handling has been improved:
Specific exception types for different errors
Better error messages and logging
Appropriate reaction indicators
Enhanced component error handling
Resource management has been enhanced:
Failed downloads are tracked and cleaned up
Temporary files are handled properly
Queue management is more robust
Concurrent downloads are better managed
Verification has been strengthened:
FFmpeg binaries are verified
Video files are validated
Compression results are checked
Component initialization is verified
Added proper ffprobe binary download alongside FFmpeg
Added proper path handling for both binaries
Added verification of both binaries
Fixed FFmpeg compression by using subprocess directly instead of ffmpeg-python
Improved URL Detection:
Switched from regex patterns to yt-dlp simulation for URL detection
Added better error handling for URL checking
Added detailed logging of URL detection results
Reduced Logging Noise:
Only log when a message contains a video URL
Only log URL check errors for actual URLs (containing http://, https://, or www.)
Removed unnecessary debug logging
