Created new shared package with centralized progress tracking:
videoarchiver/shared/progress.py
videoarchiver/shared/init.py
Refactored compression_manager.py to use shared progress module instead of importing from processor
Architecture improvements:
Better separation of concerns
Eliminated cyclic dependencies
Centralized progress tracking
Improved maintainability
- Add CommandError and EventError to imports and __all__
- Add CogError to imports and __all__
- Add TrackingError to imports and __all__
- Add helper functions for error handling and progress tracking
- Add version information and shared instances
- Add __all__ list to exceptions.py
- Fix absolute imports to use relative imports
- Fix duplicate imports
- Fix truncated code in download_core.py
- Add missing imports and type hints
- Fix indentation and formatting issues
Database components with proper schema management
FFmpeg components with process management
Queue system with state management
Processor components with proper handlers
Utility components with shared instances
Configuration components with validation
Initialization sequence is now properly ordered:
Config Manager initialization
Path setup
Database initialization
FFmpeg setup
Queue Manager initialization
Video Processor setup
Guild Components initialization
Update Checker startup
Queue Processing start
Proper cleanup handling is in place:
Component cleanup in reverse order
Resource cleanup with timeouts
Force cleanup for hung processes
System-wide FFmpeg process cleanup
Health monitoring is implemented for all components:
Database connection monitoring
Queue health checks
Processor status tracking
Component state validation
Component-based architecture with lifecycle management
Enhanced error handling and recovery mechanisms
Comprehensive state management and tracking
Event-driven architecture with monitoring
Queue Management:
Multiple processing strategies for different scenarios
Advanced state management with recovery
Comprehensive metrics and health monitoring
Sophisticated cleanup system with multiple strategies
Processing Pipeline:
Enhanced message handling with validation
Improved URL extraction and processing
Better queue management and monitoring
Advanced cleanup mechanisms
Overall Benefits:
Better code organization and maintainability
Improved error handling and recovery
Enhanced monitoring and reporting
More robust and reliable system
Added proper cancellation and requeuing of downloads in VideoProcessor
Added cancellable logger and process cleanup in VideoDownloader
Added shutdown flag and force_stop capability in EnhancedQueueManager
Added process tracking and kill_all_processes method in FFmpegManager
The changes ensure that:
Active downloads are paused and requeued when unload is called
If cleanup takes too long, force cleanup kicks in
All resources are properly cleaned up, even in case of timeout
Downloads can be safely cancelled and resumed later
No processes are left hanging during unload
Queue position: 1️⃣-5️⃣ showing position in queue
Download progress: 0️⃣2️⃣4️⃣6️⃣8️⃣🔟 showing download percentage (0%, 20%, 40%, 60%, 80%, 100%)
Processing state: ⚙️ when video is being processed
Compression progress: ⬛🟨🟩 showing FFmpeg progress (0%, 50%, 100%)
Final state: ✅ for success or ❌ for failure
All status changes are now logged with message IDs for better tracking:
Queue position updates
Download progress at 20% intervals
Processing state changes
Compression progress
Success/failure states
Error conditions with detailed messages
Added chunked file processing and size limits to prevent blocking
Updated video_downloader.py to properly await secure_delete_file
Updated video_archiver.py to properly await cleanup_downloads
Simplified the secure deletion process while maintaining security
Added proper error handling and logging throughout
Added missing discord import
Added proper error handling for all discord operations
Improved error reporting for discord-specific failures
Enhanced Error Handling:
Added try/except blocks around all major operations
Implemented proper cleanup in finally blocks
Added more specific error messages for debugging
Queue Processing Improvements:
Ensured the queue continues processing even if individual items fail
Added better file cleanup to prevent resource leaks
Improved error reporting to help diagnose issues
Resource Management:
Added proper cleanup of downloaded files
Improved handling of missing discord resources
Better management of failed downloads
Added FileCleanupError to utils/exceptions.py
Created root exceptions.py for better organization
Fixed circular imports in utils/init.py
Updated imports in video_archiver.py and update_checker.py
Fixed FFmpeg Management:
Updated extraction logic for BtbN's new archive structure
Added fallback for backward compatibility
Better binary verification and permissions handling
Improved Error Handling:
Proper exception hierarchy
Better error propagation
More detailed error messages
Enhanced cleanup on errors
Updated extraction logic to handle BtbN's new archive structure with binaries in bin directory
Added fallback to old structure for backward compatibility
Improved binary verification and permissions handling
Enhanced error handling during extraction and verification
Fixed Exception Handling:
Added missing FileCleanupError exception
Updated file operations to use proper exceptions
Improved error propagation and logging
Better cleanup error handling
Improved Resource Management:
Better cleanup of failed downloads
Proper handling of temporary files
Enhanced queue management
Improved file deletion with retries
Enhanced Error Reporting:
More detailed error messages
Better logging of cleanup operations
Proper error propagation through the system
Improved user feedback
The system now:
Properly handles FFmpeg's new archive structure
Better manages file operations and cleanup
Provides more detailed error messages
Cleans up resources properly
Shows appropriate error messages for different failure types
FFmpeg is now properly managed:
Binaries are downloaded and verified on startup
Permissions are properly set
Hardware acceleration is detected and used when available
Resources are cleaned up properly
Error handling has been improved:
Specific exception types for different errors
Better error messages and logging
Appropriate reaction indicators
Enhanced component error handling
Resource management has been enhanced:
Failed downloads are tracked and cleaned up
Temporary files are handled properly
Queue management is more robust
Concurrent downloads are better managed
Verification has been strengthened:
FFmpeg binaries are verified
Video files are validated
Compression results are checked
Component initialization is verified
Added proper ffprobe binary download alongside FFmpeg
Added proper path handling for both binaries
Added verification of both binaries
Fixed FFmpeg compression by using subprocess directly instead of ffmpeg-python
Improved URL Detection:
Switched from regex patterns to yt-dlp simulation for URL detection
Added better error handling for URL checking
Added detailed logging of URL detection results
Reduced Logging Noise:
Only log when a message contains a video URL
Only log URL check errors for actual URLs (containing http://, https://, or www.)
Removed unnecessary debug logging
- Replaced extractor.suitable() with direct regex pattern matching
- Store both pattern and site name for better debugging
- Added debug logging for URL matching
- Improved error handling
- Fixed 'str' object has no attribute 'suitable' error
- Now properly instantiating extractors before calling suitable()
- Improved error handling in URL validation
