Architecture Changes:

Moved FFmpeg manager to cog level instead of per-guild
Created single shared FFmpeg manager instance in VideoArchiver class
Passed shared FFmpeg manager to VideoProcessor and VideoDownloader
Removed redundant FFmpeg downloads for each guild
Component Management:

Updated VideoArchiver to initialize one FFmpeg manager
Modified guild components to remove FFmpeg manager
Updated component cleanup to handle shared resources
Improved resource initialization order
Resource Efficiency:

Eliminated duplicate FFmpeg binary downloads
Reduced disk space usage
Improved initialization time
Better resource sharing across guilds
Error Handling:

Added proper cleanup for shared resources
Improved error propagation
Enhanced initialization error handling
Better component lifecycle management
This commit is contained in:
pacnpal
2024-11-15 05:17:21 +00:00
parent 06fcf55d90
commit f8fe409cdd
3 changed files with 139 additions and 384 deletions

View File

@@ -263,11 +263,18 @@ class FFmpegDownloader:
def _extract_tar(self, archive_path: Path, temp_dir: str):
"""Extract from tar archive (Linux/macOS)"""
try:
# First decompress the .xz file
# First decompress the .xz file in chunks to prevent blocking
decompressed_path = archive_path.with_suffix('')
chunk_size = 1024 * 1024 # 1MB chunks
with lzma.open(archive_path, 'rb') as compressed:
with open(decompressed_path, 'wb') as decompressed:
shutil.copyfileobj(compressed, decompressed)
while True:
chunk = compressed.read(chunk_size)
if not chunk:
break
decompressed.write(chunk)
# Allow other tasks to run
time.sleep(0)
# Then extract from the tar file
with tarfile.open(decompressed_path, "r:") as tar_ref:
@@ -285,10 +292,22 @@ class FFmpegDownloader:
if not binary_files:
raise DownloadError(f"{binary_name} not found in archive")
tar_ref.extract(binary_files[0], temp_dir)
# Extract binary with progress tracking
member = tar_ref.getmember(binary_files[0])
tar_ref.extract(member, temp_dir)
extracted_path = Path(temp_dir) / binary_files[0]
target_path = self.base_dir / binary_name
shutil.copy2(extracted_path, target_path)
# Copy file in chunks
with open(extracted_path, 'rb') as src, open(target_path, 'wb') as dst:
while True:
chunk = src.read(chunk_size)
if not chunk:
break
dst.write(chunk)
# Allow other tasks to run
time.sleep(0)
logger.info(f"Extracted {binary_name} to {target_path}")
# Clean up decompressed file