4.9 KiB

Raw Permalink Blame History

YouTube Thumbnail Watcher Service

🎯 Overview

A Python backend service that automatically monitors Directus media_items collection and downloads YouTube thumbnails for items that don't have them yet. This provides a robust alternative to Directus Flows.

✅ Features Implemented

Automatic Polling: Checks Directus every 30 seconds for unprocessed YouTube items
Smart Filtering: Only processes items with type='youtube_video' or type='youtube' that have URLs but no thumbnails
Quality Fallback: Downloads best available thumbnail (maxres → high → medium → default)
Robust Error Handling: Continues processing on individual failures
Comprehensive Logging: Detailed logs with statistics and error tracking
Stateless Design: Can be restarted anytime without data loss

🏗️ Architecture

projects/youtube-automation/
├── config.py                  # Configuration management
├── src/
│   ├── directus_client.py     # Directus API wrapper
│   ├── youtube_processor.py   # YouTube thumbnail logic
│   └── watcher_service.py     # Main polling service
├── run_watcher.sh             # Startup script
├── requirements.txt           # Dependencies
└── .env                       # Environment variables

🚀 Usage

Start the Service

# Make sure you have a .env file configured
./run_watcher.sh

Environment Variables (.env)

DIRECTUS_URL="https://enias.zeabur.app/"
DIRECTUS_TOKEN="your_token"
YOUTUBE_API_KEY="optional_youtube_api_key"
POLL_INTERVAL=30
BATCH_SIZE=10
LOG_LEVEL=INFO

Monitor Logs

tail -f /tmp/youtube_watcher.log

📊 Service Statistics

The service tracks and reports:

Items processed
Success/failure rates
Uptime
Processing speed

Example output:

📊 YouTube Thumbnail Watcher Statistics
  Uptime: 0:02:15
  Items Processed: 5
  Succeeded: 3
  Failed: 2
  Success Rate: 60.0%

🔄 Processing Flow

Poll Directus: Query media_items for unprocessed YouTube videos
Extract Video IDs: Parse YouTube URLs to get video identifiers
Download Thumbnails: Try multiple quality levels until success
Upload to Directus: Create file entries in Directus files collection
Update Items: Link thumbnails to original media items
Log Results: Track success/failure for monitoring

🧪 Testing Results

Successfully tested with:

✅ Rick Roll video (dQw4w9WgXcQ) - 65KB thumbnail
✅ LLM Introduction video (zjkBMFhNj_g) - 184KB thumbnail
✅ RAG tutorial video - Full processing pipeline

Failed gracefully with:

❌ Private/deleted videos - Proper error handling
❌ Age-restricted videos - Continues to next item

🔧 Configuration

Database Query

filter = {
    "_and": [
        {"_or": [{"type": {"_eq": "youtube_video"}}, {"type": {"_eq": "youtube"}}]},
        {"url": {"_nnull": True}},
        {"youtube_thumbnail": {"_null": True}}
    ]
}

Thumbnail Quality Priority

maxres: 1280x720 (best quality)
high: 480x360
medium: 320x180
default: 120x90 (fallback)

🎉 Benefits Over Directus Flows

✅ Better Error Handling: Individual failures don't stop the service
✅ Comprehensive Logging: Full visibility into processing
✅ Easy Testing: Can test individual components
✅ Flexible Deployment: Run anywhere, not tied to Directus
✅ Stateless Recovery: Restart anytime without issues
✅ Performance Monitoring: Built-in statistics and metrics

🔍 Monitoring

Key Metrics to Watch

Processing success rate (should be >80% for public videos)
Queue size (items waiting for processing)
Error patterns (404s vs network issues)
Processing speed (items per minute)

Log Levels

INFO: Normal operation and successful processing
WARNING: Failed downloads, retries
ERROR: Critical failures, configuration issues
DEBUG: Detailed processing information

🚀 Production Deployment

Systemd Service (Linux)

# Create service file
sudo nano /etc/systemd/system/youtube-watcher.service

[Unit]
Description=YouTube Thumbnail Watcher
After=network.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/path/to/youtube-automation
ExecStart=/path/to/youtube-automation/run_watcher.sh
Restart=always

[Install]
WantedBy=multi-user.target

Docker Alternative

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "src/watcher_service.py"]

🎯 Next Steps

The core YouTube thumbnail automation is complete and working! The service successfully:

✅ Polls Directus for unprocessed YouTube items
✅ Downloads thumbnails with quality fallback
✅ Uploads files to Directus
✅ Updates media items with thumbnail references
✅ Handles errors gracefully
✅ Provides comprehensive logging and statistics

The service is ready for production use!

4.9 KiB Raw Permalink Blame History