youtube-automation/README.md

# YouTube-Directus Automation Service

Automatically extract YouTube video thumbnails and upload them to Directus when YouTube videos are created or updated in your Directus CMS.

## 🎯 MVP Features

- **Webhook Integration**: Receives Directus webhooks on item create/update events
- **YouTube Processing**: Extracts video IDs from various YouTube URL formats  
- **Thumbnail Extraction**: Downloads best quality thumbnail (maxres → high → medium → default)
- **Directus Upload**: Uploads thumbnails to Directus files collection
- **Item Association**: Updates YouTube video items with thumbnail file reference
- **Error Handling**: Robust error handling with logging and fallback mechanisms
- **Security**: Optional webhook signature verification

## 🏗️ Architecture

```
[Directus CMS] → [Webhook] → [FastAPI Service] → [YouTube API] → [Thumbnail Download]
                                    ↓
[Directus Item Update] ← [File Upload] ← [Image Processing] ← [Quality Selection]
```

## 🚀 Quick Start

### 1. Setup Environment

```bash
# Copy environment template
cp .env.example .env

# Edit .env with your configuration
nano .env
```

Required configuration:
```bash
DIRECTUS_URL="https://your-directus-instance.com/"
DIRECTUS_TOKEN="your_directus_api_token"
```

Optional but recommended:
```bash
YOUTUBE_API_KEY="your_youtube_api_key"  # For better reliability
DIRECTUS_WEBHOOK_SECRET="your_webhook_secret"  # For security
```

### 2. Start the Service

```bash
# Install dependencies and start service
./start.sh
```

The service will start on `http://localhost:8000`

### 3. Test the Service

```bash
# Run basic service tests
python3 test_service.py
```

### 4. Configure Directus Webhook

In your Directus admin panel:

1. Go to **Settings** → **Webhooks**
2. Create a new webhook:
   - **Name**: YouTube Thumbnail Automation
   - **Method**: POST
   - **URL**: `http://localhost:8000/webhook/directus` (or your deployed service URL)
   - **Status**: Active
   - **Collections**: Select your media_items collection
   - **Actions**: Create, Update

## 🔧 Configuration

### Environment Variables

| Variable | Required | Description |
|----------|----------|-------------|
| `DIRECTUS_URL` | ✅ | Your Directus instance URL |
| `DIRECTUS_TOKEN` | ✅ | Directus API token with file upload permissions |
| `YOUTUBE_API_KEY` | ⚠️ | YouTube Data API v3 key (recommended) |
| `DIRECTUS_WEBHOOK_SECRET` | ⚠️ | Secret for webhook signature verification |
| `PORT` | ❌ | Service port (default: 8000) |

### YouTube API Setup (Optional but Recommended)

1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Create a new project or select existing
3. Enable **YouTube Data API v3**
4. Create API credentials (API Key)
5. Add the API key to your `.env` file

**Benefits of YouTube API:**
- Better error handling for private/deleted videos
- Access to video metadata
- More reliable thumbnail URLs
- Rate limit information

**Without YouTube API:**
- Service uses fallback thumbnail URLs
- Works for most public videos
- Less error information available

## 📋 Collection Schema Requirements

Your Directus collection should have:

**Required Fields:**
- A field containing YouTube URLs (detected automatically from common field names)
- A file field for thumbnail association (assumes `thumbnail` field name)

**Supported YouTube URL Field Names:**
- `youtube_url`
- `url`
- `link`
- `video_url`
- `youtube_link`

**Example Collection Schema:**
```sql
-- media_items collection
id (Primary Key)
title (String)
youtube_url (String) -- YouTube URL field
thumbnail (File) -- Thumbnail association
description (Text)
created_at (DateTime)
updated_at (DateTime)
```

## 🧪 Testing

### Manual Testing

1. Start the service: `./start.sh`
2. Run tests: `python3 test_service.py`
3. Check service health: `curl http://localhost:8000/health`

### Test with Real Directus

1. Create a new item in your media_items collection
2. Add a YouTube URL (e.g., `https://www.youtube.com/watch?v=dQw4w9WgXcQ`)
3. Check service logs for processing status
4. Verify thumbnail appears in Directus

### Webhook Testing

```bash
# Test webhook endpoint directly
curl -X POST http://localhost:8000/webhook/directus \
  -H "Content-Type: application/json" \
  -d '{
    "event": "items.create",
    "payload": {
      "id": "test-123",
      "youtube_url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
    },
    "keys": ["test-123"]
  }'
```

## 📊 Monitoring

### Service Endpoints

- `GET /` - Service information
- `GET /health` - Health check with configuration status
- `POST /webhook/directus` - Webhook endpoint for Directus

### Health Check Response

```json
{
  "status": "healthy",
  "service": "youtube-directus-automation",
  "timestamp": "2025-01-11T10:30:00Z",
  "config": {
    "directus_url": "https://your-instance.com/",
    "has_directus_token": true,
    "has_youtube_api_key": false,
    "has_webhook_secret": false
  }
}
```

### Logging

Service logs include:
- Webhook event processing
- YouTube video ID extraction
- Thumbnail download status  
- Directus upload results
- Error details and stack traces

## 🔒 Security

### Webhook Signature Verification

Enable webhook security by setting `DIRECTUS_WEBHOOK_SECRET`:

1. Generate a secure random secret
2. Add to `.env`: `DIRECTUS_WEBHOOK_SECRET="your-secret-here"`
3. Configure the same secret in Directus webhook settings
4. Service will verify HMAC-SHA256 signatures automatically

### API Token Permissions

Your Directus token needs:
- Read access to your YouTube videos collection
- Create/Upload access to files collection
- Update access to your YouTube videos collection

## 🚨 Troubleshooting

### Common Issues

**❌ "DIRECTUS_TOKEN not set" error**
- Add your Directus API token to `.env`
- Verify token has correct permissions

**❌ "Could not extract video ID" error**
- Check YouTube URL format
- Ensure URL field contains valid YouTube URLs

**❌ "Failed to download thumbnail" error**
- Video might be private/deleted
- Check internet connectivity
- Try adding YouTube API key for better error handling

**❌ "Directus upload failed" error**
- Check Directus token permissions
- Verify Directus instance is accessible
- Check Directus storage configuration

**❌ Webhook not triggering**
- Verify webhook URL is correct
- Check Directus webhook configuration
- Ensure service is running and accessible
- Check webhook secret configuration

### Debug Mode

For detailed debugging, check service logs when running `./start.sh`.

## 🛠️ Development

### Project Structure

```
youtube-automation/
├── src/
│   └── main.py              # Main FastAPI application
├── requirements.txt         # Python dependencies
├── .env.example            # Environment template
├── .env                    # Your configuration (not in git)
├── start.sh               # Service startup script
├── test_service.py        # Test suite
├── README.md              # This file
└── .taskmaster/           # Task Master project management
```

### Extending the Service

The service is built with extensibility in mind:

- Add new webhook endpoints in `main.py`
- Extend `YouTubeProcessor` class for additional features
- Add new thumbnail processing logic
- Implement additional file format support

### Task Master Integration

This project uses Task Master for project management:

```bash
# View project tasks
task-master list

# Get next task
task-master next

# Update task status
task-master set-status --id=1 --status=done
```

## 🤝 Contributing

1. Check existing tasks: `task-master list`
2. Pick a task: `task-master next` 
3. Make changes and test: `python3 test_service.py`
4. Update task status: `task-master set-status --id=X --status=done`

## 📄 License

This project is part of the Personal AI Assistant ecosystem. See main project license.

---

**🎉 Happy automating!** If you run into issues, check the troubleshooting section or create a task in Task Master.