15 KiB

Raw Blame History

CLAUDE.md - YouTube Summarizer

This file provides guidance to Claude Code (claude.ai/code) when working with the YouTube Summarizer project.

Project Overview

An AI-powered web application that automatically extracts, transcribes, and summarizes YouTube videos. The application supports multiple AI models (OpenAI, Anthropic, DeepSeek), provides various export formats, and includes intelligent caching for efficiency.

Status: Development Ready - All Epic 1 & 2 stories created and ready for implementation

Epic 1: Foundation & Core YouTube Integration (Story 1.1 ✅ Complete, Stories 1.2-1.4 📋 Ready)
Epic 2: AI Summarization Engine (Stories 2.1-2.5 📋 All Created and Ready)
Epic 3: Enhanced User Experience (Future - Ready for story creation)

Quick Start Commands

# Development Setup
cd apps/youtube-summarizer
docker-compose up             # Start full development environment

# BMad Method Story Management
/BMad:agents:sm               # Activate Scrum Master agent
*draft                        # Create next story
*story-checklist              # Validate story quality

# Development Agent Implementation
/BMad:agents:dev              # Activate Development agent
# Follow story specifications in docs/stories/

# Direct Development (without BMad agents)
source venv/bin/activate      # Activate virtual environment
python backend/main.py        # Run backend (port 8000)
cd frontend && npm run dev    # Run frontend (port 3000)

# Testing
pytest backend/tests/ -v      # Backend tests
cd frontend && npm test       # Frontend tests

# Git Operations
git add .
git commit -m "feat: implement story 1.2 - URL validation"
git push origin main

Architecture

YouTube Summarizer
├── API Layer (FastAPI)
│   ├── /api/summarize - Submit URL for summarization
│   ├── /api/summary/{id} - Retrieve summary
│   └── /api/export/{id} - Export in various formats
├── Service Layer
│   ├── YouTube Service - Transcript extraction
│   ├── AI Service - Summary generation
│   └── Cache Service - Performance optimization
└── Data Layer
    ├── SQLite/PostgreSQL - Summary storage
    └── Redis (optional) - Caching layer

Development Workflow - BMad Method

Story-Driven Development Process

Current Epic: Epic 1 - Foundation & Core YouTube Integration
Current Stories:

✅ Story 1.1: Project Setup and Infrastructure (Completed)
📝 Story 1.2: YouTube URL Validation and Parsing (Ready for implementation)
⏳ Story 1.3: Transcript Extraction Service (Pending)
⏳ Story 1.4: Basic Web Interface (Pending)

1. Story Planning (Scrum Master)

# Activate Scrum Master agent
/BMad:agents:sm
*draft                        # Create next story in sequence
*story-checklist              # Validate story completeness

2. Story Implementation (Development Agent)

# Activate Development agent
/BMad:agents:dev
# Review story file: docs/stories/{epic}.{story}.{name}.md
# Follow detailed Dev Notes and architecture references
# Implement all tasks and subtasks as specified

3. Implementation Locations

Based on architecture and story specifications:

Backend API → backend/api/
Backend Services → backend/services/
Backend Models → backend/models/
Frontend Components → frontend/src/components/
Frontend Hooks → frontend/src/hooks/
Frontend API Client → frontend/src/api/

4. Testing Implementation

# Backend testing (pytest)
pytest backend/tests/unit/test_<module>.py -v
pytest backend/tests/integration/ -v

# Frontend testing (Vitest + RTL)
cd frontend && npm test
cd frontend && npm run test:coverage

# Manual testing
docker-compose up             # Full stack
# Visit http://localhost:3000 (frontend)
# Visit http://localhost:8000/docs (API docs)

5. Story Completion

Mark all tasks/subtasks complete in story file
Update story status from "Draft" to "Done"
Run story validation checklist
Update epic progress tracking

Key Implementation Areas

YouTube Integration (`src/services/youtube.py`)

# Primary: youtube-transcript-api
from youtube_transcript_api import YouTubeTranscriptApi

# Fallback: yt-dlp for metadata
import yt_dlp

# Extract video ID from various URL formats
# Handle multiple subtitle languages
# Implement retry logic for failures

AI Summarization (`src/services/summarizer.py`)

# Multi-model support
class SummarizerService:
    def __init__(self):
        self.models = {
            'openai': OpenAISummarizer(),
            'anthropic': AnthropicSummarizer(),
            'deepseek': DeepSeekSummarizer()
        }
    
    async def summarize(self, transcript, model='auto'):
        # Implement model selection logic
        # Handle token limits
        # Generate structured summaries

Caching Strategy (`src/services/cache.py`)

# Cache at multiple levels:
# 1. Transcript cache (by video_id)
# 2. Summary cache (by video_id + model + params)
# 3. Export cache (by summary_id + format)

# Use hash for cache keys
import hashlib

def get_cache_key(video_id: str, model: str, params: dict) -> str:
    key_data = f"{video_id}:{model}:{json.dumps(params, sort_keys=True)}"
    return hashlib.sha256(key_data.encode()).hexdigest()

API Endpoint Patterns

FastAPI Best Practices

from fastapi import APIRouter, HTTPException, BackgroundTasks
from pydantic import BaseModel, HttpUrl

router = APIRouter(prefix="/api", tags=["summarization"])

class SummarizeRequest(BaseModel):
    url: HttpUrl
    model: str = "auto"
    options: dict = {}

@router.post("/summarize")
async def summarize_video(
    request: SummarizeRequest,
    background_tasks: BackgroundTasks
):
    # Validate URL
    # Extract video ID
    # Check cache
    # Queue for processing if needed
    # Return job ID for status checking

Database Schema

-- Main summaries table
CREATE TABLE summaries (
    id UUID PRIMARY KEY,
    video_id VARCHAR(20) NOT NULL,
    video_title TEXT,
    video_url TEXT NOT NULL,
    transcript TEXT,
    summary TEXT,
    key_points JSONB,
    chapters JSONB,
    model_used VARCHAR(50),
    processing_time FLOAT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Cache for performance
CREATE INDEX idx_video_id ON summaries(video_id);
CREATE INDEX idx_created_at ON summaries(created_at);

Error Handling

class YouTubeError(Exception):
    """Base exception for YouTube-related errors"""
    pass

class TranscriptNotAvailable(YouTubeError):
    """Raised when transcript cannot be extracted"""
    pass

class AIServiceError(Exception):
    """Base exception for AI service errors"""
    pass

class TokenLimitExceeded(AIServiceError):
    """Raised when content exceeds model token limit"""
    pass

# Global error handler
@app.exception_handler(YouTubeError)
async def youtube_error_handler(request, exc):
    return JSONResponse(
        status_code=400,
        content={"error": str(exc), "type": "youtube_error"}
    )

Environment Variables

# Required
OPENAI_API_KEY=sk-...        # At least one AI key required
ANTHROPIC_API_KEY=sk-ant-...
DEEPSEEK_API_KEY=sk-...
DATABASE_URL=sqlite:///./data/youtube_summarizer.db
SECRET_KEY=your-secret-key

# Optional but recommended
YOUTUBE_API_KEY=AIza...       # For metadata and quota
REDIS_URL=redis://localhost:6379/0
RATE_LIMIT_PER_MINUTE=30
MAX_VIDEO_LENGTH_MINUTES=180

Testing Guidelines

Unit Test Structure

# tests/unit/test_youtube_service.py
import pytest
from unittest.mock import Mock, patch
from src.services.youtube import YouTubeService

@pytest.fixture
def youtube_service():
    return YouTubeService()

def test_extract_video_id(youtube_service):
    urls = [
        ("https://youtube.com/watch?v=abc123", "abc123"),
        ("https://youtu.be/xyz789", "xyz789"),
        ("https://www.youtube.com/embed/qwe456", "qwe456")
    ]
    for url, expected_id in urls:
        assert youtube_service.extract_video_id(url) == expected_id

Integration Test Pattern

# tests/integration/test_api.py
from fastapi.testclient import TestClient
from src.main import app

client = TestClient(app)

def test_summarize_endpoint():
    response = client.post("/api/summarize", json={
        "url": "https://youtube.com/watch?v=test123",
        "model": "openai"
    })
    assert response.status_code == 200
    assert "job_id" in response.json()

Performance Optimization

Async Everything: Use async/await for all I/O operations
Background Tasks: Process summaries in background
Caching Layers:
- Memory cache for hot data
- Database cache for persistence
- CDN for static exports
Rate Limiting: Implement per-IP and per-user limits
Token Optimization:
- Chunk long transcripts
- Use map-reduce for summaries
- Implement progressive summarization

Security Considerations

Input Validation: Validate all YouTube URLs
API Key Management: Use environment variables, never commit keys
Rate Limiting: Prevent abuse and API exhaustion
CORS Configuration: Restrict to known domains in production
SQL Injection Prevention: Use parameterized queries
XSS Protection: Sanitize all user inputs
Authentication: Implement JWT for user sessions (Phase 3)

Common Issues and Solutions

Issue: Transcript Not Available

# Solution: Implement fallback chain
try:
    transcript = await get_youtube_transcript(video_id)
except TranscriptNotAvailable:
    # Try auto-generated captions
    transcript = await get_auto_captions(video_id)
    if not transcript:
        # Use audio transcription as last resort
        transcript = await transcribe_audio(video_id)

Issue: Token Limit Exceeded

# Solution: Implement chunking
def chunk_transcript(transcript, max_tokens=3000):
    chunks = []
    current_chunk = []
    current_tokens = 0
    
    for segment in transcript:
        segment_tokens = count_tokens(segment)
        if current_tokens + segment_tokens > max_tokens:
            chunks.append(current_chunk)
            current_chunk = [segment]
            current_tokens = segment_tokens
        else:
            current_chunk.append(segment)
            current_tokens += segment_tokens
    
    if current_chunk:
        chunks.append(current_chunk)
    
    return chunks

Issue: Rate Limiting

# Solution: Implement exponential backoff
import asyncio
from typing import Optional

async def retry_with_backoff(
    func, 
    max_retries: int = 3,
    initial_delay: float = 1.0
) -> Optional[Any]:
    delay = initial_delay
    for attempt in range(max_retries):
        try:
            return await func()
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            await asyncio.sleep(delay)
            delay *= 2  # Exponential backoff

Development Tips

Start with Task 1: Setup and environment configuration
Test Early: Write tests as you implement features
Use Type Hints: Improve code quality and IDE support
Document APIs: Use FastAPI's automatic documentation
Log Everything: Implement comprehensive logging for debugging
Cache Aggressively: Reduce API calls and improve response times
Handle Errors Gracefully: Provide helpful error messages to users

Task Master Integration

This project uses Task Master for task management. Key commands:

# View current progress
task-master list

# Get detailed task info
task-master show 1

# Expand task into subtasks
task-master expand --id=1 --research

# Update task with progress
task-master update-task --id=1 --prompt="Completed API structure"

# Complete task
task-master set-status --id=1 --status=done

BMad Method Documentation Structure

Core Documentation

Project README - General project information and setup
Architecture - Complete technical architecture specification
Front-End Spec - UI/UX requirements and component specifications
Original PRD - Complete product requirements document

Epic and Story Management

Epic Index - Epic overview and progress tracking
Epic 1 - Foundation epic details
Epic 2 - AI engine epic details
Epic 3 - Advanced features epic
Stories - Individual story implementations

Current Story Files

Epic 1 - Foundation (Sprint 1):

Story 1.1 - ✅ Project setup (COMPLETED)
Story 1.2 - 📋 URL validation (READY)
Story 1.3 - 📋 Transcript extraction (READY)
Story 1.4 - 📋 Web interface (READY)

Epic 2 - AI Engine (Sprints 2-3):

Story 2.1 - 📋 OpenAI integration (READY)
Story 2.2 - 📋 Pipeline orchestration (READY)
Story 2.3 - 📋 Caching system (READY)
Story 2.4 - 📋 Multi-model AI (READY)
Story 2.5 - 📋 Export features (READY)

Development Workflow

Check Epic Progress: Review Epic Index for current status
Review Next Story: Read story file for implementation details
Follow Dev Notes: Use architecture references and technical specifications
Implement & Test: Follow story tasks/subtasks systematically
Update Progress: Mark story complete and update epic status

Story-Based Implementation Priority

Current Focus: Epic 1 - Foundation & Core YouTube Integration

Sprint 1 (Weeks 1-2) - Epic 1 Implementation:

Story 1.2 - YouTube URL Validation and Parsing (8-12 hours) ⬅️ START HERE
Story 1.3 - Transcript Extraction Service (16-20 hours)
Story 1.4 - Basic Web Interface (16-24 hours)

Sprint 2 (Weeks 3-4) - Epic 2 Core: 4. Story 2.1 - Single AI Model Integration (12-16 hours) 5. Story 2.2 - Summary Generation Pipeline (16-20 hours) 6. Story 2.3 - Caching System Implementation (12-16 hours)

Sprint 3 (Weeks 5-6) - Epic 2 Advanced: 7. Story 2.4 - Multi-Model Support (16-20 hours) 8. Story 2.5 - Export Functionality (12-16 hours)

Developer Resources:

Developer Handoff Guide - Start here for implementation
Sprint Planning - Detailed sprint breakdown
Story Files - All stories with complete Dev Notes

This guide is specifically tailored for Claude Code development on the YouTube Summarizer project.

15 KiB Raw Blame History