20 KiB

Raw Blame History

CLAUDE.md - YouTube Summarizer

This file provides guidance to Claude Code (claude.ai/code) when working with the YouTube Summarizer project.

Project Overview

An AI-powered web application that automatically extracts, transcribes, and summarizes YouTube videos. The application supports multiple AI models (OpenAI, Anthropic, DeepSeek), provides various export formats, and includes intelligent caching for efficiency.

Status: Development in Progress - Authentication complete, core features ready for implementation

Epic 1: Foundation & Core YouTube Integration (Stories 1.1-1.4 ✅ Complete, Story 1.5 📋 Ready)
Epic 2: AI Summarization Engine (Stories 2.1-2.5 📋 All Created and Ready)
Epic 3: User Authentication & Session Management (✅ Story 3.1-3.2 Complete, 📋 Story 3.3 Ready)

Quick Start Commands

# Development Setup
cd apps/youtube-summarizer
docker-compose up             # Start full development environment

# Quick Testing (No Auth Required)
open http://localhost:3002/admin    # Direct admin access - No login needed

# BMad Method Story Management
/BMad:agents:sm               # Activate Scrum Master agent
*draft                        # Create next story
*story-checklist              # Validate story quality

# Development Agent Implementation
/BMad:agents:dev              # Activate Development agent
# Follow story specifications in docs/stories/

# Direct Development (without BMad agents)
source venv/bin/activate      # Activate virtual environment
python backend/main.py        # Run backend (port 8000)
cd frontend && npm run dev    # Run frontend (port 3002)

# Testing (Comprehensive Test Runner)
./run_tests.sh run-unit --fail-fast    # Fast unit tests (229 tests in ~0.2s)
./run_tests.sh run-all --coverage      # Complete test suite with coverage
cd frontend && npm test                 # Frontend tests

# Git Operations
git add .
git commit -m "feat: implement story 1.2 - URL validation"
git push origin main

Architecture

YouTube Summarizer
├── Frontend (React + TypeScript)
│   ├── /admin - No-auth admin interface (TESTING)
│   ├── /dashboard - Protected summarizer interface 
│   ├── /login - Authentication flow
│   └── /batch - Batch processing interface
├── API Layer (FastAPI)
│   ├── /api/summarize - Submit URL for summarization
│   ├── /api/summary/{id} - Retrieve summary
│   └── /api/export/{id} - Export in various formats
├── Service Layer
│   ├── YouTube Service - Transcript extraction
│   ├── AI Service - Summary generation (DeepSeek)
│   └── Cache Service - Performance optimization
└── Data Layer
    ├── SQLite/PostgreSQL - Summary storage
    └── Redis (optional) - Caching layer

Testing & Development Access

Admin Page (No Authentication)

URL: http://localhost:3002/admin
Purpose: Direct access for testing and development
Features: Complete YouTube Summarizer functionality without login
Visual: Orange "Admin Mode" badge for clear identification
Use Case: Quick testing, demos, development workflow

Protected Routes (Authentication Required)

Dashboard: http://localhost:3002/dashboard - Main app with user session
History: http://localhost:3002/history - User's summary history
Batch: http://localhost:3002/batch - Batch processing interface

Development Workflow - BMad Method

Story-Driven Development Process

Current Epic: Epic 3 - User Authentication & Session Management
Current Stories:

✅ Epic 1 - Foundation & Core YouTube Integration (Complete)
- ✅ Story 1.1: Project Setup and Infrastructure
- ✅ Story 1.2: YouTube URL Validation and Parsing
- ✅ Story 1.3: Transcript Extraction Service (with mocks)
- ✅ Story 1.4: Basic Web Interface
- ✅ Story 1.5: Video Download and Storage Service
✅ Epic 2 - AI Summarization Engine (Complete)
- ✅ Story 2.1-2.5: All AI pipeline and summarization features
🚀 Epic 3 - User Authentication & Session Management (Current)
- ✅ Story 3.1: User Authentication System (Backend Complete)
- 📝 Story 3.2: Frontend Authentication Integration (Ready for implementation)

1. Story Planning (Scrum Master)

# Activate Scrum Master agent
/BMad:agents:sm
*draft                        # Create next story in sequence
*story-checklist              # Validate story completeness

2. Story Implementation (Development Agent)

# Activate Development agent
/BMad:agents:dev
# Review story file: docs/stories/{epic}.{story}.{name}.md
# Follow detailed Dev Notes and architecture references
# Implement all tasks and subtasks as specified

3. Implementation Locations

Based on architecture and story specifications:

Backend API → backend/api/
Backend Services → backend/services/
Backend Models → backend/models/
Frontend Components → frontend/src/components/
Frontend Hooks → frontend/src/hooks/
Frontend API Client → frontend/src/api/

4. Testing Implementation

# Backend testing (Test Runner - Fast Feedback)
./run_tests.sh run-unit --fail-fast                    # Ultra-fast unit tests (0.2s)
./run_tests.sh run-specific "test_video_service.py"    # Test specific modules  
./run_tests.sh run-integration                         # Integration & API tests
./run_tests.sh run-all --coverage --parallel          # Complete suite with coverage

# Test Discovery & Validation
./run_tests.sh list --category unit                   # See available tests (229 found)
./scripts/validate_test_setup.py                      # Validate test environment

# Frontend testing (Vitest + RTL)
cd frontend && npm test
cd frontend && npm run test:coverage

# Manual testing
docker-compose up             # Full stack
# Visit http://localhost:3000 (frontend)
# Visit http://localhost:8000/docs (API docs)

5. Story Completion

Mark all tasks/subtasks complete in story file
Update story status from "Draft" to "Done"
Run story validation checklist
Update epic progress tracking

Testing & Quality Assurance

Test Runner System

The project includes a production-ready test runner with 229 discovered unit tests and intelligent categorization.

# Fast feedback during development
./run_tests.sh run-unit --fail-fast           # Ultra-fast unit tests (~0.2s)
./run_tests.sh run-all --coverage             # Complete validation
cd frontend && npm test                       # Frontend tests

📖 Complete Testing Guide: See TESTING-INSTRUCTIONS.md for comprehensive testing standards, procedures, and troubleshooting.

Key Implementation Areas

YouTube Integration (`src/services/youtube.py`)

# Primary: youtube-transcript-api
from youtube_transcript_api import YouTubeTranscriptApi

# Fallback: yt-dlp for metadata
import yt_dlp

# Extract video ID from various URL formats
# Handle multiple subtitle languages
# Implement retry logic for failures

AI Summarization (`src/services/summarizer.py`)

# Multi-model support
class SummarizerService:
    def __init__(self):
        self.models = {
            'openai': OpenAISummarizer(),
            'anthropic': AnthropicSummarizer(),
            'deepseek': DeepSeekSummarizer()
        }
    
    async def summarize(self, transcript, model='auto'):
        # Implement model selection logic
        # Handle token limits
        # Generate structured summaries

Caching Strategy (`src/services/cache.py`)

# Cache at multiple levels:
# 1. Transcript cache (by video_id)
# 2. Summary cache (by video_id + model + params)
# 3. Export cache (by summary_id + format)

# Use hash for cache keys
import hashlib

def get_cache_key(video_id: str, model: str, params: dict) -> str:
    key_data = f"{video_id}:{model}:{json.dumps(params, sort_keys=True)}"
    return hashlib.sha256(key_data.encode()).hexdigest()

API Endpoint Patterns

FastAPI Best Practices

from fastapi import APIRouter, HTTPException, BackgroundTasks
from pydantic import BaseModel, HttpUrl

router = APIRouter(prefix="/api", tags=["summarization"])

class SummarizeRequest(BaseModel):
    url: HttpUrl
    model: str = "auto"
    options: dict = {}

@router.post("/summarize")
async def summarize_video(
    request: SummarizeRequest,
    background_tasks: BackgroundTasks
):
    # Validate URL
    # Extract video ID
    # Check cache
    # Queue for processing if needed
    # Return job ID for status checking

Database Schema

-- Main summaries table
CREATE TABLE summaries (
    id UUID PRIMARY KEY,
    video_id VARCHAR(20) NOT NULL,
    video_title TEXT,
    video_url TEXT NOT NULL,
    transcript TEXT,
    summary TEXT,
    key_points JSONB,
    chapters JSONB,
    model_used VARCHAR(50),
    processing_time FLOAT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Cache for performance
CREATE INDEX idx_video_id ON summaries(video_id);
CREATE INDEX idx_created_at ON summaries(created_at);

Error Handling

class YouTubeError(Exception):
    """Base exception for YouTube-related errors"""
    pass

class TranscriptNotAvailable(YouTubeError):
    """Raised when transcript cannot be extracted"""
    pass

class AIServiceError(Exception):
    """Base exception for AI service errors"""
    pass

class TokenLimitExceeded(AIServiceError):
    """Raised when content exceeds model token limit"""
    pass

# Global error handler
@app.exception_handler(YouTubeError)
async def youtube_error_handler(request, exc):
    return JSONResponse(
        status_code=400,
        content={"error": str(exc), "type": "youtube_error"}
    )

Environment Variables

# Required
OPENAI_API_KEY=sk-...        # At least one AI key required
ANTHROPIC_API_KEY=sk-ant-...
DEEPSEEK_API_KEY=sk-...
DATABASE_URL=sqlite:///./data/youtube_summarizer.db
SECRET_KEY=your-secret-key

# Optional but recommended
YOUTUBE_API_KEY=AIza...       # For metadata and quota
REDIS_URL=redis://localhost:6379/0
RATE_LIMIT_PER_MINUTE=30
MAX_VIDEO_LENGTH_MINUTES=180

Testing Guidelines

Test Runner Integration

The project uses a comprehensive test runner system for efficient testing:

# Run specific test modules during development
./run_tests.sh run-specific "backend/tests/unit/test_youtube_service.py"

# Fast feedback loop (discovered 229 tests)
./run_tests.sh run-unit --fail-fast

# Comprehensive testing with coverage
./run_tests.sh run-all --coverage --reports html,json

Unit Test Structure

# tests/unit/test_youtube_service.py
import pytest
from unittest.mock import Mock, patch
from src.services.youtube import YouTubeService

@pytest.fixture
def youtube_service():
    return YouTubeService()

@pytest.mark.unit  # Test runner marker for categorization
def test_extract_video_id(youtube_service):
    """Test video ID extraction from various URL formats."""
    urls = [
        ("https://youtube.com/watch?v=abc123", "abc123"),
        ("https://youtu.be/xyz789", "xyz789"),
        ("https://www.youtube.com/embed/qwe456", "qwe456")
    ]
    for url, expected_id in urls:
        assert youtube_service.extract_video_id(url) == expected_id

Integration Test Pattern

# tests/integration/test_api.py
import pytest
from fastapi.testclient import TestClient
from src.main import app

client = TestClient(app)

@pytest.mark.integration  # Test runner marker for categorization
@pytest.mark.api
def test_summarize_endpoint():
    """Test video summarization API endpoint."""
    response = client.post("/api/summarize", json={
        "url": "https://youtube.com/watch?v=test123",
        "model": "openai"
    })
    assert response.status_code == 200
    assert "job_id" in response.json()

Test Runner Categories

The test runner automatically categorizes tests using markers and file patterns:

# Test markers for intelligent categorization
@pytest.mark.unit           # Fast, isolated unit tests
@pytest.mark.integration    # Database/API integration tests  
@pytest.mark.auth          # Authentication and security tests
@pytest.mark.api           # API endpoint tests
@pytest.mark.pipeline      # End-to-end pipeline tests
@pytest.mark.slow          # Tests taking >5 seconds

# Run specific categories
# ./run_tests.sh run-integration  # Runs integration + api marked tests
# ./run_tests.sh list --category unit  # Shows all unit tests

Performance Optimization

Async Everything: Use async/await for all I/O operations
Background Tasks: Process summaries in background
Caching Layers:
- Memory cache for hot data
- Database cache for persistence
- CDN for static exports
Rate Limiting: Implement per-IP and per-user limits
Token Optimization:
- Chunk long transcripts
- Use map-reduce for summaries
- Implement progressive summarization

Security Considerations

Input Validation: Validate all YouTube URLs
API Key Management: Use environment variables, never commit keys
Rate Limiting: Prevent abuse and API exhaustion
CORS Configuration: Restrict to known domains in production
SQL Injection Prevention: Use parameterized queries
XSS Protection: Sanitize all user inputs
Authentication: Implement JWT for user sessions (Phase 3)

Common Issues and Solutions

Issue: Transcript Not Available

# Solution: Implement fallback chain
try:
    transcript = await get_youtube_transcript(video_id)
except TranscriptNotAvailable:
    # Try auto-generated captions
    transcript = await get_auto_captions(video_id)
    if not transcript:
        # Use audio transcription as last resort
        transcript = await transcribe_audio(video_id)

Issue: Token Limit Exceeded

# Solution: Implement chunking
def chunk_transcript(transcript, max_tokens=3000):
    chunks = []
    current_chunk = []
    current_tokens = 0
    
    for segment in transcript:
        segment_tokens = count_tokens(segment)
        if current_tokens + segment_tokens > max_tokens:
            chunks.append(current_chunk)
            current_chunk = [segment]
            current_tokens = segment_tokens
        else:
            current_chunk.append(segment)
            current_tokens += segment_tokens
    
    if current_chunk:
        chunks.append(current_chunk)
    
    return chunks

Issue: Rate Limiting

# Solution: Implement exponential backoff
import asyncio
from typing import Optional

async def retry_with_backoff(
    func, 
    max_retries: int = 3,
    initial_delay: float = 1.0
) -> Optional[Any]:
    delay = initial_delay
    for attempt in range(max_retries):
        try:
            return await func()
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            await asyncio.sleep(delay)
            delay *= 2  # Exponential backoff

Development Tips

Start with Task 1: Setup and environment configuration
Test Early: Write tests as you implement features
Use Type Hints: Improve code quality and IDE support
Document APIs: Use FastAPI's automatic documentation
Log Everything: Implement comprehensive logging for debugging
Cache Aggressively: Reduce API calls and improve response times
Handle Errors Gracefully: Provide helpful error messages to users

Task Master Integration

This project uses Task Master for task management. Key commands:

# View current progress
task-master list

# Get detailed task info
task-master show 1

# Expand task into subtasks
task-master expand --id=1 --research

# Update task with progress
task-master update-task --id=1 --prompt="Completed API structure"

# Complete task
task-master set-status --id=1 --status=done

BMad Method Documentation Structure

Core Documentation

Project README - General project information and setup
Architecture - Complete technical architecture specification
Front-End Spec - UI/UX requirements and component specifications
Original PRD - Complete product requirements document

Epic and Story Management

Epic Index - Epic overview and progress tracking
Epic 1 - Foundation epic details
Epic 2 - AI engine epic details
Epic 3 - Advanced features epic
Stories - Individual story implementations

Current Story Files

Epic 1 - Foundation (Sprint 1):

Story 1.1 - ✅ Project setup (COMPLETED)
Story 1.2 - ✅ URL validation (COMPLETED)
Story 1.3 - ✅ Transcript extraction (COMPLETED)
Story 1.4 - ✅ Web interface (COMPLETED)
Story 1.5 - 📋 Video download service (READY)

Epic 2 - AI Engine (Sprints 2-3):

Story 2.1 - 📋 OpenAI integration (READY)
Story 2.2 - 📋 Pipeline orchestration (READY)
Story 2.3 - 📋 Caching system (READY)
Story 2.4 - 📋 Multi-model AI (READY)
Story 2.5 - 📋 Export features (READY)

Development Workflow

Check Epic Progress: Review Epic Index for current status
Review Next Story: Read story file for implementation details
Follow Dev Notes: Use architecture references and technical specifications
Implement & Test: Follow story tasks/subtasks systematically
Update Progress: Mark story complete and update epic status

Story-Based Implementation Priority

Current Focus: Epic 1 - Foundation & Core YouTube Integration

Sprint 1 (Weeks 1-2) - Epic 1 Implementation:

✅ Story 1.2 - YouTube URL Validation and Parsing (COMPLETED)
✅ Story 1.3 - Transcript Extraction Service (COMPLETED with mocks)
✅ Story 1.4 - Basic Web Interface (COMPLETED)
Story 1.5 - Video Download and Storage Service (12-16 hours) ⬅️ START HERE

Sprint 2 (Weeks 3-4) - Epic 2 Core: 4. Story 2.1 - Single AI Model Integration (12-16 hours) 5. Story 2.2 - Summary Generation Pipeline (16-20 hours) 6. Story 2.3 - Caching System Implementation (12-16 hours)

Sprint 3 (Weeks 5-6) - Epic 2 Advanced: 7. Story 2.4 - Multi-Model Support (16-20 hours) 8. Story 2.5 - Export Functionality (12-16 hours)

Developer Resources:

Developer Handoff Guide - Start here for implementation
Sprint Planning - Detailed sprint breakdown
Story Files - All stories with complete Dev Notes

Admin Page Implementation (Latest Feature) 🚀

No-Authentication Admin Interface

A standalone admin page provides immediate access to YouTube Summarizer functionality without authentication barriers.

Key Implementation Details:

File: frontend/src/pages/AdminPage.tsx
Route: /admin (bypasses ProtectedRoute wrapper in App.tsx)
URL: http://localhost:3002/admin
Backend: CORS configured to accept requests from port 3002

Visual Design:

Orange "Admin Mode" theme with Shield icon
Status badges: "Direct Access • Full Functionality • Testing Mode"
Footer: "Admin Mode - For testing and development purposes"

Usage:

Start services: python backend/main.py + npm run dev
Visit: http://localhost:3002/admin
Test with: https://www.youtube.com/watch?v=DCquejfz04A

Technical Notes:

Uses same components as protected dashboard (SummarizeForm, ProgressTracker, TranscriptViewer)
No AuthContext dependencies - completely self-contained
Perfect for testing, demos, and development workflow

This guide is specifically tailored for Claude Code development on the YouTube Summarizer project.

20 KiB Raw Blame History