demo
/
youtube-summarizer
1
0
Fork 0
Code Issues 8 Pull Requests 1 Packages Projects Releases Wiki Activity
youtube-summarizer/CLAUDE.md

598 lines
20 KiB
Markdown
Raw Blame History

# CLAUDE.md - YouTube Summarizer
This file provides guidance to Claude Code (claude.ai/code) when working with the YouTube Summarizer project.
## Project Overview
An AI-powered web application that automatically extracts, transcribes, and summarizes YouTube videos. The application supports multiple AI models (OpenAI, Anthropic, DeepSeek), provides various export formats, and includes intelligent caching for efficiency.
**Status**: Development in Progress - Authentication complete, core features ready for implementation
- **Epic 1**: Foundation & Core YouTube Integration (Stories 1.1-1.4 ✅ Complete, Story 1.5 📋 Ready)
- **Epic 2**: AI Summarization Engine (Stories 2.1-2.5 📋 All Created and Ready)
- **Epic 3**: User Authentication & Session Management (✅ Story 3.1-3.2 Complete, 📋 Story 3.3 Ready)
## Quick Start Commands
```bash
# Development Setup
cd apps/youtube-summarizer
docker-compose up # Start full development environment
# Quick Testing (No Auth Required)
open http://localhost:3002/admin # Direct admin access - No login needed
# BMad Method Story Management
/BMad:agents:sm # Activate Scrum Master agent
*draft # Create next story
*story-checklist # Validate story quality
# Development Agent Implementation
/BMad:agents:dev # Activate Development agent
# Follow story specifications in docs/stories/
# Direct Development (without BMad agents)
source venv/bin/activate # Activate virtual environment
python backend/main.py # Run backend (port 8000)
cd frontend && npm run dev # Run frontend (port 3002)
# Testing (Comprehensive Test Runner)
./run_tests.sh run-unit --fail-fast # Fast unit tests (229 tests in ~0.2s)
./run_tests.sh run-all --coverage # Complete test suite with coverage
cd frontend && npm test # Frontend tests
# Git Operations
git add .
git commit -m "feat: implement story 1.2 - URL validation"
git push origin main
```
## Architecture
```
YouTube Summarizer
├── Frontend (React + TypeScript)
│ ├── /admin - No-auth admin interface (TESTING)
│ ├── /dashboard - Protected summarizer interface
│ ├── /login - Authentication flow
│ └── /batch - Batch processing interface
├── API Layer (FastAPI)
│ ├── /api/summarize - Submit URL for summarization
│ ├── /api/summary/{id} - Retrieve summary
│ └── /api/export/{id} - Export in various formats
├── Service Layer
│ ├── YouTube Service - Transcript extraction
│ ├── AI Service - Summary generation (DeepSeek)
│ └── Cache Service - Performance optimization
└── Data Layer
├── SQLite/PostgreSQL - Summary storage
└── Redis (optional) - Caching layer
```
## Testing & Development Access
### Admin Page (No Authentication)
- **URL**: `http://localhost:3002/admin`
- **Purpose**: Direct access for testing and development
- **Features**: Complete YouTube Summarizer functionality without login
- **Visual**: Orange "Admin Mode" badge for clear identification
- **Use Case**: Quick testing, demos, development workflow
### Protected Routes (Authentication Required)
- **Dashboard**: `http://localhost:3002/dashboard` - Main app with user session
- **History**: `http://localhost:3002/history` - User's summary history
- **Batch**: `http://localhost:3002/batch` - Batch processing interface
## Development Workflow - BMad Method
### Story-Driven Development Process
**Current Epic**: Epic 3 - User Authentication & Session Management
**Current Stories**:
- ✅ **Epic 1 - Foundation & Core YouTube Integration** (Complete)
- ✅ Story 1.1: Project Setup and Infrastructure
- ✅ Story 1.2: YouTube URL Validation and Parsing
- ✅ Story 1.3: Transcript Extraction Service (with mocks)
- ✅ Story 1.4: Basic Web Interface
- ✅ Story 1.5: Video Download and Storage Service
- ✅ **Epic 2 - AI Summarization Engine** (Complete)
- ✅ Story 2.1-2.5: All AI pipeline and summarization features
- 🚀 **Epic 3 - User Authentication & Session Management** (Current)
- ✅ Story 3.1: User Authentication System (Backend Complete)
- 📝 Story 3.2: Frontend Authentication Integration (Ready for implementation)
### 1. Story Planning (Scrum Master)
```bash
# Activate Scrum Master agent
/BMad:agents:sm
*draft # Create next story in sequence
*story-checklist # Validate story completeness
```
### 2. Story Implementation (Development Agent)
```bash
# Activate Development agent
/BMad:agents:dev
# Review story file: docs/stories/{epic}.{story}.{name}.md
# Follow detailed Dev Notes and architecture references
# Implement all tasks and subtasks as specified
```
### 3. Implementation Locations
Based on architecture and story specifications:
- **Backend API** → `backend/api/`
- **Backend Services** → `backend/services/`
- **Backend Models** → `backend/models/`
- **Frontend Components** → `frontend/src/components/`
- **Frontend Hooks** → `frontend/src/hooks/`
- **Frontend API Client** → `frontend/src/api/`
### 4. Testing Implementation
```bash
# Backend testing (Test Runner - Fast Feedback)
./run_tests.sh run-unit --fail-fast # Ultra-fast unit tests (0.2s)
./run_tests.sh run-specific "test_video_service.py" # Test specific modules
./run_tests.sh run-integration # Integration & API tests
./run_tests.sh run-all --coverage --parallel # Complete suite with coverage
# Test Discovery & Validation
./run_tests.sh list --category unit # See available tests (229 found)
./scripts/validate_test_setup.py # Validate test environment
# Frontend testing (Vitest + RTL)
cd frontend && npm test
cd frontend && npm run test:coverage
# Manual testing
docker-compose up # Full stack
# Visit http://localhost:3000 (frontend)
# Visit http://localhost:8000/docs (API docs)
```
### 5. Story Completion
- Mark all tasks/subtasks complete in story file
- Update story status from "Draft" to "Done"
- Run story validation checklist
- Update epic progress tracking
## Testing & Quality Assurance
### Test Runner System
The project includes a production-ready test runner with **229 discovered unit tests** and intelligent categorization.
```bash
# Fast feedback during development
./run_tests.sh run-unit --fail-fast # Ultra-fast unit tests (~0.2s)
./run_tests.sh run-all --coverage # Complete validation
cd frontend && npm test # Frontend tests
```
**📖 Complete Testing Guide**: See [TESTING-INSTRUCTIONS.md](TESTING-INSTRUCTIONS.md) for comprehensive testing standards, procedures, and troubleshooting.
## Key Implementation Areas
### YouTube Integration (`src/services/youtube.py`)
```python
# Primary: youtube-transcript-api
from youtube_transcript_api import YouTubeTranscriptApi
# Fallback: yt-dlp for metadata
import yt_dlp
# Extract video ID from various URL formats
# Handle multiple subtitle languages
# Implement retry logic for failures
```
### AI Summarization (`src/services/summarizer.py`)
```python
# Multi-model support
class SummarizerService:
def __init__(self):
self.models = {
'openai': OpenAISummarizer(),
'anthropic': AnthropicSummarizer(),
'deepseek': DeepSeekSummarizer()
}
async def summarize(self, transcript, model='auto'):
# Implement model selection logic
# Handle token limits
# Generate structured summaries
```
### Caching Strategy (`src/services/cache.py`)
```python
# Cache at multiple levels:
# 1. Transcript cache (by video_id)
# 2. Summary cache (by video_id + model + params)
# 3. Export cache (by summary_id + format)
# Use hash for cache keys
import hashlib
def get_cache_key(video_id: str, model: str, params: dict) -> str:
key_data = f"{video_id}:{model}:{json.dumps(params, sort_keys=True)}"
return hashlib.sha256(key_data.encode()).hexdigest()
```
## API Endpoint Patterns
### FastAPI Best Practices
```python
from fastapi import APIRouter, HTTPException, BackgroundTasks
from pydantic import BaseModel, HttpUrl
router = APIRouter(prefix="/api", tags=["summarization"])
class SummarizeRequest(BaseModel):
url: HttpUrl
model: str = "auto"
options: dict = {}
@router.post("/summarize")
async def summarize_video(
request: SummarizeRequest,
background_tasks: BackgroundTasks
):
# Validate URL
# Extract video ID
# Check cache
# Queue for processing if needed
# Return job ID for status checking
```
## Database Schema
```sql
-- Main summaries table
CREATE TABLE summaries (
id UUID PRIMARY KEY,
video_id VARCHAR(20) NOT NULL,
video_title TEXT,
video_url TEXT NOT NULL,
transcript TEXT,
summary TEXT,
key_points JSONB,
chapters JSONB,
model_used VARCHAR(50),
processing_time FLOAT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Cache for performance
CREATE INDEX idx_video_id ON summaries(video_id);
CREATE INDEX idx_created_at ON summaries(created_at);
```
## Error Handling
```python
class YouTubeError(Exception):
"""Base exception for YouTube-related errors"""
pass
class TranscriptNotAvailable(YouTubeError):
"""Raised when transcript cannot be extracted"""
pass
class AIServiceError(Exception):
"""Base exception for AI service errors"""
pass
class TokenLimitExceeded(AIServiceError):
"""Raised when content exceeds model token limit"""
pass
# Global error handler
@app.exception_handler(YouTubeError)
async def youtube_error_handler(request, exc):
return JSONResponse(
status_code=400,
content={"error": str(exc), "type": "youtube_error"}
)
```
## Environment Variables
```bash
# Required
OPENAI_API_KEY=sk-... # At least one AI key required
ANTHROPIC_API_KEY=sk-ant-...
DEEPSEEK_API_KEY=sk-...
DATABASE_URL=sqlite:///./data/youtube_summarizer.db
SECRET_KEY=your-secret-key
# Optional but recommended
YOUTUBE_API_KEY=AIza... # For metadata and quota
REDIS_URL=redis://localhost:6379/0
RATE_LIMIT_PER_MINUTE=30
MAX_VIDEO_LENGTH_MINUTES=180
```
## Testing Guidelines
### Test Runner Integration
The project uses a comprehensive test runner system for efficient testing:
```bash
# Run specific test modules during development
./run_tests.sh run-specific "backend/tests/unit/test_youtube_service.py"
# Fast feedback loop (discovered 229 tests)
./run_tests.sh run-unit --fail-fast
# Comprehensive testing with coverage
./run_tests.sh run-all --coverage --reports html,json
```
### Unit Test Structure
```python
# tests/unit/test_youtube_service.py
import pytest
from unittest.mock import Mock, patch
from src.services.youtube import YouTubeService
@pytest.fixture
def youtube_service():
return YouTubeService()
@pytest.mark.unit # Test runner marker for categorization
def test_extract_video_id(youtube_service):
"""Test video ID extraction from various URL formats."""
urls = [
("https://youtube.com/watch?v=abc123", "abc123"),
("https://youtu.be/xyz789", "xyz789"),
("https://www.youtube.com/embed/qwe456", "qwe456")
]
for url, expected_id in urls:
assert youtube_service.extract_video_id(url) == expected_id
```
### Integration Test Pattern
```python
# tests/integration/test_api.py
import pytest
from fastapi.testclient import TestClient
from src.main import app
client = TestClient(app)
@pytest.mark.integration # Test runner marker for categorization
@pytest.mark.api
def test_summarize_endpoint():
"""Test video summarization API endpoint."""
response = client.post("/api/summarize", json={
"url": "https://youtube.com/watch?v=test123",
"model": "openai"
})
assert response.status_code == 200
assert "job_id" in response.json()
```
### Test Runner Categories
The test runner automatically categorizes tests using markers and file patterns:
```python
# Test markers for intelligent categorization
@pytest.mark.unit # Fast, isolated unit tests
@pytest.mark.integration # Database/API integration tests
@pytest.mark.auth # Authentication and security tests
@pytest.mark.api # API endpoint tests
@pytest.mark.pipeline # End-to-end pipeline tests
@pytest.mark.slow # Tests taking >5 seconds
# Run specific categories
# ./run_tests.sh run-integration # Runs integration + api marked tests
# ./run_tests.sh list --category unit # Shows all unit tests
```
## Performance Optimization
1. **Async Everything**: Use async/await for all I/O operations
2. **Background Tasks**: Process summaries in background
3. **Caching Layers**:
- Memory cache for hot data
- Database cache for persistence
- CDN for static exports
4. **Rate Limiting**: Implement per-IP and per-user limits
5. **Token Optimization**:
- Chunk long transcripts
- Use map-reduce for summaries
- Implement progressive summarization
## Security Considerations
1. **Input Validation**: Validate all YouTube URLs
2. **API Key Management**: Use environment variables, never commit keys
3. **Rate Limiting**: Prevent abuse and API exhaustion
4. **CORS Configuration**: Restrict to known domains in production
5. **SQL Injection Prevention**: Use parameterized queries
6. **XSS Protection**: Sanitize all user inputs
7. **Authentication**: Implement JWT for user sessions (Phase 3)
## Common Issues and Solutions
### Issue: Transcript Not Available
```python
# Solution: Implement fallback chain
try:
transcript = await get_youtube_transcript(video_id)
except TranscriptNotAvailable:
# Try auto-generated captions
transcript = await get_auto_captions(video_id)
if not transcript:
# Use audio transcription as last resort
transcript = await transcribe_audio(video_id)
```
### Issue: Token Limit Exceeded
```python
# Solution: Implement chunking
def chunk_transcript(transcript, max_tokens=3000):
chunks = []
current_chunk = []
current_tokens = 0
for segment in transcript:
segment_tokens = count_tokens(segment)
if current_tokens + segment_tokens > max_tokens:
chunks.append(current_chunk)
current_chunk = [segment]
current_tokens = segment_tokens
else:
current_chunk.append(segment)
current_tokens += segment_tokens
if current_chunk:
chunks.append(current_chunk)
return chunks
```
### Issue: Rate Limiting
```python
# Solution: Implement exponential backoff
import asyncio
from typing import Optional
async def retry_with_backoff(
func,
max_retries: int = 3,
initial_delay: float = 1.0
) -> Optional[Any]:
delay = initial_delay
for attempt in range(max_retries):
try:
return await func()
except RateLimitError:
if attempt == max_retries - 1:
raise
await asyncio.sleep(delay)
delay *= 2 # Exponential backoff
```
## Development Tips
1. **Start with Task 1**: Setup and environment configuration
2. **Test Early**: Write tests as you implement features
3. **Use Type Hints**: Improve code quality and IDE support
4. **Document APIs**: Use FastAPI's automatic documentation
5. **Log Everything**: Implement comprehensive logging for debugging
6. **Cache Aggressively**: Reduce API calls and improve response times
7. **Handle Errors Gracefully**: Provide helpful error messages to users
## Task Master Integration
This project uses Task Master for task management. Key commands:
```bash
# View current progress
task-master list
# Get detailed task info
task-master show 1
# Expand task into subtasks
task-master expand --id=1 --research
# Update task with progress
task-master update-task --id=1 --prompt="Completed API structure"
# Complete task
task-master set-status --id=1 --status=done
```
## BMad Method Documentation Structure
### Core Documentation
- **[Project README](README.md)** - General project information and setup
- **[Architecture](docs/architecture.md)** - Complete technical architecture specification
- **[Front-End Spec](docs/front-end-spec.md)** - UI/UX requirements and component specifications
- **[Original PRD](docs/prd.md)** - Complete product requirements document
### Epic and Story Management
- **[Epic Index](docs/prd/index.md)** - Epic overview and progress tracking
- **[Epic 1](docs/prd/epic-1-foundation-core-youtube-integration.md)** - Foundation epic details
- **[Epic 2](docs/prd/epic-2-ai-summarization-engine.md)** - AI engine epic details
- **[Epic 3](docs/prd/epic-3-enhanced-user-experience.md)** - Advanced features epic
- **[Stories](docs/stories/)** - Individual story implementations
### Current Story Files
**Epic 1 - Foundation (Sprint 1)**:
- **[Story 1.1](docs/stories/1.1.project-setup-infrastructure.md)** - ✅ Project setup (COMPLETED)
- **[Story 1.2](docs/stories/1.2.youtube-url-validation-parsing.md)** - ✅ URL validation (COMPLETED)
- **[Story 1.3](docs/stories/1.3.transcript-extraction-service.md)** - ✅ Transcript extraction (COMPLETED)
- **[Story 1.4](docs/stories/1.4.basic-web-interface.md)** - ✅ Web interface (COMPLETED)
- **[Story 1.5](docs/stories/1.5.video-download-storage-service.md)** - 📋 Video download service (READY)
**Epic 2 - AI Engine (Sprints 2-3)**:
- **[Story 2.1](docs/stories/2.1.single-ai-model-integration.md)** - 📋 OpenAI integration (READY)
- **[Story 2.2](docs/stories/2.2.summary-generation-pipeline.md)** - 📋 Pipeline orchestration (READY)
- **[Story 2.3](docs/stories/2.3.caching-system-implementation.md)** - 📋 Caching system (READY)
- **[Story 2.4](docs/stories/2.4.multi-model-support.md)** - 📋 Multi-model AI (READY)
- **[Story 2.5](docs/stories/2.5.export-functionality.md)** - 📋 Export features (READY)
### Development Workflow
1. **Check Epic Progress**: Review [Epic Index](docs/prd/index.md) for current status
2. **Review Next Story**: Read story file for implementation details
3. **Follow Dev Notes**: Use architecture references and technical specifications
4. **Implement & Test**: Follow story tasks/subtasks systematically
5. **Update Progress**: Mark story complete and update epic status
### Story-Based Implementation Priority
**Current Focus**: Epic 1 - Foundation & Core YouTube Integration
**Sprint 1 (Weeks 1-2)** - Epic 1 Implementation:
1. ✅ **Story 1.2** - YouTube URL Validation and Parsing (COMPLETED)
2. ✅ **Story 1.3** - Transcript Extraction Service (COMPLETED with mocks)
3. ✅ **Story 1.4** - Basic Web Interface (COMPLETED)
4. **Story 1.5** - Video Download and Storage Service (12-16 hours) ⬅️ **START HERE**
**Sprint 2 (Weeks 3-4)** - Epic 2 Core:
4. **Story 2.1** - Single AI Model Integration (12-16 hours)
5. **Story 2.2** - Summary Generation Pipeline (16-20 hours)
6. **Story 2.3** - Caching System Implementation (12-16 hours)
**Sprint 3 (Weeks 5-6)** - Epic 2 Advanced:
7. **Story 2.4** - Multi-Model Support (16-20 hours)
8. **Story 2.5** - Export Functionality (12-16 hours)
**Developer Resources**:
- [Developer Handoff Guide](docs/DEVELOPER_HANDOFF.md) - Start here for implementation
- [Sprint Planning](docs/SPRINT_PLANNING.md) - Detailed sprint breakdown
- [Story Files](docs/stories/) - All stories with complete Dev Notes
## Admin Page Implementation (Latest Feature) 🚀
### No-Authentication Admin Interface
A standalone admin page provides immediate access to YouTube Summarizer functionality without authentication barriers.
**Key Implementation Details**:
- **File**: `frontend/src/pages/AdminPage.tsx`
- **Route**: `/admin` (bypasses ProtectedRoute wrapper in App.tsx)
- **URL**: `http://localhost:3002/admin`
- **Backend**: CORS configured to accept requests from port 3002
**Visual Design**:
- Orange "Admin Mode" theme with Shield icon
- Status badges: "Direct Access • Full Functionality • Testing Mode"
- Footer: "Admin Mode - For testing and development purposes"
**Usage**:
1. Start services: `python backend/main.py` + `npm run dev`
2. Visit: `http://localhost:3002/admin`
3. Test with: `https://www.youtube.com/watch?v=DCquejfz04A`
**Technical Notes**:
- Uses same components as protected dashboard (SummarizeForm, ProgressTracker, TranscriptViewer)
- No AuthContext dependencies - completely self-contained
- Perfect for testing, demos, and development workflow
---
*This guide is specifically tailored for Claude Code development on the YouTube Summarizer project.*
Reference in New Issue View Git Blame Copy Permalink