# Changelog

All notable changes to the YouTube Summarizer project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added
- **⚡ Faster-Whisper Integration - MAJOR PERFORMANCE UPGRADE** - 20-32x speed improvement with large-v3-turbo model
  - **FasterWhisperTranscriptService** - Complete replacement for OpenAI Whisper with CTranslate2 optimization
  - **Large-v3-Turbo Model** - Best accuracy/speed balance with advanced AI capabilities  
  - **Performance Benchmarks** - 2.3x faster than realtime (3.6 min video in 94 seconds vs 30+ minutes)
  - **Quality Metrics** - Perfect transcription accuracy (1.000 quality score, 0.962 confidence)
  - **Intelligent Optimizations** - Voice Activity Detection, int8 quantization, GPU acceleration
  - **Native MP3 Support** - Direct processing without audio conversion overhead
  - **Advanced Configuration** - 8+ configurable parameters via environment variables
  - **Production Features** - Comprehensive metadata, error handling, performance tracking
- **🔧 Development Tools & Server Management** - Professional development workflow improvements
  - **Server Restart Scripts** - `./scripts/restart-backend.sh`, `./scripts/restart-frontend.sh`, `./scripts/restart-both.sh`
  - **Automated Process Management** - Health checks, logging, status reporting
  - **Development Logs** - Centralized logging to `logs/` directory with proper cleanup
- **🔐 Flexible Authentication System** - Configurable auth for development and production
  - **Development Mode** - No authentication required by default (perfect for development/testing)
  - **Production Mode** - Automatic JWT-based authentication in production builds
  - **Environment Controls** - `VITE_FORCE_AUTH_MODE`, `VITE_AUTH_DISABLED` for fine-grained control
  - **Unified Main Page** - Single component adapts to auth requirements with admin mode indicators
  - **Conditional Protection** - Smart wrapper applies authentication only when needed
- **📋 Persistent Job History System** - Comprehensive history management from existing storage
  - **High-Density Views** - Grid view (12+ jobs), list view (15+ jobs) meeting user requirements
  - **Smart File Discovery** - Automatically indexes existing files from `video_storage/` directories  
  - **Enhanced Detail Modal** - Tabbed interface with transcript viewer, file downloads, metadata
  - **Rich Metadata** - File status indicators, processing times, word counts, storage usage
  - **Search & Filtering** - Real-time search with status, date, and tag filtering
  - **History API** - Complete REST API with pagination, sorting, and CRUD operations
- **🤖 Epic 4: Advanced Intelligence & Developer Platform - Core Implementation** - Complete multi-agent AI and enhanced export systems
  - **Multi-Agent Summarization System** - Three perspective agents (Technical, Business, UX) + synthesis agent
  - **Enhanced Markdown Export** - Executive summaries, timestamped sections, professional formatting
  - **RAG-Powered Video Chat** - ChromaDB semantic search with DeepSeek AI responses
  - **Database Schema Extensions** - 12 new tables supporting all Epic 4 features
  - **DeepSeek AI Integration** - Cost-effective alternative to Anthropic with async processing
  - **Comprehensive Service Layer** - Production-ready services for all new features
  - **✅ Story 4.4: Custom AI Models & Enhanced Export** - Professional document generation with AI-powered intelligence
    - **ExecutiveSummaryGenerator** - Business-focused summaries with ROI analysis and strategic insights
    - **TimestampProcessor** - Semantic section detection with clickable YouTube navigation
    - **EnhancedMarkdownFormatter** - Professional document templates with quality scoring
    - **6 Domain-Specific Templates** - Educational, Business, Technical, Content Creation, Research, General
    - **Advanced Template Manager** - Custom prompts, A/B testing, domain recommendations
    - **Enhanced Export API** - Complete REST endpoints for template management and export generation

### Changed
- **🏗️ Frontend Architecture Simplification** - Eliminated code duplication and improved maintainability
  - **Unified Authentication Routes** - Replaced separate Admin/Dashboard pages with configurable single page
  - **Conditional Protection Pattern** - Smart wrapper component applies auth only when required
  - **Configuration-Driven UI** - Single codebase adapts to development vs production requirements
  - **Pydantic Compatibility** - Updated regex to pattern for Pydantic v2 compatibility
- **📋 Epic 4 Scope Refinement** - Enhanced stories with multi-agent focus
  - **Story 4.3**: Enhanced to "Multi-video Analysis with Multi-Agent System" (40 hours)
  - **Story 4.4**: Enhanced to "Custom Models & Enhanced Markdown Export" (32 hours) 
  - **Story 4.6**: Enhanced to "RAG-Powered Video Chat with ChromaDB" (20 hours)
  - Moved Story 4.5 (Advanced Analytics Dashboard) to new Epic 5
  - Removed Story 4.7 (Trend Detection & Insights) from scope
  - Total Epic 4 effort: 146 hours (54 hours completed, 92 hours enhanced implementation)

### Technical Implementation
- **Backend Services**:
  - `MultiAgentSummarizerService` - Orchestrates three analysis perspectives with synthesis
  - `EnhancedExportService` - Executive summaries and timestamped navigation
  - `RAGChatService` - ChromaDB integration with semantic search and conversation management
  - `DeepSeekService` - Async AI service with cost estimation and error handling
  
- **Database Migration**: `add_epic_4_features.py`
  - Agent summaries, playlists, chat sessions, prompt templates, export metadata
  - 12 new tables with proper relationships and indexes
  - Extended summaries table with Epic 4 feature flags

- **AI Agent System**:
  - Technical Analysis Agent - Implementation, architecture, tools focus
  - Business Analysis Agent - ROI, strategic insights, market implications
  - User Experience Agent - Usability, accessibility, user journey analysis  
  - Synthesis Agent - Unified comprehensive summary combining all perspectives

### Added  
- **📊 Epic 5: Analytics & Business Intelligence** - New epic for analytics features
  - Story 5.1: Advanced Analytics Dashboard (24 hours)
  - Story 5.2: Content Intelligence Reports (20 hours)
  - Story 5.3: Cost Analytics & Optimization (16 hours)
  - Story 5.4: Performance Monitoring (18 hours)
  - Total effort: 78 hours across 4 comprehensive analytics stories

## [5.1.0] - 2025-08-27

### Added
- **🎯 Comprehensive Transcript Fallback Chain** - 9-tier fallback system for reliable transcript extraction
  - YouTube Transcript API (primary method)
  - Auto-generated Captions fallback
  - Whisper AI Audio Transcription
  - PyTubeFix alternative downloader
  - YT-DLP robust video/audio downloader
  - Playwright browser automation
  - External tool integration
  - Web service fallback
  - Transcript-only final fallback
  
- **💾 Audio File Retention System** - Save audio for future re-transcription
  - Audio files saved as MP3 (192kbps) for storage efficiency
  - Automatic WAV to MP3 conversion after transcription
  - Audio metadata tracking (duration, quality, download date)
  - Re-transcription without re-downloading
  - Configurable retention period (default: 30 days)

- **📁 Organized Storage Structure** - Dedicated directories for all content types
  - `video_storage/videos/` - Downloaded video files
  - `video_storage/audio/` - Audio files with metadata
  - `video_storage/transcripts/` - Text and JSON transcripts
  - `video_storage/summaries/` - AI-generated summaries
  - `video_storage/cache/` - Cached API responses
  - `video_storage/temp/` - Temporary processing files

### Changed
- Upgraded Python from 3.9 to 3.11 for better Whisper compatibility
- Updated TranscriptService to use real YouTube API and Whisper services
- Modified WhisperTranscriptService to preserve audio files
- Enhanced VideoDownloadConfig with audio retention settings

### Fixed
- Fixed circular state update in React transcript selector hook
- Fixed missing API endpoint routing for transcript extraction
- Fixed mock service configuration defaulting to true
- Fixed YouTube API integration with proper method calls
- Fixed auto-captions extraction with real API implementation

## [5.0.0] - 2025-08-27

### Added
- **🚀 Advanced API Ecosystem** - Comprehensive developer platform
  - **MCP Server Integration**: FastMCP server with JSON-RPC interface for AI development tools
  - **Native SDKs**: Production-ready Python and JavaScript/TypeScript SDKs
  - **Agent Framework Support**: LangChain, CrewAI, and AutoGen integrations
  - **Webhook System**: Real-time event notifications with HMAC authentication
  - **Autonomous Operations**: Self-managing rule-based automation system
  - **API Authentication**: Enterprise-grade API key management and rate limiting
  - **OpenAPI 3.0 Specification**: Comprehensive API documentation
  - **Developer Tools**: Advanced MCP tools for batch processing and analytics
  - **Production Monitoring**: Health checks, metrics, and observability

### Features Implemented
- **Backend Infrastructure**:
  - `backend/api/developer.py` - Developer API endpoints with rate limiting
  - `backend/api/autonomous.py` - Webhook and automation management
  - `backend/mcp_server.py` - FastMCP server with comprehensive tools
  - `backend/services/api_key_service.py` - API key generation and validation
  - `backend/middleware/api_auth.py` - Authentication middleware

- **SDK Development**:
  - `sdks/python/` - Full async Python SDK with error handling
  - `sdks/javascript/` - TypeScript SDK with browser/Node.js support
  - Both SDKs feature: authentication, rate limiting, retry logic, streaming

- **Agent Framework Integration**:
  - `backend/integrations/langchain_tools.py` - LangChain-compatible tools
  - `backend/integrations/agent_framework.py` - Multi-framework orchestrator
  - Support for LangChain, CrewAI, AutoGen with unified interface

- **Autonomous Operations**:
  - `backend/autonomous/webhook_system.py` - Secure webhook delivery
  - `backend/autonomous/autonomous_controller.py` - Rule-based automation
  - Scheduled, event-driven, threshold-based, and queue-based triggers

### Documentation
- Comprehensive READMEs for all new components
- API endpoint documentation with examples
- SDK usage guides and integration examples
- Agent framework integration tutorials
- Webhook security best practices

## [4.1.0] - 2025-01-25

### Added
- **🎯 Dual Transcript Options (Story 4.1)** - Complete frontend and backend implementation
  - **Frontend Components**: Interactive TranscriptSelector and TranscriptComparison with TypeScript safety
  - **Backend Services**: DualTranscriptService orchestration and WhisperTranscriptService integration
  - **Three Transcript Sources**: YouTube captions (fast), Whisper AI (premium), or compare both
  - **Quality Analysis Engine**: Punctuation, capitalization, and technical term improvement analysis
  - **Processing Time Estimates**: Real-time estimates based on video duration and hardware
  - **Smart Recommendations**: Intelligent source selection based on quality vs speed trade-offs
  - **API Endpoints**: RESTful dual transcript extraction with background job processing
  - **Demo Interface**: `/demo/transcript-comparison` showcasing full functionality with mock data
  - **Production Ready**: Comprehensive error handling, resource management, and cleanup
  - **Hardware Optimization**: Automatic CPU/CUDA detection for optimal Whisper performance
  - **Chunked Processing**: 30-minute segments with overlap for long-form content
  - **Quality Comparison**: Side-by-side analysis with difference highlighting and metrics

### Changed
- **Enhanced TranscriptService Integration**: Seamless connection with existing YouTube transcript extraction
- **Updated SummarizeForm**: Integrated transcript source selection with backward compatibility
- **Extended Data Models**: Comprehensive Pydantic models with quality comparison support
- **API Architecture**: Extended transcripts API with dual extraction endpoints

### Technical Implementation
- **Frontend**: React + TypeScript with discriminated unions and custom hooks
- **Backend**: FastAPI with async processing, Whisper integration, and quality analysis
- **Performance**: Parallel transcript extraction and intelligent time estimation
- **Developer Experience**: Complete TypeScript interfaces matching backend models
- **Documentation**: Comprehensive implementation guides and API documentation

### Planning
- **Epic 4: Advanced Intelligence & Developer Platform** - Comprehensive roadmap created
  - ✅ Story 4.1: Dual Transcript Options (COMPLETE)
  - 6 remaining stories: API Platform, Multi-video Analysis, Custom AI, Analytics, Q&A, Trends
  - Epic 4 detailed document with architecture, dependencies, and risk analysis
  - Implementation strategy with 170 hours estimated effort over 8-10 weeks

## [3.5.0] - 2025-08-27

### Added
- **Real-time Updates Feature (Story 3.5)** - Complete WebSocket-based progress tracking
  - WebSocket infrastructure with automatic reconnection and recovery
  - Granular pipeline progress tracking with sub-task updates
  - Real-time progress UI component with stage visualization
  - Time estimation based on historical processing data
  - Job cancellation support with immediate termination
  - Connection status indicators and heartbeat monitoring
  - Message queuing for offline recovery
  - Exponential backoff for reconnection attempts

### Enhanced
- **WebSocket Manager** with comprehensive connection management
  - ProcessingStage enum for standardized stage tracking
  - ProgressData dataclass for structured updates
  - Message queue for disconnected clients
  - Automatic recovery with message replay
  - Historical data tracking for time estimation
  
- **SummaryPipeline** with detailed progress reporting
  - Enhanced `_update_progress` with sub-progress support
  - Cancellation checks at each pipeline stage
  - Integration with WebSocket manager
  - Time elapsed and remaining calculations

### Frontend Components
- Created `ProcessingProgress` component for real-time visualization
- Enhanced `useWebSocket` hook with reconnection and queuing
- Added connection state management and heartbeat support

## [3.4.0] - 2025-08-27

### Added
- **Batch Processing Feature (Story 3.4)** - Complete implementation of batch video processing
  - Process up to 100 YouTube videos in a single batch operation
  - File upload support for .txt and .csv files containing URLs
  - Sequential queue processing to manage API costs effectively
  - Real-time progress tracking via WebSocket connections
  - Individual item status tracking with error messages
  - Retry mechanism for failed items with exponential backoff
  - Batch export as organized ZIP archive with JSON and Markdown formats
  - Cost tracking and estimation at $0.0025 per 1k tokens
  - Job cancellation and deletion support

### Backend Implementation
- Created `BatchJob` and `BatchJobItem` database models with full relationships
- Implemented `BatchProcessingService` with sequential queue management
- Added 7 new API endpoints for batch operations (`/api/batch/*`)
- Database migration `add_batch_processing_tables` with performance indexes
- WebSocket integration for real-time progress updates
- ZIP export generation with multiple format support

### Frontend Implementation
- `BatchProcessingPage` with tabbed interface for job management
- `BatchJobStatus` component for real-time progress display
- `BatchJobList` component for historical job viewing
- `BatchUploadDialog` for file upload and URL input
- `useBatchProcessing` hook for complete batch management
- `useWebSocket` hook with auto-reconnect functionality

## [3.3.0] - 2025-08-27

### Added
- **Summary History Management (Story 3.3)** - Complete user summary organization
  - View all processed summaries with pagination
  - Advanced search and filtering by title, date, model, tags
  - Star important summaries for quick access
  - Add personal notes and custom tags for organization
  - Bulk operations for managing multiple summaries
  - Generate shareable links with unique tokens
  - Export summaries in multiple formats (JSON, CSV, ZIP)
  - Usage statistics dashboard

### Backend Implementation
- Added history management fields to Summary model
- Created 12 new API endpoints for summary management
- Implemented search, filter, and sort capabilities
- Added sharing functionality with token generation
- Bulk operations support with transaction safety

### Frontend Implementation
- `SummaryHistoryPage` with comprehensive UI
- Search bar with multiple filter options
- Bulk selection with checkbox controls
- Export dialog for multiple formats
- Sharing interface with copy-to-clipboard

## [3.2.0] - 2025-08-26

### Added
- **Frontend Authentication Integration (Story 3.2)** - Complete auth UI
  - Login page with validation and error handling
  - Registration page with password confirmation
  - Forgot password flow with email verification
  - Email verification page with token handling
  - Protected routes with authentication guards
  - Global auth state management via AuthContext
  - Automatic logout on token expiration
  - Persistent auth state across page refreshes

### Frontend Implementation
- Complete authentication page components
- AuthContext for global state management
- ProtectedRoute component for route guards
- Token storage and refresh logic
- Auto-redirect after login/logout

## [3.1.0] - 2025-08-26

### Added
- **User Authentication System (Story 3.1)** - Complete backend authentication infrastructure
  - JWT-based authentication with access and refresh tokens
  - User registration with email verification workflow
  - Password reset functionality with secure token generation
  - Database models for User, RefreshToken, APIKey, EmailVerificationToken, PasswordResetToken
  - Complete FastAPI authentication endpoints (`/api/auth/*`)
  - Password strength validation and security policies
  - Email service integration for verification and password reset
  - Authentication service layer with proper error handling
  - Protected route middleware and dependencies

### Fixed
- **Critical SQLAlchemy Architecture Issue** - Resolved "Multiple classes found for path 'RefreshToken'" error
  - Implemented Database Registry singleton pattern to prevent table redefinition conflicts
  - Added fully qualified module paths in model relationships
  - Created automatic model registration system with `BaseModel` mixin
  - Ensured single Base instance across entire application
  - Production-ready architecture preventing SQLAlchemy conflicts

### Technical Details
- Created `backend/core/database_registry.py` - Singleton registry for database models
- Updated all model relationships to use fully qualified paths (`backend.models.*.Class`)
- Implemented `backend/models/base.py` - Automatic model registration system
- Added comprehensive authentication API endpoints with proper validation
- String UUID fields for SQLite compatibility
- Proper async/await patterns throughout authentication system
- Test fixtures with in-memory database isolation (conftest.py)
- Email service abstraction ready for production SMTP integration

## [2.5.0] - 2025-08-26

### Added
- **Export Functionality (Story 2.5)** - Complete implementation of multi-format export system
  - Support for 5 export formats: Markdown, PDF, HTML, JSON, and Plain Text
  - Customizable template system using Jinja2 engine
  - Bulk export capability with ZIP archive generation
  - Template management API with CRUD operations
  - Frontend export components (ExportDialog and BulkExportDialog)
  - Progress tracking for export operations
  - Export status monitoring and download management

### Fixed
- Duration formatting issues in PlainTextExporter, HTMLExporter, and PDFExporter
- File sanitization to properly handle control characters and null bytes
- Template rendering with proper Jinja2 integration

### Changed
- Updated MarkdownExporter to use Jinja2 templates instead of simple string replacement
- Enhanced export service with better error handling and retry logic
- Improved bulk export organization with format, date, and video grouping options

### Technical Details
- Created `ExportService` with format-specific exporters
- Implemented `TemplateManager` for template operations
- Added comprehensive template API endpoints (`/api/templates/*`)
- Updated frontend with React components for export UI
- Extended API client with export and template methods
- Added TypeScript definitions for export functionality
- Test coverage: 90% (18/20 unit tests passing)

## [2.4.0] - 2025-08-25

### Added
- Multi-model AI support (Story 2.4)
- Support for OpenAI, Anthropic, and DeepSeek models

## [2.3.0] - 2025-08-24

### Added
- Caching system implementation (Story 2.3)
- Redis-ready caching architecture
- TTL-based cache expiration

## [2.2.0] - 2025-08-23

### Added
- Summary generation pipeline (Story 2.2)
- 7-stage async pipeline for video processing
- Real-time progress tracking via WebSocket

## [2.1.0] - 2025-08-22

### Added
- Single AI model integration (Story 2.1)
- Anthropic Claude integration

## [1.5.0] - 2025-08-21

### Added
- Video download and storage service (Story 1.5)

## [1.4.0] - 2025-08-20

### Added
- Basic web interface (Story 1.4)
- React frontend with TypeScript

## [1.3.0] - 2025-08-19

### Added
- Transcript extraction service (Story 1.3)
- YouTube transcript API integration

## [1.2.0] - 2025-08-18

### Added
- YouTube URL validation and parsing (Story 1.2)
- Support for multiple YouTube URL formats

## [1.1.0] - 2025-08-17

### Added
- Project setup and infrastructure (Story 1.1)
- FastAPI backend structure
- Database models and migrations
- Docker configuration

## [1.0.0] - 2025-08-16

### Added
- Initial project creation
- Basic project structure
- README and documentation

---

[Unreleased]: https://eniasgit.zeabur.app/demo/youtube-summarizer/compare/v3.1.0...HEAD
[3.1.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v3.1.0
[2.5.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v2.5.0
[2.4.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v2.4.0
[2.3.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v2.3.0
[2.2.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v2.2.0
[2.1.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v2.1.0
[1.5.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v1.5.0
[1.4.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v1.4.0
[1.3.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v1.3.0
[1.2.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v1.2.0
[1.1.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v1.1.0
[1.0.0]: https://eniasgit.zeabur.app/demo/youtube-summarizer/releases/tag/v1.0.0