youtube-summarizer/TESTING-INSTRUCTIONS.md

Testing Instructions - YouTube Summarizer

This document provides comprehensive testing guidelines, standards, and procedures for the YouTube Summarizer project.

Table of Contents

1. Test Runner System
2. Quick Commands
3. Testing Standards
4. Test Structure
5. Unit Testing
6. Integration Testing
7. Frontend Testing
8. Quality Checklist
9. Test Runner Development Standards
10. Troubleshooting

Test Runner System 🚀

Overview

The YouTube Summarizer includes a production-ready test runner system with intelligent test discovery, parallel execution, and comprehensive reporting. The test runner discovered 229 unit tests across all project modules and provides ultra-fast feedback for development.

Quick Commands

# ⚡ Fast Feedback Loop (Primary Development Workflow)
./run_tests.sh run-unit --fail-fast           # Ultra-fast unit tests (~0.2s)
./run_tests.sh run-specific "test_auth*.py"   # Test specific patterns

# 🔍 Test Discovery & Validation  
./run_tests.sh list --category unit           # Show all 229 discovered tests
./scripts/validate_test_setup.py              # Validate test environment

# 📊 Comprehensive Testing
./run_tests.sh run-all --coverage --parallel  # Full suite with coverage
./run_tests.sh run-integration                # Integration & API tests
./run_tests.sh run-coverage --html            # Generate HTML coverage reports

Test Categories Discovered

• Unit Tests: 229 tests across all service modules (auth, video, cache, AI, etc.)
• Integration Tests: API endpoints, database operations, external service integration
• Pipeline Tests: End-to-end workflow validation
• Authentication Tests: JWT, session management, security validation

Test Runner Features

🎯 Intelligent Test Discovery

• Automatically categorizes tests by type (unit, integration, API, auth, etc.)
• Analyzes test files using AST parsing for accurate classification
• Supports pytest markers for advanced filtering

⚡ Performance Optimized

• Ultra-fast execution: 229 tests discovered in ~0.2 seconds
• Parallel execution support with configurable workers
• Smart caching and dependency management

📊 Multiple Report Formats

# Generate different report formats
./run_tests.sh run-all --reports html,json,junit
# Outputs:
# - test_reports/test_report.html (Interactive dashboard)
# - test_reports/test_report.json (CI/CD integration)  
# - test_reports/junit.xml (Standard format)

🔧 Developer Tools

• One-time setup: ./scripts/setup_test_env.sh
• Environment validation: ./scripts/validate_test_setup.py
• Test runner CLI with comprehensive options
• Integration with coverage.py for detailed analysis

Testing Standards

Test Runner System Integration

Production-Ready Test Runner: The project includes a comprehensive test runner with intelligent discovery, parallel execution, and multi-format reporting.

Current Test Coverage: 229 unit tests discovered across all modules

• Video service tests, Authentication tests, Cache management tests
• AI model service tests, Pipeline tests, Export service tests

# Primary Testing Commands (Use These Instead of Direct pytest)
./run_tests.sh run-unit --fail-fast           # Ultra-fast feedback (0.2s discovery)
./run_tests.sh run-all --coverage --parallel  # Complete test suite
./run_tests.sh run-specific "test_auth*.py"   # Test specific patterns
./run_tests.sh list --category unit           # View all discovered tests

# Setup and Validation
./scripts/setup_test_env.sh                   # One-time environment setup  
./scripts/validate_test_setup.py              # Validate test environment

Test Coverage Requirements

• Minimum 80% code coverage
• 100% coverage for critical paths
• All edge cases tested
• Error conditions covered

# Run tests with coverage
./run_tests.sh run-all --coverage --html

# Coverage report should show:
# src/services/youtube.py      95%
# src/services/summarizer.py   88%
# src/api/routes.py            92%

Test Structure

tests/
├── unit/ (229 tests discovered)
│   ├── test_youtube_service.py       # Video URL parsing, validation
│   ├── test_summarizer_service.py    # AI model integration
│   ├── test_cache_service.py         # Caching and performance
│   ├── test_auth_service.py          # Authentication and JWT
│   └── test_*_service.py             # All service modules covered
├── integration/
│   ├── test_api_endpoints.py         # FastAPI route testing
│   └── test_database.py              # Database operations
├── fixtures/
│   ├── sample_transcripts.json       # Test data
│   └── mock_responses.py             # Mock API responses
├── test_reports/                     # Generated by test runner
│   ├── test_report.html              # Interactive dashboard
│   ├── coverage.xml                  # Coverage data
│   └── junit.xml                     # CI/CD integration
└── conftest.py

Unit Testing

Unit Test Example

# tests/unit/test_youtube_service.py
import pytest
from unittest.mock import Mock, patch, AsyncMock
from src.services.youtube import YouTubeService

class TestYouTubeService:
    @pytest.fixture
    def youtube_service(self):
        return YouTubeService()
    
    @pytest.fixture
    def mock_transcript(self):
        return [
            {"text": "Hello world", "start": 0.0, "duration": 2.0},
            {"text": "This is a test", "start": 2.0, "duration": 3.0}
        ]
    
    @pytest.mark.asyncio
    @pytest.mark.unit  # Test runner marker for categorization
    async def test_extract_transcript_success(
        self, 
        youtube_service, 
        mock_transcript
    ):
        with patch('youtube_transcript_api.YouTubeTranscriptApi.get_transcript') as mock_get:
            mock_get.return_value = mock_transcript
            
            result = await youtube_service.extract_transcript("test_id")
            
            assert result == mock_transcript
            mock_get.assert_called_once_with("test_id")
    
    @pytest.mark.unit
    def test_extract_video_id_various_formats(self, youtube_service):
        test_cases = [
            ("https://www.youtube.com/watch?v=abc123", "abc123"),
            ("https://youtu.be/xyz789", "xyz789"),
            ("https://youtube.com/embed/qwe456", "qwe456"),
            ("https://www.youtube.com/watch?v=test&t=123", "test")
        ]
        
        for url, expected_id in test_cases:
            assert youtube_service.extract_video_id(url) == expected_id

Test Markers for Intelligent Categorization

# Test markers for intelligent categorization
@pytest.mark.unit           # Fast, isolated unit tests
@pytest.mark.integration    # Database/API integration tests  
@pytest.mark.auth          # Authentication and security tests
@pytest.mark.api           # API endpoint tests
@pytest.mark.pipeline      # End-to-end pipeline tests
@pytest.mark.slow          # Tests taking >5 seconds

# Run specific categories
# ./run_tests.sh run-integration  # Runs integration + api marked tests
# ./run_tests.sh list --category unit  # Shows all unit tests

Integration Testing

Integration Test Example

# tests/integration/test_api_endpoints.py
import pytest
from fastapi.testclient import TestClient
from src.main import app

@pytest.fixture
def client():
    return TestClient(app)

class TestSummarizationAPI:
    @pytest.mark.asyncio
    @pytest.mark.integration
    @pytest.mark.api
    async def test_summarize_endpoint(self, client):
        response = client.post("/api/summarize", json={
            "url": "https://youtube.com/watch?v=test123",
            "model": "openai",
            "options": {"max_length": 500}
        })
        
        assert response.status_code == 200
        data = response.json()
        assert "job_id" in data
        assert data["status"] == "processing"
    
    @pytest.mark.asyncio
    @pytest.mark.integration
    async def test_get_summary(self, client):
        # First create a summary
        create_response = client.post("/api/summarize", json={
            "url": "https://youtube.com/watch?v=test123"
        })
        job_id = create_response.json()["job_id"]
        
        # Then retrieve it
        get_response = client.get(f"/api/summary/{job_id}")
        assert get_response.status_code in [200, 202]  # 202 if still processing

Frontend Testing

Frontend Test Commands

# Frontend testing (Vitest + RTL)
cd frontend && npm test
cd frontend && npm run test:coverage

# Frontend test structure
frontend/src/
├── components/
│   ├── SummarizeForm.test.tsx    # Component unit tests
│   ├── ProgressTracker.test.tsx  # React Testing Library
│   └── TranscriptViewer.test.tsx # User interaction tests
├── hooks/
│   ├── useAuth.test.ts           # Custom hooks testing
│   └── useSummary.test.ts        # State management tests
└── utils/
    └── helpers.test.ts           # Utility function tests

Frontend Test Example

// frontend/src/components/SummarizeForm.test.tsx
import { render, screen, fireEvent, waitFor } from '@testing-library/react';
import { SummarizeForm } from './SummarizeForm';

describe('SummarizeForm', () => {
  it('should validate YouTube URL format', async () => {
    render(<SummarizeForm onSubmit={jest.fn()} />);
    
    const urlInput = screen.getByPlaceholderText('Enter YouTube URL');
    const submitButton = screen.getByText('Summarize');
    
    // Test invalid URL
    fireEvent.change(urlInput, { target: { value: 'invalid-url' } });
    fireEvent.click(submitButton);
    
    await waitFor(() => {
      expect(screen.getByText('Please enter a valid YouTube URL')).toBeInTheDocument();
    });
    
    // Test valid URL
    fireEvent.change(urlInput, { 
      target: { value: 'https://youtube.com/watch?v=test123' } 
    });
    fireEvent.click(submitButton);
    
    await waitFor(() => {
      expect(screen.queryByText('Please enter a valid YouTube URL')).not.toBeInTheDocument();
    });
  });
});

Quality Checklist

Before marking any task as complete:

• All tests pass (./run_tests.sh run-all)
• Code coverage > 80% (./run_tests.sh run-all --coverage)
• Unit tests pass with fast feedback (./run_tests.sh run-unit --fail-fast)
• Integration tests validated (./run_tests.sh run-integration)
• Frontend tests pass (cd frontend && npm test)
• No linting errors (ruff check src/)
• Type checking passes (mypy src/)
• Documentation updated
• Task Master updated
• Changes committed with proper message

Test Runner Development Standards

For AI Agents

When working on this codebase:

1. Always use Test Runner: Use ./run_tests.sh commands instead of direct pytest
2. Fast Feedback First: Start with ./run_tests.sh run-unit --fail-fast for rapid development
3. Follow TDD: Write tests before implementation using test runner validation
4. Use Markers: Add proper pytest markers for test categorization
5. Validate Setup: Run ./scripts/validate_test_setup.py when encountering issues
6. Full Validation: Use ./run_tests.sh run-all --coverage before completing tasks

Development Integration

Story-Driven Development

# 1. Start story implementation
cat docs/stories/2.1.single-ai-model-integration.md

# 2. Fast feedback during development
./run_tests.sh run-unit --fail-fast           # Instant validation

# 3. Test specific modules as you build
./run_tests.sh run-specific "test_anthropic*.py"

# 4. Full validation before story completion
./run_tests.sh run-all --coverage

BMad Method Integration

• Seamlessly integrates with BMad agent workflows
• Provides fast feedback for TDD development approach
• Supports continuous validation during story implementation

Test Runner Architecture

Core Components:

• TestRunner: Main orchestration engine (400+ lines)
• TestDiscovery: Intelligent test categorization (500+ lines)
• TestExecution: Parallel execution engine (600+ lines)
• ReportGenerator: Multi-format reporting (500+ lines)
• CLI Interface: Comprehensive command-line tool (500+ lines)

Configuration Files:

• pytest.ini - Test discovery and markers
• .coveragerc - Coverage configuration and exclusions
• backend/test_runner/config/ - Test runner configuration

Required Test Patterns

Service Tests

@pytest.mark.unit
@pytest.mark.asyncio
async def test_service_method(self, mock_dependencies):
    # Test business logic with mocked dependencies
    pass

API Tests

@pytest.mark.integration
@pytest.mark.api
def test_api_endpoint(self, client):
    # Test API endpoints with TestClient
    pass

Pipeline Tests

@pytest.mark.pipeline
@pytest.mark.slow
async def test_end_to_end_pipeline(self, full_setup):
    # Test complete workflows
    pass

Troubleshooting

Common Issues

Tests not found

./run_tests.sh list --verbose                 # Debug test discovery
./scripts/validate_test_setup.py              # Check setup

Environment issues

./scripts/setup_test_env.sh                   # Re-run setup
source venv/bin/activate                      # Check virtual environment

Performance issues

./run_tests.sh run-all --no-parallel          # Disable parallel execution
./run_tests.sh run-unit --fail-fast           # Use fast subset for development

Import errors

# Check PYTHONPATH and virtual environment
echo $PYTHONPATH
which python3
pip list | grep -E "(pytest|fastapi)"

Quick Fixes

• Test discovery issues → Run validation script
• Import errors → Check PYTHONPATH and virtual environment
• Slow execution → Use --parallel flag or filter tests with --category
• Coverage gaps → Use --coverage --html to identify missing areas

Debug Commands

# Test runner debugging
./run_tests.sh run-specific "test_auth" --verbose    # Debug specific tests
./run_tests.sh list --category integration           # List tests by category
./scripts/validate_test_setup.py --verbose           # Detailed environment check

# Performance analysis
time ./run_tests.sh run-unit --fail-fast             # Measure execution time
./run_tests.sh run-all --reports json                # Generate performance data

Advanced Features

Parallel Execution

# Enable parallel testing (default: auto-detect cores)
./run_tests.sh run-all --parallel

# Specify number of workers
./run_tests.sh run-all --parallel --workers 4

# Disable parallel execution for debugging
./run_tests.sh run-all --no-parallel

Custom Test Filters

# Run tests matching pattern
./run_tests.sh run-specific "test_youtube" --pattern "extract"

# Run tests by marker combination
./run_tests.sh run-all --markers "unit and not slow"

# Run tests for specific modules
./run_tests.sh run-specific "backend/tests/unit/test_auth_service.py"

CI/CD Integration

# Generate CI-friendly outputs
./run_tests.sh run-all --coverage --reports junit,json
# Outputs: junit.xml for CI, test_report.json for analysis

# Exit code handling
./run_tests.sh run-all --fail-fast --exit-on-error
# Exits immediately on first failure for CI efficiency

---

This testing guide ensures comprehensive, efficient testing across the YouTube Summarizer project. Always use the test runner system for optimal development workflow.