16 KiB

Raw Permalink Blame History

Testing Instructions - YouTube Summarizer

This document provides comprehensive testing guidelines, standards, and procedures for the YouTube Summarizer project.

Test Runner System
Quick Commands
Testing Standards
Test Structure
Unit Testing
Integration Testing
Frontend Testing
Quality Checklist
Test Runner Development Standards
Troubleshooting

Test Runner System 🚀

Overview

The YouTube Summarizer includes a production-ready test runner system with intelligent test discovery, parallel execution, and comprehensive reporting. The test runner discovered 229 unit tests across all project modules and provides ultra-fast feedback for development.

Quick Commands

# ⚡ Fast Feedback Loop (Primary Development Workflow)
./run_tests.sh run-unit --fail-fast           # Ultra-fast unit tests (~0.2s)
./run_tests.sh run-specific "test_auth*.py"   # Test specific patterns

# 🔍 Test Discovery & Validation  
./run_tests.sh list --category unit           # Show all 229 discovered tests
./scripts/validate_test_setup.py              # Validate test environment

# 📊 Comprehensive Testing
./run_tests.sh run-all --coverage --parallel  # Full suite with coverage
./run_tests.sh run-integration                # Integration & API tests
./run_tests.sh run-coverage --html            # Generate HTML coverage reports

Test Categories Discovered

Unit Tests: 229 tests across all service modules (auth, video, cache, AI, etc.)
Integration Tests: API endpoints, database operations, external service integration
Pipeline Tests: End-to-end workflow validation
Authentication Tests: JWT, session management, security validation

Test Runner Features

🎯 Intelligent Test Discovery

Automatically categorizes tests by type (unit, integration, API, auth, etc.)
Analyzes test files using AST parsing for accurate classification
Supports pytest markers for advanced filtering

⚡ Performance Optimized

Ultra-fast execution: 229 tests discovered in ~0.2 seconds
Parallel execution support with configurable workers
Smart caching and dependency management

📊 Multiple Report Formats

# Generate different report formats
./run_tests.sh run-all --reports html,json,junit
# Outputs:
# - test_reports/test_report.html (Interactive dashboard)
# - test_reports/test_report.json (CI/CD integration)  
# - test_reports/junit.xml (Standard format)

🔧 Developer Tools

One-time setup: ./scripts/setup_test_env.sh
Environment validation: ./scripts/validate_test_setup.py
Test runner CLI with comprehensive options
Integration with coverage.py for detailed analysis

Testing Standards

Test Runner System Integration

Production-Ready Test Runner: The project includes a comprehensive test runner with intelligent discovery, parallel execution, and multi-format reporting.

Current Test Coverage: 229 unit tests discovered across all modules

Video service tests, Authentication tests, Cache management tests
AI model service tests, Pipeline tests, Export service tests

# Primary Testing Commands (Use These Instead of Direct pytest)
./run_tests.sh run-unit --fail-fast           # Ultra-fast feedback (0.2s discovery)
./run_tests.sh run-all --coverage --parallel  # Complete test suite
./run_tests.sh run-specific "test_auth*.py"   # Test specific patterns
./run_tests.sh list --category unit           # View all discovered tests

# Setup and Validation
./scripts/setup_test_env.sh                   # One-time environment setup  
./scripts/validate_test_setup.py              # Validate test environment

Test Coverage Requirements

Minimum 80% code coverage
100% coverage for critical paths
All edge cases tested
Error conditions covered

# Run tests with coverage
./run_tests.sh run-all --coverage --html

# Coverage report should show:
# src/services/youtube.py      95%
# src/services/summarizer.py   88%
# src/api/routes.py            92%

Test Structure

tests/
├── unit/ (229 tests discovered)
│   ├── test_youtube_service.py       # Video URL parsing, validation
│   ├── test_summarizer_service.py    # AI model integration
│   ├── test_cache_service.py         # Caching and performance
│   ├── test_auth_service.py          # Authentication and JWT
│   └── test_*_service.py             # All service modules covered
├── integration/
│   ├── test_api_endpoints.py         # FastAPI route testing
│   └── test_database.py              # Database operations
├── fixtures/
│   ├── sample_transcripts.json       # Test data
│   └── mock_responses.py             # Mock API responses
├── test_reports/                     # Generated by test runner
│   ├── test_report.html              # Interactive dashboard
│   ├── coverage.xml                  # Coverage data
│   └── junit.xml                     # CI/CD integration
└── conftest.py

Unit Testing

Unit Test Example

# tests/unit/test_youtube_service.py
import pytest
from unittest.mock import Mock, patch, AsyncMock
from src.services.youtube import YouTubeService

class TestYouTubeService:
    @pytest.fixture
    def youtube_service(self):
        return YouTubeService()
    
    @pytest.fixture
    def mock_transcript(self):
        return [
            {"text": "Hello world", "start": 0.0, "duration": 2.0},
            {"text": "This is a test", "start": 2.0, "duration": 3.0}
        ]
    
    @pytest.mark.asyncio
    @pytest.mark.unit  # Test runner marker for categorization
    async def test_extract_transcript_success(
        self, 
        youtube_service, 
        mock_transcript
    ):
        with patch('youtube_transcript_api.YouTubeTranscriptApi.get_transcript') as mock_get:
            mock_get.return_value = mock_transcript
            
            result = await youtube_service.extract_transcript("test_id")
            
            assert result == mock_transcript
            mock_get.assert_called_once_with("test_id")
    
    @pytest.mark.unit
    def test_extract_video_id_various_formats(self, youtube_service):
        test_cases = [
            ("https://www.youtube.com/watch?v=abc123", "abc123"),
            ("https://youtu.be/xyz789", "xyz789"),
            ("https://youtube.com/embed/qwe456", "qwe456"),
            ("https://www.youtube.com/watch?v=test&t=123", "test")
        ]
        
        for url, expected_id in test_cases:
            assert youtube_service.extract_video_id(url) == expected_id

Test Markers for Intelligent Categorization

# Test markers for intelligent categorization
@pytest.mark.unit           # Fast, isolated unit tests
@pytest.mark.integration    # Database/API integration tests  
@pytest.mark.auth          # Authentication and security tests
@pytest.mark.api           # API endpoint tests
@pytest.mark.pipeline      # End-to-end pipeline tests
@pytest.mark.slow          # Tests taking >5 seconds

# Run specific categories
# ./run_tests.sh run-integration  # Runs integration + api marked tests
# ./run_tests.sh list --category unit  # Shows all unit tests

Integration Testing

Integration Test Example

# tests/integration/test_api_endpoints.py
import pytest
from fastapi.testclient import TestClient
from src.main import app

@pytest.fixture
def client():
    return TestClient(app)

class TestSummarizationAPI:
    @pytest.mark.asyncio
    @pytest.mark.integration
    @pytest.mark.api
    async def test_summarize_endpoint(self, client):
        response = client.post("/api/summarize", json={
            "url": "https://youtube.com/watch?v=test123",
            "model": "openai",
            "options": {"max_length": 500}
        })
        
        assert response.status_code == 200
        data = response.json()
        assert "job_id" in data
        assert data["status"] == "processing"
    
    @pytest.mark.asyncio
    @pytest.mark.integration
    async def test_get_summary(self, client):
        # First create a summary
        create_response = client.post("/api/summarize", json={
            "url": "https://youtube.com/watch?v=test123"
        })
        job_id = create_response.json()["job_id"]
        
        # Then retrieve it
        get_response = client.get(f"/api/summary/{job_id}")
        assert get_response.status_code in [200, 202]  # 202 if still processing

Frontend Testing

Frontend Test Commands

# Frontend testing (Vitest + RTL)
cd frontend && npm test
cd frontend && npm run test:coverage

# Frontend test structure
frontend/src/
├── components/
│   ├── SummarizeForm.test.tsx    # Component unit tests
│   ├── ProgressTracker.test.tsx  # React Testing Library
│   └── TranscriptViewer.test.tsx # User interaction tests
├── hooks/
│   ├── useAuth.test.ts           # Custom hooks testing
│   └── useSummary.test.ts        # State management tests
└── utils/
    └── helpers.test.ts           # Utility function tests

Frontend Test Example

// frontend/src/components/SummarizeForm.test.tsx
import { render, screen, fireEvent, waitFor } from '@testing-library/react';
import { SummarizeForm } from './SummarizeForm';

describe('SummarizeForm', () => {
  it('should validate YouTube URL format', async () => {
    render(<SummarizeForm onSubmit={jest.fn()} />);
    
    const urlInput = screen.getByPlaceholderText('Enter YouTube URL');
    const submitButton = screen.getByText('Summarize');
    
    // Test invalid URL
    fireEvent.change(urlInput, { target: { value: 'invalid-url' } });
    fireEvent.click(submitButton);
    
    await waitFor(() => {
      expect(screen.getByText('Please enter a valid YouTube URL')).toBeInTheDocument();
    });
    
    // Test valid URL
    fireEvent.change(urlInput, { 
      target: { value: 'https://youtube.com/watch?v=test123' } 
    });
    fireEvent.click(submitButton);
    
    await waitFor(() => {
      expect(screen.queryByText('Please enter a valid YouTube URL')).not.toBeInTheDocument();
    });
  });
});

Quality Checklist

Before marking any task as complete:

All tests pass (./run_tests.sh run-all)
Code coverage > 80% (./run_tests.sh run-all --coverage)
Unit tests pass with fast feedback (./run_tests.sh run-unit --fail-fast)
Integration tests validated (./run_tests.sh run-integration)
Frontend tests pass (cd frontend && npm test)
No linting errors (ruff check src/)
Type checking passes (mypy src/)
Documentation updated
Task Master updated
Changes committed with proper message

Test Runner Development Standards

For AI Agents

When working on this codebase:

Always use Test Runner: Use ./run_tests.sh commands instead of direct pytest
Fast Feedback First: Start with ./run_tests.sh run-unit --fail-fast for rapid development
Follow TDD: Write tests before implementation using test runner validation
Use Markers: Add proper pytest markers for test categorization
Validate Setup: Run ./scripts/validate_test_setup.py when encountering issues
Full Validation: Use ./run_tests.sh run-all --coverage before completing tasks

Development Integration

Story-Driven Development

# 1. Start story implementation
cat docs/stories/2.1.single-ai-model-integration.md

# 2. Fast feedback during development
./run_tests.sh run-unit --fail-fast           # Instant validation

# 3. Test specific modules as you build
./run_tests.sh run-specific "test_anthropic*.py"

# 4. Full validation before story completion
./run_tests.sh run-all --coverage

BMad Method Integration

Seamlessly integrates with BMad agent workflows
Provides fast feedback for TDD development approach
Supports continuous validation during story implementation

Test Runner Architecture

Core Components:

TestRunner: Main orchestration engine (400+ lines)
TestDiscovery: Intelligent test categorization (500+ lines)
TestExecution: Parallel execution engine (600+ lines)
ReportGenerator: Multi-format reporting (500+ lines)
CLI Interface: Comprehensive command-line tool (500+ lines)

Configuration Files:

pytest.ini - Test discovery and markers
.coveragerc - Coverage configuration and exclusions
backend/test_runner/config/ - Test runner configuration

Required Test Patterns

Service Tests

@pytest.mark.unit
@pytest.mark.asyncio
async def test_service_method(self, mock_dependencies):
    # Test business logic with mocked dependencies
    pass

API Tests

@pytest.mark.integration
@pytest.mark.api
def test_api_endpoint(self, client):
    # Test API endpoints with TestClient
    pass

Pipeline Tests

@pytest.mark.pipeline
@pytest.mark.slow
async def test_end_to_end_pipeline(self, full_setup):
    # Test complete workflows
    pass

Troubleshooting

Common Issues

Tests not found

./run_tests.sh list --verbose                 # Debug test discovery
./scripts/validate_test_setup.py              # Check setup

Environment issues

./scripts/setup_test_env.sh                   # Re-run setup
source venv/bin/activate                      # Check virtual environment

Performance issues

./run_tests.sh run-all --no-parallel          # Disable parallel execution
./run_tests.sh run-unit --fail-fast           # Use fast subset for development

Import errors

# Check PYTHONPATH and virtual environment
echo $PYTHONPATH
which python3
pip list | grep -E "(pytest|fastapi)"

Quick Fixes

Test discovery issues → Run validation script
Import errors → Check PYTHONPATH and virtual environment
Slow execution → Use --parallel flag or filter tests with --category
Coverage gaps → Use --coverage --html to identify missing areas

Debug Commands

# Test runner debugging
./run_tests.sh run-specific "test_auth" --verbose    # Debug specific tests
./run_tests.sh list --category integration           # List tests by category
./scripts/validate_test_setup.py --verbose           # Detailed environment check

# Performance analysis
time ./run_tests.sh run-unit --fail-fast             # Measure execution time
./run_tests.sh run-all --reports json                # Generate performance data

Advanced Features

Parallel Execution

# Enable parallel testing (default: auto-detect cores)
./run_tests.sh run-all --parallel

# Specify number of workers
./run_tests.sh run-all --parallel --workers 4

# Disable parallel execution for debugging
./run_tests.sh run-all --no-parallel

Custom Test Filters

# Run tests matching pattern
./run_tests.sh run-specific "test_youtube" --pattern "extract"

# Run tests by marker combination
./run_tests.sh run-all --markers "unit and not slow"

# Run tests for specific modules
./run_tests.sh run-specific "backend/tests/unit/test_auth_service.py"

CI/CD Integration

# Generate CI-friendly outputs
./run_tests.sh run-all --coverage --reports junit,json
# Outputs: junit.xml for CI, test_report.json for analysis

# Exit code handling
./run_tests.sh run-all --fail-fast --exit-on-error
# Exits immediately on first failure for CI efficiency

This testing guide ensures comprehensive, efficient testing across the YouTube Summarizer project. Always use the test runner system for optimal development workflow.

16 KiB Raw Permalink Blame History