14 KiB

Raw Blame History

Test Runner Guide

Overview

The YouTube Summarizer Test Runner is a comprehensive testing framework that provides intelligent test discovery, parallel execution, multiple report formats, and advanced features for maintaining high code quality.

Quick Start

Setup

# Run the setup script (from project root)
./scripts/setup_test_env.sh

# Activate virtual environment
source venv/bin/activate

# Run all tests
./run_tests.sh run-all

Basic Usage

# Run all tests with coverage
python3 -m backend.test_runner run-all --coverage

# Run only unit tests
python3 -m backend.test_runner run-unit

# Run integration tests
python3 -m backend.test_runner run-integration

# Run tests in parallel
python3 -m backend.test_runner run-all --parallel

# Generate HTML coverage report
python3 -m backend.test_runner run-coverage

Test Runner Architecture

Core Components

TestRunner (backend/test_runner/core/test_runner.py)
- Main orchestration engine
- Supports multiple execution modes
- Handles parallel processing and result aggregation
TestDiscovery (backend/test_runner/core/test_discovery.py)
- Intelligent test categorization
- Dependency analysis
- Pattern matching for test files
TestExecution (backend/test_runner/core/test_execution.py)
- pytest and frontend test execution
- Environment management
- Real-time progress monitoring
ReportGenerator (backend/test_runner/core/reporting.py)
- Multiple report formats (HTML, JSON, JUnit, Markdown, CSV)
- Historical tracking and trend analysis
- Quality metrics and coverage analysis
Configuration (backend/test_runner/config/test_config.py)
- Centralized configuration management
- Environment-specific settings
- Validation and defaults

Test Categories

The test runner automatically categorizes tests based on file patterns and markers:

Backend Tests

Unit Tests (backend/tests/unit/): Fast, isolated tests with no external dependencies
Integration Tests (backend/tests/integration/): Tests with database or API dependencies
API Tests: FastAPI endpoint testing with TestClient
Performance Tests: Load and performance validation
Database Tests: Database-specific functionality

Frontend Tests

Component Tests (frontend/src/test/): React component unit tests
Hook Tests: Custom React hook testing
Integration Tests: Component integration scenarios
E2E Tests: End-to-end user workflows

Test Markers

Tests can be marked with pytest markers for targeted execution:

import pytest

@pytest.mark.unit
@pytest.mark.asyncio
async def test_youtube_service():
    """Fast unit test for YouTube service."""
    pass

@pytest.mark.integration
@pytest.mark.database
def test_user_creation():
    """Integration test requiring database."""
    pass

@pytest.mark.slow
@pytest.mark.performance
def test_large_transcript_processing():
    """Performance test for large data."""
    pass

Command Line Interface

Primary Commands

`run-all`

Executes the complete test suite with comprehensive reporting.

python3 -m backend.test_runner run-all [options]

Options:
  --coverage           Generate coverage report
  --parallel           Run tests in parallel
  --fail-fast          Stop on first failure
  --reports FORMAT     Report formats (html,json,junit,markdown,csv)
  --verbose           Detailed output
  --timeout SECONDS    Test timeout

`run-unit`

Executes only unit tests for fast feedback.

python3 -m backend.test_runner run-unit [options]

`run-integration`

Executes integration tests with database and external service dependencies.

python3 -m backend.test_runner run-integration [options]

`run-coverage`

Specialized coverage analysis with detailed reporting.

python3 -m backend.test_runner run-coverage [options]

Options:
  --min-coverage NUM   Minimum coverage percentage (default: 80)
  --html              Generate HTML coverage report
  --xml               Generate XML coverage report for CI

`run-frontend`

Executes frontend tests using npm/Vitest.

python3 -m backend.test_runner run-frontend [options]

`discover`

Lists all discovered tests without execution.

python3 -m backend.test_runner discover [options]

Options:
  --category CATEGORY  Filter by test category
  --pattern PATTERN    Filter by file pattern
  --verbose           Show test details

`validate`

Validates test environment and configuration.

python3 -m backend.test_runner validate [options]

Advanced Options

Filtering and Selection

# Run tests matching pattern
python3 -m backend.test_runner run-all --pattern "test_auth*"

# Run specific categories
python3 -m backend.test_runner run-all --category unit,api

# Run tests with specific markers
python3 -m backend.test_runner run-all --markers "unit and not slow"

Parallel Execution

# Use specific number of workers
python3 -m backend.test_runner run-all --parallel --workers 4

# Auto-detect workers based on CPU cores
python3 -m backend.test_runner run-all --parallel --workers auto

Output Control

# Minimal output
python3 -m backend.test_runner run-all --quiet

# Detailed debugging
python3 -m backend.test_runner run-all --verbose --debug

# Custom output format
python3 -m backend.test_runner run-all --output-format json

Configuration

Configuration Files

`pytest.ini`

Primary pytest configuration with test discovery and markers:

[tool:pytest]
testpaths = backend/tests
python_files = test_*.py *_test.py
python_classes = Test*
python_functions = test_*

markers =
    unit: Unit tests (fast, isolated)
    integration: Integration tests (database, APIs)
    slow: Tests taking more than 5 seconds
    auth: Authentication-related tests

`.coveragerc`

Coverage configuration with exclusions and reporting:

[run]
source = backend
omit = */tests/*, */test_*, */migrations/*

[report]
exclude_lines =
    pragma: no cover
    def __repr__
    raise AssertionError

[html]
directory = test_reports/coverage_html

Test Configuration (`backend/test_runner/config/test_config.py`)

Runtime configuration with environment-specific settings:

config = TestConfig(
    test_timeout=300,
    parallel_workers=4,
    coverage_threshold=80,
    fail_fast=False,
    reports_dir="test_reports",
    mock_external_apis=True
)

Environment Variables

# Test database
DATABASE_URL=sqlite:///:memory:
TEST_DATABASE_URL=sqlite:///./test_data/test.db

# Test mode flags
TESTING=true
TEST_MODE=true
MOCK_EXTERNAL_APIS=true

# API keys for testing
ANTHROPIC_API_KEY=test_key
JWT_SECRET_KEY=test_secret_key

# Timeouts and limits
TEST_TIMEOUT=300
NETWORK_TIMEOUT=30

Report Formats

HTML Reports

Comprehensive interactive reports with charts and navigation:

python3 -m backend.test_runner run-all --reports html
# Output: test_reports/test_report.html

Features:

Interactive test results with filtering
Coverage visualization
Performance metrics
Historical trends
Failure analysis

JSON Reports

Machine-readable reports for CI/CD integration:

python3 -m backend.test_runner run-all --reports json
# Output: test_reports/test_report.json

Schema:

{
  "summary": {
    "total_tests": 150,
    "passed": 145,
    "failed": 3,
    "skipped": 2,
    "duration": 45.2,
    "coverage_percentage": 87.5
  },
  "suites": [...],
  "failures": [...],
  "coverage": {...}
}

JUnit XML

Standard JUnit format for CI systems:

python3 -m backend.test_runner run-all --reports junit
# Output: test_reports/junit.xml

Markdown Reports

Human-readable reports for documentation:

python3 -m backend.test_runner run-all --reports markdown
# Output: test_reports/test_report.md

CSV Reports

Data export for analysis and tracking:

python3 -m backend.test_runner run-all --reports csv
# Output: test_reports/test_report.csv

Advanced Features

Smart Test Selection

The runner can intelligently select tests based on code changes:

# Run tests affected by recent changes
python3 -m backend.test_runner run-smart --since HEAD~1

# Run tests for specific files
python3 -m backend.test_runner run-smart --changed-files "backend/services/auth.py"

Historical Tracking

Track test performance and coverage over time:

# Generate trend report
python3 -m backend.test_runner run-trends --days 30

# Compare with previous run
python3 -m backend.test_runner run-all --compare-with previous

Quality Gates

Enforce quality standards with automatic failure:

# Fail if coverage below 80%
python3 -m backend.test_runner run-all --min-coverage 80

# Fail if performance regression > 20%
python3 -m backend.test_runner run-all --max-duration-increase 20

Test Environment Management

Automatic test environment setup and cleanup:

# Custom environment setup
from backend.test_runner.utils.environment import TestEnvironmentManager

manager = TestEnvironmentManager()
manager.setup_test_environment(config)
try:
    # Run tests
    pass
finally:
    manager.cleanup_test_environment()

Integration Workflows

Local Development

# Quick feedback loop
python3 -m backend.test_runner run-unit --fail-fast

# Pre-commit validation
python3 -m backend.test_runner run-all --coverage --min-coverage 80

# Full validation
python3 -m backend.test_runner run-all --reports html,junit --parallel

Continuous Integration

# GitHub Actions example
- name: Run Tests
  run: |
    source venv/bin/activate
    python3 -m backend.test_runner run-all \
      --reports junit,json \
      --coverage \
      --parallel \
      --min-coverage 80

Pre-commit Hooks

# .pre-commit-config.yaml
- repo: local
  hooks:
    - id: test-runner
      name: Run unit tests
      entry: python3 -m backend.test_runner run-unit --fail-fast
      language: system
      pass_filenames: false

Troubleshooting

Common Issues

Tests Not Discovered

# Check test discovery
python3 -m backend.test_runner discover --verbose

# Verify test patterns
python3 -m backend.test_runner discover --pattern "test_*"

Environment Issues

# Validate environment
python3 -m backend.test_runner validate

# Check dependencies
python3 -c "from backend.test_runner.utils.environment import check_test_dependencies; print(check_test_dependencies())"

Performance Issues

# Disable parallel execution
python3 -m backend.test_runner run-all --no-parallel

# Reduce workers
python3 -m backend.test_runner run-all --workers 2

# Skip slow tests
python3 -m backend.test_runner run-all --markers "not slow"

Database Issues

# Use memory database
export DATABASE_URL="sqlite:///:memory:"

# Reset test database
rm -f test_data/test.db

Debug Mode

# Enable debug logging
python3 -m backend.test_runner run-all --debug --verbose

# Show internal state
python3 -m backend.test_runner run-all --show-internals

Best Practices

Writing Tests

Use appropriate markers (@pytest.mark.unit, @pytest.mark.integration)
Follow naming conventions (test_*.py, Test* classes)
Mock external dependencies in unit tests
Use fixtures for common setup
Write descriptive test names and docstrings

Test Organization

Group related tests in classes
Use separate files for different components
Keep test files close to source code
Use consistent directory structure

Performance

Keep unit tests fast (< 100ms each)
Use parallel execution for large test suites
Mock external services and APIs
Cache expensive setup operations

Maintenance

Run tests regularly during development
Monitor coverage trends
Review and update slow tests
Clean up obsolete tests

API Reference

TestRunner Class

from backend.test_runner.core.test_runner import TestRunner

runner = TestRunner(config=config)
result = await runner.run_tests(
    mode=TestRunnerMode.ALL,
    coverage=True,
    parallel=True,
    reports=[ReportFormat.HTML, ReportFormat.JSON]
)

Configuration API

from backend.test_runner.config.test_config import TestConfig

config = TestConfig.load_from_file("test_config.yaml")
config.update_from_env()
config.validate()

Report Generation API

from backend.test_runner.core.reporting import ReportGenerator

generator = ReportGenerator(config)
report_path = await generator.generate_report(
    result=test_result,
    format_type=ReportFormat.HTML,
    output_path=Path("custom_report.html")
)

Examples

Custom Test Runner Script

#!/usr/bin/env python3
"""Custom test runner script."""

import asyncio
from pathlib import Path
from backend.test_runner.core.test_runner import TestRunner, TestRunnerMode
from backend.test_runner.config.test_config import TestConfig

async def main():
    config = TestConfig.load_default(Path.cwd())
    runner = TestRunner(config)
    
    result = await runner.run_tests(
        mode=TestRunnerMode.UNIT,
        coverage=True,
        parallel=True,
        fail_fast=True
    )
    
    print(f"Tests: {result.total_tests}")
    print(f"Passed: {result.passed}")
    print(f"Failed: {result.failed}")
    print(f"Coverage: {result.coverage_percentage:.1f}%")
    
    return 0 if result.success else 1

if __name__ == "__main__":
    exit(asyncio.run(main()))

Integration with pytest

# conftest.py
import pytest
from backend.test_runner.utils.environment import TestEnvironmentManager

@pytest.fixture(scope="session", autouse=True)
def setup_test_environment():
    """Setup test environment for entire test session."""
    manager = TestEnvironmentManager()
    config = TestConfig.load_default(Path.cwd())
    
    manager.setup_test_environment(config)
    yield
    manager.cleanup_test_environment()

Contributing

Adding New Features

Update relevant core modules
Add comprehensive tests
Update CLI interface
Update documentation
Add examples and troubleshooting

Testing the Test Runner

# Self-test the test runner
python3 -m pytest backend/test_runner/tests/ -v

# Integration test with sample project
python3 -m backend.test_runner run-all --config test_configs/sample_project.yaml

This test runner provides a robust foundation for maintaining high code quality in the YouTube Summarizer project with comprehensive testing capabilities, intelligent automation, and detailed reporting.

14 KiB Raw Blame History