164 lines
6.2 KiB
Markdown
164 lines
6.2 KiB
Markdown
# Testing Issues Documentation
|
|
|
|
**Date**: 2025-08-27
|
|
**Status**: Active Issues Documented
|
|
**Reporter**: Claude Code Agent
|
|
|
|
This document captures the testing problems encountered and their solutions for future reference.
|
|
|
|
## Issue #1: SQLAlchemy Relationship Resolution Error
|
|
|
|
**Problem**: Test runner and direct pytest failing with SQLAlchemy errors:
|
|
```
|
|
NameError: Module 'models' has no mapped classes registered under the name 'batch_job'
|
|
sqlalchemy.exc.InvalidRequestError: When initializing mapper Mapper[User(users)], expression 'backend.models.batch_job.BatchJob' failed to locate a name
|
|
```
|
|
|
|
**Root Cause**: The `BatchJob` and `BatchJobItem` models existed in `backend/models/batch_job.py` but were not imported in the models `__init__.py` file. This caused SQLAlchemy to fail when resolving relationships defined in the User model.
|
|
|
|
**Solution**: Added missing imports to `backend/models/__init__.py`:
|
|
```python
|
|
# Before (missing imports)
|
|
from .user import User, RefreshToken, APIKey, EmailVerificationToken, PasswordResetToken
|
|
from .summary import Summary, ExportHistory
|
|
|
|
# After (fixed)
|
|
from .user import User, RefreshToken, APIKey, EmailVerificationToken, PasswordResetToken
|
|
from .summary import Summary, ExportHistory
|
|
from .batch_job import BatchJob, BatchJobItem # Added this line
|
|
|
|
__all__ = [
|
|
# ... existing models ...
|
|
"BatchJob", # Added
|
|
"BatchJobItem", # Added
|
|
]
|
|
```
|
|
|
|
**Verification**: Auth service tests went from 20/21 failing to 21/21 passing.
|
|
|
|
## Issue #2: Pydantic Configuration Validation Errors
|
|
|
|
**Problem**: Tests failing with Pydantic validation errors:
|
|
```
|
|
pydantic_core._pydantic_core.ValidationError: 19 validation errors for VideoDownloadConfig
|
|
anthropic_api_key: Extra inputs are not permitted [type=extra_forbidden]
|
|
openai_api_key: Extra inputs are not permitted [type=extra_forbidden]
|
|
...
|
|
```
|
|
|
|
**Root Cause**: The `VideoDownloadConfig` class extends `BaseSettings` and automatically loads environment variables. However, the environment contained many API keys and configuration variables that weren't defined in the model schema. Pydantic 2.x defaults to `extra="forbid"` which rejects unknown fields.
|
|
|
|
**Solution**: Modified the `VideoDownloadConfig` class configuration:
|
|
```python
|
|
# File: backend/config/video_download_config.py
|
|
class VideoDownloadConfig(BaseSettings):
|
|
# ... model fields ...
|
|
|
|
class Config:
|
|
env_file = ".env"
|
|
env_prefix = "VIDEO_DOWNLOAD_"
|
|
case_sensitive = False
|
|
extra = "ignore" # Added this line to allow extra environment variables
|
|
```
|
|
|
|
**Verification**: Enhanced video service tests went from collection errors to 13/23 passing.
|
|
|
|
## Issue #3: Test Runner Result Parsing Bug
|
|
|
|
**Problem**: The custom test runner consistently reports "0/X passed" even when tests are actually passing.
|
|
|
|
**Evidence**:
|
|
- Test runner shows: `Completed unit tests: 0/229 passed`
|
|
- Direct pytest shows: `182 passed, 61 failed, 38 errors` (75% pass rate)
|
|
- Individual test files run successfully when tested directly
|
|
|
|
**Root Cause**: Bug in the test runner's pytest result parsing logic. The `TestExecutor` class in `backend/test_runner/core/test_execution.py` is not correctly parsing the pytest output or exit codes.
|
|
|
|
**Current Status**: **UNRESOLVED** - Test runner display bug exists but does not affect actual test functionality.
|
|
|
|
**Workaround**: Use direct pytest commands for accurate results:
|
|
```bash
|
|
# Instead of: ./run_tests.sh run-unit
|
|
# Use: PYTHONPATH=/path/to/project python3 -m pytest backend/tests/unit/
|
|
|
|
# For specific tests:
|
|
PYTHONPATH=/Users/enias/projects/my-ai-projects/apps/youtube-summarizer python3 -m pytest backend/tests/unit/test_auth_service.py -v
|
|
```
|
|
|
|
## Issue #4: Test Environment Setup
|
|
|
|
**Problem**: Tests may fail if run without proper PYTHONPATH configuration.
|
|
|
|
**Solution**: Always set PYTHONPATH when running tests directly:
|
|
```bash
|
|
export PYTHONPATH="/Users/enias/projects/my-ai-projects/apps/youtube-summarizer"
|
|
# or
|
|
PYTHONPATH=/Users/enias/projects/my-ai-projects/apps/youtube-summarizer python3 -m pytest
|
|
```
|
|
|
|
**Verification Script**: Use `./scripts/validate_test_setup.py` to check environment.
|
|
|
|
## Current Test Status (as of 2025-08-27 01:45 UTC)
|
|
|
|
### ✅ Working Test Categories:
|
|
- **Authentication Tests**: 21/21 passing (100%)
|
|
- **Core Service Tests**: Most passing
|
|
- **Database Model Tests**: Working after BatchJob fix
|
|
- **Basic Integration Tests**: Many passing
|
|
|
|
### ❌ Known Failing Areas:
|
|
- **Enhanced Video Service**: 10/23 failing (test implementation issues)
|
|
- **Video Downloader Tests**: Multiple failures (mocking issues)
|
|
- **AI Service Tests**: Some import/dependency errors
|
|
- **Complex Integration Tests**: Various issues
|
|
|
|
### Overall Stats:
|
|
- **Total Discovered**: 241 tests
|
|
- **Passing**: 182 tests (75%)
|
|
- **Failing**: 61 tests
|
|
- **Errors**: 38 tests
|
|
|
|
## Recommended Next Steps
|
|
|
|
1. **Fix Test Runner Parsing Bug**:
|
|
- Investigate `backend/test_runner/core/test_execution.py`
|
|
- Fix pytest result parsing logic
|
|
- Ensure proper exit code handling
|
|
|
|
2. **Address Remaining Test Failures**:
|
|
- Fix mocking issues in video downloader tests
|
|
- Resolve import/dependency errors in AI service tests
|
|
- Update test implementations for enhanced video service
|
|
|
|
3. **Improve Test Environment**:
|
|
- Create more reliable test fixtures
|
|
- Improve test isolation
|
|
- Add better error reporting
|
|
|
|
4. **Documentation**:
|
|
- Update TESTING-INSTRUCTIONS.md with current status
|
|
- Document workarounds for known issues
|
|
- Create debugging guide for test failures
|
|
|
|
## Commands for Testing Tomorrow
|
|
|
|
```bash
|
|
# Quick verification (works reliably)
|
|
PYTHONPATH=/Users/enias/projects/my-ai-projects/apps/youtube-summarizer python3 -m pytest backend/tests/unit/test_auth_service.py -v
|
|
|
|
# Full unit test run (accurate results)
|
|
PYTHONPATH=/Users/enias/projects/my-ai-projects/apps/youtube-summarizer python3 -m pytest backend/tests/unit/ --tb=no -q
|
|
|
|
# Debug specific failures
|
|
PYTHONPATH=/Users/enias/projects/my-ai-projects/apps/youtube-summarizer python3 -m pytest backend/tests/unit/test_enhanced_video_service.py -v --tb=short
|
|
|
|
# Test runner (has display bug but still useful for discovery)
|
|
./run_tests.sh list --category unit
|
|
```
|
|
|
|
---
|
|
|
|
**Last Updated**: 2025-08-27 01:45 UTC
|
|
**Next Review**: When test runner parsing bug is resolved
|
|
|
|
*This document should be updated as issues are resolved and new problems are discovered.* |