# Database Management Rules

## Core Principles
1. **Use PostgreSQL**: Multi-service support, JSONB for flexibility
2. **Registry Pattern**: Prevent SQLAlchemy initialization conflicts
3. **Migration First**: Every schema change through Alembic
4. **Test Migrations**: Up and down paths must work

## Schema Design Rules

### Table Structure
```sql
-- REQUIRED: Primary tables
CREATE TABLE media_files (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    source_path TEXT NOT NULL,
    local_path TEXT,
    file_hash VARCHAR(64) UNIQUE,
    file_size BIGINT,
    mime_type VARCHAR(100),
    duration_seconds FLOAT,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

CREATE TABLE transcripts (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    media_file_id UUID REFERENCES media_files(id) ON DELETE CASCADE,
    version VARCHAR(10) NOT NULL, -- v1, v2, v3, v4
    raw_content JSONB NOT NULL,
    enhanced_content JSONB,
    multipass_content JSONB,
    diarized_content JSONB,
    confidence_scores JSONB,
    speaker_profiles JSONB,
    processing_time FLOAT,
    model_config JSONB,
    created_at TIMESTAMP DEFAULT NOW(),
    
    -- Version tracking
    parent_transcript_id UUID REFERENCES transcripts(id),
    
    -- Indexes
    INDEX idx_media_version (media_file_id, version),
    INDEX idx_created (created_at)
);

CREATE TABLE processing_jobs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    job_type VARCHAR(50) NOT NULL,
    status VARCHAR(20) NOT NULL,
    media_file_id UUID REFERENCES media_files(id),
    config JSONB DEFAULT '{}',
    result JSONB,
    error TEXT,
    started_at TIMESTAMP,
    completed_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT NOW(),
    
    INDEX idx_status (status),
    INDEX idx_type_status (job_type, status)
);
```

### JSONB Structure Rules
```python
# REQUIRED: Transcript JSON structure
transcript_json = {
    "text": str,  # Full text
    "language": str,  # Detected or specified
    "duration": float,  # Total duration
    "segments": [
        {
            "id": int,
            "start": float,
            "end": float,
            "text": str,
            "confidence": float,
            "words": [  # Optional
                {
                    "word": str,
                    "start": float,
                    "end": float,
                    "confidence": float
                }
            ],
            "speaker": str  # Optional, for v4
        }
    ],
    "metadata": {
        "model": str,
        "version": str,
        "processing_date": str,
        "parameters": dict
    }
}
```

## Migration Rules

### Creating Migrations
```bash
# REQUIRED: Always use descriptive names
alembic revision -m "add_speaker_profiles_to_transcripts"

# REQUIRED: Test before applying
alembic upgrade head --sql  # Preview SQL
alembic upgrade head        # Apply
alembic downgrade -1        # Test rollback
alembic upgrade head        # Re-apply
```

### Migration Template
```python
"""Add speaker profiles to transcripts

Revision ID: ${revision_id}
Create Date: ${create_date}
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql

def upgrade():
    """REQUIRED: Forward migration"""
    op.add_column('transcripts',
        sa.Column('speaker_profiles', postgresql.JSONB, nullable=True)
    )
    op.create_index(
        'idx_transcripts_speakers',
        'transcripts',
        [sa.text("(speaker_profiles->'count')")]
    )

def downgrade():
    """REQUIRED: Backward migration"""
    op.drop_index('idx_transcripts_speakers', 'transcripts')
    op.drop_column('transcripts', 'speaker_profiles')
```

## Model Registry Pattern

### CRITICAL: Prevent Import Conflicts
```python
# models/__init__.py - REQUIRED PATTERN
"""
Database Model Registry
CRITICAL: This prevents SQLAlchemy initialization errors
"""

from typing import Dict, Type
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

# Model registry
_model_registry: Dict[str, Type[Base]] = {}

def register_model(name: str, model: Type[Base]):
    """Register a model in the central registry"""
    if name in _model_registry:
        raise ValueError(f"Model {name} already registered")
    _model_registry[name] = model
    return model

def get_model(name: str) -> Type[Base]:
    """Get a model from the registry"""
    if name not in _model_registry:
        raise ValueError(f"Model {name} not found")
    return _model_registry[name]

# Import all models to register them
from .media_file import MediaFile
from .transcript import Transcript
from .processing_job import ProcessingJob

# Export for convenience
__all__ = ['Base', 'MediaFile', 'Transcript', 'ProcessingJob']
```

### Model Definition Rules
```python
# models/transcript.py
from sqlalchemy import Column, String, Float, ForeignKey, Index
from sqlalchemy.dialects.postgresql import UUID, JSONB
from . import Base, register_model

@register_model("Transcript")
class Transcript(Base):
    """REQUIRED: Use registry pattern"""
    __tablename__ = 'transcripts'
    
    # REQUIRED: UUID primary keys
    id = Column(UUID(as_uuid=True), primary_key=True)
    
    # REQUIRED: Foreign keys with CASCADE
    media_file_id = Column(
        UUID(as_uuid=True),
        ForeignKey('media_files.id', ondelete='CASCADE'),
        nullable=False
    )
    
    # REQUIRED: JSONB for flexible data
    raw_content = Column(JSONB, nullable=False)
    enhanced_content = Column(JSONB)
    
    # REQUIRED: Indexes for performance
    __table_args__ = (
        Index('idx_media_version', 'media_file_id', 'version'),
        Index('idx_created', 'created_at'),
    )
```

## Connection Management Rules

### Database Configuration
```python
# core/database.py
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.orm import sessionmaker
from contextlib import asynccontextmanager

# REQUIRED: Use async PostgreSQL
DATABASE_URL = "postgresql+asyncpg://user:pass@localhost/trax"

# REQUIRED: Connection pooling
engine = create_async_engine(
    DATABASE_URL,
    pool_size=20,
    max_overflow=10,
    pool_pre_ping=True,  # Check connections
    echo=False  # Set True for debugging
)

# REQUIRED: Async session factory
AsyncSessionLocal = sessionmaker(
    engine,
    class_=AsyncSession,
    expire_on_commit=False
)

@asynccontextmanager
async def get_session():
    """REQUIRED: Context manager for sessions"""
    async with AsyncSessionLocal() as session:
        try:
            yield session
            await session.commit()
        except Exception:
            await session.rollback()
            raise
        finally:
            await session.close()
```

## Query Optimization Rules

### Index Requirements
```sql
-- REQUIRED: Performance indexes
CREATE INDEX idx_transcripts_text_search 
ON transcripts USING gin(to_tsvector('english', raw_content->>'text'));

CREATE INDEX idx_media_hash ON media_files(file_hash);
CREATE INDEX idx_jobs_pending ON processing_jobs(status) 
WHERE status = 'pending';

-- JSONB indexes
CREATE INDEX idx_transcript_segments 
ON transcripts USING gin((raw_content->'segments'));

CREATE INDEX idx_confidence_scores
ON transcripts ((confidence_scores->>'average')::float);
```

### Query Patterns
```python
# REQUIRED: Efficient queries
async def get_transcript_with_media(transcript_id: UUID):
    """Use joined loading for relationships"""
    async with get_session() as session:
        result = await session.execute(
            select(Transcript)
            .options(selectinload(Transcript.media_file))
            .where(Transcript.id == transcript_id)
        )
        return result.scalar_one_or_none()

# REQUIRED: Bulk operations
async def bulk_insert_segments(segments: List[dict]):
    """Use bulk inserts for performance"""
    async with get_session() as session:
        await session.execute(
            insert(Segment),
            segments
        )
        await session.commit()
```

## Testing Rules

### Database Test Setup
```python
# tests/conftest.py
@pytest.fixture
async def test_db():
    """REQUIRED: Isolated test database"""
    # Create test database
    test_engine = create_async_engine(
        "postgresql+asyncpg://test:test@localhost/trax_test"
    )
    
    # Create all tables
    async with test_engine.begin() as conn:
        await conn.run_sync(Base.metadata.create_all)
    
    # Yield for tests
    yield test_engine
    
    # Clean up
    async with test_engine.begin() as conn:
        await conn.run_sync(Base.metadata.drop_all)
```

### Migration Testing
```python
# tests/test_migrations.py
def test_migration_up_down():
    """REQUIRED: Test all migrations"""
    # Apply all migrations
    alembic.command.upgrade(config, "head")
    
    # Get current revision
    current = get_current_revision()
    
    # Downgrade all
    alembic.command.downgrade(config, "base")
    
    # Upgrade again
    alembic.command.upgrade(config, current)
    
    # Verify schema intact
    assert verify_schema_complete()
```

## Monitoring Rules

### Required Metrics
```python
database_metrics = {
    # Performance
    "query_time_avg": avg_query_duration_ms,
    "slow_queries": queries_over_100ms,
    "connection_pool_size": active_connections,
    "connection_wait_time": pool_wait_ms,
    
    # Health
    "table_sizes": {table: size_mb},
    "index_usage": {index: hit_rate},
    "dead_tuples": vacuum_needed_count,
    "replication_lag": slave_delay_seconds,
    
    # Business
    "transcripts_per_day": daily_transcript_count,
    "storage_growth_rate": mb_per_day,
    "average_jsonb_size": avg_transcript_kb
}
```

### Maintenance Tasks
```sql
-- REQUIRED: Regular maintenance
-- Daily
VACUUM ANALYZE transcripts;
REINDEX INDEX idx_transcripts_text_search;

-- Weekly  
VACUUM FULL media_files;
ANALYZE;

-- Monthly
REINDEX DATABASE trax;
```

## Backup Rules

### Backup Strategy
```bash
# REQUIRED: Daily backups
pg_dump -d trax -f backup_$(date +%Y%m%d).sql

# REQUIRED: Test restore monthly
createdb trax_restore
psql trax_restore < backup_20240101.sql
# Verify data integrity
dropdb trax_restore

# REQUIRED: Keep backups
# - Daily for 7 days
# - Weekly for 4 weeks  
# - Monthly for 12 months
```

## Security Rules

### Access Control
```sql
-- REQUIRED: Separate users
CREATE USER trax_app WITH PASSWORD 'secure_password';
GRANT CONNECT ON DATABASE trax TO trax_app;
GRANT USAGE ON SCHEMA public TO trax_app;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES TO trax_app;

CREATE USER trax_readonly WITH PASSWORD 'readonly_password';
GRANT CONNECT ON DATABASE trax TO trax_readonly;
GRANT USAGE ON SCHEMA public TO trax_readonly;
GRANT SELECT ON ALL TABLES TO trax_readonly;
```

### SQL Injection Prevention
```python
# NEVER: String concatenation
BAD = f"SELECT * FROM transcripts WHERE id = '{user_input}'"

# ALWAYS: Parameterized queries
GOOD = select(Transcript).where(Transcript.id == user_input)
```

---

*These rules ensure reliable, performant database operations with PostgreSQL.*