demo
/
trax
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
trax/scratchpad.md

11 KiB
Raw Permalink Blame History

Trax Project Rewrite - Context Engineering Brief

Note: This is more like a prompt/context engineering document rather than a final specification. The content here provides the framework and requirements for the next developer to work from.

Executive Summary

Objective: Complete rewrite of ../app/trax with focus on deterministic, reliable AI coding agents and robust context management.

Key Problem: Previous project failed due to insufficient context engineering, unclear rules, and non-systematic development.

Solution: Implement systematic, permission-based development process with comprehensive reporting.

Core Concept: Trax (and the YouTube summarizer it evolved from) is a media content processing system that:

  • Starts with YouTube transcripts as the foundation
  • Runs content through various AI workflows
  • Produces summaries, glossaries, study guides, and other educational content
  • Uses AI agents to transform raw media into structured, educational outputs

1. Project Context & Requirements

1.1 Core Trax Project Goals

  • Package Manager: Migrate from pip to uv
  • Documentation: Consolidate CLAUDE.md and AGENTS.md (600 LOC limit)
  • Context Engineering: Establish robust AI agent context management for media processing workflows
  • Backend-First: Start with CLI transcription service, then Directus, then frontend
  • Database Reliability: Excellent testing for migrations
  • Media Processing: Build AI workflows for transforming YouTube transcripts into educational content

1.2 Development Philosophy

  • Context Engineering: Previous project failed due to insufficient context engineering
  • Rule Setting: Poor rule establishment led to project chaos
  • Systematic Approach: Need to shift from ad-hoc to systematic development
  • Sequential Development: Avoid simultaneous backend/frontend development
  • Modular Design: Ensure workflows and pipelines are modular

1.3 Documentation Constraints

  1. Strict LOC Limits: All documentation must remain under 600 lines of code
  2. Approval Process: Changes to CLAUDE.md or AGENTS.md require explicit approval
  3. Changelog Requirements: Comprehensive changelog for all modifications
  4. Update Protocol: Request approval before starting new work or after completing tasks

2. Comprehensive Report Methodology (General Context Engineering)

2.1 Interactive Report Process

Expectation: A comprehensive report on how to start over, including deep repo search and breaking up reports for manageable review.

2.2 Six-Checkpoint Process (Permission Required at Each Stage)

Phase 1: Current State Analysis

CHECKPOINT 1: Repository Inventory Report

  • Complete file structure analysis, codebase assessment, documentation review
  • Configuration system analysis, dependencies and technical debt
  • Media processing pipeline analysis, YouTube API integration assessment
  • REQUIRES APPROVAL before proceeding

CHECKPOINT 2: Historical Context Report

  • Analysis of built/discarded media processing features, development patterns
  • Failed approaches to content generation, lessons learned, success patterns to preserve
  • YouTube summarizer evolution analysis, educational content generation experiments
  • REQUIRES APPROVAL before proceeding

Phase 2: Strategic Planning

CHECKPOINT 3: Architecture Design Report

  • Modular backend architecture for media processing, database migration strategy
  • Testing framework design, context engineering system design for AI workflows
  • Content generation pipeline architecture, educational output formatting system
  • REQUIRES APPROVAL before proceeding

CHECKPOINT 4: Team Structure Report

  • Role definitions for media processing team, skill requirements, collaboration workflows
  • Communication protocols and decision-making distribution for content generation pipeline
  • Educational content specialist roles, AI workflow coordination protocols
  • REQUIRES APPROVAL before proceeding

Phase 3: Implementation Roadmap

CHECKPOINT 5: Technical Migration Report

  • uv package manager migration, documentation consolidation
  • Code quality standards, development environment setup for media processing
  • YouTube API integration migration, content generation workflow setup
  • REQUIRES APPROVAL before proceeding

CHECKPOINT 6: Product Vision Report

  • Feature prioritization matrix for educational content types, development phases and milestones
  • Success metrics for content generation quality, KPIs for user engagement with educational outputs
  • Risk mitigation strategies for AI content generation, media processing reliability
  • FINAL APPROVAL required before implementation

3. Trax-Specific Implementation Plan

3.1 Development Roadmap

Phase 1: CLI Transcription Service

  • Goal: Iterate back to CLI enhanced transcription service for YouTube content
  • Focus: Backend-first approach with modular workflows for transcript processing
  • Requirements: Robust testing, clean architecture, YouTube API integration

Phase 2: Directus Integration

  • Goal: Add connection to Directus CMS for content management
  • Focus: Database reliability and migration testing for media content storage
  • Requirements: Excellent migration testing suite, content metadata management

Phase 3: Frontend Development

  • Goal: Develop frontend interface for content viewing and management
  • Focus: Tailwind + Vanilla JS approach for educational content display
  • Requirements: Separate from backend development, responsive design for study materials

Phase 4: AI Content Generation

  • Goal: Add AI-powered content generation (summaries, glossaries, study guides)
  • Focus: Context engineering and AI agent integration for educational content creation
  • Requirements: Strong context management system, multiple AI workflow pipelines

3.2 Required Team Structure

  • Backend Python Developer (and separate researcher) - for transcript processing and AI workflows
  • Audio Engineer Specialist - for media processing and quality assurance
  • Tailwind + Vanilla JS Researcher (and separate frontend developer) - for educational content display
  • AI/Machine Learning Deep Researcher (and separate developer) - for content generation algorithms

3.3 Deliverables Required

  1. PRODUCT-VISION.md Report: Historical analysis of media processing features, lessons learned, clear product vision for educational content generation
  2. Team Structure Recommendation: Role definitions and collaboration protocols for media processing team
  3. Development Roadmap: Phased implementation plan with milestones for transcript-to-educational-content pipeline

4. Interactive Development Process

4.1 Permission-Based Workflow

CRITICAL: Each checkpoint requires explicit approval before proceeding. Ask clarifying questions and wait for confirmation.

4.2 Phase 1: Analysis & Discovery

CHECKPOINT 1: Repository Inventory

  • Task: Deep dive into current codebase and documentation, especially media processing components
  • Deliverable: Comprehensive technical analysis report including YouTube integration assessment
  • Questions to Ask:
    • What aspects of current media processing architecture should be preserved?
    • Which dependencies are critical for content generation vs. replaceable?
    • What technical debt in the transcript processing pipeline should be prioritized?

CHECKPOINT 2: Historical Context

  • Task: Research project evolution and lessons learned, especially YouTube summarizer development
  • Deliverable: Historical analysis and pattern recognition for media processing workflows
  • Questions to Ask:
    • Which failed approaches to content generation should be avoided?
    • What successful patterns in educational content creation should be replicated?
    • What media processing features are still desired but need better implementation?

4.3 Phase 2: Strategic Planning

CHECKPOINT 3: Architecture Design

  • Task: Design modular backend architecture for media processing and content generation
  • Deliverable: Technical architecture proposal for transcript-to-educational-content pipeline
  • Questions to Ask:
    • What level of modularity is desired for content generation workflows?
    • Which architectural patterns align with your vision for educational content processing?
    • What are the critical non-functional requirements for media processing reliability?

CHECKPOINT 4: Team Structure

  • Task: Define roles, responsibilities, and workflows for media processing team
  • Deliverable: Team structure and collaboration plan for content generation pipeline
  • Questions to Ask:
    • Are the proposed roles sufficient for your vision of educational content creation?
    • What collaboration patterns work best for media processing workflows?
    • How should decision-making be distributed across content generation pipeline?

4.4 Phase 3: Implementation Planning

CHECKPOINT 5: Technical Migration

  • Task: Plan technical implementation and migration for media processing system
  • Deliverable: Detailed implementation roadmap for transcript processing and content generation
  • Questions to Ask:
    • What migration approach minimizes risk for YouTube API integration?
    • Which technical decisions need your input for content generation workflows?
    • What rollback strategies should be planned for media processing pipeline?

CHECKPOINT 6: Product Vision

  • Task: Define product roadmap and success metrics for educational content generation
  • Deliverable: Comprehensive product vision document for media processing platform
  • Questions to Ask:
    • Does this vision align with your long-term goals for educational content creation?
    • Are the success metrics meaningful for content quality and user engagement?
    • What risks or concerns need additional attention for AI content generation reliability?

5. Quality Assurance & Success Criteria

5.1 Quality Assurance Requirements

  1. Documentation Standards: Ensure all docs meet 600 LOC limit
  2. Approval Workflows: Implement change management processes
  3. Testing Framework: Create comprehensive test suite
  4. Development Processes: Establish systematic workflows

5.2 Success Criteria

  • Clean Architecture: Maintainable, modular codebase with clear context for media processing
  • Reliable Agents: Robust AI agent system with strong context management for content generation
  • Systematic Development: Prevent future chaos through proper processes for educational content creation
  • Scalable Team: Clear documentation and structure for team growth in media processing
  • Comprehensive Reports: Detailed analysis and planning documents for content generation pipeline
  • Interactive Process: Maintain your control and input throughout development
  • Permission-Based Workflow: No major decisions made without your approval
  • Content Quality: High-quality educational outputs (summaries, glossaries, study guides)
  • Media Processing Reliability: Robust YouTube transcript processing and content transformation

5.3 Communication Protocol

  • Checkpoint Reviews: Each checkpoint requires your review and approval
  • Clarifying Questions: Developer must ask specific questions at each stage
  • Decision Points: All architectural and strategic decisions need your input
  • Progress Updates: Regular status updates between checkpoints
  • Risk Escalation: Immediate notification of any blockers or concerns