🧬 Multi-Tier Memory & Context Scheduling

Overview

Implement hierarchical memory system with intelligent context selection to overcome LLM context limits and enable persistent learning across sessions.

Problem

Current context is limited to single conversation. No persistent memory of project patterns, user preferences, or lessons learned. Context selection is naive and wastes tokens on irrelevant information.

Solution

Build multi-tier memory system with smart context scheduling:

Memory Tiers

• Session Memory: Ephemeral, conversation-scoped
• Project Memory: Durable, repo-specific patterns and lessons
• Global Memory: Cross-project skills, preferences, patterns
• Team Memory: Shared knowledge within organizations

Memory Store Trait

#[async_trait]
trait MemoryStore: Send + Sync {
    async fn put(&self, item: MemoryItem) -> Result<MemoryId>;
    async fn search(&self, query: &str, k: usize) -> Result<Vec<MemoryItem>>;
    async fn summarize(&self, scope: MemoryScope) -> Result<String>;
    async fn evict(&self, policy: EvictionPolicy) -> Result<usize>;
}

Context Scheduling

• Relevance Scoring: Rank context by task relevance
• Token Budget Management: Optimal allocation across context types
• AST-Aware Compression: Preserve semantics while reducing tokens
• Citation Tracking: Link generated content to source context

Memory Item Types

• Code Patterns: Common implementations and gotchas
• Build/Test Configurations: Project-specific commands and flags
• Bug Reports: Historical issues and solutions
• User Preferences: Coding style, tool preferences
• Lessons Learned: What worked/failed in similar contexts

Privacy & Security

• Local-only mode: No external memory storage
• Tenant isolation: Strict namespace separation
• Encryption at rest: Sensitive data protection
• Retention policies: Automatic cleanup and archival

Implementation Plan

1. Design MemoryItem schema and storage backend
2. Build ContextScheduler with relevance scoring
3. Add memory persistence with encryption
4. Implement eviction and cleanup policies
5. Add privacy controls and tenant isolation
6. Build memory query and retrieval system

Context Scheduling Algorithm

1. Level 1: Current task, plan, specification (always included)
2. Level 2: Impacted files' AST chunks, ranked by relevance
3. Level 3: Related tests, historical patterns, common pitfalls
4. Level 4: Broader project context, documentation, conventions

Success Criteria

• Context relevance improves 40%+ vs naive selection
• Memory queries return results in <50ms
• 90%+ reduction in repeated questions about project setup
• Zero memory leaks between tenants/projects

Timeline

6-8 weeks for core memory system, 10-12 weeks with full scheduling