AI Distiller MCP

AI Distiller (aid)

> Note: This is the very first version of this tool. We would be very grateful for any feedback in the form of a discussion or by creating an issue on GitHub. Thank you!

🚀 MCP Server Available: Install the Model Context Protocol server for AI Distiller from NPM: @janreges/ai-distiller-mcp - seamlessly integrate with Claude, Cursor, and other MCP-compatible AI tools!

Detection priority:

1. .aidrc file - Create this empty file to explicitly mark your project root
2. Language markers - go.mod, package.json, pyproject.toml, etc.
3. Version control - .git directory
4. Environment variable - AID_PROJECT_ROOT (fallback if no markers found)
5. Current directory - Final fallback with warning

# Mark a specific directory as project root (recommended)
touch /my/project/.aidrc

# Run from anywhere in your project - outputs always go to project root
cd deep/nested/directory
aid ../../../src  # Output: <project-root>/.aid/aid.src.txt

# Use environment variable as fallback (useful for CI/CD)
AID_PROJECT_ROOT=/build/workspace aid src/

🌍 Language Support

Currently supports 12 languages via tree-sitter:

• Full Support: Python, Go, JavaScript, PHP, Ruby
• Beta: TypeScript, Java, C#, Rust, Kotlin, Swift, C++
• Coming Soon: Zig, Scala, Clojure

Language-Specific Documentation:

• C++ - C++11/14/17/20 support with templates, namespaces, modern features
• C# - Complete C# 12 support with records, nullable reference types, pattern matching
• Go - Full Go support with interfaces, goroutines, generics (1.18+)
• Java - Java 8-21 support with records, sealed classes, pattern matching
• JavaScript - ES6+ support with classes, modules, async/await
• Kotlin - Kotlin 1.x support with coroutines, data classes, sealed classes
• PHP - PHP 7.4+ with PHP 8.x features (attributes, union types, enums)
• Python - Full Python 3.x support with type hints, async/await, decorators
• Ruby - Ruby 2.x/3.x support with blocks, modules, metaprogramming
• Rust - Rust 2018/2021 editions with traits, lifetimes, async
• Swift - Swift 5.x support with protocols, extensions, property wrappers
• TypeScript - TypeScript 4.x/5.x with generics, decorators, type system

🎯 How It Works

1. Scans your codebase recursively for supported file types (10+ languages)
2. Parses each file using language-specific tree-sitter parsers (all bundled, no dependencies)
3. Extracts only what you need: public APIs, type signatures, class hierarchies
4. Outputs in your preferred format: compact text, markdown, or structured JSON

All tree-sitter grammars are compiled into the aid binary - zero external dependencies!

🚀 Transform Massive Codebases Into AI-Friendly Context

> The Problem: Modern codebases contain thousands of files with millions of lines. But for AI to understand your code architecture, suggest improvements, or help with development, it doesn't need to see every implementation detail - it needs the structure and public interfaces.

> The Solution: AI Distiller extracts only what matters - public APIs, types, and signatures - reducing codebase size by 90-98% while preserving all essential information for AI comprehension.

🎯 Why This Matters for AI-Assisted Development

Large codebases are overwhelming for AI models. A typical web framework like Django has ~10 million tokens of source code. Even with Claude's 200K context window, you'd need to split it into 50+ chunks, losing coherence and relationships between components.

But here's the good news: Most real-world projects that teams have invested hundreds to thousands of hours developing are much smaller. Thanks to AI Distiller, the vast majority of typical business applications, SaaS products, and internal tools can fit entirely within AI context windows, enabling unprecedented AI assistance quality.

⚠️ The Hidden Problem with AI Coding Tools

Most AI agents and IDEs are "context misers" - they try to save tokens at the expense of actual codebase knowledge. They rely on:

• 🔍 Grep/search to find relevant code snippets
• 📄 Limited context showing only 10-50 lines around matches
• 🎲 Guessing interfaces based on partial information

This is why AI-generated code often fails on first attempts - the AI is literally guessing method signatures, parameter types, and return values because it can't see the full picture.

AI Distiller changes the game by giving AI complete knowledge of:

• ✅ Exact interfaces of all classes, methods, and functions
• ✅ All parameter types and their expected values
• ✅ Return types and data structures
• ✅ Full inheritance hierarchies and relationships

Instead of playing "code roulette", AI can now write correct code from the start.

Result: Django's 10M tokens compress to just 256K tokens - suddenly the entire framework fits in a single AI conversation, leading to:

• 🎯 More accurate suggestions - AI sees all available APIs at once
• 🚀 Less hallucination - No more inventing methods that don't exist
• 💡 Better architecture advice - AI understands the full system design
• ⚡ Faster development - Especially for "vibe coding" with AI assistance
• 💰 40x cost reduction - Pay for 256K tokens instead of 10M on API calls

🔧 Flexible for Different Use Cases

# Process entire codebase (default: public APIs only)
aid ./my-project

# Process specific directory or module
aid ./my-project/src/auth
aid ./my-project/src/api

# Process a directory
aid ./my-project/core/

# Process individual file
aid src/main.py

# Include protected/private for deeper analysis
aid ./my-project --private=1 --protected=1

# Include implementations for small projects
aid ./my-small-lib --implementation=1

# Everything for complete understanding
aid ./micro-service --private=1 --protected=1 --implementation=1

Granular Control: Process your entire codebase, specific modules, directories, or even individual files. Perfect for focusing AI on the exact context it needs - whether that's understanding the whole system architecture or diving deep into a specific authentication module.

📈 Full benchmark details | 🧪 Reproduce these results

🚀 Quick Start

One-Line Installation

macOS / Linux / WSL:

# Install to ~/.aid/bin (recommended, no sudo required)
curl -sSL https://raw.githubusercontent.com/janreges/ai-distiller/main/install.sh | bash

# Install to /usr/local/bin (requires sudo)
curl -sSL https://raw.githubusercontent.com/janreges/ai-distiller/main/install.sh | bash -s -- --sudo

Windows PowerShell:

iwr https://raw.githubusercontent.com/janreges/ai-distiller/main/install.ps1 -useb | iex

The installer will:

• Detect your OS and architecture automatically
• Download the appropriate pre-built binary
• Install to ~/.aid/bin/aid by default (no sudo required)
• Or to /usr/local/bin/aid with --sudo flag
• Guide you through PATH configuration if needed

Basic Usage

# Basic usage
aid .                                   # Current directory, output is saved to file in ./aid
aid . --stdout                          # Current directory, output is printed to STDOUT
aid src/                                # Specific directory
aid main.py                             # Specific file

📖 Example Output

class Car:
    """A car with basic attributes and methods."""
    
    def __init__(self, make: str, model: str):
        self.make = make
        self.model = model
        self._mileage = 0  # Private
    
    def drive(self, distance: int) -> None:
        """Drive the car."""
        if distance > 0:
            self._mileage += distance

<file path="car.py">
class Car:
    +def __init__(self, make: str, model: str)
    +def drive(self, distance: int) -&gt; None
</file>

export interface User {
  id: number;
  name: string;
  email?: string;
}

export class UserService {
  private cache = new Map<number, user="">();
  
  async getUser(id: number): Promise<user null="" |=""> {
    return this.cache.get(id) || null;
  }
}

<file path="api.ts">
export interface User {
  id: number;
  name: string;
  email?: string;
}

export class UserService {
  +async getUser(id: number): Promise<user null="" |="">
}
</user></file>

📖 Guides & Examples

Deep Code Analysis Prompt Generation

AI Distiller generates sophisticated analysis prompts that AI assistants can execute for comprehensive codebase understanding:

aid internal \
   --private=1 --protected=1 --implementation=1 \
   --ai-action=flow-for-deep-file-to-file-analysis

✅ AI Analysis Task List generated successfully!
📋 Task List: .aid/ANALYSIS-TASK-LIST.internal.2025-06-20.md
📊 Summary File: .aid/ANALYSIS-SUMMARY.internal.2025-06-20.md
📁 Analysis Reports Directory: .aid/analysis.internal/2025-06-20
🤖 Ready for AI-driven analysis workflow!
📂 Files to analyze: 158

💡 If you are an AI agent, please read the Task List above and carefully follow all instructions to systematically analyze each file.

What AI Distiller generates:

• 📋 Task list prompt - A structured checklist for AI to follow (.aid/ANALYSIS-TASK-LIST.PROJECT.DATE.md)
• 🎯 Analysis instructions - Detailed prompts guiding AI through security, performance, and quality checks
• 📊 Code structure - Distilled code included in the prompt files for AI to analyze
• 📁 Directory structure - Pre-created folders where AI agents can save their analysis results

How to use the generated prompts:

1. For AI agents: Direct the agent to read the generated task list file and follow instructions
2. For web AI tools: Copy the content of generated files and paste into Gemini (best for large codebases due to 1M context)
3. For small codebases: Use --stdout to get prompt directly without saving to file

Note: The analysis dimensions (Security, Performance, Maintainability, Readability) are part of the prompts that guide the AI - AI Distiller itself doesn't perform any analysis.

🤖 Use with Claude Code/Desktop (MCP)

AI Distiller now integrates seamlessly with Claude Code/Desktop through the Model Context Protocol (MCP), enabling AI agents to analyze and understand codebases directly within conversations.

# One-line installation
claude mcp add aid -- npx -y @janreges/ai-distiller-mcp

📦 NPM Package: @janreges/ai-distiller-mcp - Full documentation and examples available

Available MCP Tools

🔍 Code Structure Tools:

• distill_file - Extract structure from a single file
• distill_directory - Extract structure from entire directories
• list_files - Browse directories with file statistics
• get_capabilities - Get info about AI Distiller capabilities

🎯 Specialized AI Analysis Tools:

• aid_hunt_bugs - Generate bug-hunting prompts with distilled code
• aid_suggest_refactoring - Create refactoring analysis prompts
• aid_generate_diagram - Produce diagram generation prompts (Mermaid)
• aid_analyze_security - Generate security audit prompts (OWASP Top 10)
• aid_generate_docs - Create documentation generation prompts
• aid_deep_file_analysis - Systematic file-by-file analysis workflow
• aid_multi_file_docs - Multi-file documentation workflow
• aid_complex_analysis - Enterprise-grade analysis prompts
• aid_performance_analysis - Performance optimization prompts
• aid_best_practices - Code quality and best practices prompts

🔧 Core Analysis Engine:

• aid_analyze - Direct access to all AI actions for custom workflows

Important: AI Distiller generates analysis prompts with distilled code - it does NOT perform the actual analysis! The output is a specialized prompt + distilled code that AI agents (like Claude) then execute. For large codebases, you can copy the output to tools like Gemini 2.0 with 1M context window.

Smart Context Management: AI agents can analyze your entire project for understanding the big picture, then zoom into specific modules (auth, API, database) for detailed work. No more overwhelming AI with irrelevant code!

📖 Complete CLI Reference

Command Synopsis

aid <path> [OPTIONS]

Core Arguments and Options

🎯 Primary Arguments

📁 Output Options

🤖 AI Actions

👁️ Visibility Filtering

📝 Content Filtering

🎛️ Alternative Filtering Syntax

📂 File Selection

🔧 Processing Options

📍 Path Control

⚡ Performance Options

📊 Summary Output Options

📜 Git Mode Options (when path is .git)

🐛 Diagnostic Options

AI Actions Detailed

AI actions generate pre-configured prompts combined with distilled code that AI agents can then execute for specific analysis tasks:

Summary Types

Exit Codes

Examples

# Basic usage - distill with default settings (public APIs only)
aid ./src

# Include all visibility levels and implementation
aid ./src --private=1 --protected=1 --internal=1 --implementation=1

# Generate security analysis prompt (AI agent will execute the analysis)
aid --ai-action prompt-for-security-analysis ./api --private=1

# Process only Python and Go files, exclude tests
aid --include "*.py,*.go" --exclude "*test*,*spec*" ./

# Git history analysis with AI insights
aid .git --with-analysis-prompt --git-limit=500

# Raw text processing for documentation
aid ./docs --raw

# Force single-threaded processing for debugging (-v, -vv, -vvv)
aid ./complex-code -w 1 -vv

# Custom output with absolute paths
aid ./lib --output=/tmp/analysis.txt --file-path-type=absolute

# CI/CD integration with clean output
aid ./internal --summary-type=ci-friendly --no-emoji

⚠️ Limitations

• Syntax Errors: Files with syntax errors may be skipped or partially processed
• Dynamic Features: Runtime-determined types/interfaces in dynamic languages are not resolved
• Macro Expansion: Complex macros (Rust, C++) show pre-expansion source
• Generated Code: Consider using .aidignore to skip generated files

🔒 Security Considerations

⚠️ Important: AI Distiller extracts code structure which may include:

• Function and variable names that could reveal business logic (e.g., processPayment, calculateTaxEvasion)
• API endpoints and internal routes (e.g., /api/v1/internal/user-data)
• Type information and data structures
• Comments and docstrings (unless stripped)
• File paths revealing project structure or codenames

Recommendations:

1. Always review output before sending to external services
2. Use --comments=0 to remove potentially sensitive documentation
3. Consider running a secrets scanner on your codebase first
4. For maximum security, run AI Distiller in an isolated environment
5. Future: We're exploring an --obfuscate flag to anonymize sensitive identifiers

🛠️ Advanced Usage

⚡ Parallel Processing

AI Distiller now supports parallel processing for significantly faster analysis of large codebases:

# Use default parallel processing (80% of CPU cores)
aid ./src

# Force serial processing (original behavior)
aid ./src -w 1

# Use specific number of workers
aid ./src -w 16

# Check performance with verbose output
aid ./src -v  # Shows: "Using 25 parallel workers (32 CPU cores available)"

Performance Benefits:

• React packages: 3.5s → 0.5s (7x faster)
• Large codebases: Near-linear speedup with CPU cores
• Maintains identical output order as serial processing

Processing from stdin

AI Distiller can process code directly from stdin, perfect for:

• Quick code snippet analysis
• Pipeline integration
• Testing without creating files
• Dynamic code generation workflows

# Auto-detect language from stdin
echo 'class User { getName() { return this.name; } }' | aid --format text

# Explicit language specification
cat mycode.php | aid --lang php --private=0 --protected=0

# Use "-" to explicitly read from stdin
aid - --lang python < snippet.py

# Pipeline example: extract structure from generated code
generate-code.sh | aid --lang typescript --format json

Language Detection: When using stdin without --lang, AI Distiller automatically detects the language based on syntax patterns. Supported languages for auto-detection: python, typescript, javascript, go, ruby, swift, rust, java, c#, kotlin, c++, php.

Integration with AI Tools

# Create a context file for Claude or GPT
aid ./src --format text --implementation=0 > context.txt

# Generate a codebase summary for RAG systems
aid . --format json | jq -r '.files[].symbols[].name' > symbols.txt

# Extract API surface for documentation
aid ./api --comments=0 --implementation=0 --format md > api-ref.md

🚫 Ignoring Files with .aidignore

AI Distiller respects .aidignore files for excluding files and directories from processing. The syntax is similar to .gitignore.

Important: What AI Distiller Processes

AI Distiller only processes source code files with these extensions:

• Python: .py, .pyw, .pyi
• JavaScript: .js, .mjs, .cjs, .jsx
• TypeScript: .ts, .tsx, .d.ts
• Go: .go
• Rust: .rs
• Ruby: .rb, .rake, .gemspec
• Java: .java
• C#: .cs
• Kotlin: .kt, .kts
• C++: .cpp, .cc, .cxx, .c++, .h, .hpp, .hh, .hxx, .h++
• PHP: .php, .phtml, .php3, .php4, .php5, .php7, .phps, .inc
• Swift: .swift

Note: Files like .log, .txt, .md, images, PDFs, and other non-source files are automatically ignored by AI Distiller, so you don't need to add them to .aidignore.

Default Ignored Directories

AI Distiller automatically ignores these common dependency and build directories:

• node_modules/ - npm packages
• vendor/ - Go and PHP dependencies
• target/ - Rust build output
• build/, dist/ - Common build directories
• __pycache__/, .pytest_cache/, venv/, .venv/, env/, .env/ - Python
• .gradle/, gradle/ - Java/Kotlin
• Pods/ - Swift/iOS dependencies
• .bundle/ - Ruby bundler
• bin/, obj/ - Compiled binaries
• .vs/, .idea/, .vscode/ - IDE directories
• coverage/, .nyc_output/ - Test coverage
• bower_components/ - Legacy JavaScript
• .terraform/ - Terraform
• .git/, .svn/, .hg/ - Version control

You can override these defaults using ! patterns in .aidignore (see Advanced Usage below).

Basic Syntax

Create a .aidignore file in your project root or any subdirectory:

# Comments start with hash
*.test.js          # Ignore test files
*.spec.ts          # Ignore spec files
temp/              # Ignore temp directory
build/             # Ignore build directory
/secrets.py        # Ignore secrets.py only in root
node_modules/      # Ignore node_modules everywhere
**/*.bak           # Ignore .bak files in any directory
src/test_*         # Ignore test_* files in src/
!important.test.js # Don't ignore important.test.js (negation)

How It Works

• .aidignore files work recursively - place them in any directory
• Patterns are relative to the directory containing the .aidignore file
• Use / prefix for patterns relative to the .aidignore location
• Use ** for recursive matching
• Directory patterns should end with /
• Use ! prefix to negate a pattern (re-include previously ignored files)

Examples

# .aidignore in project root
node_modules/       # Excludes all node_modules directories
*.test.js          # Excludes all test files
*.spec.ts          # Excludes all spec files
dist/              # Excludes dist directory
.env.py            # Excludes environment config files
vendor/            # Excludes vendor directory

# More specific patterns
src/**/test_*.py   # Test files in src subdirectories
!src/test_utils.py # But include this specific test file
/config/*.local.py # Local config files in root config dir
**/*_generated.go  # Generated Go files anywhere

Advanced Usage: Including Normally Ignored Content

Include Default-Ignored Directories

Use ! patterns to include directories that are ignored by default:

# Include vendor directory for analysis
!vendor/

# Include specific node_modules package
!node_modules/my-local-package/

# Include Python virtual environment
!venv/

Include Non-Source Files

You can also include files that AI Distiller normally doesn't process:

# Include all markdown files
!*.md
!**/*.md

# Include configuration files
!*.yaml
!*.json
!.env

# Include specific documentation
!docs/**/*.txt
!README.md
!CHANGELOG.md

When you include non-source files with ! patterns, AI Distiller will include their raw content in the output.

Nested .aidignore Files

You can place .aidignore files in subdirectories for more specific control:

# project/.aidignore
*.test.py
!vendor/            # Include vendor in this project

# project/src/.aidignore
test_*.go
*.mock.ts
!test_helpers.ts   # Exception: include test_helpers.ts

🎯 Git History Analysis Mode

AI Distiller includes a special mode for analyzing git repositories. When you pass a .git directory, it switches to git log mode:

# View formatted git history
aid .git

# Limit to recent commits (default is 200)
aid .git --git-limit=500

# Include AI analysis prompt for comprehensive insights
aid .git --git-limit=1000 --with-analysis-prompt

The --with-analysis-prompt flag adds a sophisticated prompt combined with git history that AI agents can use to generate:

• Contributor statistics with expertise areas and collaboration patterns
• Timeline analysis with development phases and activity visualization
• Functional categorization of commits (features, fixes, refactoring)
• Codebase evolution insights including technology shifts
• Actionable recommendations based on discovered patterns

The output file contains both the analysis prompt and formatted git history, ready for AI agents to process. Perfect for understanding project history, identifying knowledge silos, or generating impressive development reports.

❓ FAQ

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Development Setup

# Clone and setup
git clone https://github.com/janreges/ai-distiller
cd ai-distiller
make dev-init    # Initialize development environment

# Run tests
make test         # Unit tests
make test-integration  # Integration tests

# Build binary
make build        # Build for current platform

Building Release Binaries

AI Distiller requires CGO for full language support via tree-sitter parsers. To build release binaries for all supported platforms:

Prerequisites

Ubuntu/Debian:

# Install cross-compilation toolchains
sudo apt-get update
sudo apt-get install -y gcc-aarch64-linux-gnu gcc-mingw-w64-x86-64

# For macOS cross-compilation, you need osxcross:
# 1. Clone osxcross: git clone https://github.com/tpoechtrager/osxcross tools/osxcross
# 2. Obtain macOS SDK (see https://github.com/tpoechtrager/osxcross#packaging-the-sdk)
# 3. Place SDK in tools/osxcross/tarballs/
# 4. Build osxcross: cd tools/osxcross && ./build.sh

Build All Platforms

# Build release archives for all platforms
./scripts/build-releases.sh

# This creates:
# - aid-linux-amd64.tar.gz    (Linux 64-bit)
# - aid-linux-arm64.tar.gz    (Linux ARM64)
# - aid-darwin-amd64.tar.gz   (macOS Intel)
# - aid-darwin-arm64.tar.gz   (macOS Apple Silicon)
# - aid-windows-amd64.zip     (Windows 64-bit)

The script automatically detects available toolchains and builds for all possible platforms. Each archive contains the aid binary (or aid.exe for Windows) with full language support.

Note: Without proper toolchains, only the native platform will be built.

📄 License

MIT License - see LICENSE for details.

🙏 Acknowledgments

• Built on tree-sitter for accurate parsing
• Inspired by the need for better AI-code interaction
• Created with ❤️ by Claude Code & Ján Regeš from SiteOne (Czech Republic).