RAG On-Demand: How to Query 300,000+ Emails Without Melting Your Server

RAG On-Demand: How to Query 300,000+ Emails Without Melting Your Server

We've all faced the nightmare of searching for a specific piece of information, a forgotten password, a system decision, or the details of a retired server hidden deep within a historical archive of over 300,000 emails (roughly 30 GB in Maildir format) accumulated over years of service.

Traditional clients like Outlook or Thunderbird crawl to a halt or take precious minutes to search. The modern, trendy solution would be to feed everything into a local LLM (Large Language Model) via RAG (Retrieval-Augmented Generation). However, chunking and embedding 300,000 emails takes days of CPU computation, requires heavy vector databases, and keeps your GPU at 100% load just to keep the index up to date.

We designed a hybrid, lightweight RAG On-Demand architecture that solves this problem brilliantly. By combining a lightning-fast local SQLite FTS5 index, incremental synchronization via Docker, and remote GPU-powered Ollama inference only when a query is made, we reduced the response time from over 80 seconds to a mere 11 seconds.

Here is how we built this system, and how you can do it too.

Overall System Architecture

Instead of using expensive, memory-resident vector databases (like ChromaDB or PGVector) that require calculating embeddings for 30 GB of text on every sync, we went for a hybrid approach based on structured metadata and Full-Text Search (FTS5):

flowchart TD
    subgraph LocalHost ["🖥️ Local Host (Laptop / Server)"]
        IMAP["IMAP Server\n(imap.example.com)"] -->|mbsync / ISYNC| Maildir["./maildir/\n(300.000+ .eml files)"]
        Maildir -->|Phase 1: Quick Index| SQLite["./data/emails.db\n(SQLite + FTS5 Index)"]
        
        UserQuery["👤 User: python3 ask_emails.py 'question'"] -->|1. FTS5 Search 0.01s| SQLite
        SQLite -->|2. Returns Top 20 Candidates| AskScript["ask_emails.py Engine"]
        Maildir -->|3. Read Full Text via IMAP UID| AskScript
    end

    subgraph RemoteServer ["⚡ Remote GPU Server (NVIDIA RTX 3060 12GB)"]
        AskScript -->|4. HTTP REST API (Payload Context 16k tokens)| Ollama["Ollama Server @ OLLAMA_URL"]
        Ollama -->|5. Model: ornith:9b| Inference["GPU Inference (10-15 sec)"]
        Inference -->|6. Stream Response| UserQuery
    end

Core Components

1. Incremental IMAP Sync (mail-sync)

To keep local emails updated without bottlenecking network bandwidth, we use a Docker container running isync (mbsync). Emails are saved to disk in Maildir format (a single .eml file for each email).

A key feature of Maildir is that files change names when their read/status flags change (e.g., from :2,RS to :2,S). Therefore, we cannot map emails in the database using the physical filename; instead, we map them using the IMAP UID, which remains constant and unique over time.

2. Fast Indexing Daemon (mail-analyzer)

A Python daemon runs in the background every 5 minutes, scanning for new files in the ./maildir/ directory. Instead of parsing the entire text (which would take too long), the daemon performs a lightning-fast extraction of the main metadata headers only:

• Sender (From)
• Recipient (To)
• Subject (Subject)
• Date (Date)
• IMAP UID (UID)

The parser processes 500 emails in just 0.5 seconds, writing the structured fields into a SQLite table enabled with FTS5 (Full-Text Search 5).

The Magic of the RAG Engine (ask_emails.py)

When a user asks a question like python3 ask_emails.py "What credentials did Matthew set up for the Antares server in 2025?", the script executes the following steps:

1. Stemming and Normalization

The query is normalized to support English and Italian grammar (e.g., incidente/incidenti are stemmed to the root incident). This ensures that the FTS5 search matches different inflections of the query terms.

2. User Identity Injection

If the user uses first-person pronouns in the question (e.g., "What do I need to do..." or "What did I send..."), the script reads the USER_NAME environment variable and automatically injects the user's real name/surname into the SQLite search query, filtering results to emails sent or received by that user.

3. Automatic Temporal Filtering

If the algorithm detects a year in the query (e.g., "in 2025"), it instantly appends a strict SQL clause: AND date_iso LIKE '2025%'. This prevents the LLM's context window from being flooded with irrelevant emails from other years.

4. Dynamic UID Resolution

Once FTS5 finds the 20 most relevant emails in SQLite, the script needs to read their full text to feed the LLM. Since the physical files on disk might have changed names due to IMAP flag updates, the script resolves paths dynamically using glob to locate the unique UID on the filesystem:

def resolve_maildir_path(uid, maildir_root):
    # Searches for the file containing "U=109655" (the UID) regardless of trailing flags
    pattern = os.path.join(maildir_root, "**", f"*U={uid}*")
    matches = glob.glob(pattern, recursive=True)
    return matches[0] if matches else None

5. HTML Parsing and Text Clean-up

Many messages (especially those from Outlook Exchange) are composed entirely of heavy, redundant HTML markup. The script strips all HTML tags and extracts only the plain text body, reducing the token payload sent to the GPU by up to 90%.

6. Large Context on a Dedicated GPU

The 20 most relevant, cleaned email texts are formatted as context and sent to a remote server equipped with a dedicated GPU (NVIDIA RTX 3060 12GB) running Ollama. Thanks to the generous VRAM, we can utilize a large 16,384-token context window with the ornith:9b model, streaming rich and detailed answers directly back to the user's terminal.

Performance Comparison

Conclusion

This approach demonstrates that you don't always need complex and resource-heavy vector databases to perform RAG on large datasets. Often, combining traditional, highly optimized system administration tools (like SQLite FTS5 and mbsync) with the flexibility of modern local LLMs via Ollama delivers superior results, saving computing resources while ensuring absolute privacy for your archives.

Discussion & Support

Discussion and support channel. To discuss configurations or other topics, join us on fluxer.dreadful.work.