AllMail/walkthrough.md

Phase 1 Implementation Walkthrough

The foundational Phase 1 (Retrieval-Only Architecture) for your Unified Email Semantic Search system is now complete. The repository structure, Docker configurations, backend API, indexing daemon, and frontend have been successfully scaffolded in /home/adipu/AllMail.

Repository Structure Overview

graph TD
    A[AllMail/] --> B(db_data/)
    A --> C(Maildir/)
    A --> D(mail-sync/)
    A --> E(api/)
    A --> F(webmail/)
    A --> G(.gitea/)
    A --> H(docker-compose.yml)
    A --> I(.env.example)

    D --> D1[Dockerfile]
    D --> D2[entrypoint.sh]
    D --> D3[mbsyncrc.template]

    E --> E1[Dockerfile]
    E --> E2[main.py]
    E --> E3[indexer.py]
    E --> E4[database.py]
    E --> E5[run.sh]

    F --> F1[Dockerfile]
    F --> F2[nginx.conf]
    F --> F3[entrypoint.sh]
    F --> F4[spa/]

Key Components Implemented

1. Docker Infrastructure & Resource Management

• Created docker-compose.yml with memory limits applied specifically to respect the 16GB total memory overhead:
  • db is restricted to 1G (and Postgres tuned via command arguments shared_buffers=256MB).
  • api is restricted to 512M.
  • mail-sync is restricted to 256M.
  • webmail is restricted to 512M.
• Uses real filesystem mounts for persistent data: ./db_data for PostgreSQL and ./Maildir for the email files.

2. Database & API (api/)

• A Python 3.11 container running both a FastAPI server (main.py) and a background Watchdog daemon (indexer.py).
• Integrates pgvector and uses SQLAlchemy to manage vector embeddings.
• Automatically connects to OpenAI for text-embedding-3-small and indexes new files arriving in ./Maildir.

3. Synchronization (mail-sync/)

• A lightweight Alpine image utilizing isync (mbsync).
• Contains an entrypoint that initializes a 5-minute cron schedule.
• A mbsyncrc.template was created to serve as your configuration starting point.

4. Frontend & SnappyMail (webmail/)

• A custom php:8.2-fpm-alpine container running Nginx.
• Serves a React/Vite SPA on the root (/) for querying the Unified Search API.
• Proxies /mail/ to SnappyMail for standard webmail consumption.
• The Dockerfile handles building the React app automatically during docker compose build.

5. Deployment CI/CD (.gitea/)

• A GitHub-Actions-compatible workflow is present at .gitea/workflows/deploy.yml which triggers on pushes to main and runs the requested commands (docker compose up -d --build & docker image prune -f).

Next Steps for You

Important

Action Required: The system is ready to be started, but you need to configure your secrets and accounts before running docker compose up -d.

1. Configure Environment Variables: Copy .env.example to .env and fill in your passwords and the OpenAI API Key.

   cp /home/adipu/AllMail/.env.example /home/adipu/AllMail/.env
   # Edit .env
   

2. Configure Mail Accounts: Copy the mbsyncrc template to an actual config file and configure your IMAP servers.

   cp /home/adipu/AllMail/mail-sync/mbsyncrc.template /home/adipu/AllMail/mail-sync/mbsyncrc
   # Edit mbsyncrc
   

3. Deploy Phase 1: After configuring the above files, you can start the system locally to ensure it builds correctly:

   cd /home/adipu/AllMail
   docker compose up -d --build