RAG Agent is a local-first MVP for a headless document agent that ingests files from a mounted directory, stores normalized chunks in PostgreSQL with pgvector, and exposes FastAPI endpoints plus a small web console for cited question answering.
- Local filesystem ingestion from a mounted document directory, including Nextcloud-backed folders
- Text extraction for
PDF,DOCX,TXT, andMarkdown - PostgreSQL full-text search plus semantic search with
pgvector - Agent-selected keyword or semantic retrieval with cited answers through OpenAI or a local
llama.cppOpenAI-compatible endpoint - Separate configuration for chat and embedding providers, including configurable embedding prefixes
- OCR-aware parsing for image receipts and scanned PDFs, with configurable thresholds
- Optional Docling-backed normalization phase that writes Markdown plus JSON metadata sidecars before database indexing
- Long-running normalization watcher for incremental raw-document changes
- Long-running embed watcher for incremental normalized Markdown indexing
POST /reindex,POST /agent/query, and debug retrieval endpoints
- Advanced schedule or receipt metadata extraction
- Reranking and date-aware retrieval
- Copy
.env.exampleto.envand fill inOPENAI_API_KEYwhen using OpenAI. After signing in, choose chat and embedding providers, model names, and endpoints in Settings → Models. Those values are stored in the runtime settings file and apply to new requests. UseEMBEDDING_QUERY_PREFIXandEMBEDDING_DOCUMENT_PREFIXto match your embedding model's expected prompt format. The container image prefetches the embedding tokenizer at build time and defaultsEMBEDDING_TOKENIZER_LOCAL_FILES_ONLY=trueso runtime chunking does not depend on network access. UseRAG_ENABLED_SUFFIXESto control which raw file types the normalizer watches. Docling handles OCR for image files and scanned PDFs in the normalizer image. Copyconfig/runtime-settings.example.jsontoconfig/runtime-settings.jsonto customize non-secret API, embedding, and normalization settings. Docker mounts this file's directory at/config, so it persists across container recreation. - Put source files under
./data/nextcloudor mount your Nextcloud storage there. - Start the stack:
docker compose up --build- Reindex documents:
curl -X POST http://localhost:8000/reindexOr run the reconciliation worker directly:
python -m app.embed.workerYou can also run the first normalization phase directly. It writes Markdown artifacts under NORMALIZED_OUTPUT_DIR.
By default NORMALIZATION_BACKEND=docling. Structured documents such as PDFs, DOCX files, and images are normalized through Docling; .txt and .md files use a lightweight passthrough.
python -m app.normalize.workerThe normalizer image runs as a long-lived watcher by default. For local development, you can run the watcher directly:
python -m app.normalize.mainThe embed image runs as a long-lived watcher by default. It watches normalized Markdown and metadata artifacts, then indexes only the changed normalized documents:
python -m app.embed.mainFor a one-shot manual reindex, run:
python -m app.embed.worker- Ask a question through the agent. It iteratively chooses read-only tools, inspects each result, and decides whether to investigate further before answering:
curl -X POST http://localhost:8000/agent/query \
-H "Content-Type: application/json" \
-d '{"question":"What is my latest glasses prescription?"}'The agent can search indexed passages and document metadata, grep normalized Markdown, and read bounded line ranges from
candidate documents. Each request is limited by RAG_AGENT_MAX_STEPS, which defaults to six tool calls. The web console presents replies as a conversation,
keeps recent history in browser local storage, and sends recent user/assistant turns with follow-up requests. Thinking,
tool calls, tool results, and controller decisions are shown as a folded chronological trace; saved reasoning is not sent
back to the model as conversation context. See the agent runtime walkthrough for the controller,
protocol, and tool boundaries.
Web routes:
/is the agent-only conversation./debugcontains retrieval modes, raw retrieval debugging, pipeline status, and agent inspection./settingsmanages non-secret chat and retrieval settings for this installation./docscontains the generated FastAPI API documentation.
app/agent/: agent runtime, prompt and memory state, retrieval tools, and its FastAPI/web interfaceapp/normalize/: Phase 1 raw document normalization to Markdown and JSON metadata, using Docling by defaultapp/embed/: normalized Markdown watching/scanning, mismatch reconciliation, chunking, embedding, and DB upsertsapp/core/: shared settings, DB, parser, tokenizer/chunking, and model client helpersapp/agent/config.py,app/embed/config.py,app/normalize/config.py: runtime-specific settings grouped with the code that uses themsql/init.sql: schema and indexesDockerfile: multi-target runtime image forapi,embed, andnormalize
The Dockerfile has three role-specific targets so the API does not need to carry the heavy Docling/OCR runtime:
api: FastAPI query/debug service, retrieval, database access, and LLM clientsembed: normalized Markdown indexing daemon, chunking, embedding, and database upsertsnormalize: Docling/OCR raw-document normalization watcher, plus the batch worker for manual backfills
Build them locally with:
export GHCR_OWNER=<owner>
make docker-build-api
make docker-build-embed
make docker-build-normalizeRelease builds publish separate GHCR packages:
ghcr.io/<owner>/rag-agent-api
ghcr.io/<owner>/rag-agent-embed
ghcr.io/<owner>/rag-agent-normalize
RAG_ENABLED_SUFFIXES is the normalizer allowlist. It is intersected with the formats the code knows how to normalize today: .pdf, .docx, .txt, .md, .jpg, .jpeg, and .png.