Enterprise Agent Platform
Hybrid all the way down. A five-engine parser chain that falls back until a document actually reads, dense and BM25 retrieval fused inside the vector database, and a cross-encoder reranker that decides what survives into context. On top of it, any team assembles their own tool-using agent — instructions, model, tools, knowledge, guardrails — and the tenant ACL is enforced inside the search itself, not filtered afterwards.
- Role
- AI Engineer — retrieval & agents
- Domain
- Enterprise knowledge assistants
- Sources
- Drive · Gmail · Slack · uploads · S3
- Retrieval
- Hybrid parse → hybrid search → rerank
- Status
- Live in production
Outcomes
What was
broken
The problem as it actually presented itself, and the constraints that shaped every decision after it.
Enterprise knowledge does not live in one place. It is scattered across Drive, Gmail, Slack, ad-hoc uploads, and a decade of PDFs, HWP files, and spreadsheets. A single shared chatbot over a single index solves none of it: every org needs its own agent behaviour, its own document set, and tenant isolation you can actually defend.
Naive RAG failed on this corpus in three specific ways. Dense retrieval missed exact identifiers — part numbers, document codes, Korean proper nouns — which are precisely what people search by. Scanned PDFs returned OCR noise that looked like text and poisoned the index. And a chunk of spreadsheet cells means nothing once it has been separated from its header row.
It was also expensive. Answers were accurate but cost ~21k tokens per turn against ~51k characters of assembled context, most of which the model never used. Cutting that without losing answer quality was the constraint that shaped the whole retrieval layer.
What it
runs on
Models, retrieval, serving, and the operational layer that keeps it honest in production.
- Docling
- Upstage
- DeepSeek-OCR / Qwen-VL
- PDF repair + content router
- Milvus 2.6 (dense + sparse)
- Server-side BM25
- WeightedRanker fusion
- ko-dic analyzer
- Cross-encoder
- Hosted reranker API
- Lexical + filename boosts
- Per-doc dedup
- LangGraph StateGraph
- ReAct tool loop
- Postgres checkpointer
- LiteLLM proxy
- LangSmith
- Prometheus
- Sentry
- E2B sandbox
- Redis
How it
works
The pipeline end to end — each stage, and why it earns its place in the latency budget.
- 01
Hybrid parsing — five engines, one fallback chain
A dispatcher picks a primary parser per document and walks a fallback chain until one succeeds: Docling for structured layout, a hosted document-AI parser for hard PDFs, two vision-language models for scanned and image-heavy pages, and a lightweight text path underneath. PDFs get a health check and an automatic repair pass before anyone tries to read them, a content router decides text-route vs. vision-route per file, and an OCR noise filter drops the garbage that survives extraction. No single parser wins on every document — the chain is the product.
- 02
Structure-aware chunking
Two-stage splitting: a Markdown header split preserves document structure, small fragments merge up to a target size, then a recursive or token splitter enforces the ceiling. Identifier prefixes keep part numbers and document codes attached to their chunk, and every spreadsheet chunk carries its sheet name and header row as metadata — because a block of cells separated from its header is noise with good embeddings.
- 03
Hybrid retrieval inside the vector DB
Every chunk carries a dense vector and a BM25-generated sparse vector in the same Milvus collection, searched in parallel and fused by a weighted ranker that keeps scores in [0,1] so one similarity threshold still means something. A Korean morphological analyser tokenises the CJK-heavy corpus. Critically, the tenant and ACL filter is applied to both arms — fusion can never rank a candidate the requester is not allowed to read.
- 04
Reranking the fused candidates
The fused set goes through a cross-encoder that scores query and candidate together, with a hosted reranker API as an alternative backend and cosine similarity as the floor if neither is configured. Around it sit the cheap signals that a bi-encoder cannot see: exact lexical match boosts, filename-match boosts for 'the Q3 deck' style queries, a recency boost, and per-document deduplication that keeps the top three chunks per source instead of letting one verbose document flood the context.
- 05
Adaptive retrieval by query type
Queries are classified into seven types — simple lookup, detailed question, calculation, comparison, multi-part, procedural, chit-chat — and each gets its own k, chunks-per-document, similarity threshold, and multi-hop budget. A lookup pulls ten tight chunks; a detailed question fans out and runs a second hop. Chit-chat skips retrieval entirely and never touches the index.
- 06
Managed agent runtime
Each configured agent compiles to a LangGraph ReAct loop with a Postgres checkpointer. Tools resolve per agent from a registry. The system prompt, learned user facts, canvas state, and pre-fetched knowledge are assembled as an ephemeral preamble that is rebuilt every request and never written to the checkpoint, so reloading a conversation cannot duplicate context.
- 07
Per-agent memory
After each stream, a lightweight model extracts atomic facts about the user, fire-and-forget and debounced. Facts are filtered by importance, deduplicated against existing ones at cosine 0.92, and capped per agent-user pair. On the next request they are retrieved by a blend of similarity, importance, and recency — so a support agent and a code-review agent learn different things about the same person.
Decisions
and their cost
Every choice below bought something and gave something up. The second half is the part worth reading.
BM25 in the database, not in the request
The previous 'hybrid' arm built an in-memory BM25 index over only the ~60 candidates dense search had already returned. It could reorder them, but it could never recall a document dense search missed — and its IDF statistics came from 60 texts instead of the whole collection. Moving BM25 into Milvus as a server-side function turned it into a real recall arm. The cost: a sparse field cannot be retrofitted onto an existing collection, so it needed a migration that copies rows with their existing embeddings — zero re-embedding spend, but a genuine cutover with a rollback flag.
A parser chain, not a parser
Every parser has a document class it is bad at. Layout-aware parsers mangle scanned pages; vision models are slow and expensive on clean text; the cheap text path silently returns whitespace for an image-only PDF. Chaining them with a health check, a content router, and an explicit fallback order costs more moving parts and more failure paths to reason about — and it is the difference between 'we support PDF' and documents that actually make it into the index.
Rerank in the hot path, and pay for it elsewhere
A cross-encoder scoring query and candidate together adds latency that a bi-encoder lookup does not. It earns it twice: irrelevant chunks stop reaching the prompt, which cuts input tokens, and per-document deduplication after reranking stops a single verbose file from occupying the whole context. The reranker made the context smaller, not larger — which is why the token numbers moved down while answer quality held.
A context budget object instead of magic numbers
Truncation limits used to be constants: 30k characters total, 15k for documents, 2k per chunk. That silently starved 200k-context models and overflowed small ones. Now the budget is computed from the model's reported input window, with an explicit priority order when it gets tight — system prompt, then learned user facts, then knowledge, then recent history, and the old-history summary is sacrificed first.
Guardrails in three layers, none of them a paid API call
Length, injection patterns, blocked keywords, and PII run as local checks in under 10ms and block before a single token is spent. Denied topics and custom rules are injected into the system prompt above the tenant's own instructions, so they ride the LLM call already being made. Output PII redaction runs on the way back. Regex misses semantics and prompts miss misspellings — running both is what makes the pair useful.
Half the context, same answers
Retrieval over-fetch dropped from k×5 to k×3, per-document chunk selection from five to three, and the similarity threshold moved to 0.45 to buy back recall. Context per turn halved. The tradeoff is real — coverage per document went from roughly 10-12% to 6-8% — which is exactly why the eval suite had to exist before the change, not after.
How it was
measured
Nothing shipped on intuition. Each number below is produced by a repeatable harness that gates deploys.
Layout, document-AI, two VLMs, and a text path — with PDF repair in front.
Dense and BM25 searched in parallel, fused, then cross-encoder reranked.
21.3k → ~12k per turn after retrieval, rerank, and context-budget work.
Per-document dedup after reranking, measured on the same query set.
Halved without a measurable drop in answer quality.
Local guardrail layer; a blocked message costs $0 in model spend.
Unit and integration coverage over retrieval, tools, and parsers.
Versioned datasets per domain, scored before and after each rollout.
Guardrails
and safety
What stands between a good demo and something you can leave running unattended.
The ACL filter is applied to both the dense and sparse arms of every hybrid search, so fusion cannot rank a chunk the requester is not permitted to read.
Layer one — input length, prompt-injection patterns, blocked keywords, and PII detect/redact/block — runs locally before any model call. A blocked message costs nothing.
Layer two injects denied topics and custom rules above the tenant's own agent instructions, so a tenant cannot instruct their way past platform policy.
Layer three redacts PII and filters blocked terms on the response before it leaves the stream.
Agent-generated code runs in an isolated E2B sandbox; the local subprocess runner exists for development only.
Error events carry a correlation id that matches the server log; the exception text and traceback never reach the client. The gateway additionally screens every request for leaked secrets.
What I'd
carry forward
The parts that generalise — earned the expensive way, on this build.
Recall is upstream of ranking. A reranker that only sees what dense search returned is a reranker, not hybrid search — and it will happily report good scores on the documents it never found.
Parsing is retrieval. Every point of recall you lose at ingestion is unrecoverable downstream, and it is invisible in your metrics because the document was never a candidate in the first place.
Some decisions are migrations. The BM25 analyser is baked in at collection creation, so choosing it is a schema decision disguised as a config value. Evaluate before you write, not after.
Budget the context window explicitly. Every hardcoded truncation constant is a silent quality ceiling on your best model and a silent overflow on your cheapest.
The cheapest guardrail is the one that runs before the model call. Ordering safety checks by cost is free architecture.