Companion Chat Runtime
A tool-calling companion agent over a dual-layer memory system, with every provider call — chat, embeddings, image, voice — routed through a self-hosted LiteLLM gateway. Relationship state drives both the persona and the sampling temperature; a DPO-fine-tuned persona model sits behind the same alias as the frontier models, so changing a character's voice is a config change.
- Role
- AI Engineer — backend & agent runtime
- Domain
- Consumer AI companion chat
- Surface
- Chat · voice · image · live cards
- Runtime
- FastAPI · LangChain · pgvector · LiteLLM
- Status
- Live, shipping on iOS & Android
Outcomes
What was
broken
The problem as it actually presented itself, and the constraints that shaped every decision after it.
A companion that forgets is not a companion. Stuffing the whole transcript into context does not fix it — cost scales with session length, and the model starts attending to a conversation from three weeks ago instead of the sentence in front of it.
Personality also has to be stable and earned. A character that is instantly warm has nothing to give later, and one that never warms up is unusable. That is a state machine problem, not a prompt problem, and it needed to be visible in both the instructions and the sampling parameters.
Then there is the money. Image generation, TTS, and frontier-model turns each cost real money per interaction on a consumer product with a free tier — so the caching, routing, and fallback layers were not optimisations, they were the business model.
What it
runs on
Models, retrieval, serving, and the operational layer that keeps it honest in production.
- GPT-class + Claude + Gemini + Grok
- DPO fine-tuned persona model
- text-embedding-3-small
- LangChain create_agent
- Tool calling
- LangSmith tracing
- Postgres FIFO buffer
- pgvector long-term
- LLM summariser
- Fal (image)
- Fish Audio (TTS)
- S3 / R2
- Semantic prompt cache
How it
works
The pipeline end to end — each stage, and why it earns its place in the latency budget.
- 01
Gateway-first provider access
Every outbound model call goes through a self-hosted LiteLLM gateway, and exactly one module in the codebase imports an LLM SDK. Model selection, cost accounting, retries, and rate limits become configuration. Swapping a character onto a different model — including a fine-tuned one — never touches application code.
- 02
Tool-calling companion agent
The agent is built per character and cached, with tools for semantic memory recall, writing a new fact about the user, generating a selfie, synthesising voice, and adjusting trust. The model decides when to reach for them; the platform owns what they are allowed to do.
- 03
Dual-layer memory
A Postgres FIFO buffer holds the last twenty messages and feeds the ten most recent into the prompt. Overflow compresses through a summariser into pgvector long-term storage. Compression runs under a Redis lock with a short TTL and a failure cooldown — without it, a gateway outage turns retry-on-next-turn into one summarisation attempt per active conversation per minute, platform-wide.
- 04
Relationship state as a control signal
Accumulated trust maps to a stage — cold, warm, peak — and the stage drives both the persona instructions and the sampling temperature, from guarded and low-variance to open and expressive. Trust moves in small negative-skewed increments: slow to earn, fast to lose.
- 05
Turn directives out of band
The model may end a reply with a single scene-transition tag proposing a location move, time skip, or situation change. The tag is stripped from every user-visible and persisted surface and travels to the client as a structured card payload instead. Parsing is owned in code; the wording that triggers it is owned by the content team, and the boundary is marked in the source.
- 06
Semantic image cache
Before any image generation, an ANN query runs over stored prompt embeddings, scoped to the character, provider, and reference-image set. Byte-identical prompts land at distance zero and skip the provider entirely. The cache fails open by design — an embedding timeout, gateway error, or database error all return nothing and fall through to normal generation. The module never raises.
Decisions
and their cost
Every choice below bought something and gave something up. The second half is the part worth reading.
One module owns the SDK
Every provider call goes through a single client. It is more indirection than the first version needed and it is the reason adding a fifth provider, a fine-tuned model, or a new fallback chain is a config edit rather than a refactor.
Cache image prompts semantically — but scope it tightly
The cache key is the prompt embedding scoped by character, provider, and reference-image hash. Widening that scope would raise the hit rate and would also serve one character's face for another. The cheaper cache is the wrong cache.
Verify prompt caching, do not assume it
I built a probe harness that sends the same prefix with and without cache-control markers, streaming and non-streaming, across every route, and prints the usage fields back. It found two things assumption would have missed: repeated-paragraph filler produces false cache hits on implicit prefix matching, and one provider's streaming route rejects a cached-content block that its non-streaming route accepts. The feature flag stayed off until both were understood.
Fallback chains that cross providers
Every model has a short fallback chain, and each chain includes at least one entry from a different upstream. Same-provider fallbacks look tidy and fail together. The tradeoff is tail latency — every hop adds it — so the chains are deliberately kept to two or three entries behind a gateway-wide concurrency breaker.
How it was
measured
Nothing shipped on intuition. Each number below is produced by a repeatable harness that gates deploys.
First week to last full week of the 30-day report window.
Probe harness across every provider route, streaming and non-streaming.
Repeat prompts served from pgvector at distance 0, no provider call.
10 recent messages + retrieved memories — flat regardless of history length.
Circuit breaker returns 429 upstream rather than stampeding providers.
Plus integration smoke tests against DB, Redis, and the gateway.
Guardrails
and safety
What stands between a good demo and something you can leave running unattended.
Secret-pattern blockers screen input before it reaches the gateway, and the gateway enforces a global concurrency ceiling rather than letting a burst reach providers.
Per-user concurrency caps via Redis counters, plus per-route rate limits, so one client cannot monopolise generation capacity.
The image cache fails open on every error path — a cache failure degrades to a normal generation, never to a failed request.
Summarisation is guarded by a distributed lock and a failure cooldown so a provider outage cannot amplify into a retry storm.
Scene tags and other control directives are stripped from everything user-visible and everything persisted; only the parsed structure ships to the client.
Media is served through presigned URLs from private buckets rather than public objects.
What I'd
carry forward
The parts that generalise — earned the expensive way, on this build.
Provider behaviour is not portable. The same cache-control block is accepted, ignored, or rejected depending on the provider and whether you are streaming — the only way to know is to probe each route and read the usage fields.
Personality is state, not prompting. Once trust drove both the instructions and the temperature, the character became consistent in a way no amount of prompt rewriting achieved.
Fail-open and fail-closed are product decisions. A cache should fail open. A moderation layer should not. Deciding per component beats a single global posture.
Draw the line between content-owned and code-owned text, and mark it in the source. It stopped a whole category of merge conflicts between the writers and the engineers.