draft-imran-systems-and-arguments-45Internet-Draft
← draft-imran-systems-and-arguments-45
Off the Bit Implementation 3.1 Category: System State: production Share: https://mosthofaimran.com/l/3-1
Mevrik: agentic customer experience platform 3M+ conversations/month 99.9% against contracted SLAs

Mevrik: agentic customer experience platform

Multi-tenant conversational automation for banks and telecom operators. One image ships to public cloud and to air-gapped estates, and configuration alone decides which.

Stack

Go · Python · TypeScript / Next.js · Flutter · gRPC + protobuf · PostgreSQL + pgvector · ClickHouse · Redis · NATS JetStream · MinIO / S3 · Docker · Kubernetes · Helm · OpenTelemetry

Text

What this is. A conversational platform where an AI agent handles customer contact end to end across chat, messaging, email and voice, and a human supervises what the agent escalates. It runs multi-tenant in public cloud for most customers and inside the customer's own estate, including fully air-gapped, for banks and regulated operators who cannot let a conversation leave their network. The same build serves both.

1. The constraint that shaped everything

One sentence decided most of the architecture: the same product has to run in a shared cloud and in a bank’s air-gapped data centre, and neither can be a port of the other.

Paper 5.5 argues that sovereignty is cheap when it is adopted at design time and ruinous when it is retrofitted. This is the system that argument came from. Every decision below is downstream of refusing to maintain two builds.

2. Tenancy is a schema property, not a query habit

The failure everyone fears in multi-tenant software is one tenant seeing another’s data, and the usual defence is discipline in the application layer. Discipline is not a control.

The contract here is structural and has four parts:

The part that makes it hold is not the rule, it is the enforcement. A new table without tenant_id fails the build. Every migration pull request spins an ephemeral database, applies the full history from zero and runs a row-level security smoke test against it. A policy that is only in a document decays; a policy that fails continuous integration does not.

3. Two services, and the discipline was not splitting into more

The backend is two services: a modular monolith carrying the gateway, identity, tenancy, conversations and the agent runtime, and a second service carrying AI compute.

The interesting decision is the one not taken. Splitting the first service into gateway, identity, conversations and runtime would have looked more modern and bought nothing: they share a database, a tenant context and an auth context, so separating them adds deployment and failure surface without adding scaling headroom. The seams are kept clean so extraction is available later, and extraction happens on a measured signal rather than on taste.

AI compute is separate from the first day for a reason that survives scrutiny: it has a genuinely different scaling profile. Bursty GPU work, vector index memory, model caches and high-cost-per-call operations scale on a different axis from a chat gateway holding websockets. Scaling those independently is a real saving rather than an architectural preference.

Upgrade triggers are written down in advance, with the signal that fires each one, so growing the system is a decision taken calmly rather than a reaction. Paper 5.12 is about architectures that follow the org chart; this is the attempt not to.

4. Own the interface, rent the engine

The agent runtime is built on six primitive interfaces defined in-house: tool calling, skills, a guardrail hook pipeline, memory, context curation and model routing. Those interfaces are the contract and they were frozen early.

Underneath them, the runtime today wraps a vendor agent framework through an adapter. That is deliberate. Writing the loop from scratch in the first month would have been a way of spending a quarter on something a vendor had already shipped. Wrapping it without owning the interface would have been worse: every skill, tool and integration would have been written against somebody else’s abstractions, and replacing the runtime later would mean rewriting all of them.

So the loop is rented and the surface is owned. When the framework stops fitting, the implementation behind the interfaces changes and the skills, tools, hooks and evaluation suites written against them do not.

5. Guardrails are a pipeline, not a policy

Every conversational turn runs through the same ordered hooks, and the ordering is the design.

EVERY TURN, IN THIS ORDER inbound message pre-flight PII redacted here, before the model is ever called agent runtime reason, call tools, observe, decide post-flight grounding, tone, sanitise, meter customer tool gateway injects tenant id at dispatch the model cannot construct a cross-tenant call THE AUDIT LANE the harness writes the trace at every stage, not the agent append only · hash chained · replicated to object-locked storage No prompt, tool input or model output leaves a stage without a record written by something other than the thing being recorded.
Figure 1. Three decisions in one picture. Redaction happens before the model, not after. The gateway injects tenancy rather than trusting the model to scope its own calls. The audit is written by the harness, so the record does not come from the process it describes.

Three of those are worth stating on their own.

Redaction runs before the model, not after. A post-hoc scrub of a response is a cleanup; redacting on the way in means personal data was never in the prompt. The second is a control, the first is a hope.

The gateway injects tenancy at dispatch. Tool calls do not carry a tenant chosen by the model. The gateway attaches it when the call is dispatched, so no output the model can emit constructs a cross-tenant request. This is the single control I would keep if I could only keep one.

The audit is written by the harness. Paper 5.21 argues that a record authored by the process being observed is testimony rather than evidence, and that the fix is a boundary. This is that boundary, built before I had written the argument down: the trace is emitted by the runtime around the agent, append-only and hash-chained, and the agent has no write path to it.

6. Actions are planned, approved, then executed

Any action that changes the world runs through a fixed loop. The agent emits a structured plan naming intent, target, side effects and whether the action is reversible. The surface renders it in plain language. A person approves at a chosen scope. Only then does the call dispatch, with audit written before and after.

The part I am most pleased with is the smallest. Every tool must declare whether it is reversible, and a pre-commit hook rejects any new tool that does not. Irreversible actions require a second confirmation in which the operator types the resource name. The rule is enforced by the thing that will not let you commit rather than by a paragraph in a wiki.

Where an agent assists a human rather than acting, suggestion-only is structural too: its tool allow-list cannot contain a write tool, enforced at registration, and a write tool tagged for that agent fails the build.

7. One image, four ways to run it

Cloud multi-tenant, isolated multi-tenant, single-tenant in the customer’s cloud, and air-gapped on-premise. Same container images, same deployment charts, same database schema. Configuration decides which mode a deployment is in.

No build-time forks and no schema forks per edition. The reason is maintenance arithmetic rather than elegance: one security patch has to reach every edition, and reproducible builds across editions are an audit requirement rather than a nicety. A fork per deployment mode is a promise to do every fix four times, and that promise is always broken quietly.

8. What I would do differently

I would put the tool-surface governance in from the first week. It arrived after the tool registry did, which meant a period where adding an endpoint to an agent’s reach was a one-line change nobody reviewed. Grouping several hundred endpoints into a handful of domains turned out to be the single biggest lever on answer quality, and I found that out later than I should have. It is failure 6.1 above and it is the one I would tell someone else to front-load.

I would have built the evaluation gate before the guardrails. Guardrails stop bad output reaching a customer. An evaluation suite tells you whether a change made the system better, and without one the guardrail pass rate becomes the quality metric by default, which is paper 5.19’s argument arriving from the inside.

The numbers in this note are of two kinds and I have marked which is which. Conversation volume, availability and recovery time are measured. Latency, kill-switch activation and throughput are targets the build is held to. Publishing a target as though it were a measurement is the specific dishonesty this site exists to avoid, and the temptation is real because targets are always rounder.

Measurements

MetricValueNote
Conversations per month3,000,000+measured, across banking and telecom tenants
Availability99.9%measured, against contracted SLAs
Mean time to recoveryunder 30 minmeasured, not targeted
First token, design targetunder 800 ms p95a target the build is held to, not a published measurement
Full resolution, design targetunder 3 s p95same: target, not measurement
Kill switch activationunder 5 starget, from command to all in-flight conversations stopped
Audit retention7 yearsconfigured retention, not a measurement: append-only, hash-chained, object-locked in object storage

Falls over at: Tool surface breadth rather than request volume. Grounding quality degrades as the number of individually registered tools grows, which is why promotion to an agent's tool surface is a reviewed step rather than a decorator.

Known failure modes

6.1, accepted. Tool sprawl degrades grounding. Every endpoint added to an agent's reachable surface widens the space the model chooses from, and past some breadth the accuracy won earlier evaporates. Mitigated by governance rather than solved: endpoints are proposed, reviewed by a person and grouped into a handful of domains before an agent can see them. It remains the thing most likely to quietly get worse.
6.2, fixed. Writing to the database and publishing to the event bus in the same operation loses events when one of the two fails. Replaced with a transactional outbox: the publish is a row written inside the same transaction and drained by a relay. Designed against in advance rather than learned from an incident.
6.3, fixed. Inbound channel webhooks retry. Without an idempotency key per inbound message a retry becomes a duplicate message, and a duplicate message becomes a duplicate billable record, which is an integrity problem rather than a cosmetic one. Keys are enforced at ingest.
6.4, open. Nothing in the design distinguishes a model that has degraded from a model that is being asked harder questions. Anomaly thresholds on cost, error, escalation and grounding rate will fire on both, and a person still has to read the conversations to tell which happened.
ImranImplementation 3.1production