Why the question isn’t “which one wins” — it’s “which one owns this boundary”
In design reviews for AI infrastructure, someone always asks “REST or gRPC?” and someone else says “why not just Kafka?” Both are right, and both are asking the wrong question — these three don’t compete on the same axis.
REST and gRPC are request/response protocols: how does service A ask B for something and wait for an answer. Kafka is a durable, replayable event log: how does data move through a system without the producer knowing or caring who’s listening, or when. Treat them as competitors and your architecture fights itself. Treat them as three tools that own different boundaries, and the design writes itself.
Why AI systems break the classic playbook
- Latency is bimodal. A cached embedding lookup takes 2ms; a cold-start GPU call takes 400ms; an agent chain takes seconds. One protocol shouldn’t have to absorb that whole spread.
- Payloads are heavy and oddly shaped — tensors, embeddings, token streams, not tidy JSON.
- Streaming is the default UX, not an edge case — token-by-token generation, partial tool results.
- Most “traffic” isn’t request/response at all — feature writes, drift events, retraining triggers.
- Failure needs to be replayable, not just retryable. A downstream outage shouldn’t mean lost work.
REST: the boundary for humans and the outside world
REST owns the edge — browsers, partners, webhooks, model registry CRUD — anywhere a stable, curl-able, human-debuggable contract matters more than raw efficiency.
It quietly hurts you internally: token streaming over REST means bolting on SSE, which is one-way, fights buffering proxies, and re-encodes JSON per chunk. HTTP/1.1 also head-of-line-blocks — a slow agent call can queue behind a health check on the same connection. REST also tends to accumulate undocumented schema drift with no compile-time contract.
Rule of thumb: REST at the edge. Rarely the protocol two of your own services use to talk to each other at high QPS.
gRPC: internal, latency-sensitive service calls
gRPC (HTTP/2 + protobuf) gives typed contracts, native bidirectional streaming, multiplexing, and per-call deadlines. Triton, TorchServe, and KServe all use gRPC as their primary high-throughput interface for exactly this reason — when an orchestrator calls an inference service hundreds of times a second, binary encoding and stream multiplexing aren’t optional extras.
protobuf
service GenerationService {
rpc StreamGenerate (GenerationRequest) returns (stream Token);
}That stream Token is the point: tokens arrive as generated, over one connection, with HTTP/2 flow control handling backpressure instead of hand-rolled chunking.
Costs: you can’t curl it (needs grpcurl or a client), protobuf punishes careless field renumbering, Python teams resent maintaining _pb2.py stubs, and browsers need gRPC-Web plus a proxy.
Rule of thumb: gRPC for synchronous, low-latency, internal calls where you control both ends and streaming or strict typing earns its complexity.
Kafka: everything that isn’t a conversation
Kafka doesn’t answer “how do two services talk” — it answers “how does data move when the producer shouldn’t care who’s downstream or when they read it.” A huge share of AI-system work fits this: one embedding event needs a feature-store update, a vector DB upsert, an audit log entry, and a drift check — four independent consumers, none of which should block the producer.
python
producer.send("inference.completed", key=request_id, value={
"model_version": "gen-v14", "latency_ms": 214, "user_feedback": None,
})
# drift monitor, training-data curator, and cost aggregator all read
# this topic independently, and can replay from any offset.Kafka is right for feature pipelines, embedding backfills, drift monitoring, feedback loops feeding training days later, and retrain triggers — anywhere replay and fan-out beat millisecond latency.
It’s wrong as a slow RPC substitute. If a caller is waiting right now for an answer, routing it through Kafka just adds consumer-lag as a new failure mode you didn’t need — you end up rebuilding request/reply (correlation IDs, timeouts) on top of a system that wasn’t built for it.
How it composes
REST at the edge → gRPC for the synchronous internal path → Kafka for anything fanning out or without a waiting caller:
| Dimension | REST | gRPC | Kafka |
|---|---|---|---|
| Model | Sync req/response | Sync + native streaming | Async pub/sub, durable log |
| AI use case | Edge APIs, registry CRUD | Inference mesh, streaming gen | Feature/drift/retrain pipelines |
| Coupling | Producer knows consumers | Producer knows consumers | Producer knows nothing |
| Replay | None | None | Core feature |
| Failure mode to watch | Timeout cascades | Deadline propagation | Consumer lag |
Real failure pattern: a team ran embedding ingestion → vector DB writes over synchronous REST. During a 40M-document backfill, writer latency crept up, REST timeouts cascaded, and retries silently duplicated and dropped writes for hours. The fix wasn’t gRPC (still no durable buffer) — it was Kafka between the two stages: ingestion publishes and moves on, the writer consumes at its own pace, and a backfill becomes “replay from offset zero” instead of “re-run millions of synchronous calls and hope.” When the real problem is lost or duplicated work under load, that’s a durability problem — only a log fixes it.
The decision framework
- Is a caller waiting synchronously right now? No → Kafka.
- Is the caller outside your trust boundary (browser, partner)? Yes → REST.
- Do you control both ends and need streaming, strict typing, or sub-50ms overhead? Yes → gRPC. No → REST is fine internally too.
- Does more than one independent system need to react to this same data? Yes → Kafka, even if it started out feeling like an RPC call.
None of these three is the “modern” choice. They answer different questions — and the AI systems that hold up under real load are the ones where someone drew these boundaries deliberately instead of defaulting to whatever the last service used.
