engineering · 2025-12-05 · 9 min read

Why we moved document routing off Kafka and onto Postgres (and lived)

Everyone told us we were wrong. Six months later: p99 latency 11s → 2.4s, infra bill down 68%, on-call quieter. Here's the trade-off analysis nobody let us skip.

When we started unstructred, we did the respectable thing and put Kafka between every stage. Documents in, parse jobs out, results onward. It's what you do — a queue means you're serious. Two years later we deleted almost all of it and replaced it with Postgres tables, and this post is our confession plus the receipts.

What our workload actually looked like

Kafka is built for high-throughput, fan-out streaming: many consumers, each needing the same events, replayed from wherever. Our workload is nothing like that:

• Fan-out of one. Each document goes to exactly one parser worker. No second consumer wanted it.
• Bursty, not steady. Customers upload 10,000 invoices at 9am and nothing until lunch. Our brokers idled at 4% utilization except for twice-daily thunderstorms — a capacity-planning joke.
• Document-shaped state, not stream-shaped. A document isn't an event; it's a state machine (ingested → parsing → partial → done). We already had the state in Postgres for the API to serve. The Kafka layer was a second source of truth about the same state, and every bug where they disagreed was a support ticket.
• Pausing, not replaying. The worst mismatch. When OCR fails on page 34 of 212, the right move is pause, retry that page, resume — with document-level bookkeeping. Kafka's answer is "retry the whole message from offset N," which for us meant re-parsing 33 pages to redo one. We were fighting the semantics we'd adopted for respectability.

The Postgres design

The core is boring, which is the point. One table, roughly:

CREATE TABLE parse_jobs (
    id            uuid PRIMARY KEY,
    document_id   uuid NOT NULL,
    state         text NOT NULL,  -- queued | running | partial | done | failed
    attempt       int  NOT NULL DEFAULT 0,
    lease_expires timestamptz,
    enqueued_at   timestamptz NOT NULL DEFAULT now()
);

Workers claim jobs with the classic pattern:

UPDATE parse_jobs
SET state = 'running',
    attempt = attempt + 1,
    lease_expires = now() + interval '5 minutes'
WHERE id IN (
    SELECT id FROM parse_jobs
    WHERE state = 'queued'
       OR (state = 'running' AND lease_expires < now())
    ORDER BY enqueued_at
    FOR UPDATE SKIP LOCKED
    LIMIT 10
)
RETURNING *;

FOR UPDATE SKIP LOCKED does the load balancing: workers grab whatever isn't already claimed and never block each other. The lease expiry is the crash detector — a worker that dies mid-job has its row released automatically five minutes later, and attempt tells the next worker what happened. Total code: one UPDATE, one index. Total new infrastructure: zero.

Blobs never went through the queue in the first place — object storage then, object storage now. Only the pointers move.

The numbers, six months later

The latency win isn't Postgres being magic — it's removing hops. Produce, commit, consumer poll, rebalance, consume: each step added tail latency, and tail latency is what p99 is made of. A single indexed table lookup doesn't have those tails.

The bill is the same story with money. Broker nodes sized for the 9am thunderstorm, running 24/7, doing almost nothing at 4pm. Postgres was already there, already sized, already backed up.

What we gave up

This isn't a free lunch, and we'd be lying by omission if we stopped here:

• Replay. Kafka's log is genuinely great for "reprocess everything from Tuesday." With Postgres you can do it (UPDATE jobs SET state='queued' WHERE ...) but it's heavier, and there's no clever offset arithmetic. We use it maybe twice a year. Fine trade.
• Very high steady throughput. Our sustained rate is thousands of jobs a minute, which Postgres handles with headroom. At hundreds of thousands a second — actual stream scale — this design would fall over and Kafka would be right again. We are not at that scale and may never be.
• Fan-out. The moment a document needs to go to multiple consumers (say, extraction and an audit hook), a table with one state column stops being enough. You end up inventing per-consumer state, then cursing, then buying a queue. Our advice below covers when to accept that.

When you should keep Kafka anyway

1. You're fanning out to more than about three independent consumers per event, and they care about the same bytes. (Extraction + webhook + analytics? Buy the queue.)
2. You need replay as a product feature — event sourcing, audit logs, "reprocess the last month against the new model."
3. Your throughput is steady and genuinely high. Bursts don't count; steady 100k events/sec does.

If none of those describe you, a table with SKIP LOCKED will hold more than you think, and you already know how to operate it. That last part — "already know how to operate it" — is worth more than any benchmark.

Questions, or want to argue about queue semantics at length? Our favorite. hello@unstructred.io.