Stop Calling LLM APIs from Your Request Handler — Queue Them

Every backend I've touched this year has grown an AI feature: summaries, auto-tagging, content moderation, chat. And almost every one of them started with the same mistake — calling the LLM API directly inside the request handler.

It works in the demo. It falls over in production. Here's why, and the architecture that fixes it.

---

Why Inline LLM Calls Break in Production

LLM APIs have three properties that make them unlike any other dependency you call inline:

Now put that inside a request handler:

• A 30-second generation holds an HTTP connection, a DB connection, and server memory the whole time
• Ten users clicking at once blows through your provider's requests-per-minute tier — everyone gets 429s
• A naive retry loop doubles your bill on flaky calls
• One provider incident cascades into your API's uptime

The fix is the same one you already use for emails and PDFs: the request enqueues, a worker generates.

---

The Architecture

POST /summarize
     ↓
API validates + enqueues job → returns 202 { jobId } in ~10ms
     ↓
BullMQ (Redis)
     ↓
Worker (concurrency-capped) → LLM API
     ↓
Result stored → client polls /jobs/:id, or gets a websocket/webhook push

The API stays fast and stateless. The queue absorbs bursts. The worker is the only place that talks to the provider — which means rate limiting, retries, and cost control all live in exactly one file.

---

The Code (NestJS + BullMQ)

1. Enqueue instead of calling

// summarize.controller.ts
@Post("summarize")
async summarize(@Body() dto: SummarizeDto) {
  const job = await this.llmQueue.add(
    "summarize",
    { articleId: dto.articleId, tenantId: dto.tenantId },
    {
      jobId: `summarize:${dto.articleId}`, // idempotency: same article → same job
      attempts: 4,
      backoff: { type: "exponential", delay: 3000 }, // 3s, 6s, 12s, 24s
      removeOnComplete: { age: 3600 },
    },
  );
  return { status: "queued", jobId: job.id };
}

The jobId line matters more than it looks: if the user double-clicks or your client retries, BullMQ deduplicates instead of paying for the same generation twice.

2. The worker — the only file that touches the provider

// llm.processor.ts
@Processor("llm", {
  concurrency: 5, // match your provider tier, not your CPU
  limiter: { max: 50, duration: 60_000 }, // 50 calls/min across all workers
})
export class LlmProcessor extends WorkerHost {
  async process(job: Job) {
    await this.budget.assertWithinBudget(job.data.tenantId); // check BEFORE spending

    const article = await this.articles.findById(job.data.articleId);

    const response = await this.anthropic.messages.create(
      {
        model: "claude-sonnet-5",
        max_tokens: 1024,
        messages: [{ role: "user", content: `Summarize:\n\n${article.body}` }],
      },
      { timeout: 60_000 }, // never wait forever
    );

    await this.budget.recordUsage(job.data.tenantId, response.usage);
    await this.articles.saveSummary(article.id, response.content[0].text);
  }
}

Two things do the heavy lifting here:

• concurrency + limiter — your requests-per-minute ceiling is enforced in one place. Bursts of 500 jobs don't 429; they wait.
• Retries live in the queue, not in your code. A 429 or overload error just throws; BullMQ re-runs it with exponential backoff. No hand-rolled retry loops, no accidental double-spend (the budget check runs on every attempt).

3. Cost control: budget before, not invoice after

// budget.service.ts — Redis counters per tenant per month
async assertWithinBudget(tenantId: string) {
  const spent = await this.redis.get(`llm:spend:${tenantId}:${thisMonth()}`);
  if (Number(spent ?? 0) >= this.limits.for(tenantId)) {
    throw new UnrecoverableError("LLM budget exceeded"); // BullMQ: don't retry this
  }
}

async recordUsage(tenantId: string, usage: Usage) {
  const cost = usage.input_tokens * INPUT_RATE + usage.output_tokens * OUTPUT_RATE;
  await this.redis.incrbyfloat(`llm:spend:${tenantId}:${thisMonth()}`, cost);
}

UnrecoverableError is the detail teams miss: budget exhaustion is not a transient failure, so it must not retry. Provider errors retry; business-rule failures don't.

Add prompt caching on top — hash the prompt, check Redis before enqueueing — and identical requests cost you a Redis lookup instead of an API call.

---

When You Should NOT Queue

One honest exception: interactive chat. If the user is watching tokens stream into a UI, a queue between them and the model adds latency for no benefit. Stream those directly — but keep the guardrails:

Interactive chat (user watching)   → direct call, streamed, with timeout + budget check
Everything else                    → queue
  summaries, tagging, moderation,
  embeddings, enrichment, batch
  generation, scheduled reports

Even for chat, the budget service and the timeout stay. Only the queue goes.

---

Decision Framework

Is a human watching the response render live?
  → Yes → direct + streaming
  → No  → queue it

Can the work tolerate a 5–60 second delay?
  → Yes → queue it (that's almost everything)

Are you calling the LLM per-item over a collection?
  → Always queue — one job per item, let the limiter pace it

Did you just get your first 429 in production?
  → You needed the queue yesterday

---

What This Looks Like in Practice

The chat assistant on this site runs on exactly these guardrails — hard timeout, per-session budget, no unbounded retries. And the queue pattern is the same one from my notification engine build: the only thing that changed between "send 10,000 WhatsApp messages" and "generate 10,000 summaries" is what the worker does. The infrastructure — BullMQ, Redis, backoff, rate limiter — is identical.

That's the real argument for this architecture: LLM calls aren't special. They're slow, flaky, expensive jobs — and we've known how to run those safely for a decade. Treat them like jobs.

If you're weighing whether BullMQ is enough for this or you need something heavier, I've written a full comparison: BullMQ vs Kafka — which one you actually need.

---

Adding AI features to a production backend and want them to survive contact with real traffic? Drop a message — this is exactly the kind of system I build.