LLM APIs are slow, rate-limited, and expensive — three things that destroy inline request handling. Here's the queue-based architecture I use for AI features in production Node.js backends, with BullMQ code you can copy.
Queue them unless the user is actively watching the response stream. Chat UIs need synchronous streaming, but summaries, enrichment, moderation, and batch generation should go through a queue with retries and rate limiting.
Treat 429s as retryable: put calls in a BullMQ queue with exponential backoff and a concurrency limiter matched to your provider tier, so bursts wait in the queue instead of failing.
Enforce budgets before the call, not after the invoice: track token usage per tenant in Redis, reject or degrade jobs over budget, cache identical prompts, and make jobs idempotent so retries never double-spend.
I'm a freelance NestJS & Node.js backend developer. Let's ship your MVP or SaaS.
Explore other insights in the gallery.