Six LLM calls down to one: an NL→SQL analytics endpoint
Zebra Technologies, analytics for a global sales organisation
Select a step to see the decision behind it.
Calls per question, before and after (schematic, not to scale)
Problem
Sales leaders ask questions in plain English and get back numbers and a chart. The endpoint answered each question with a chain of 4–6 sequential LLM calls, then ran its queries one after another. Every hop added latency and input tokens, and every answer waited on the slowest link.
Constraints
Answers are checked against live BigQuery data, charts must always render, and the cost per question could not go up.
Decisions
- Collapse the chain into one structured Gemini Pro call that returns the original SQL plus two decomposed insight queries.
- Put a small, fast table classifier first, so the big model only sees the relevant tables and rule blocks.
- Run every BigQuery execution in parallel with asyncio.gather instead of in sequence.
- Cut the Pro model's thinking budget to minimal and move the retry path to a fast model.
- Return chart specs (Vega-Lite) typed from the BigQuery result schema, so a chart never references a column that isn't there.
Trade-off
One bigger prompt is a single point of failure: a malformed response now costs the whole answer, so retries go to a fast model instead of repeating the expensive call. A classifier misroute would hide a table the question needs, so the fallback is the full schema.
Result
4–6 sequential LLM calls became 1. The classifier cut input tokens by 60–80%. All BigQuery executions now run concurrently.
What I'd do next
Build an offline eval set of real questions with known-good results, so every prompt change is scored before it ships.