Agent Cost and Latency SLOs for Production Queues
An agent SLO should bind cycle time and cost per completed case to a named queue. Token charts without an error budget do not tell operators when to shed load.
Direct Answer
Set the SLO on completed cases, not on isolated model calls.
Agent cost and latency SLOs are targets for how long a production queue may take to finish a case and how much inference and tool spend that case may consume. The service level indicator is measured at the business object: a screening packet completed, pended, or failed. A fast model call that still leaves the case unfinished is not success.
Google's SRE material is blunt about this shape. An SLO is a target on an indicator that tracks user happiness. For an operational agent, the user is the operator waiting on the queue and the downstream control that consumes the packet. Token totals and raw model latency are inputs to the indicator. They are not the objective.
Our bias is to write the first SLO in queue language before anyone debates model choice. If the team cannot say what a completed case is, they are not ready to set a budget.
Dashboard Trap
Token charts look like operations and behave like trivia.
A typical pilot dashboard shows tokens per day, average model latency, and maybe tool-error rate. Those numbers are easy to export. They do not tell a trade-compliance lead whether denied-party screening still finishes before the shipment cut-off, or whether a retry storm spent the week's budget on one bad batch of bills of lading.
The old pattern treats the model as the service. The new pattern treats the screening lane as the service. The model, the retrieval step, the watchlist API, and the reviewer pause are dependencies inside that lane.
Latency without a case clock is incomplete
A 400-millisecond completion can sit behind a 20-minute tool timeout and still miss the cut-off.
Cost without a unit of work is unowned
A spike in tokens is only actionable when it is attached to a lane, a document class, and a retry reason.
Availability without a stop rule hides failure
An agent that keeps retrying a watchlist timeout can look 'up' while the dock clock runs out.
Queue Example
Denied-party screening needs three indicators, not one token graph.
Use a trade-compliance team at a mid-size manufacturer. The systems of record are the shipment or order record, the denied-party and sanctions screening service, the customer master, and the export-hold queue. The agent reads the party list, calls the screening service, drafts a clear or hold packet, and routes possible matches to a licensed reviewer. It should not release a hold.
Write indicators that a compliance lead and an SRE can both read.
Case-cycle SLI
Time from shipment-ready event to a clear, hold, or failed-open packet. Target the cut-off the dock actually uses, not an average model time.
Cost-per-case SLI
Inference tokens, screening-API calls, and retries attributed to one shipment ID. A useful first SLO is a weekly budget per lane, not a company-wide token cap.
Completeness SLI
Share of cases that finish with a decision class and evidence, rather than timing out, looping, or dropping to a blank ticket.
Error budget
How many late or over-budget cases the lane can absorb before new prompt and model experiments pause. Google's workbook treats that budget as the decision tool, not a vanity percentage.
Instrumentation
Keep model metrics, then roll them up to the shipment.
OpenTelemetry's generative-AI semantic conventions define client histograms for token usage and operation duration, plus streaming timings such as time to first chunk. Those names are still marked development, so pin the convention version you emit. They are the right raw signals. They become an SLO only when each span carries the shipment ID, lane, and decision class.
The observability runbook at https://solzero.com/blog/agent-observability-runbooks-for-production-workflows tells an on-call person what to do after a miss. The SLO tells them whether the miss is a budget burn or a one-off.
Propagate the business ID
Every model span, tool span, and retry should carry the shipment or order identifier. Orphaned token counts cannot support a cost SLO.
Separate wait classes
Model time, screening-API time, and human-review wait are different delays. Mixing them into one average hides the lever you can actually pull.
Count retries as spend
A timeout that retries three times is three charges and one late case. The SLO should see both.
Shed load on purpose
When the error budget is gone, stop expanding autonomy and stop expensive research loops. Fall back to deterministic screening plus human review.
Operating Rules
The budget should change the week's work.
An SLO that nobody uses to say no is a report. The release gate at https://solzero.com/blog/agent-release-gates-for-production-workflows should include the current budget status. A prompt change that doubles tokens on the hold lane is a production change even if the model sounds the same.
Start with wide targets from last month's actuals, then tighten. The SRE workbook's advice still holds: pick a target users can feel, measure it consistently, and iterate. Do not advertise four nines on a queue that still has a human review step measured in hours.
Owner pair
Trade compliance owns the case-cycle target. Platform or SRE owns the measurement. Neither can set the number alone.
Lane-specific budgets
Auto-clear of a clean domestic shipment is not the same lane as a possible match on a new foreign consignee.
No silent model upgrade
A cheaper or slower model is an SLO event. Promote it only if the golden cases and the budget both still hold.
Scoreboard
Measure cut-off misses and cost per finished packet.
The SLO is working when a late screening or a spend spike changes the plan for the next change window. Useful measures include cases finished before cut-off, cost per completed packet by lane, retry share of spend, human-wait share of cycle time, and change freezes triggered by a burned error budget.
Do not treat lower token volume as a win if more shipments miss the dock. Cheap and late is still a failed export control.
The SolZero take is that agent SLOs are queue contracts. If a screening lane already has traces but no budget, the operating sequence is at https://solzero.com/#how-it-works.
Further reading