Prompt caching on a public chat endpoint: what a load test taught me
The chat assistant on this site, Patricia, is open to anyone. That makes it the one part of the stack where a stranger can spend my money, so I load tested it before leaving it on.
What the test showed
Each guest call carried roughly 3.2K tokens of scaffold: the system prompt, the persona and the tool definitions. None of that changes between calls. At the model's input price, resending it every time was most of the bill, about $9.70 per thousand guest calls.
The fix
The system prompt now goes to Amazon Bedrock as cache-pointed blocks, so the stable prefix is read from cache instead of reprocessed. The first call in a window pays full price; repeat calls within the five-minute cache window cost a fraction of that.
I confirmed it on the live endpoint rather than trusting the docs: the second call's
usage log showed 3,179 tokens read from cache. A chat.usage log line now records
cache reads on every call, so the saving is visible, not assumed.
What I'd tell someone else
- Load test the public endpoint before you promote it. The cost problem was invisible at normal traffic.
- Put the stable part first. Caching only helps when the cached prefix is byte-identical, so per-user text goes after the cache point.
- Log cache reads. If the log line never shows a hit, the change did nothing.
- Cap the ceiling separately. Caching lowers the unit cost. Rate limits and input size caps bound the total.