Where Does the Breakpoint Go?

A support bot can cache the start of every request. Pick where the cached part should end.

The Situation

A support bot answers customers' questions about the Lumen smart thermostat. Each customer's question is a new request to an AI model. The model is the program that writes the answer. The bot keeps no chat history. Each request contains only the three parts below.

Models read text in tokens. A token is a small piece of a word. You pay for every token you send.

Each request has three parts, always in this order:

1. The instructions: 1,300 tokens. They're the same on every request.
2. The Lumen smart thermostat manual: 24,000 tokens. It's the same on every request.
3. The customer's question. It's different on every request.

A cache stores the start of a request. When a later request starts with exactly the stored text, the stored part costs a tenth of the normal price. If the start differs at all, there's no discount.

A breakpoint marks where the stored part ends. Everything before the breakpoint goes into the cache. You can place one breakpoint.

Right now nothing is cached, so the bot pays full price for every token.