GLOSSARY
Prompt Caching
Reusing the computed prefix of a long system prompt so the next call does not pay the full input-token bill.
A 50-page system prompt plus a one-line user question still bills the 50 pages every time unless the provider caches the prefix. Prompt caching stores the KV state (or a hash of it) for a window of minutes, so subsequent turns with the same preamble are cheaper and often faster.
Anthropic, OpenAI and several gateways now expose this as an explicit feature; Helicone-style proxies cache at the HTTP layer instead. The catch is cache locality: a one-token change at the start of the prompt busts the prefix. Put the stable instructions first and the volatile user text last, or you will pay full price and wonder why the cache never hits.