Semantic Caching

Semantic caching is a caching strategy for LLM applications that decides whether to serve a stored response by comparing the meaning of the incoming request to previous ones, rather than by matching the request string exactly. A conventional cache is keyed on a hash of the input: change one character and it misses. A semantic cache embeds the incoming query, searches a vector index of previously answered queries, and returns the cached answer if the nearest neighbour sits above a configured similarity threshold. The motivation is that natural language requests repeat constantly in meaning and almost never in form — "how do I cancel my plan", "where do I cancel the subscription" and "stop my billing" are three cache misses to an exact-match cache and one hit to a semantic one. Because the cached response costs nothing to generate, hit rate translates directly into lower inference spend and a response that returns in retrieval time rather than generation time. The design has three parameters that matter more than the choice of vector store: the similarity threshold, the scope of the cache key, and the eviction policy. The threshold is the dangerous one. Set it too high and the cache almost never hits, so it adds latency and infrastructure without benefit; set it too low and semantically distinct questions collide — "can I get a refund" and "can I get a discount" are close in embedding space and have different correct answers. The scope matters because a response is only reusable for requests that share the same context: a cache keyed on the user question alone will happily serve one tenant's answer to another tenant, or serve a stale answer that was correct under a document version since replaced. In practice the key should include tenant, locale, model, prompt version and any retrieved-document version, so that changing any of them produces a miss rather than a wrong hit. Eviction is usually time-based, because the underlying knowledge changes even when the question does not. For SaaS builders the honest framing is that semantic caching is a cost and latency optimisation with a correctness cost attached, and it belongs on high-volume, low-variance surfaces — support deflection, FAQ answering, repeated internal lookups — rather than on anything personalised or transactional. Practical note: log every hit with its similarity score and the response it displaced, and sample them for review. Without that record, a threshold that is quietly too permissive shows up as a slow drift in answer quality that no error rate will report.