Caching Basics

Cache hits and cache misses

A cache keeps a copy of data that can be obtained from another source. When the required key is already in the cache, we get a cache hit — the fast layer returns the value immediately. If the key is absent, it is a cache miss: the system has to go to the original source.

request
   |
   v
 cache ---- hit ----> response
   |
  miss
   v
primary source

The higher the share of useful hits, the more the cache reduces average latency and load on the primary source. But a large cache alone does not guarantee a high hit rate: what matters is whether requests repeat for the same data.

Why a cache speeds up the system

A cache is usually chosen so that reading from it is cheaper than the original operation. For example, access to memory or a nearby network cache can be faster than a database query, while a ready computed result can be cheaper than repeating the computation.

Suppose a read from the primary source takes 40 ms and a cache read takes 2 ms. With many repeated reads, the benefit appears not only in response time: the primary store performs fewer operations and gains additional load capacity.

The key defines what exactly we reuse

A cached value is usually found by key. The key must identify the data unambiguously, otherwise different requests may accidentally receive the same value.

product:42
user:17
catalog:category:books:page:3

If the result depends on language, user, page, or another parameter, that parameter usually needs to be represented in the key as well.

A cache does not fit every kind of data

The greatest benefit comes from data that is read frequently and changed much less often. If a value is almost never requested twice, the cache only consumes memory. If it changes constantly, the cost of keeping it fresh may exceed the benefit of fast reads.