Distributed Cache

One shared cache for several application instances

A local in-process cache is very fast, but every application instance keeps its own copy. When requests are distributed across several servers, their contents can differ, and total memory is spent on duplicates.

A distributed cache moves data into a separate network layer that all application instances access.

app-1 ----\
app-2 -----+----> distributed cache
app-3 ----/

A shared layer makes data reuse easier

If one instance populates a key, the next request on another instance can already receive the same cached value. This is especially useful when the application scales horizontally and requests from one user are not pinned to a particular server.

At the same time, a network cache is usually slower than local process memory, so “distributed” does not mean “free.” It trades some latency for shared capacity and one key space.

Data has to be distributed across nodes

A large distributed cache usually consists of several nodes. Keys need to be spread across them so that memory and load are reasonably balanced. If one very popular key receives a huge share of requests, a particular node can become a hot spot.

Additional copies of data can be used to survive failures, but they consume memory and add work for maintaining consistent state between nodes.

The cache can fail too

The application should not assume that a network cache is always available. On a timeout or partial cluster outage, the system should already know whether it can access the primary source, return a degraded response, or temporarily reject the request.

A large-scale fallback to the primary source after a cache outage is especially dangerous: load that was previously absorbed by the cache suddenly reaches the database. That is why cache failure is part of capacity planning for the whole system.