Caching: how systems remember answers so they don't redo work

1 What is caching & why we do it

A cache is a small, fast store that holds copies of data that's expensive to fetch or compute, kept closer to whoever needs it. Caching is the act of putting answers in that store the first time, then reusing them. The goal is simple and always the same: cut latency (make things feel fast) and cut load (stop hammering the slow thing behind it).

The "slow thing behind it" has a name: the origin (or source of truth) — the real, authoritative place the data lives, like a database or another service. A cache never replaces the origin; it just sits in front of it and answers the easy, repeated questions.

Hits, misses, and the hit ratio

Every time something asks the cache for data, exactly one of two things happens:

• A cache hit — the data was already in the cache. Fast. We avoided the slow trip.
• A cache miss — the data wasn't there. Now we do make the slow trip to the origin, fetch it, and usually store a copy in the cache so the next request hits.

The single most important number for any cache is its hit ratio (also called hit rate): the fraction of requests that were hits.

# hit ratio = hits / total requests
hit_ratio = 9200 / 10000   # = 0.92  → 92%
miss_ratio = 800 / 10000   # = 0.08  →  8%
miss_ratio = 1 - hit_ratio  # always true

# average = (hit% × hit_cost) + (miss% × miss_cost)
avg = (0.92 * 1) + (0.08 * 50)
avg = 0.92 + 4.0 = 4.92 ms   # vs 50 ms with no cache

Latency-numbers intuition: why "closer and faster" matters so much

To feel why caching pays off, you need a gut sense of how wildly different "fast" and "slow" are inside a computer. These are the famous latency numbers every engineer should know — rounded to make the ratios obvious, not to be exact:

Why caching exists at all: memory is always hierarchical

Here's the deep reason caching is unavoidable — it shows up even in the human brain. Think about making a cup of tea: you keep the tea, sugar, and a mug on the counter (instant), the spare packets in a cupboard (a few steps), and the bulk supply in a shop across town (a whole trip). You naturally keep what you use most in the closest, smallest spot.

Your brain does the same thing with three tiers of memory:

When you sleep (during REM), your brain consolidates short-term memories into long-term storage — exactly like flushing a fast cache down to slow, durable storage.

The same hierarchy inside a computer

Your machine stacks memory the same way, from blistering and tiny to vast and slow:

So a CPU register is roughly 500× faster than RAM, and RAM is roughly 100,000× faster than a spinning disk. Climbing each rung of this ladder — disk → RAM → CPU cache → registers — is an order-of-magnitude win, which is the whole reason caching pays off.

2 The caching hierarchy: layers from browser to backend

Caching doesn't happen in just one place. A single request can pass through several caches stacked in a line, each closer to the user than the last. This stack is the caching hierarchy. The closer a layer is to the user, the faster and cheaper a hit there is — so we want to answer requests as early (as far "left") as possible.

The three big layers

Two terms worth pinning down here. A CDN (Content Delivery Network) is a fleet of cache servers scattered across the globe; each location is called an edge because it sits at the edge of the network, close to users. When someone in Tokyo and someone in Berlin request the same image, each is served from a nearby edge rather than from one origin server on the other side of the planet — turning a ~150 ms cross-continent trip (Topic 1) into a few milliseconds. (We go much deeper on CDNs in Topic 3.)

Zooming into the browser layer (client side)

The browser cache is really several caches. Some of them a frontend developer cannot touch, and some they can:

End-to-end: a request falling through the layers

Here's the journey of a single request when each layer in turn misses, all the way down to the origin:

The data then flows back up the chain, and each layer can store a copy on the way so future requests stop earlier. The very next identical request might be answered right at the browser and never touch the network at all.

3 CDNs deep dive: serving the world from the edge

What does a CDN cache, and what problem does it solve?

A CDN caches large (> ~10 KB), mostly static data — content that changes rarely:

• Multimedia: images, audio, videos, PDFs, zip files, …
• Code: JS, HTML, CSS.
• Essentially all user-uploaded content (images, videos, reels, avatars) is served via CDNs.

A CDN solves two problems at once: it acts as a cache and it reduces latency by serving from a server close to the user.

Why "close" matters: the US ⇄ India round trip

Imagine you're watching a Scaler lecture recording. Scaler's servers are in Mumbai (AWS South-East Asia), but you live in the US. How long does just one packet take to make the round trip?

circumference of earth  ≈ 40,000 km
speed of light (vacuum) c = 299,792,458 m/s ≈ 3 × 10^8 m/s
but inside fiber-optic cable, light is slower ≈ 2 × 10^8 m/s

RTT = 40,000 km / (2 × 10^8 m/s)
    = (4 × 10^4 × 10^3 m) / (2 × 10^8 m/s)
    = 4 / 20 seconds
    = 0.2 s = 200 ms

Two definitions to keep straight:

• Latency = the delay between sending a request and receiving a response.
• Bandwidth = bytes per second (throughput).

Fetching from far away hurts both: latency is high, and bandwidth is low too. Data crosses many hops, and your bandwidth is limited by the slowest server on the route. The intercontinental fiber-optic cables that carry trans-Atlantic and trans-Pacific traffic are a real bottleneck — all cross-continent traffic squeezes through them.

The fix: what if you fetched that video from a server near your home instead? Latency would drop to a few ms and bandwidth would be far higher. A server physically/geographically close to the user is called an edge node (or edge server); "the edge" simply means close to the user.

How do you find the nearest edge server?

Are CDNs databases?

How does the CDN get the data in the first place?

Walk through an upload, using the handout's cast. Abdul uploads a video:

Concretely, the backend asks the CDN: "give me a CDN URL for this database URL s3.aws.com/1234/video.mp4", and the CDN returns something like cdn.akamai.com/a3b2fd5.mp4, storing the mapping CDN URL → database URL internally.

Now Upinta wants to watch the video:

• She requests the page from Facebook's backend, which returns an HTML+JS+CSS page.
• The crucial choice is which URL the backend embeds in that page:

<html>
  <body>
    <!-- BAD: the raw database URL — far away, slow, high latency -->
    <video src="s3.aws.com/1234/video.mp4" />

    <!-- GOOD: the CDN URL — served from the nearest edge node -->
    <video src="cdn.akamai.com/a3b2fd5.mp4" />
  </body>
</html>

If the embedded URL is the database URL, Upinta (far from the DB servers) gets slow downloads and high latency. So the backend must embed the CDN URL. The edge node then:

Finally, Pankaj (Upinta's neighbour) wants the same video. He hits the same edge node, but this time the file is already cached — because Upinta just watched it — so the edge simply serves it. Same edge, instant delivery.

When does the CDN actually cache the file?

Can a CDN be a bottleneck?

4 Types of backend caches: local vs distributed

Inside the datacenter, backend caches come in two fundamental shapes.

Local (in-process) cache — the private notebook

A local cache lives inside the application process itself, in that server's own memory (RAM). Because it's in-process — the same program, the same memory — a lookup is just a memory read: blisteringly fast, no network involved.

The catch: it's per-server. If you run 10 app servers behind a load balancer (recall load balancing from Session 2), you have 10 separate caches that don't know about each other. Each holds its own copy, and they can disagree.

# A simple per-server, in-process cache
cache = {}  # lives in THIS server's RAM only

def get_user(user_id):
    if user_id in cache:        # cache hit
        return cache[user_id]
    user = db.query("SELECT * FROM users WHERE id = ?", user_id)  # miss → origin
    cache[user_id] = user        # store copy for next time
    return user

Global / distributed cache — the shared whiteboard

A global cache (also called a distributed cache) is a separate, shared service that all your app servers talk to over the network. The two famous examples are Redis and Memcached — fast in-memory key-value stores that run as their own servers. Every app server reads and writes the same cache, so they all see one consistent copy.

The cost: every lookup is now a network round trip (a fraction of a millisecond inside a datacenter — slower than local RAM, but still vastly faster than a database query or disk read). "Distributed" also hints that the cache itself can be spread across many machines — and the trick for spreading keys evenly across them is consistent hashing, which you met in Session 2.

A global cache sits as a separate layer between the app servers and the DB, and all app servers can reach it. The classic cache-aside read pattern against a Redis cache looks like this:

def get_user_preferences(request):
    user_id = request.user_id
    preferences = redisClient.getKey('pref:user_id')
    if preferences is None:                 # cache miss
        preferences = sqlClient.getPreferencesForUser(user_id)
        redisClient.setKey('pref:user_id', preferences)
    return preferences

A local cache uses the exact same logic, but the store is just an in-memory dictionary that lives in this server's RAM — no network call:

user_preferences_cache = {}   # stored in THIS server's RAM

def get_user_preferences(request):
    user_id = request.user_id
    preferences = user_preferences_cache.getKey('pref:user_id')
    if preferences is None:                 # cache miss
        preferences = sqlClient.getPreferencesForUser(user_id)
        user_preferences_cache.setKey('pref:user_id', preferences)
    return preferences

Single vs distributed cache

A quick clarification of words you'll see: a single cache means one cache node (everything fits on one machine — simplest, but limited by that one box's memory and a single point of failure). A distributed cache splits the data across many nodes — you reach for it when one cache server isn't enough, either because you need to store more data or improve read throughput.

Pros and cons, side by side

5 The two hard problems (an introduction)

Caching sounds easy — keep a copy, reuse it. But two questions turn out to be deep, and they haunt every cache ever built. A famous joke captures it: "There are only two hard things in computer science: cache invalidation and naming things." Let's meet both problems so you know exactly what Session 4 is about to solve.

Both stem from the same root fact: a cache is much smaller than the database.

Hard problem #1 — Eviction: what do we remove when the cache is full?

A cache is small on purpose (fast memory is limited and costs money). So it can't hold everything. When it's full and a new item needs space, the cache must throw something out. That decision is called eviction, and the rule it follows is an eviction policy.

Why is choosing hard? Because you're guessing the future: you want to evict the item you're least likely to need again — but you can't see the future, so you use clues from the past. Common policies you'll meet next session include:

Evict too aggressively and your hit ratio tanks (you keep throwing out things you'll need). Keep too much and you run out of memory. There's no policy that's best for every workload — which is exactly why it's a "hard problem."

Hard problem #2 — Invalidation: keeping the copy fresh

The cache holds a copy. The origin can change at any moment. The instant it does, every cached copy is potentially stale — out of date. Cache invalidation is the job of making sure stale copies get refreshed or removed, so users don't see old data.

This is the tug-of-war at the centre of caching:

★ Putting it all together

You just built the entire foundation of caching. Here's the one-paragraph story that connects all four topics:

Quick self-check

📚 References & Further Reading

Class material

• 📄 Original class notes / handout (Google Doc) — open the shared class material for this session.
• 📝 HLD101 Assignment — Search Typeahead — the assignment shared for this session.
• Class handout: "[SST-2028] Intro to Caching".

Papers, docs & deep dives

• Latency Numbers Every Programmer Should Know — the latency ladder behind why caching pays off.
• Redis developer docs — using Redis as a fast in-memory distributed cache.
• MDN: HTTP caching — how browser and proxy caches work via HTTP headers.
• Cloudflare: What is a CDN? — edge caching and serving content close to users.

CDN docs & deep dives (from the handout)

• AWS: What is a CDN?
• Akamai: What is a CDN?
• Cloudflare: AnyCast network — how the nearest edge is reached.
• Quick CDN setup guide (CloudFront).
• CloudFront: invalidate files to remove content — CDN cache invalidation.

How storage & memory work (videos / extras)

• "How do Hard Disk Drives Work?", "How do SSDs Work?", "How does Computer Memory (RAM) Work?" — the storage tiers behind the latency ladder.
• "Information Storage and the Brain: Learning and Memory", "How We Make Memories (Crash Course Psychology #13)", "Tools to Enhance Working Memory & Attention" — the biological memory hierarchy.
• "CppCon 2016: Timur Doumler — Want fast C++? Know your hardware!" — optimizing code for efficient CPU-cache usage.
• "JavaScript Cookies vs Local Storage vs Session Storage" — the browser-side stores compared.