Picking the right NoSQL database — and using it for chat

1 The NoSQL landscape

First, a quick reminder of why NoSQL exists at all. Recall from our earlier database sessions that a relational database (SQL — like PostgreSQL or MySQL) stores data in tables with fixed columns, enforces a strict schema, and is brilliant at JOINs and transactions. That model dominated for decades. So what went wrong?

Why NoSQL came along

Three pressures pushed teams past the limits of a single SQL server:

• Scale. A relational database is easiest to run on one big machine (vertical scaling — buy a bigger server). But there's a ceiling: the biggest machine money can buy. The web needed to spread data across many cheap machines (horizontal scaling — add more servers), and JOINs across machines are painfully slow.
• Flexible shape. A strict schema means every row has the same columns. But a product catalogue (a book vs. a TV vs. a t-shirt) or a user profile that keeps gaining fields doesn't fit neatly. A flexible/schemaless model lets each record carry its own shape.
• Speed for specific patterns. If you always look data up by one key (a user ID, a session token), you don't need the full power of SQL — you need a lookup that's blisteringly fast. Specialised stores win here.

The four families

Almost every NoSQL database falls into one of four families, organised by the shape of data they store:

Addressability — the unit you read and write

A subtle but important idea ties the whole family together: every storage layer has a unit of addressability — the smallest chunk it can read or write at once. Why? Because all physical memory (HDD, RAM, CPU cache, registers) is byte-addressable, never bit-addressable. That's why a boolean — logically 1 bit — actually takes 1 byte (8 bits, 7 wasted) in Java/C/C++/Rust, and 4 bytes in Python: the hardware simply cannot read or write a single bit. Each database family inherits its own version of this:

2 Key-Value & Document stores

Part A — Key-Value stores (Redis, DynamoDB)

A key-value store is the simplest possible database: a giant dictionary. You store a value under a unique key, and you get it back by handing over that key. The database treats the value as an opaque blob — it usually doesn't look inside it or let you query by its contents.

In the pure model, both the key and the value are just plain strings — the database doesn't know and doesn't care what's inside them. (Most modern key-value stores violate this: Redis values can be arrays, JSON, sets, sorted sets, bloom filters, even custom data structures, while the key stays a string.) A toy example:

Because the only thing it does is "look up by key," it can be extraordinarily fast — and because it's typically in RAM, ridiculously so. The numbers are striking:

Three famous examples sit at different points on the in-memory/on-disk spectrum:

• Redis — in-memory + optional disk persistence, so reads and writes are microseconds. The go-to for caching, session storage, rate limiting, and leaderboards. It also offers richer value types (lists, sets, sorted sets, hashes, bloom filters) — handy, as we'll see in the messaging follow-up.
• Memcached — purely in-memory, no persistence; a classic, very simple cache.
• DynamoDB (AWS) — disk persistence, managed and infinitely scalable. You give it a key, it spreads your data across countless machines automatically, and it stays fast at any size. Great for huge, predictable key lookups like shopping carts and user sessions.

The core API is just three operations:

get(key)            # ⇒ value
set(key, value)     # ⇒ ack / failure
delete(key)         # ⇒ ack / failure

# DON'T do this — it has a race condition
value = key_value_db.get(key)
value += 1
key_value_db.set(key, value)

INCR key            # atomic: no lost updates

# Store a session, auto-expiring after 1 hour (3600 seconds)
SET session:a1b2c3 "{\"userId\":42,\"name\":\"Mei\"}" EX 3600

# Later, on every request: who owns this session token?
GET session:a1b2c3
→ "{\"userId\":42,\"name\":\"Mei\"}"   (returned in microseconds)

# A leaderboard using a Redis sorted set
ZADD leaderboard 980 "Mei"
ZADD leaderboard 1240 "Raj"
ZREVRANGE leaderboard 0 2 WITHSCORES   # top 3 players

Weaknesses (the price of that simplicity): no complex queries (no joins, no filtering), no search, no indexing, no relations. There's also no separate sharding key — the data is automatically sharded by hash(key), so the sharding key = primary key = the key itself.

When to use key-value: you always (or almost always) look up data by a single known key, you need extreme speed, and you don't need to query inside the value. Caching (global, single, or distributed), and storing very simple key→value data that's queried extremely frequently: user preferences, rules, rate-limiter bucket counts, view counts.

Part B — Document stores (MongoDB)

A document store keeps each record as a self-contained document — typically JSON (or its binary cousin BSON in MongoDB). Unlike key-value, the database understands the document's structure, so you can query by any field and even index those fields. Documents are grouped into collections (loosely like SQL tables, but without a forced schema).

The superpower of documents is nesting: related data can live inside one document instead of being split across many tables. A blog post can carry its author info, tags, and comments all in one place — no JOIN needed to read it. The other examples in this family are ElasticSearch, Couchbase, and Firestore. Think of a document store as a collection of JSON/JSONB files distributed across many servers.

{                              {
  _id: uuidv4                    _id: uuidv4
  product_id: int                product_id: int
  name: string                   name: string
  type: "t-shirt"                  type: "laptop"
  brand: string                  brand: string
  color: string                  color: string
  neck_type: byte                ram: {
  sleeve_length: byte              size: integer
  image_url: string                technology: "ddr4/ddr5"
}                                  cas_latency: string
                                 }
                                 cpu: string
                                 image_url: [string, string, string]
                               }

{
  "_id": "post_8842",
  "title": "Why we chose NoSQL",
  "author": { "id": 42, "name": "Mei" },
  "tags": ["databases", "nosql", "backend"],
  "published": true,
  "views": 1503,
  "comments": [
    { "user": "Raj",  "text": "Great post!", "likes": 4 },
    { "user": "Ana",  "text": "Helpful, thanks.", "likes": 1 }
  ]
}

And because the store understands fields, you can query them richly. MongoDB's query language is itself JSON-shaped:

// Find all published posts tagged "nosql", newest first
db.posts.find(
  { published: true, tags: "nosql" }
).sort({ views: -1 })

// Add a new comment to a post (push into the nested array)
db.posts.updateOne(
  { _id: "post_8842" },
  { $push: { comments: { user: "Sam", text: "Nice!", likes: 0 } } }
)

// Index the tags field so the query above is fast
db.posts.createIndex({ tags: 1 })

When to use document stores: your data is naturally nested/object-shaped, the schema evolves often, you need full-text search, and you mostly read a whole record at once but still need to query by its fields. Amazon product listings, social-media posts, user notes, LinkedIn job postings, user reviews, content management, and (very commonly) the "main object store" for a web/mobile app.

3 Wide-Column & Graph stores

Part A — Wide-Column stores (Cassandra, HBase)

"Wide-column," "column-family," and "columnar" all name the same thing. The data is still tabular like a relational DB, but stored in wide-column format (column-wise) instead of SQL's row-wise layout — which makes aggregate queries (sum/avg over a few columns across many rows) very fast. There are still no joins, and no relations between tables. Time-series databases are just a subset (a special type) of wide-column databases.

The headline examples are Cassandra (popular, peer-to-peer, no single master), BigTable (the first NoSQL database ever), ScyllaDB, and HBase (built on Hadoop). They absorb enormous write volumes across hundreds of machines with no single point of failure. Note that every column-family database models data quite differently; the common thread is that all of them store data in tables that are partitioned (sharded) across servers.

The whole model revolves around two kinds of keys baked into the primary key:

CREATE TABLE readings (
  device_id   text,        -- partition key: groups a device's data
  ts          timestamp,   -- clustering key: sorts within the device
  temperature float,
  PRIMARY KEY ((device_id), ts)
) WITH CLUSTERING ORDER BY (ts DESC);   -- newest first

-- Writes scale across the whole cluster (different devices → different nodes)
INSERT INTO readings (device_id, ts, temperature)
VALUES ('sensor-7', '2026-06-20 09:01:00', 21.4);

-- Read is a single fast hit: one partition, already time-sorted
SELECT ts, temperature FROM readings
WHERE device_id = 'sensor-7'
LIMIT 20;

When to use wide-column: you have a huge volume of writes (high write throughput), a known and limited set of query patterns, and you need to scale across many machines with high availability. Analytics, sensor data (GPS coordinates, IoT telemetry), event/activity logs, paginated / time-based queries ("fetch a user's location history for the last month"), and — as we'll see — chat messages.

Part B — Graph databases (Neo4j)

A graph database stores data as a network: nodes (the things — people, products, places) connected by edges (the relationships — "FOLLOWS," "PURCHASED," "FRIENDS_WITH"), and both nodes and edges can carry properties. The star example is Neo4j.

Why a whole separate family? Because relationship-heavy questions are painful in other models. "Friends of friends of friends" in SQL means JOINing a table to itself three times — each JOIN gets exponentially slower. In a graph database, you simply walk the edges, and the cost depends on how many connections you actually traverse, not on the total size of the database. This is sometimes called index-free adjacency: each node directly points at its neighbours.

// Create some people and friendships
CREATE (mei:Person {name: 'Mei'}),
       (raj:Person {name: 'Raj'}),
       (ana:Person {name: 'Ana'}),
       (mei)-[:FRIENDS_WITH]->(raj),
       (raj)-[:FRIENDS_WITH]->(ana);

// "People my friends know, that I'm not already friends with"
MATCH (me:Person {name: 'Mei'})-[:FRIENDS_WITH]->(friend)-[:FRIENDS_WITH]->(suggested)
WHERE NOT (me)-[:FRIENDS_WITH]->(suggested) AND me <> suggested
RETURN suggested.name, count(*) AS mutualFriends
ORDER BY mutualFriends DESC;
// → Ana (1 mutual friend: Raj)

When to use graph databases: when your queries require "path-finding." Recommender systems (Amazon, Netflix), shortest route between two places (Uber, Google Maps), fraud detection (rings of connected accounts), and knowledge graphs.

4 Large-File / Object & other families

Large File / Object storage (S3, GCS, HDFS)

This family stores flat files directly on disk. A file is chunked and distributed across servers, and a single file can even be split across multiple machines. Examples: Amazon S3, Google Cloud Storage, Git Large File Storage (Git LFS), and the Hadoop Distributed File System (HDFS).

When to use: any user-generated multimedia (PDFs, images, videos, audio, CSVs, zips, …); static client-side HTML/CSS/JS; and in general any large files (> ~10 KB) that are mostly static.

Other / niche families

Beyond the big families, "every kid and their grandmother have their own NoSQL database type." A few worth knowing:

5 Choosing the right database

There's no magic formula, but a few questions get you 90% of the way. Ask them in order:

The decision framework

• 1. Access pattern — how do you fetch the data? Always by one key? → key-value. By fields inside a record? → document or SQL. By following relationships? → graph. By a partition + time range, write-heavy? → wide-column.
• 2. Consistency — do reads need to be perfectly up-to-date? Money, inventory, and bookings usually need strong consistency (often a relational DB or a strongly-consistent NoSQL config). A "like" count or a social feed can tolerate eventual consistency (it's fine if it's a second stale). Recall the CAP theorem trade-offs from our earlier sessions: under a network partition you choose consistency or availability.
• 3. Scale — how much data, and how many reads/writes per second? If one beefy SQL server comfortably handles it, you may not need NoSQL at all. If you need to spread across many machines and absorb a firehose of writes, lean toward wide-column or DynamoDB.
• 4. Shape — what does the data look like? Tabular with lots of relations → relational. Self-contained nested objects → document. A web of connections → graph. A simple value behind a key → key-value.

The all-families comparison table

Full worked example: storing tweets per hashtag

Let's run the framework end-to-end on a real Twitter feature.

Now we test each family against those requirements:

• SQL — no. The scale is too large. For a popular hashtag like #US-Elections-2025 or #Diwali, a single server can't even store all the tweets for one hashtag.
• Key-Value — no. You'd use a key like "#Diwali2025:popular" or "#Diwali2025:recent" and the value would be the tweets. But the value is far too large (#Diwali can have 100M+ tweets → multiple GBs), pagination is impossible (you can't fetch "the next 20" out of an opaque blob), and inserting a tweet means rewriting the whole "recent" value — far too costly.
• Document DB — no, for the same reasons. Two shapes both fail: one giant document per hashtag holding a tweets: [...] array hits the document size cap and rewrites the whole document on every tweet; one document per tweet (__doc_id == tweet_id, with hashtag, content, author, like/view counts) fixes the size issue but still leaves you without the cheap time-ordered pagination you need.
• Column Family — yes! (e.g. HBase) — all the requirements match exactly with the strengths of wide-column DBs: huge write throughput, and fast, time-ordered, paginated reads per hashtag.

{                                  {
  __doc_id: ....                     __doc_id: == tweet_id
  hashtag: "Diwali 2025"           hashtag: "Diwali 2025"
  tweets: [                          content: "Crackerless diwali <3"
    {user_id: ..., ...},             author_id: "Krishna"
    {user_id: ..., ...},             likeCount: 1234
    {user_id: ..., ...},             viewCount: 12322
  ]                                }
}                                  // one doc per tweet: better, but
// (too big to fit)                    still no cheap pagination

6 Messaging app follow-up

Back in Session 14 we sketched the architecture of a messaging app (think WhatsApp / Messenger). Now that we know the four NoSQL families, let's make the storage decisions concrete. We'll take three core pieces of data and choose a store for each — a perfect, real example of polyglot persistence.

Piece 1 — Messages: wide-column (Cassandra)

Messages are the heart of the system, and they have a very clear access pattern: an enormous volume of writes (billions a day), almost always read as "the recent messages in this conversation, newest first." That's the textbook case for a wide-column store. We partition by conversation and cluster by time — exactly the pattern from Topic 3.

CREATE TABLE messages_by_conversation (
  conversation_id  uuid,       -- partition key: all of one chat lives together
  message_id       timeuuid,   -- clustering key: time-ordered & unique
  sender_id        uuid,
  body             text,
  PRIMARY KEY ((conversation_id), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);  -- newest first

-- Send a message: a single fast write, lands on the conversation's partition
INSERT INTO messages_by_conversation (conversation_id, message_id, sender_id, body)
VALUES (conv_uuid, now(), mei_uuid, 'Hey, are you free tonight?');

-- Open a chat: read the latest 50 messages in one partition hit
SELECT sender_id, body, message_id FROM messages_by_conversation
WHERE conversation_id = conv_uuid
LIMIT 50;

Piece 2 — Presence: key-value in memory (Redis)

Presence is "who is online right now." It's tiny per user, changes constantly, is read extremely often, and — importantly — it's disposable: if it's a second stale, or even lost on a restart, nobody is harmed. That screams in-memory key-value (Redis) with an automatic expiry (TTL), so a user silently drops to "offline" if their app stops sending heartbeats.

# App sends a heartbeat every ~30s; key auto-expires after 60s
SET presence:user:42 "online" EX 60

# Is Mei online? (microsecond lookup)
GET presence:user:42
→ "online"   (or nil if her heartbeat stopped → treated as offline)

# Optional: keep online friends in a set for "who's online" lists
SADD online:friends:42 7 19 88

Piece 3 — Inbox / conversation list: document store (MongoDB or DynamoDB)

A user's inbox is the list of their conversations with a bit of metadata each: the other participants, a preview of the last message, an unread count, and a timestamp for sorting. It's read every time the app opens, it's naturally an object-per-conversation, and the shape can vary (group chats vs. one-to-one). That's a great fit for a document store, fetched by the user's ID and sorted by most-recent activity.

{
  "_id": "inbox:user42:conv889",
  "userId": 42,
  "conversationId": "conv889",
  "participants": ["Mei", "Raj"],
  "lastMessage": { "text": "Hey, are you free tonight?", "from": "Mei" },
  "unreadCount": 3,
  "updatedAt": "2026-06-20T09:01:00Z"
}

// Load a user's inbox: their conversations, most recent first
db.inbox.find({ userId: 42 }).sort({ updatedAt: -1 })

The whole picture

★ Putting it all together

You can now navigate the whole NoSQL world and make real storage decisions. Here's the one-paragraph story that connects every topic:

Quick self-check

📚 References & Further Reading

Class material

• 📄 Original class notes / handout (Google Doc) — open the shared class material for this session.
• Class handout: "[SST-2028] Types of npSql Databases".

Papers, docs & deep dives

• 📄 MongoDB Documentation — the document-store model, querying nested fields, and indexing.
• 📄 Apache Cassandra — Data Modeling — wide-column partition/clustering keys and query-first design.
• 📄 Redis — Data Types — strings, hashes, sets, and sorted sets behind caching, presence, and leaderboards.
• 📄 Neo4j Documentation — graph data modeling and the Cypher query language for relationship traversal.