mini-redis: Building Redis From Scratch to Understand What It Actually Does

The Problem

Using Redis and understanding Redis are two different things. I had used Redis for caching, rate limiting, and task queues across several projects, but all of that sat on top of a library call. I wanted to answer three questions concretely instead of accepting them as facts I had read somewhere: what does "Redis speaks a protocol" actually mean, byte for byte. How does a key expire when nothing is running on a timer. And how does data survive a restart when it all lives in RAM.

The only way to actually know is to build the thing and hit the problems it hits.

What I Built

A TCP server that parses the real RESP protocol (the same length prefixed format actual Redis clients send), applies commands against an in memory store, and replies in the correct RESP format for each reply type.

redis-cli -p 6399 SET name malahim
redis-cli -p 6399 GET name
"malahim"

That is the real redis-cli, talking to code I wrote, with zero compatibility shims in between.

Architecture

• Protocol layer: a hand written RESP parser. Reads a command like *3\r\n$3\r\nSET\r\n$3\r\nfoo\r\n$3\r\nbar\r\n off the wire and turns it into ['SET', 'foo', 'bar']. Uses length prefixed reads rather than splitting on whitespace, since a value containing spaces or newlines would break a naive parser.
• Storage layer: a plain dictionary, plus a second dictionary tracking expiry timestamps. Expiry is lazy: a key's TTL is only checked, and only cleaned up, when something actually asks about that key, not on a background timer. This mirrors a real design choice Redis itself makes.
• Persistence layer: an append only log. Every write command gets appended to a file, RESP encoded, in the same bytes a real client would have sent. On startup, the log replays command by command to rebuild state. The same parser function used for reading commands off a live socket gets reused unchanged to read commands back out of the log file, since both are just something with readline and read.
• Concurrency layer: each client connection runs on its own thread, protected by a re entrant lock around every function that touches shared state, so two clients hitting the same key at once can't corrupt it.

Why this approach

I deliberately avoided any existing Redis client library or protocol helper. The entire point was to hit the actual problems (parsing wire bytes, deciding how expiry works without a timer, deciding what happens to a log file that only ever grows) rather than reading about how someone else solved them.

Hard Parts / What I Learned

Two real bugs, both worth remembering.

The append only log re logging itself on every restart. My replay function reused the same execute function used for live client commands, which also logs every write. That meant every restart replayed N commands from the log and re logged all of them, so the log doubled in size every time the server restarted. A few restarts later the file had over a million lines. The fix was splitting the command handler into two functions: one that only touches the store and never logs, and one that logs and then calls the first. Replay only ever calls the non logging version. This is a real trap in anything built around an append only log or event sourcing pattern: replaying history must never generate new history.

A lock that deadlocked itself. Once I added threading, get_value needed to check expiry, which is its own function, and both needed the same lock. A plain lock deadlocks the moment the same thread tries to acquire it twice in a row, which is exactly what happens when one locked function calls another locked function on the same thread. Switching to a re entrant lock, which allows the same thread to reacquire a lock it already holds, fixed it.

Neither bug would show up from reading about Redis. Both only show up from building it and watching it misbehave.

What This Demonstrates, Honestly

This is not a Redis replacement, and I am direct about that in the repo. There is no fsync control, no memory eviction policy, no authentication, and threading only helps with I/O waits rather than giving true parallel CPU work under Python's GIL. It is genuinely useful as a local development dependency for small personal projects, or as a demo, not as something serving real production traffic.

What it actually proves is different from what it does. I now know why Redis is single threaded for command execution by design, not limitation, because I felt how much simpler the logic gets without two commands racing on the same dictionary. I know the actual tradeoff between an append only log and snapshotting, because I broke the append only version and had to fix it. That is a different kind of knowing than having read it in documentation.

Try It / Links

• Repo: github.com/MalahimHaseeb/mini-redis
• Stack: Python, raw sockets, threading, hand written RESP protocol