What it is
KeplerFS is a peer-to-peer distributed file system. Every node runs the same Bun binary, and that binary acts as an HTTP gateway, a gossip peer (it shares cluster state by sending small messages to random peers), and a chunk store. Any node can accept an upload or serve a download. A SvelteKit web app shows the cluster, the files, and a failure simulator. A team of four built it at the HackX 4.0 hackathon.
The problem
The hackathon problem statement was a distributed file system. Many such systems depend on a master or metadata server, which becomes a single point of failure. KeplerFS removes that role: every node holds the same code and can take over any request. The cost is that nodes must agree on membership and file metadata without a coordinator.
How it works
Each node exposes three interfaces at the same time.
| Interface | Transport | Job |
|---|---|---|
| HTTP gateway | TCP | Uploads, downloads, file listing, cluster status, event stream |
| Gossip | UDP | Peer discovery, failure detection, file manifest and delete notices |
| Chunk transfer | TCP | Direct chunk streaming between nodes |
An upload follows these steps:
- The receiving node reads the upload as a stream and splits it into chunks (4 MiB by default). It hashes each chunk with SHA-256. The hash is the chunk ID.
- The node writes each chunk to disk through a
.tmpfile and a rename, so a crash cannot leave a half-written chunk under its final name. - After the last chunk, the node hashes the whole file to get the file ID. If that file already exists, the node returns the existing record and stores nothing new.
- In one SQLite transaction, the node records the chunks and the file manifest. For each chunk, it asks the consistent hash ring for the owners: the next distinct nodes clockwise from the chunk's hash, up to the replication factor (3 by default).
- For every owner that is not the receiving node, the transaction adds a row to the
replication_queuetable. The node then answers with HTTP 201. - A background pipeline drains the queue and sends each chunk over the transfer port. A row is removed only after the target acknowledges the chunk.
Cluster membership works as follows:
- A node joins through the nodes listed in
SEEDS. On a local network, a UDP multicast beacon (src/discovery/mdns.ts) can also introduce peers. - The gossip protocol is SWIM-inspired. SWIM (Scalable Weakly-consistent Infection-style process group Membership) spreads membership news by periodic random contact. Each node pings three random peers every second. A peer silent for 5 seconds becomes
suspect. After 15 seconds it becomesdead. - When the set of ring members changes, the pipeline re-queues chunks for owners that have not yet confirmed them. It tracks confirmations in a
chunk_locationstable. - A node deletes a chunk it no longer owns only after every current owner has confirmed a copy.
- Deleting a file creates a tombstone that gossip spreads to all nodes. Tombstones expire after 24 hours.
The repository is a pnpm monorepo. packages/node holds the node binary, with one directory per concern: ring/, gossip/, replication/, storage/, transfer/, db/, http/, and chaos/. apps/frontend holds the web app, with dashboard, cluster, files, and chaos views.
What I built
I built the SWIM-inspired gossip protocol for membership and failure detection. I also built the multi-node Docker emulation and its failure-mode demo, plus the log and event streams the dashboard shows. I set up Tailscale so nodes on different devices could reach each other.
Design decisions
No master node. The alternative is a metadata server that tracks where chunks live. KeplerFS instead copies file manifests between nodes by gossip. This removes the single point of failure. The trade-off is eventual consistency: a node can briefly lack a manifest that another node already has.
Consistent hashing with virtual nodes. Each node appears on the ring at 64 positions, derived from the first four bytes of a SHA-256 hash. When a node joins or leaves, only the chunks near its positions change owner. Plain modulo placement would remap most chunks. The trade-off is that every node must hold the full ring, which gossip keeps current.
A SQLite replication queue. An in-memory queue would lose pending copies on a crash. The queue table survives a restart, so the pipeline resumes where it stopped. Failed sends retry with exponential backoff, and a task is dropped after 10 attempts. The trade-off is that a peer which stays down past that limit relies on the ring-change re-queue to receive its chunks later.
Asynchronous replication. The upload handler replies after the local write and the queue insert, not after remote acknowledgements. This keeps uploads fast. The trade-off is a short window where a new file exists on only one node.
A failure simulator in the product. The Chaos Lab routes (/chaos/trigger, /chaos/recover, /chaos/reset) mark nodes as dead in a gossip kill list, block them from rejoining for 30 seconds, and show recovery progress. This makes fault tolerance visible in a demo instead of leaving it as a claim.
Run it locally
You need Bun 1.2 or later and pnpm 10 or later.
-
Install dependencies from the repository root:
pnpm install -
Start a seed node in the first terminal. Each node needs its own
DATA_DIR, because the default./dataholds the node's SQLite file:cd packages/node NODE_ID=node-a HTTP_PORT=3001 GOSSIP_PORT=4001 TRANSFER_PORT=5001 DATA_DIR=/tmp/kepler-a \ pnpm run dev -
Start two more nodes in separate terminals, both from
packages/node:NODE_ID=node-b HTTP_PORT=3002 GOSSIP_PORT=4002 TRANSFER_PORT=5002 DATA_DIR=/tmp/kepler-b \ SEEDS=127.0.0.1:4001 pnpm run devNODE_ID=node-c HTTP_PORT=3003 GOSSIP_PORT=4003 TRANSFER_PORT=5003 DATA_DIR=/tmp/kepler-c \ SEEDS=127.0.0.1:4001 pnpm run dev -
Start the web app in a fourth terminal. Point its proxy at the first node:
cd apps/frontend NODE_HTTP_PORT=3001 pnpm run devOpen
http://localhost:5173.
To run a 20-node cluster in Docker instead, run docker compose up -d --build from the repository root. Node 1 serves its UI at http://localhost:3001, node 2 at port 3002, and so on up to 3020.
To compile the node into one executable, run pnpm run build in packages/node. The output is dist/kepler-node. To run the tests, run bun test in the same directory.
Reference
The node reads these environment variables (packages/node/src/config.ts).
| Variable | Default | Purpose |
|---|---|---|
NODE_ID | random UUID | Node identity on the ring |
HTTP_PORT | 3000 | HTTP gateway |
GOSSIP_PORT | 4000 | UDP gossip |
TRANSFER_PORT | 5000 | TCP chunk transfer |
DATA_DIR | ./data | Chunk files and SQLite database |
SEEDS | empty | Comma-separated host:gossipPort of known peers |
REPLICATION_FACTOR | 3 | Owners per chunk |
CHUNK_SIZE_BYTES | 4194304 | Chunk size, clamped to 256 KiB-64 MiB |
ADVERTISE_HOST | detected | Address announced to peers; prefers private LAN addresses |
MDNS_ENABLED | true | LAN multicast discovery |
The HTTP API accepts requests with or without an /api prefix.
| Endpoint | Purpose |
|---|---|
GET /health, /peers, /stats, /chunks, /replication | Node and cluster state |
GET /events | Server-sent event stream for the web app |
GET /files | List file manifests |
POST /file | Upload a file |
GET /file/<id>, GET /file/<id>/preview, DELETE /file/<id> | Download, preview, delete |
The stack:
| Layer | Choice | Use |
|---|---|---|
| Node runtime | Bun, TypeScript | HTTP server, UDP and TCP sockets, single-binary build |
| Local state | SQLite | Chunks, manifests, peers, tombstones, replication queue |
| Web app | SvelteKit, Svelte 5, shadcn-svelte | Cluster, file, and chaos views |
Outcome
The team placed second at HackX 4.0, a 24-hour hackathon held on 28 February 2026 at NMIMS, Navi Mumbai. The repository was created on 28 February 2026 and has 76 commits through 2 March 2026. It uses the MIT license. The team was Yash Kolekar, Jay Kerkar, Sundaram Krishnan, and Aayush Nair.