ml-infra-platform

Three correct systems
do not make one correct system.

A Raft consensus engine, an HNSW vector index, and a distributed training framework, each already tested on its own, composed into one platform: models train, their embeddings are replicated through consensus into a vector store, and a control plane runs both. The engines are unmodified. Everything interesting happens at the seams between them, so that is where this project does its testing.

Abhineeth Duddela generated 2026-08-26T20:39:00+00:00 python 3.13.0 numpy 2.5.1 90.9s to reproduce

Watch the recorded session → Source

8/8named scenarioscross-layer fault sequences

60/60fuzz seeds cleanrandomized fault schedules

2,032embeddingswritten through Raft, then read back

0acked but lostthe failure this project exists to detect

488invariant checksrun continuously, not at the end

5overlapping faultsin two layers at once, measured

3bugs foundin the composition, all fixed

3tick read costlinearizable over stale

Every figure on this page was produced by python run_experiments.py and read out of site/results.json. Nothing here is typed in by hand or estimated; where a value was not measured, the page says so.

the claim

What is actually being proven

The three engines come with their own suites, and they pass: 23 tests for Raft, 28 for HNSW, 68 for the training framework. None of them says anything about what happens when a training worker dies at the same moment the storage cluster splits in half.

That is the gap. A component proof is a statement about a component under its own assumptions; composing components creates states that neither one's assumptions cover. The first bug below is exactly that shape: Raft behaved correctly, HNSW behaved correctly, and the platform crashed every replica in the cluster.

The write path, the read path, and the acknowledgement path. An embedding is acknowledged to the training job only once it is quorum-committed under the leader's current term; each replica applies the same committed log into its own HNSW index, so the indices are built independently and must agree by construction rather than by copying.

consistency

The guarantee, stated exactly

The storage engine on its own offers reads from local state, which can lag. Linearizable reads are implemented in this platform as a ReadIndex protocol layered on top of it, not inherited from it.

operation	guarantee	cost	under partition
write	Linearizable. Acknowledged only after quorum commit in the leader's current term.	27 ticks p50	Minority side cannot commit; the write returns unacknowledged and is never claimed as durable.
read: linearizable	Reflects every write acknowledged before the read began. Leadership is confirmed by a heartbeat quorum round, then the read waits for `applied ≥ read index`.	3 ticks, 0.563 ms	Refuses. Leadership cannot be confirmed, so the read fails rather than serving something it cannot vouch for.
read: stale	Served from any replica's local index with no coordination. May lag arbitrarily. Opt-in, never the default.	0 ticks, 0.421 ms	Succeeds, possibly with old data. That is the mode's contract, so it is not counted as a violation.

A linearizable read pays a heartbeat quorum round to confirm leadership; a stale read pays nothing and may lag. There is no leader lease here, so every linearizable read pays the full round trip -- a real system amortises that.

the recorded run

five nodes, one training job, 3 injected faults across both layers

130 frames captured from one real execution: a 4-worker job publishing embeddings through consensus while a node dies, the cluster splits, and a training worker is killed mid-partition. The dashboard replays these frames; it is the same interface that runs live against the control plane, with recorded frames substituted for polled ones.

Cluster health per frame (green healthy, amber degraded), with each injected fault marked in red and each recovery in green, and the running count of acknowledged embeddings in violet. The violet curve never falls: no acknowledged embedding was ever withdrawn, including across the partition and the leadership change.

acknowledged

71 embeddings

Every one still queryable from the leader after the faults healed: 71 of 71.

in flight when a node died

1 lost

Never acknowledged, so never claimed. This is correct behaviour and the direct analogue of an uncommitted Raft entry, so it is reported rather than counted as a violation.

after settling

converged

All five replicas returned identical index digests, having each built their HNSW graph independently from the same committed log.

the job itself

0.9900 accuracy

The model still trained. Losing a worker mid-partition reshards onto the survivors rather than failing the job.

Open the replay →

invariants

Six properties that only exist at the seams

Checked after every round of every scenario and every fuzz seed, not once at the end. 488 checks across the named scenarios alone.

1 · durability

Acknowledged writes survive leadership

An embedding acknowledged to a training job is present on every leader elected afterwards.

2 · convergence

Replicas agree on their common prefix

Two in-sync replicas hold identical applied sequences. Catches seed divergence and apply-order drift.

3 · watermark

The index matches what the replica claims

A replica's vector count matches the committed prefix it says it applied. Catches a reset that drops one but not the other.

4 · no phantoms

Nothing searchable was uncommitted

Every vector in an index traces to a committed log entry.

5 · exactly once

No embedding stored twice

Re-applying a log prefix after restart must not duplicate. This is the invariant bug 1 violated.

6 · read honesty

A linearizable read never lies

The state served is no older than any write acknowledged before the read began, or the read fails.

Deliberately not violations

A stale read lagging — that is the mode's contract, and counting it would make the mode meaningless.
A minority-side linearizable read failing — refusing is the correct answer and the entire point of ReadIndex.
An unacknowledged in-flight write vanishing when its node dies — correct, and reported separately as lost uncommitted so it stays visible rather than hidden.

Each of the six has a mutation test that breaks it deliberately and asserts the checker fires and names the right one. A correctness harness never observed to fail is not evidence; every “0 violations” on this page is worth exactly as much as those tests.

results

Cross-layer scenarios

Named fault sequences, each run against a live training job rather than an idle cluster. The overlaps column is measured from the schedule and counts fault windows that were open in both layers at the same instant — so it is worth reading it honestly: most of these rows are a storage fault landing while training continues, and only the ones with a non-zero count are genuinely simultaneous cross-layer failures. Reporting that column is what stops the headline claim from going quietly vacuous.

scenario	overlaps	acked	queryable	lost in flight	converged	result
No faults none	0	40	40	0	yes	pass
Worker dies while storage is partitioned Partition [n1,n2]\|[n3,n4,n5] rounds 15-45; KillWorker at 20	1	40	40	0	yes	pass
Storage node dies mid-write KillNode n2@10, KillNode on the leader@25	0	40	40	0	yes	pass
Slow link during publication DelayLink n1->n3 and n1->n4 by 8 ticks, rounds 12-40	0	40	40	0	yes	pass
Cascading failures across both layers KillNode n5@8; KillWorker@14; Partition rounds 20-38; KillWorker@26; RestartNode n5@42	1	39	39	1	yes	pass
Leader isolated on the minority side Partition [n1]\|[n2..n5] rounds 18-50	0	40	40	0	yes	pass
Replica rejoins after missing many writes KillNode n4@6, RestartNode n4@45	0	40	40	0	yes	pass
Simultaneous faults in both layers KillNode n5@10; KillWorker@10; Partition rounds 10-40; DelayLink n1->n2 rounds 10-40	3	40	40	0	yes	pass

Randomized sweep

60/60seeds cleaneach a different fault schedule

1,713embeddingsacknowledged across the sweep

0acked, not queryablethe headline failure mode

0.9240lowest accuracyworst-case model, under faults

Faults are bounded so a quorum always survives; a cluster deliberately driven below quorum stops being able to commit, which tests nothing about the composition. A failing seed replays exactly: python -m mlplat.chaos.fuzz.

Cross-validated against real sockets

Everything above runs on a deterministic queued transport. That is what makes a fault reproducible from a seed, but it is also a modelling assumption, and an assumption nobody checks is just a hope: if the storage layer only behaves because messages arrive in a tidy order no real network would produce, these results describe the harness rather than the system.

So the same VectorStateMachine is also run under the Raft engine's real asyncio TCP server — three processes on localhost, real sockets, real election timeouts — and driven through the engine's own client. Replicas converge to identical index digests, acknowledged writes survive a node dying, and the duplicate-id fix from bug 1 holds. Reverting that fix reproduces the original ValueError inside the server's apply path over TCP too, which is how these three tests are known not to be vacuous.

64tests passingmeasured by running them, not recalled

0failingat the time this page was generated

3over real TCPcross-validating the deterministic transport

6mutation testsone per invariant, each expects a catch

honestly

Bugs found in the composition

Three, all fixed, each with a regression test. Two were mine. One was in the correctness checker itself, which is worth stating plainly: the only violation the sweep ever reported turned out to be the harness being wrong, not the platform.

1. The same embedding id committed at two log indices

found by: randomized cross-layer sweep, during a partition
symptom: ValueError: id 'job:s24@48' already exists raised inside apply, on every replica at once.
cause: The client proposed a write, timed out waiting for the commit, and re-proposed it. When the partition healed, both entries committed. The state machine deduped by log-index watermark, which cannot help here: both indices are above the watermark, so both look new. VectorDB.insert then refused the duplicate id and raised.
whose bug: Neither engine. Raft committed two entries because it was asked to twice; HNSW refused a duplicate because it should. The defect was treating an unconfirmed proposal as safe to retry, which is a claim about at-most-once delivery that Raft never made.
fix: Two halves, both necessary. The client now returns an unconfirmed result with its log index instead of re-proposing. The state machine additionally suppresses a repeated id, because a state machine over a replicated log must be total: an exception during apply does not reject one bad write, it stops that replica applying anything further while its peers continue.
regression: test_regression_same_id_committed_twice_does_not_kill_a_replica, test_regression_client_does_not_repropose_an_unconfirmed_write
file: storage/client.py + storage/statemachine.py

2. A shared event log silently replaced by an empty one

found by: the cross-layer timeline test, which saw zero event kinds
symptom: Storage and training events landed in three separate logs, so the unified timeline the whole dashboard depends on was empty for every layer but one.
cause: EventLog defines __len__, so an empty log is falsy. Three constructors used self.events = events or EventLog(), which discards the caller's log precisely when it is new and empty, which is always at startup.
whose bug: Mine, in all three places, from one idiom copied forward.
fix: events if events is not None else EventLog(), everywhere an optional shared object is accepted.
regression: test_events_from_both_layers_share_one_timeline
file: storage/cluster.py, training/job.py, training/publisher.py

3. The correctness checker was testing the wrong property

found by: fuzz seed 57, which the sweep reported as a violation
symptom: A linearizable read was flagged as dishonest: 8 acknowledged ids, only 6 returned by a k-NN query at k=13.
cause: The invariant asserted that a k-NN query returns every acknowledged id. HNSW is an approximate index and a k-NN query is not an enumeration; replaying the seed showed all 8 ids present in the serving replica. The check conflated recall (an approximation property) with staleness (a consistency property).
whose bug: The harness, not the platform. Worth stating plainly: the one violation this project's sweep ever reported was the checker's own bug, which is exactly why the mutation tests exist.
fix: Invariant 6 now asserts containment in the state actually read from, which is the consistency claim. Recall is recorded as an observation alongside it rather than asserted.
regression: test_regression_read_honesty_tests_containment_not_search_recall
file: chaos/checker.py, invariant 6

measured

Benchmarks

Wall-clock figures are from one machine and one process, and are the least transferable numbers here. Tick counts are protocol costs and are the ones that would carry to a real deployment, so they are reported alongside everywhere.

End-to-end: training step to queryable embedding

38.50ms p50checkpoint published and read back

73.33ms p95worst of the sampled checkpoints

27ticks p50the protocol cost

12/12queryableevery checkpoint, confirmed by a real read

wall_ms is this machine; consensus_ticks is the protocol cost and is what would carry to a real deployment. Each sample covers one checkpoint of embeddings plus a linearizable read confirming them.

Scaling

Storage width — more nodes, 4 workers fixed

nodes	quorum	acked	ticks	ticks/write	wall s
3	2	24	119	4.96	0.061
5	3	24	117	4.88	0.089
7	4	24	117	4.88	0.123
9	5	24	116	4.83	0.153

Training width — more workers, 5 nodes fixed

workers	rounds	wall s	accuracy	acked
2	30	0.096	0.9676	24
4	30	0.106	0.9726	24
8	30	0.165	0.9689	24

Storage and training width are swept separately: more nodes make a write costlier, more workers make a step cheaper, and sweeping together would cancel them out.

Recovery

fault	recovered	median ticks	max ticks
follower killed	3/3	3	3
leader killed	3/3	15	16
one node partitioned off	3/3	3	3
leader isolated	3/3	108	108
leader + one follower killed	3/3	32	40

Ticks from the fault to the first write that COMMITS after it, retrying with a fresh id each attempt. Time to elect a leader would be smaller and less honest: a cluster with a leader that cannot yet accept writes has not recovered. The client's commit timeout is 25 ticks and is part of the number -- at the default 200 the client's own patience dominated and the rows stopped being comparable.

limits

What this does not do

No snapshots. Neither the Raft engine nor this platform truncates the log, so a rejoining replica catches up by full replay and the log grows without bound. Fine at these sizes; the first thing a real deployment would need.
No leader lease. Every linearizable read pays a full heartbeat quorum round, which is why the measured overhead is 3 ticks rather than zero. A real system amortises that across a lease interval.
Deterministic transport, not a network. The cluster runs on a queued transport with modelled latency, drops and partitions. That is what makes a fault reproducible from a seed, and it is also why the wall-clock numbers are not deployment numbers.
Deletion is soft. Inherited from the vector engine. Vectors are tombstoned, not reclaimed.
One shard. Every replica holds the entire index, so storage capacity is one node's capacity. Sharding would be a replication-group-per-shard change, not a change to any of this.

reproduce

Run it yourself

# the three engines are cloned in as read-only dependencies
git clone https://github.com/abho7/ml-infra-platform && cd ml-infra-platform
pip install numpy pytest

# every test, including the mutation tests behind every claim above
pytest -q

# the live control plane, with a real dashboard at localhost:8950
python -m mlplat.control.cli serve

# regenerate every number on this page
python run_experiments.py && python build_site.py

Dependencies are numpy and pytest. The Raft engine, the HNSW index, the training framework, the transport, the control plane and its HTTP API are all from scratch.