AI restart tests need to prove the next step can continue

Original research record

We connect new OTR records on saved llama.cpp context, Ray training input and Codex worker reload tests. Each asks a different version of the same question: does the saved state contain what the next action needs? A model’s final state, a list of delivered batches and a worker’s visible history describe different parts of progress. We see a reason to test the first resumed action as carefully as the save itself.

The llama.cpp change makes the distinction clear. A saved final model state cannot recreate earlier recurrent checkpoints, which are saved restore points, so the server now appends the checkpoints needed to roll back when a later prompt diverges. It reports an incomplete checkpoint write as a failed save and can fall back to processing the full prompt when a restored checkpoint cannot load. Ray Data addresses a different layer: its new iterator checkpointer records row identifiers, persistent labels on input rows, and flushes them when a state snapshot is requested. The source explicitly calls this the data half of a larger training-resumption project, and the caller must ensure delivered batches have actually been consumed before saving model and iterator state. Codex adds a separate scenario test that unloads an idle worker and checks the resumed worker’s prior context and unread messages. We infer a shared acceptance rule: identify the state needed for the next step, then test that step after interruption. These are different systems and different kinds of resumption, not evidence of a common defect.

llama.cpp now preserves rollback checkpoints alongside its saved slot payload. Byte counts include the appended state. Damaged or incompatible restored checkpoint data may lead to full prompt processing, so a successful restore response alone does not prove reuse of the checkpoint. Ray’s new data checkpointer requires unique row IDs that persist through data operations. It records delivered batches and assumes the application has consumed them before taking the state snapshot. Ray marks the work as part one and its public configuration as alpha. Model-state coordination and the wider training restart remain separate responsibilities; the implementation also retains a TODO for a barrier across training workers. The Codex source is a regression test for idle eviction and follow-up. It checks retained context and queued messages without starting an extra model turn. It is not a new measurement of durable recovery after an operating-system or machine crash.

We would narrow this view for workloads that do not need recurrent rollback, mid-epoch data resumption or idle-worker reload. We have not run the code or verified stable packages. A later Ray integration may complete more of the training contract, and an application may already coordinate its own saved state. The examples do not quantify data loss, repeated work, recovery time or production reliability.

We would test a save, an interruption and the first resumed action together. For serving, check whether the intended prompt prefix is reused or rebuilt. For training, check that the restored model and data position refer to the same completed step. For an idle agent worker, check that previous context and queued messages are still in the right order. A clear fallback can be safer than pretending incomplete state is a complete recovery. No investment direction applies.

We connect the 11:13 UTC OTR Technology file with its 13:08 UTC file and reopen all three primary records. Ray’s iterator-checkpoint groundwork was committed at 18:29:40 UTC on 7 October and discovered later; we retain that original clock. The llama.cpp save/restore change was committed at 10:18:20 UTC on 8 October, and the Codex scenario test at 10:53:45. These are three project origins. They illustrate separate state requirements and do not count as three reproductions of one failure.

Not a price forecast. Earlier OTR research covered users’ uncertainty about stopping remote work and older model-checkpoint loading costs. This update concerns the minimum state required to resume the next operation. We distinguish saving a model’s weights or current state, saving earlier rollback points, saving which input rows were consumed and preserving queued agent messages. The new evidence supports that technical distinction rather than a new claim about user demand.

The sources make full recovery conditional. llama.cpp can ignore damaged checkpoint additions or drop incompatible draft data and reprocess the prompt. That is a fallback, not proof that all saved fast paths were recovered. Ray’s API is alpha, automatic row-ID generation is not yet supported, and the source requires applications to align consumed input with the model snapshot. Its state_dict code still marks a cross-worker barrier as unfinished. The Codex record adds a test and does not independently establish crash durability or a production success rate.

No investment call. This is a new operational thesis about restart-state completeness. It does not revise the prior tool-discovery or permission-expiry cards. Those asked what can continue and under which authority; this asks what information is needed when work resumes after state has been unloaded or saved.

We avoid turning words such as saved, restored or resumed into a claim that every part of the original workflow survived. Here the evidence is code and tests, not a representative survey of user failures. We have no denominator for lost state, repeated prompts or failed training restarts, and we do not infer that an unrelated user report has the same root cause.

We found no matched resolved forecast sample for this restart-state thesis and provide no success percentage. The three projects are examples with different contracts, so their test cases cannot be pooled into a single probability. This signal has no investible bullish or bearish call. A market-direction percentage does not apply.

OTR Database Data

https://github.com/ggml-org/llama.cpp/commit/033df86b69ec1a333eb241f0c16325a4d43dcff5

https://github.com/ray-project/ray/commit/5ca60cf9ccfa90b7b8a873f88c4628bd105b463d

https://github.com/openai/codex/commit/48bd7667ca270a8ce7e3e1aa57f88a0f54aee15b

For this Technology update we retrieved and hash-checked all 23 newly assigned files. We fully read four findings files and eleven checkpoint, receipt and recovery files. All eight JSONL files were parsed, but only four selected observation records were read in full. Earlier whole-corpus and archive reading gaps remain.

We reopened the cited first-party commits and checked their relevant implementation and tests. We did not execute the code, reproduce benchmarks, verify stable release inclusion or measure production adoption and incident rates.

Code, tests, developer-reported results and our inference remain separate. Different projects provide distinct examples of a shared design question; they do not establish one common bug or an industry-wide failure rate.

This is a non-investment research thesis. Market-direction percentages, asset prices and price-performance results do not apply.

The check date is a suggested point to review new evidence. This card does not create an automated follow-up.