An AI answer can score 100% when only one option is allowed

Original research record

vLLM’s new decision API separates an option’s share of the listed answers from the model’s wider token probabilities. A 100% share within a one-option list does not establish that the answer is correct. We connect this with an earlier change that makes empty token lists explicit in streamed replies. Both give an automated client evidence it needs to interpret carefully.

We see a new integration risk where AI answer scores become rules for taking action. vLLM’s new structured-decision API labels the supplied options A, B, C and so on, then reads probabilities for the first reply token. It rescales the listed labels to total 100%. It also returns their total probability across the vocabulary and a confidence field for the chosen label. The choice function picks the leading listed option, with no low-confidence stop rule in that function. We connect this with a separate vLLM streaming fix: when token IDs are requested, every reply chunk now contains that field, including an empty list where there are no output tokens. Both changes let clients see what their machine-readable evidence actually contains. Our inference is that clients should preserve empty-output states and what each percentage is out of, then use a separately tested rule for acting or deferring. These changes concern separate endpoints; neither measures real-world decision accuracy.

In an upstream unit-test example, one allowed option is given a listed-answer probability of 1.0 and a label mass of 0.4. The expected confidence is 0.4. These are supplied test values and expected results, not a measured 40% success rate or a forecast.

The API defines label_mass as the total first-token probability assigned to the supplied labels across the full vocabulary. Its confidence is the chosen option’s share within those labels multiplied by label_mass. Neither field is a measured chance that an external event will happen.

The source also reports whether the model’s top token belongs to the listed labels. We found no low-label-mass rejection in the choice function. A client can use the diagnostics to defer, but the inspected code does not establish a safe threshold for a particular task.

The separate Rust streaming test expects three chunks: an empty token list, the generated token IDs, then another empty list. The fix makes those states explicit when requested. It does not prove that a client records them or uses them correctly.

The API already exposes the wider probability information, so we are not claiming it hides that information. A well-defined option list may be useful for a bounded task. We have not found a documented customer decision caused by confusing these fields, measured adoption, or run the upstream tests. We would revise the operational advice after task-specific testing shows how diagnostics and defer rules affect errors and useful coverage.

We would label the score as a share of the supplied options, retain label_mass and token diagnostics, and provide a clear defer path when the task needs one. Test the client with a one-option list and with a likely answer missing. Measure decisions against known outcomes before setting an action threshold. A neat response and a 100% score inside a list do not supply that outcome record.

This signal has no investible bullish or bearish call. A market-direction percentage does not apply. Bayesian · what changed the estimate? The API provides model token scores; we have no event-specific prior, likelihood model or fitted outcome calibration to turn them into a real-world forecast. de Finetti · are the comparisons fair? One-option tests, different prompts and different deployments are not interchangeable forecast trials. We do not pool them into a success rate. Kolmogorov · do the numbers agree? The supplied label probabilities total one because they are rescaled over that list. That is a coherent distribution over labels, not proof the list covers every possible real-world answer. Horizon, price basis and market neutral band: Not applicable. Research gaps: outcome calibration, client behaviour and release reach remain unverified.

We connect two OTR Database Data research batches and reopen their exact sources. A vLLM Rust completion-stream change was committed at 11:11:55 UTC on 8 October. The structured-decision endpoint was committed at 14:48:12 UTC. We checked the streaming serialization test, plus the new endpoint’s routing, strategy, answer construction, diagnostics and relevant unit and end-to-end test definitions. Both changes come from one project, so they are one publisher cluster and do not independently validate each other. Their shared question is whether a client can tell what the returned evidence means.

Not applicable to past price moves. An earlier OTR Technology record covered missing log probabilities in a Cohere wrapper. A separate General card asked whether adding an AI model improves an existing forecast. This view adds a different, newly implemented issue: the same reply contains scores based on the supplied labels and scores based on the whole vocabulary. Restoring a field or adding a model does not decide how a client should interpret those quantities.

The source already provides label_mass, confidence and a top-token diagnostic. These show when most first-token probability went outside the labels, but do not establish why. A low label mass alone does not prove the correct real-world answer was missing. The new endpoint is off by default, supports a defined set of model architectures and rejects raw-logit modes. The checked strategy and end-to-end test reject requests that turn thinking on, because the method reads the first reply token. We do not treat it as an assessment of a complete reasoning process. The test fixture demonstrates a score relationship; it does not prove miscalibration or customer harm.

No investment direction applies. We add a new API-interpretation thesis from later source changes and preserve the older log-probability and forecast-comparison records. The new information is the separation between a relative option score, total label probability and the client’s action rule. We have not changed any existing market forecast, deadline or outcome.

A field named confidence can sound like a chance of being right. In this API its meaning is defined by token probabilities. We have not measured whether users or software clients confuse those meanings. We make no claim about public sentiment or widespread misuse. Our practical concern follows from the checked response contract and must be tested in the intended task.

No market forecast is made. We found no task-matched record of decisions and outcomes that could turn these token scores into a validated success probability. The upstream one-option test is not such a record. A client would need to assess error rates, score reliability and how often it defers on held-out tasks; none of those results is supplied by the inspected commits.

OTR Database Data

https://github.com/vllm-project/vllm/commit/d33872dfeb79db89b734ce13e07af4630e8bdce2

https://github.com/vllm-project/vllm/commit/b49c921a27083c25c16f308be1326055dbc87bbf

We read both newly assigned Technology findings documents and the two raw observation records used by this card. The two raw observation files contain 37 parsed records; parsing is not full semantic review. We do not claim a full semantic read of the wider database.

We checked the relevant primary implementation, documentation and test definitions for both changes. We did not run the tests, call the new API, verify a stable release or measure client decisions. Numerical examples are explicitly attributed test inputs and expectations.

Both commits are from vLLM. They are separate changes in separate API paths, with a shared interpretation question. They are not independent replications, and the streaming fix is not represented as an implementation dependency of the decision API.