OpenJev: When an LLM Stops Talking and Starts Deciding
A clean, local inference stack that reads logits directly, reuses KV cache across branches, and turns consumer GPUs into decision engines.
- OpenJev treats a causal LLM like a decision runtime by reading logits directly instead of entering the decoding loop.
- Its speed comes from two moves together: one-token categorical scoring and KV-cache reuse across many branches.
- The repo is most compelling where generation is overkill, such as routing, validation, and other semantic yes-or-no tasks.
- The browser demo matters because it shows the same pattern can run locally, not just in a server-side research harness.
The fastest answer is no answer. OpenJev is built around a simple reversal of the usual LLM workflow: it does not ask the model to write anything. It asks for a small categorical choice, reads the logits directly, and stops there. That makes it useful anywhere the task is a decision, not a paragraph.
On the Jev waitlist? You can use not Jev (but close enough) right here.
Why this exists at all
Most LLM apps still force a generation loop onto problems that do not need one. Routing, validation, scoring, and classification often end with a single label, but the stack still pays for token-by-token decoding, JSON parsing, and retry logic. OpenJev argues that this is the wrong shape for the job.
| Approach | What the model does | Failure mode | Latency profile | Best use case |
|---|---|---|---|---|
| Standard generation + JSON parsing | Writes a structured response token by token | Malformed output, parser churn, retries | Highest and most variable | Open-ended assistant tasks |
| Direct logit readout | Chooses among fixed labels in one pass | Needs carefully constrained labels | Low and predictable | Routing, validation, classification |
| Reranker-style yes/no | Scores document-query pairs | Narrow label space | Fast, but specialized | Retrieval validation |
| OpenJev | Reads typed option probabilities and reuses prefix state | Depends on good slot design and tokenization discipline | Low latency across branches | Local decision engines |
What makes OpenJev different
Two design choices do most of the work. First, the model is used as a scorer, not a speaker. Second, the prefix state is reused instead of being recomputed for every branch. That is what turns a single expensive prompt into a batch of cheap decisions.
Inside the scoring engine
The core files split the job cleanly. direct.py and core.py handle direct readout, pulling the model’s final scores instead of asking it to decode. serial.py and shared.py focus on cache reuse, so one prefill can support many branches. reranker.py provides a more specialized yes-or-no baseline for retrieval-style tasks.
# Simplified shape of the idea
# 1. Prefill once
outputs = model(**inputs, use_cache=True, logits_to_keep=1)
# 2. Read the final-position logits directly
scores = outputs.logits[:, -1, :]
choice = scores[:, slot_ids].argmax(dim=-1)
# 3. Reuse the cached prefix for multiple branches
branched_cache = copy.deepcopy(outputs.past_key_values)
for question in questions:
branch_scores = model(**question, past_key_values=branched_cache, use_cache=True)
The implementation detail that matters most is the slot discipline. OpenJev checks that the answer tokens stay stable and one-token wide, so the decision layer stays deterministic. That avoids a common failure mode in logit-based systems: the label looks simple until tokenization makes it messy.
| Mechanism | What it buys | What it costs |
|---|---|---|
| Direct logits | No decoding loop, cleaner control flow | You must constrain labels carefully |
| KV-cache branching | Shared prefix compute across many questions | Cache handling gets more complex |
| Reranker baseline | Useful for retrieval-style yes/no scoring | Narrower than general decision scoring |
The browser demo proves the point
The webgpu-demo/ directory matters because it moves the pattern out of the server. A local browser demo says this is not just a backend trick or a benchmark stunt. It is a deployable primitive for decision-heavy UIs, where latency and locality are part of the product.
How it compares to the usual LLM stack
Compared with the standard approach, OpenJev is narrower and more deliberate. It gives up open-ended generation in exchange for speed, determinism, and a simpler runtime. That tradeoff is exactly why it feels useful: it matches the shape of the task instead of forcing every problem through chat.
| Approach | Strength | Weakness | Where it fits |
|---|---|---|---|
| OpenJev | Fast categorical decisions on local hardware | Not a general text generator | Agent middleware, routing, validation |
| Standard LLM generation | Flexible and familiar | Slower and more failure-prone for simple choices | Open-ended assistant flows |
| Proprietary Jev-style APIs | Purpose-built decision UX | Closed and harder to reproduce locally | Teams that want the interface, not the stack |
jev does not replace gpt / claude, jev is just a really smart switch statement — like if 2016 ml classifiers got 2026 levels of intelligence. it's a new* type of tool…
What this changes for agents
The bigger implication is architectural. If a decision can happen in one forward pass, synchronous agent middleware becomes much easier to reason about. The system stops behaving like a chat interface wrapped around a model and starts behaving like runtime control flow. That is a different category.