SemIf: The Open Source Project That Turns an LLM Into a Logit Reader

Instead of generating JSON and fixing it afterward, SemIf reads option probabilities directly, reuses cached prefixes across many questions, and makes local inference feel like a typed decision engine.

8 min read • View on GitHub • More from TheoLeeCJ

A mechanical voting chamber with a long evidence scroll feeding into a central machine, while discrete option gauges labeled as choices emerge on the other side. The image explains that SemIf reads model scores for declared options instead of asking the model to write a full answer.
SemIf changes the unit of inference. The model does not write a paragraph and then get parsed. It exposes scores for the choices you already defined.
Key Takeaways

The model is not writing. It is voting.

That is the move SemIf makes. It does not ask a model to draft an answer, then clean up the mess. It asks the model to expose which option it prefers, directly from the logits.

That sounds subtle until you compare it with the usual structured-output stack. The default pattern is still some variation of prompt, generate, parse, repair, retry. SemIf cuts across that workflow and treats the model more like a typed decision engine.

Most agent decisions are small: route this, retry that, does the evidence support X? A chat model can answer them, but it spends time generating text that software immediately parses back into an if statement.

Theodore Lee, Author/Maintainer · GitHub - TheoLeeCJ/SemIf

Why that matters for structured decisions

SemIf is aimed at workloads where the answer is one of a few declared options. Think route this, retry that, accept or reject, support this claim or not. In those cases, full text generation is mostly overhead.

The usual costs show up fast. You pay for decoding tokens you do not need. You spend time repairing malformed output. You build retry logic for failures that are not really model failures, just interface friction.

ToolCore methodGenerates text?Best fitMain tradeoff
SemIfReads declared option logits directlyNoBinary and N-way decisionsSpecialized, not general text generation
OutlinesConstrains decoding with schema rulesYesStructured text outputStill pays the generation cost
InstructorParses model output into typed objectsYesPydantic-style workflowsNeeds validation and retries
LM Format EnforcerFilters tokens during generationYesSchema-constrained completionStill runs the decode loop

That comparison is the whole point. SemIf is not trying to be a better JSON parser. It is trying to make JSON unnecessary for this class of task.

From OpenJev to SemIf

The repo is framed as an open-source reproduction of a closed hosted pattern, but the important detail is the stance. It is local-first, hardware-aware, and explicit about its constraints. It is built for commodity GPUs and Apple Silicon, not for a glossy hosted abstraction.

That positioning matters because the project is not selling general intelligence. It is selling a narrow interface: fast, typed decisions on top of smaller open models.

How the direct scorer works

The core files make the design unusually legible. In core.py, SemIf validates a row so every decision has an id, a state, a question, and a list of options. In direct.py, it maps those options to slot tokens such as A, B, C, and then reads the logits for those tokens from a single forward pass.

def score(model, tokenizer, state, question, options):
    prompt = build_prompt(state, question, options)
    outputs = model(**tokenizer(prompt, return_tensors="pt"), logits_to_keep=1)
    slot_ids = [tokenizer.convert_tokens_to_ids(letter) for letter in option_letters(options)]
    logits = outputs.logits[0, -1, slot_ids]
    probs = softmax(logits)
    return probs

The key mechanism is not constrained generation. It is direct readout: one forward pass, then a probability distribution over the options you already declared.

The important detail is what is missing. There is no decoding loop. There is no sentence to repair. The model is being used as a scorer, not a narrator.

The hidden superpower is prefix reuse

SemIf becomes more interesting once the evidence block gets large. A long document can be prefetched once, then reused across many downstream questions. That is where shared.py and serial.py matter.

The machine stays warm on the same prefix, then branches into short question suffixes. In practice, that turns one expensive read into many cheaper decisions.

A single thick document block being fed into a branching machine, then splitting into several parallel rails that each reuse the same fixed prefix before attaching short question suffixes. The image explains how one evidence state can support many downstream decisions without recomputing the prefix each time.
Prefix reuse is the second big idea. SemIf does not just avoid generation. It reuses the expensive part of the prompt across many questions.

That is also why the project is more than a one-off benchmark trick. It is a pattern for repeated decisions over shared context.

What SemIf beats, and where it does not

SemIf beats tools that still insist on generating text first. If your task is fundamentally a choice among declared options, bypassing the generation loop is cleaner and usually faster.

But that specialization is also the boundary. If you want open-ended text, rich extraction, or flexible schema filling, a constrained generator like Outlines or LM Format Enforcer is still the better fit. SemIf is sharper, not broader.

That tradeoff is honest. The repo is not pretending to replace the whole structured-output ecosystem. It is carving out the slice where direct scoring is the right primitive.

Why the repo feels research-grade

The engineering discipline shows up in the boring places, which is usually a good sign. The project includes benchmark data, raw results, pinned revisions, and SHA256 checksums. The model path is not meant to drift underneath the numbers.

That kind of reproducibility matters because the central claim is performance. If you are arguing that one forward pass is materially better than a decode-and-parse pipeline, you need the receipts.

SemIf has them, at least in structure. That makes the repo feel like a small research system, not just a clever demo.

The browser demo matters more than it looks

The WebGPU demo is doing more than showing off. It proves the interface is not tied to one GPU stack. The idea can travel into a browser, which is a strong signal that the pattern is portable.

That widens the story from a benchmark on a workstation to a deployable decision layer. The point is not just speed. It is that the interface stays simple enough to move.

SemIf, renamed from OpenJev, reads the probability of each answer you allow straight out of a small open model in a single forward pass. Nothing is written out — no answer sentence, no JSON to repair, no decoding loop.

Theodore Lee, Author/Maintainer · SemIf: Structured Output Layer