SemIf: Turning a Language Model Into an If Statement

A local-first research baseline that reads logits instead of generating text, so LLMs can make fast typed decisions on consumer hardware.

8 to 10 min read • View on GitHub • More from TheoLeeCJ

A mechanical switchboard with a model core in the center routing power into labeled decision terminals. The machine sends signals to Yes, No, Route, and Hold without emitting any speech bubble or text stream, explaining that SemIf reads choice probabilities directly instead of generating prose.
SemIf treats a language model like a decision engine. One forward pass, many branches.
Key Takeaways

The Model That Refuses to Talk

SemIf starts with a simple provocation: for branching logic, text generation is often wasted motion. If you only need a yes, no, route, or score, why ask a language model to compose prose and then parse it back into structure?

That shift changes the job description. The model is no longer a storyteller. It becomes a typed decision surface, exposing its preference distribution directly through logits. The win is not elegance. It is fewer tokens, less latency, and fewer opportunities for output weirdness.

Direct Mode Is the Real Trick

The core implementation lives in direct.py. Its move is blunt: map options to tokens, run a single forward pass, and read the logits for those slots. There is no decoding loop, no JSON repair, and no wandering text to clean up afterward.

# Conceptually: one pass, then read the option logits
outputs = model(**inputs, logits_to_keep=1)
last_logits = outputs.logits[:, -1, :]
choice_scores = last_logits[:, slot_ids]
selected = choice_scores.argmax(dim=-1)

That tiny shape matters. In a normal generation path, the model spends compute producing the next token, then the next one, then the next one again. SemIf uses the model at the exact point where it already knows enough to decide.

A close-up of a token selector with a single dial feeding several small option gauges. The evidence enters once, then the model's last-position scores fan out into multiple choices, explaining why direct logit readout is different from decoding text.
Direct mode reads the first decision boundary, not the whole sentence after it.

The Cache Problem, Solved Cleanly

The clever part is not that SemIf can score one prompt. It is that it can score many questions against the same evidence without paying for the prefix every time. That is what SerialPrefixScorer, past_key_values, and _state_prefix are really for.

The boundary matters. Once the evidence is encoded, the repo freezes that state in the KV cache and only processes the suffix for each new query. In other words, the document becomes reusable context, not repeated work.

The cache boundary is the whole trick. Encode the evidence once, then reuse it for every suffix question.

# Simplified mental model
state = encode(prefix)
for question in questions:
    logits = decode_suffix(state, question)
    yield logits

# The prefix is frozen in cache, so only the suffix moves.

This is where the repo stops feeling like a neat demo and starts feeling engineered. The boundary logic is doing real work: it makes repeated semantic branching cheap without pretending the model is stateless.

Why the Browser Demo Matters

The WebGPU demo turns the idea into something broader than a Python research repo. It says the pattern belongs in the browser too, with local execution and no backend dependency. That widens the use cases immediately: privacy-sensitive routing, interactive tools, and lightweight agent flows.

It also changes the audience. A browser demo is not a garnish here. It is proof that typed decision models can live inside normal product surfaces, not just in a notebook or a server process.

Direct vs. Reranker

SemIf is best understood alongside reranking. Both try to avoid free-form generation. They differ in where the cost lands.

PatternPasses requiredSpeedBest use caseFailure modeLocal hardware friendliness
Autoregressive generationOne decode loop per answerSlowestOpen-ended text or JSONParsing errors and hallucinated structureWeak when you only need a decision
Direct logit readoutOne pass for many optionsFastestRouting, classification, typed branchingDepends on clean token-to-option mappingStrong on consumer GPUs and Macs
RerankerOne pass per candidateSlower than direct, often sturdierHigh-stakes option selectionCost grows with every extra choiceGood when correctness beats throughput

Reranking can be more robust because it scores full candidate answers against the evidence. But direct readout scales better with option count because it turns one pass into many decisions. If you care about throughput, that difference is the story.

What SemIf Is Really For

The sweet spot is any workflow where the answer needs to be machine-readable more than poetic. Routing, automation, browser agents, smart home control, and code-adjacent triage all fit that shape.

That is also why local-first matters. If the task is a branching decision, a consumer GPU or an Apple Silicon laptop is often enough. You do not need a cloud model with a conversation budget. You need a fast decision engine you can trust to stay inside your app.

On the Jev waitlist? You can use not Jev (but close enough) right here.

Theo Lee, Creator of openjev / SemIf · openjev + openjev.com by @theoleecj

Reproducibility as a Product Choice

The repo is unusually serious about evidence. Benchmarks, scripts, results folders, and hashes are all part of the product, not just the appendix. That matters because performance claims around LLM inference are easy to wave at and hard to verify.

SemIf does not just say it is faster. It leaves a trail that lets you check the claim. In a field full of vibes, that is a meaningful differentiator.