SemIf: Turning a Language Model Into an If Statement
A local-first research baseline that reads logits instead of generating text, so LLMs can make fast typed decisions on consumer hardware.
- SemIf's main idea is to skip generation entirely and read a model's preference distribution as a typed decision surface.
- The repo's cache reuse turns one long evidence block into many cheap follow-up questions, which is what makes the pattern practical.
- The browser demo matters because it proves typed decision models can run locally without a backend or a cloud round trip.
- SemIf is strongest where the answer must be machine-readable, auditable, and fast enough to sit inside routing, automation, or agent loops.
The Model That Refuses to Talk
SemIf starts with a simple provocation: for branching logic, text generation is often wasted motion. If you only need a yes, no, route, or score, why ask a language model to compose prose and then parse it back into structure?
That shift changes the job description. The model is no longer a storyteller. It becomes a typed decision surface, exposing its preference distribution directly through logits. The win is not elegance. It is fewer tokens, less latency, and fewer opportunities for output weirdness.
Direct Mode Is the Real Trick
The core implementation lives in direct.py. Its move is blunt: map options to tokens, run a single forward pass, and read the logits for those slots. There is no decoding loop, no JSON repair, and no wandering text to clean up afterward.
# Conceptually: one pass, then read the option logits
outputs = model(**inputs, logits_to_keep=1)
last_logits = outputs.logits[:, -1, :]
choice_scores = last_logits[:, slot_ids]
selected = choice_scores.argmax(dim=-1)
That tiny shape matters. In a normal generation path, the model spends compute producing the next token, then the next one, then the next one again. SemIf uses the model at the exact point where it already knows enough to decide.
The Cache Problem, Solved Cleanly
The clever part is not that SemIf can score one prompt. It is that it can score many questions against the same evidence without paying for the prefix every time. That is what SerialPrefixScorer, past_key_values, and _state_prefix are really for.
The boundary matters. Once the evidence is encoded, the repo freezes that state in the KV cache and only processes the suffix for each new query. In other words, the document becomes reusable context, not repeated work.
# Simplified mental model
state = encode(prefix)
for question in questions:
logits = decode_suffix(state, question)
yield logits
# The prefix is frozen in cache, so only the suffix moves.
This is where the repo stops feeling like a neat demo and starts feeling engineered. The boundary logic is doing real work: it makes repeated semantic branching cheap without pretending the model is stateless.
Why the Browser Demo Matters
The WebGPU demo turns the idea into something broader than a Python research repo. It says the pattern belongs in the browser too, with local execution and no backend dependency. That widens the use cases immediately: privacy-sensitive routing, interactive tools, and lightweight agent flows.
It also changes the audience. A browser demo is not a garnish here. It is proof that typed decision models can live inside normal product surfaces, not just in a notebook or a server process.
Direct vs. Reranker
SemIf is best understood alongside reranking. Both try to avoid free-form generation. They differ in where the cost lands.
| Pattern | Passes required | Speed | Best use case | Failure mode | Local hardware friendliness |
|---|---|---|---|---|---|
| Autoregressive generation | One decode loop per answer | Slowest | Open-ended text or JSON | Parsing errors and hallucinated structure | Weak when you only need a decision |
| Direct logit readout | One pass for many options | Fastest | Routing, classification, typed branching | Depends on clean token-to-option mapping | Strong on consumer GPUs and Macs |
| Reranker | One pass per candidate | Slower than direct, often sturdier | High-stakes option selection | Cost grows with every extra choice | Good when correctness beats throughput |
Reranking can be more robust because it scores full candidate answers against the evidence. But direct readout scales better with option count because it turns one pass into many decisions. If you care about throughput, that difference is the story.
What SemIf Is Really For
The sweet spot is any workflow where the answer needs to be machine-readable more than poetic. Routing, automation, browser agents, smart home control, and code-adjacent triage all fit that shape.
That is also why local-first matters. If the task is a branching decision, a consumer GPU or an Apple Silicon laptop is often enough. You do not need a cloud model with a conversation budget. You need a fast decision engine you can trust to stay inside your app.
On the Jev waitlist? You can use not Jev (but close enough) right here.
Reproducibility as a Product Choice
The repo is unusually serious about evidence. Benchmarks, scripts, results folders, and hashes are all part of the product, not just the appendix. That matters because performance claims around LLM inference are easy to wave at and hard to verify.
SemIf does not just say it is faster. It leaves a trail that lets you check the claim. In a field full of vibes, that is a meaningful differentiator.





