# OpenJev Gets Within 3.8 Points of Jev, With a 102-Row Asterisk **Summary:** OpenJev's 4B baseline reaches 84.5% agreement against Jev's published 88.3% on a selected 102-row subset. I see a credible test of the typed-decision interface, not proof that Jev itself has been reproduced. - Canonical: https://markhuang.ai/news/openjev-3-8-point-102-row-asterisk - Language: en - Author: [Mark Huang](https://markhuang.ai/about) - Published: 2026-09-18 - Section: News - Tags: OpenJev, Jev, Local AI, AI Agents, Model Evaluation - Source: [OpenJev](https://openjev.com/) - License: https://creativecommons.org/licenses/by-nc/4.0/ --- ![A compact local decision engine sends paired probability tokens toward a glass boundary and a sealed black-box system](https://cdn.markhuang.ai/news/openjev-3-8-point-102-row-asterisk/hero.webp) *OpenJev makes the decision interface inspectable. The glass boundary is the part I would keep in view: matching an interface is different from reproducing the model behind it.* [OpenJev's browser demo](https://openjev.com/) puts a small local model behind the kind of typed decision interface TypeSafe introduced with Jev. Its published ladder reports 84.5% agreement for a 3.01 GB Qwen3.5 4B browser artifact on a selected TypeSafe subset, compared with Jev's published 88.3%. The 3.8 percentage point gap is close enough to be interesting, but the comparison comes with a large asterisk. My read is that OpenJev matters less as a Jev substitute than as a teardown of the product idea. It shows that developers can get much of the interface pattern from an ordinary open model: provide state, define allowed options, and read probabilities without waiting for the model to write an answer. What it does not show is that Jev's undisclosed architecture or training has been reproduced. That distinction changes the decision for me. I would use OpenJev to test whether this new interface belongs in my stack. I would not use its closest benchmark number as proof that the underlying systems are interchangeable. ## The interface escaped first The [OpenJev repository](https://github.com/TheoLeeCJ/openjev) is unusually direct about its scope. It calls Jev a closed service and says the project reproduces the interface pattern with open models, not Jev's model or training. The baseline feeds unstructured state, runtime criteria, and typed options into Qwen3.5 4B, then reads the declared option logits and normalizes them into probabilities. No answer sentence or JSON object has to be generated. That is already useful. Many agent decisions are bounded: choose a queue, approve a retry, or decide whether evidence supports a claim. Asking a chat model to write prose or JSON for such a choice creates output that software must parse back into a branch. OpenJev turns the branch into the model interface. ```mermaid flowchart LR A[State and runtime criteria] --> B[Open model] C[Declared options] --> B B --> D[Read option logits] D --> E[Conditional probabilities] E --> F{Application policy} F -->|Act| G[Bounded action] F -->|Abstain| H[Review] ``` The last branch is mine, not the model's. OpenJev warns that its scores are conditional on the options supplied. They are not calibrated confidence, and they do not account for an answer the developer forgot to include. A faster probability still needs a policy for action, abstention, and review. ## The 102-row comparison needs its label The headline comparison is encouraging, but it is easy to read more into it than the evidence permits. OpenJev reports 0.845 modal agreement for direct 4B logits and 0.883 for published Jev on 102 rows across 20 cases. The project did not call a live Jev endpoint. It aligned the rows it could reconstruct from TypeSafe's public records, and it says the comparison does not cover TypeSafe's reported 711-row aggregate. Modal agreement is also not a universal accuracy score. It says how often the OpenJev choice matched the modal result in that selected material. It does not tell me the cost of a wrong action in my workflow, whether the probabilities are calibrated on my traffic, or how either system behaves after the input distribution shifts. This extends the concern I raised in [my first read on Jev](https://markhuang.ai/news/jev-schema-guarantee-wrong-decision). A guaranteed output shape can remove parsing failures. It cannot guarantee that the selected option is right. OpenJev makes that boundary easier to inspect because its prompts, revisions, row-level outputs, known failures, and checksums are public. ## The speed result is the sharper argument OpenJev's systems comparison is more persuasive because it keeps the model and workload fixed. On one RTX 3090, the same Qwen3.5 4B model answered 21 binary criteria in 1.023 seconds by reading typed logits. Generating a compact array took 5.332 seconds and 111 output tokens. The direct path was 5.21 times faster. There is a catch worth preserving. The generated choices agreed with direct argmax on 18 of the 21 criteria. Changing the readout method changed three decisions, even though both paths used the same model and state. That is a warning against treating inference plumbing as a neutral optimization. Prefix reuse adds another promising result. On an owned workload of 37 states by 21 criteria, fresh scoring handled 2.33 decisions per second, while parallel suffixes reached 20.03. Yet the project also reports that the experimental BF16 reuse paths changed five or six of 777 argmaxes relative to fresh scoring. I like that disclosure. A production team would need to decide whether those changes are noise or unacceptable drift. ## The browser demo is a sizing exercise The browser page offers three local tiers. Qwen3 0.6B downloads 639 MB, MiniCPM5 2B downloads 1.56 GB, and Qwen3.5 4B downloads 3.01 GB. Their reported authored balanced accuracy rises from 44.0% to 68.6% to 81.3%. The page says weights stay in the browser cache and inputs do not leave the page, while also warning that larger models may not fit some devices and quantization can change quality and speed. I see a practical product lesson there. Local privacy and zero output tokens sound clean, but the device budget still buys capability. The smallest browser model is convenient, not equivalent. Teams evaluating this pattern should choose a model against their error cost, not against the nicest download experience. ## Copying the boundary is still a real result Public reaction has split around the word "Jev." In one [LocalLLaMA discussion](https://www.reddit.com/r/LocalLLaMA/comments/1whxf90/localjev/), commenters pointed to existing zero-shot classifiers and warned that a Qwen wrapper is not TypeSafe's claimed architecture. Another commenter argued that probability outputs and parallel inference are natural capabilities of language models, and hoped for a convenient local interface. Both reactions make sense to me. OpenJev should not inherit claims that belong to Jev's private system. It also does not need to. If a reproducible 4B baseline can make typed decisions quickly enough for a routing or verification job, developers gain another option and TypeSafe gains a more meaningful comparison. > **Info:** > > My evaluation rule: use OpenJev to test the interface on labeled local data. Compare it with existing rules, classifiers, and narrow LLM calls. Keep the action threshold and abstention path outside the model. The next benchmark I want is not a broader imitation contest. It is a workload where mistakes have prices: missed fraud, needless escalation, a bad retry, or an answer sent without enough evidence. OpenJev has made the machinery inspectable. Now the application owner has to prove that the decisions deserve to run.