Jev Lab
Twenty-five decisions an AI product makes every day: classify, gate, route, filter, verify. Each one decided three ways: by Jev, by an LLM, and by plain code. Same rows, real API calls, every mistake counted.
New to Jev? What it is and where to find it →
Unofficial tests by Prateek Hitli, not affiliated with TypeSafe AI. Run with his own OpenRouter API key; OpenRouter is the provider for the Jev model.
Every experiment: accuracy vs cost(latest saved run of each project; click a point to open it)
Graded projects plot pass rate; the rest plot accuracy against the labels. No-model baselines cost $0 and sit in the “free” column.
The 25 Experiments
How Each Experiment Is Run
One decision call, typed. A Choice, a Noul (yes/no) or a Score, with calibrated probabilities. Code turns the answer into the action, so counting, facts and hard rules are never asked of a model.
The same question, as a prompt. The fast model with a plain or structured-output prompt, given the same policy and inputs. Where it matters, a frontier model too.
What you would write without AI. Regex, deny-lists, keyword filters, points tables, “store everything”. Free, and blind to meaning. Every dataset has rows written to test exactly that.
Against labels, on real calls. Accuracy or a graded pass rate, unsafe actions allowed, cost per passing answer, latency. Saved runs live in History; the re-test diffs them row by row.
uv run streamlit run app.py