How it works

The gold set is 80 repositories labelled by hand from README, manifests and code-search evidence: a first pass by an assistant, every label reviewed by the maintainer. It was sampled in strata so the interesting cases are over-represented: noise that keyword search drags in, repositories Jev scored near the gate, and genuine projects spread across category, stars and language. Two fields are labelled separately on purpose: genuine asks whether the repository is about Jev at all, substance how real it is. A one-line scaffold with a TypeSafe dependency is genuine and empty.

The full write-up, including every gate disagreement and what each one says about the evidence or the question set, is in docs/calibration.md.

Threshold

The listed gate is genuine ≥ 0.5 with a substance floor beside it, and those two are the whole gate. The sweep above over-states the miss rate: repositories near the gate were deliberately over-sampled.

Two repositories a human called noise now score above the gate — one whose entire content is a 117-character README, one with no files at all — so precision on the gold set reads 0.97 where an earlier run read 1.00. Both are empty repositories that the substance floor exists to hold back and no longer does; that is a problem with the substance score, not with the gate, and moving the gate to cover for it would cost real projects. The working through is in docs/calibration.md §3.

Category confidence used to be a third condition and is not one any more: it answers which shelf an entry belongs on, not whether it belongs on any, and the two are independent enough that a personal blog with no Jev in it scores 0.96. A repository Jev is sure about is now listed with its category marked uncertain rather than held back.

Category

Jev and the human label agree on 92% of 66 categorised repositories. Where they disagree:

Three quarters of judgments come back at 0.9 or above and every one of those is right; below that the cells are small and the curve is not monotonic. The uncertainty threshold reads this: under it the category is shown with a question mark instead of stated plainly. Listing is unaffected either way.

Substance

Jev's 0–3 substance score against the human one: Spearman 0.75, mean absolute error 0.31 (n=54). Good enough to rank within a category, not to be read as a grade.

Where Jev and the humans disagree on the gate

Every disagreement at the current gate. Most are repositories whose README says nothing about Jev while the code uses it; README-only evidence cannot see those.

Question set v1 → v2

Measured on the 33 gold repositories judged under both sets. v2 removed the "official" category (now an owner flag in code) and rewrote the gate question to name what counts: uses, wraps, evaluates or reimplements Jev.

Cost and latency

Calls stored

3683

Input tokens per call

3235

Latency p50 / p90

210 / 337 ms

Estimated spend to date

$0.50 at $0.042/1M input