Jevify

Rubric first: Jev scores EVERY unit in a complete pass, not repeat-until-no-changes subagent sweeps. The orchestrator decides and reads only flagged units; subagents edit, never scan. This generalises Can Bölük's Jevict (thread; skill://jevict if installed) beyond tests; scores are not proof.

Contents: use · quick path · 0–8 loop · groups · API · facts · recipes A–F · write-time rule · guardrails.

When to use

• Use for roughly 20+ homogeneous items: changed hunks, migration calls, existing

diagnostics, sanitized logs, findings or test units. Below ~20, read them directly.

• Use a compiler, grep count, ast-grep rule or test instead when it answers deterministically.

Cross-unit reasoning needs sibling/group context; otherwise read directly.

Quick path: read-only answers (do this first)

You loaded this skill for a 20+ item job, so judge the items; do not read them yourself first. Reading every item by hand is the failure this skill exists to prevent. For a read-only answer ("which of these stand out?"), this is the whole job. Steps 1–8 below (pilot, calibration, re-judge) are for deletions, bulk edits or 150+ items.

%load skill://jevify/jevify.py

Q = {"verdict": jv.choice(
    "One changed hunk from a commit. Is it explained by the commit message below?",
    explained="Every line changed is stated by the message or mechanically follows from it.",
    mixed="Mostly explained, plus at least one added or changed line the message does not mention.",
    unexplained="Changes behavior, config or code the message does not mention.",
    unclear="Cannot tell from the hunk and message alone.")}
import subprocess  # the full message, not just the title
msg = subprocess.run(["git", "log", "-1", "--format=%B", "HEAD"], capture_output=True, text=True).stdout
units = jv.git_units("HEAD", by="hunk")        # or rg_units / ast_units / line_units
states = jv.states(units, lambda u: {"message": msg, **u})
rows = await jv.run(states, Q, intent="hunks vs commit message")
jv.tally(rows, "verdict")
flagged = jv.flag(rows, lambda r: r["verdict"] != "explained" or r["verdict.explained"] < 0.7)
jv.show(flagged, states)

Then read each flagged item's source, confirm or overturn it, and answer with the counts and the confirmed items. Change only Q, the unit source and the render; keep the rest.

0. Load the helper

In the Python kernel, %load defines jv and prints its helper names; judge_batch, completion and wait come from the kernel. Re-%load resets jv.frozen/jv.stats. Never %pip: the kernel Python may lack pip, so jv.ast_units installs ast-grep-py through jv.dep. The notice's wait(judge(...)) example fails here.

%load skill://jevify/jevify.py

1. Decide the rubric before seeing data

Write questions against a concrete contract, before data. Labels are mutually exclusive; UNKNOWN needs investigation. Fix the 0.7 policy now, not after seeing pilot scores.

TARGET = "legacyApi"
paths = ["**/*.ts", "**/*.tsx"]
SKIP_PREFIX = "generated/"  # deterministic pre-filter; keep both probes aligned
STATE_CAP = jv.CAP
Q = {"migration": jv.choice(
    "For this call site, decide whether it still depends on legacyApi's old behavior. "
    "A renamed alias or wrapper is not a successful migration; missing context is UNKNOWN.",
    NEEDS_CHANGE="The old behavior or obsolete contract still reaches a consumer.",
    SAFE="The call is already migrated or is unrelated despite its name.",
    UNKNOWN="The available unit cannot prove either conclusion.")}
ACTION_AT = 0.7
def need(r):
    top = max(r.get(f"migration.{k}", 0) for k in ("NEEDS_CHANGE", "SAFE", "UNKNOWN"))
    if top < ACTION_AT or r.get("migration") == "UNKNOWN": return "uncertain"
    return "obsolete call" if r.get("migration.NEEDS_CHANGE", 0) >= ACTION_AT else False

2. Extract units and check coverage

Use git_units for diffs, rg_units for hits, ast_units for syntax, line_units for signatures or Jevict's parser for tests. Count naive heads against parser units in the same paths; inspect unmapped comments/strings versus real missed calls. Equal totals alone prove nothing.

raw = jv.ast_units(f"{TARGET}($$$ARGS)", lang="typescript", globs=paths)
units = {k: u for k, u in raw.items() if not u["file"].startswith(SKIP_PREFIX)}
heads = {k: h for k, h in jv.rg_units(TARGET, fixed=True, globs=paths).items()
         if not h["file"].startswith(SKIP_PREFIX)}
print(f"pre-filter removed={len(raw)-len(units)}")
uncovered = [(h["file"], h["line"]) for h in heads.values() if not any(
    u["file"] == h["file"] and u["line"] <= h["line"] <= u["end_line"]
    for u in units.values())]
print(f"naive={len(heads)} mapped={len(heads)-len(uncovered)} units={len(units)}", uncovered[:20])

3. Build states, spot-check extraction

A state is per-unit text/JSON. jv.states caps strings and drops empties (one rejects a batch); jv.MAX_STATE warns on overflow. Inspect boundaries; use render to attach helpers/siblings when AST call text lacks context (jv.hunk_states does this for diff hunks).

states = jv.states(units, cap=STATE_CAP)
print(f"extracted={len(units)} states={len(states)}", jv.stats["states"])
jv.show(list(states)[:5], states, width=1200, limit=5)

4. Pilot; edit WORDING, not the threshold

Pilot 40–150 units (for 20–39, read all). Read every flag and near misses; name each misread idiom: helper, span, near-miss, allow-list, transition or duplicate. Fix Q's wording, never the threshold; rejudge the same pilot for precision/recall. Two or three rounds are normal; after five fix extraction/context. Freeze when satisfied.

pilot = jv.sample(states, n=min(100, len(states)))
prows = await jv.run(pilot, Q, intent="jevify pilot: migration calls")
pflags = jv.flag(prows, need)
jv.show(pflags, pilot, fields=["migration", "migration.NEEDS_CHANGE"],
        width=STATE_CAP, limit=len(pflags))
jv.show([r for r in prows if .5 <= r.get("migration.NEEDS_CHANGE", 0) < ACTION_AT], pilot, limit=20)

5. Freeze; judge every state

Freeze Q to JSON (12-hex hash), then judge all states. The helper drains in bounded slices, retries errors/undrained keys once and keeps failures as error rows. A post-freeze wording change invalidates every verdict; re-freeze and re-judge all.

seal = jv.freeze(Q)
rows = await jv.run(states, Q, intent="jevify: complete migration sweep")
print(jv.report(rows, Q, title="Migration call-site sweep"))

6. Calibrate before bulk or destructive action

Get up to 20 slow-model opinions per choice-probability bin. Agreement should rise monotonically to ~100% in the top bin and ≥80% in [0.7, 0.9). Flat bins mean a broken state build; top-bin <80% means return to the pilot and re-judge all. The threshold table estimates true/false flags; empty bins are forward-filled estimates, not proof. Calibrate before bulk edits/deletion; optional for read-only reports. No file authority.

assert not jv.flag(rows), "Resolve failures/fallbacks first"
cal = jv.calibrate(rows, states, Q, qid="migration", label="NEEDS_CHANGE",
                   agree={"NEEDS_CHANGE"}, per_bin=20,
                   rubric="Migration requires consumer-equivalent behavior, not a rename.")

7. Escalate exact flagged units and act

jv.flag includes errors/fallback models regardless of rule. Read every flagged source in full (jv.show may truncate), including UNKNOWN. Give task subagents the exact confirmed ID list grouped by file to edit, never scan. Do not act on error/fallback rows; re-run or read. Deletion needs authority and jv.delete_spans dry-run first; preserve dirty work. Use LSP references for exported-symbol refactors.

flags = jv.flag(rows, need)
jv.show(flags, states, fields=["migration", "migration.NEEDS_CHANGE"], limit=len(flags))
candidates = {}
for r in flags:
    if r["why"] == ["obsolete call"]:
        candidates.setdefault(units[r["id"]]["file"], []).append(r["id"])
print("Review before assigning:", candidates)

8. Re-extract, re-judge SAME Q, report

After edits, re-extract and remap naive heads; re-judge with identical frozen Q. Closure needs zero flags/errors/fallbacks/unknowns and no uncovered executable hits. Report counts then reviewed IDs: "judge output tuned on an N-unit pilot, calibrated on M, not a full read." Zero units is not a model verdict.

again_raw = jv.ast_units(f"{TARGET}($$$ARGS)", lang="typescript", globs=paths)
again = {k: u for k, u in again_raw.items() if not u["file"].startswith(SKIP_PREFIX)}
heads_after = {k: h for k, h in jv.rg_units(TARGET, fixed=True, globs=paths).items()
               if not h["file"].startswith(SKIP_PREFIX)}
uncovered_after = [(h["file"], h["line"]) for h in heads_after.values() if not any(
    u["file"] == h["file"] and u["line"] <= h["line"] <= u["end_line"]
    for u in again.values())]
print(f"after: heads={len(heads_after)} units={len(again)} skipped={len(again_raw)-len(again)}", uncovered_after[:20])
now = jv.states(again)
post = await jv.run(now, Q, intent="jevify: after action") if now else []
assert all(r["qhash"] == seal for r in post)
remaining = jv.flag(post, need)
print(jv.report(post, Q, title="After action"), f"remaining flags={len(remaining)}")
assert not remaining, "Flagged units remain; inspect uncovered heads separately"

Grouped mode: one shared header, multiple per-unit questions

Group independently judgeable units sharing context. jv.group chunks by key into ≤8 slots; jv.run(..., slots=slots) returns per-unit rows. The header is billed once: eight units with a 3k-character header cost 5.4× less; a real 646-line sweep (1.6k header, 111 groups averaging 5.8 units) cost 1.4× less and ran 4× faster. Keep groups below jv.MAX_STATE to avoid silent fallback.

gstates, slots = jv.group(units, by=lambda u: u["file"], k=8,
                          render=lambda u: u["text"], shared=lambda file: {"file": file})
grows = await jv.run(gstates, Q, intent="jevify: grouped call sites", slots=slots)
gflags = jv.flag(grows, need)

Grouping is a different instrument, not a cheaper copy: on that sweep grouped and flat labels agreed 73%, bool P(yes) correlated 0.66, and ≥0.5 flags overlapped 24 of 112; a slow tie-break on 16 disagreements sided with flat 6, grouped 7, neither 3. Pilot and calibrate grouped rows separately; never mix grouped and flat rows in one decision.

Helper reference: jv (exact public API)

jv owns all public names; units are ordered {id: unit} with collisions #2, #3.

git_units: no-color/no-ext-diff, paths after --; file id path → {file,status,diff}, hunk id path:+NEW_START (or -OLD_START) → {file,status,header,hunk,line,old_line}; binary/rename/mode-only → path:meta. rg_units: explicit . and DEVNULL stdin; path:line → {file,line,match,context}. ast_units: rg --files, path:line → {file,line,end_line,text} (1-based); .tsx/.jsx parse as tsx/jsx and files with parse errors are still searched. A def/function pattern starts at the keyword (decorators excluded) and also matches nested definitions; use a rule (recipe F) for deletable spans. hunk_states: {file,status,hunk,nearest_sibling_hunks,hunks_using_names_defined_here}; users come from names defined at the hunk's outermost indentation, dunders excluded. line_units: UUID/hex/timestamp/number-normalized sha1(signature)[:10] → {signature,count,examples}, most frequent first.

jv.flag infers modal primary; string rule results label why, other truthy values become "rule", error rows skip the rule. jv.calibrate returns {bins,thresholds, verdicts} with verdicts[id]={slow,reason,p}; slow errors are ERR, excluded from agreement. jv.delete_spans requires inclusive 1-based {file,line,end_line}; missing ids are reported and dry-run is the default.

Harness facts (measured 2026-09-23, typesafe/jev-1.13.0)

• judge(state, questions) is a coroutine: await judge(...) returns answers;

wait([judge(...)]) raises TypeError. The built-in jevify notice's wait(handles) example fails on this build; use jv.run/judge_batch for bulk work.

• `judge_batch(states, questions, *, concurrency=None, retries=None, min_ok=None,

intent=None) has id,intent,total,status(),drain(timeout),drain_iter(timeout), results(),failed(),cancel(),close(). Status: {id,intent,total,done,failed,cost, running,model,elapsedS}; item: {key,ok,answers,error,model}. Choice returns {type,choice,confidence,probabilities}, bool {type,bool: P(yes)}, score {type,score,confidence,legend,probabilities}` (weighted level index).

• One empty "" state rejects the entire batch at submit. Jev ceiling ~32k tokens:

130k source chars stayed on Jev (0.6 s); 200k/300k source chars and 60k dense chars silently fell back here to the session's default model (3–7 s/unit versus <1 s). Fallback is unpriced in batch cost, calibrated differently; status has one model, only per-item item.model/row model identifies it. Shrink, re-judge, re-calibrate.

• State cost ~$0.03–0.05/M tokens (~$0.01/M ordinary chars, ~$0.04/M dense);

questions are cheap. Can Bölük reported ~$2 for 27k tests; not a guarantee.

• completion(prompt, *, model="default"|"smol"|"slow", system=None, schema=None)

returns id,kind,status,done,wait,cancel; schema makes .wait() return a dict. wait([...], raise_errors=False) supports handles. One slow call took ~6 s.

• The kernel runs system Python 3.12 without pip: %pip install fails with

No module named pip. jv.dep installed ast-grep-py via uv into the cache dir in ~1 s. Present: rg, git, uv; absent: numpy, matplotlib, tree-sitter. %load skill://jevify/jevify.py also resolves skills created mid-session.

• Stability: re-judging 100 hunks with the identical frozen Q flipped 0 labels

(median |ΔP| 0, max 0.07), yet 5 rows crossed the 0.7 uncertainty line. Churn at the threshold is noise, not wording drift.

• Throughput here: 646 flat states in 5.4 s (~120/s, $0.025); 100 hunks in ~1 s ($0.006).

jv.calibrate ran 21 slow calls in 20 s (waves of 16).

Recipes: source, complete question, escalation, action

Freeze Q before data; escalate top P<0.7, ambiguous bools (0.3–0.7), failures/fallbacks and named labels; labels grant no authority.

A. Diff scope audit (commit, PR, branch, or worktree)

Source: jv.git_units(rev, by='hunk') (keep binary/meta units) and the full commit message or PR body as the boundary, not its title. Give every declared purpose an operational label (what its hunks look like) plus undeclared and mixed. Render with jv.hunk_states: a setup, helper or parametrization hunk serves the purpose its tests assert, which only neighbouring hunks reveal.

import subprocess
REV = "HEAD"  # or 'origin/main...HEAD' with MESSAGE from `gh pr view --json body`
MESSAGE = subprocess.run(["git", "log", "-1", "--format=%B", REV],
                         capture_output=True, text=True, check=True).stdout
Q = {"purpose": jv.choice(
    "Assign THIS hunk to the declared purpose it serves. Sibling hunks and hunks using names "
    "it defines are context: helpers, fixtures and parametrizations serve the purpose their "
    "tests assert; a change's own proposal/tasks/metadata serve that change.\n"
    f"MESSAGE:\n{MESSAGE}",
    feature="<declared purpose 1, operationally: its code, tests, docs and reporting>",
    validation_repair="<declared repair: formatter blank lines, type-narrowing asserts, mock fixes>",
    undeclared="Serves none of the declared purposes.",
    mixed="Separable changes serving different purposes, including an undeclared one."),
    "behaviour_undeclared": jv.yes(
        "Does this hunk change runtime (non-test) behavior for a reason OTHER than the declared "
        "purposes? A behavior change that implements a declared purpose answers no."),
    "debug_leftover": jv.yes("Does this hunk add debug output, commented-out code or TODOs?")}
units = jv.git_units(REV, by="hunk")
states = jv.hunk_states(units)

Escalate undeclared/mixed, top label P < 0.7, either bool ≥ 0.5 and errors/fallbacks; read every flag in full. Measured on a 100-hunk, three-purpose commit: a title-only in/out-of-scope rubric flagged 41–42 hunks (declared fixture repairs, a package upgrade, change metadata); declared purposes plus hunk_states flagged 21–24. A double-negative behaviour question sat at P≈0.5 on 7 core hunks until reworded as above. Fast undeclared at P 0.5–0.9 matched the slow model on 1 of 6; the slow model matched hand reading on 8 of 8 reviewed flags, leaving one truly undeclared spec reflow. Preserve user work, repair authorized scope, re-extract against the same base and re-judge SAME Q.

B. Refactor or migration call-site completeness

Source: LSP references first for exports; rg_units or ast_units plus a naive grep count. Include aliases/wrappers; syntax matches alone cannot prove completeness.

Q = {"migration": jv.choice(
    "Classify THIS call against the new API contract; a rename alone is not migration.",
    migrated="New contract used with equivalent consumer-visible behavior.",
    needs_change="Obsolete contract or lost required behavior remains reachable.",
    not_applicable="Lookalike hit not governed by this migration.",
    unsure="Alias, type or surrounding context is insufficient.")}
units = jv.ast_units("oldApi($$$ARGS)", lang="typescript")

Escalate migration.needs_change ≥ 0.7, unsure, errors/fallbacks; read consumers. Fan confirmed IDs by file for edits; re-extract/re-judge SAME Q to zero residual heads.

C. Existing compiler/diagnostic output triage

Source: parse captured tsc/eslint/cargo output into one unit per diagnostic {file,line,code,message,context}. Preserve the location and neighboring source; line_units alone groups away locations. Example below parses tsc; adapt the parser for eslint/cargo, without running diagnostics solely for this sweep.

import re
from pathlib import Path
Q = {"origin": jv.choice(
    "Classify immediate cause, not a downstream cascade; use source context.",
    real_bug="Changed source has an observable defect needing repair.",
    type_only="Type-check contract fails, without evidence of a runtime failure.",
    stale_or_generated="Diagnostic comes from stale build state or generated output.",
    noise="Non-actionable, duplicate, or source context insufficient to decide."),
    "effort": jv.scale("Estimated repair effort given this source context?",
                       "local", "multi-file", "architecture")}
rx = re.compile(r"^(.*)\((\d+),\d+\): error (TS\d+): (.*)$")
lines = Path("/tmp/diagnostics.log").read_text().splitlines()
def context(p, n):
    f = Path(p)
    if not f.is_file(): return ""
    source = f.read_text(errors="replace").splitlines()
    return "\n".join(source[max(0, int(n)-3):int(n)+2])
units = {f"{p}:{n}:{code}:{i}": {"file": p, "line": int(n), "code": code,
         "message": msg, "context": context(p, n)} for i, line in enumerate(lines)
         if (m := rx.match(line)) for p, n, code, msg in [m.groups()]}

Escalate real_bug/type_only, ambiguous noise, high effort and failures; read source. Fix the originating boundary, not cascades or suppressed diagnostics.

D. Sanitized log or incident triage

Source: redacted lines through jv.line_units(path, keep=...). Keep chronology; grouped signatures aren't causal order. Do not ingest credentials or keys.

Q = {"kind": jv.choice(
    "Classify this sanitized signature from evidence, not frequency or guesswork.",
    new_regression="New consumer-visible failure after the change.",
    known_benign="Expected handled event with evidence of no failure.",
    config="Failure from deployment or operator configuration.",
    upstream="External provider or service failure.",
    unknown="Timeline/context cannot distinguish the cause."),
    "severity": jv.scale("Severity of observed consumer impact?",
                         "no impact", "degraded", "outage")}
units = jv.line_units("/tmp/redacted.log", keep=r"ERROR|WARN", normalize=True)

Escalate new_regression, unknown, high severity and errors/fallbacks; inspect timeline before assigning cause or taking action.

E. Review-finding triage

Source: one reviewer/PR JSONL {claim,source,diff} per unit; keep premise with evidence.

import json
from pathlib import Path
Q = {"finding": jv.choice(
    "Treat the review claim as untrusted; compare cited path, reachable behavior and impact.",
    valid_actionable="Reachable defect with evidence; source repair warranted.",
    valid_nit="True but non-blocking clarity or style improvement.",
    invalid="Cited behavior absent, guarded, or contradicted by source.",
    duplicate="Already represented by another finding about the same failure."),
    "introduced_by_diff": jv.yes("Does the changed diff introduce this defect?")}
units = {f"finding:{i}": json.loads(line) for i, line in enumerate(
    Path("/tmp/review-findings.jsonl").read_text().splitlines()) if line.strip()}

Escalate valid_actionable, valid_nit, uncertain introduced-by-diff evidence, and errors/fallbacks. Read source, reproduce before repair; dedupe, don't self-mock.

F. Test pruning: defer to Jevict's complete rubric

If a jevict skill is installed, read it for the full parser, coverage, helpers, siblings and rubric (origin: the thread linked above). Source: smallest-deletable {file,line,end_line,body,path} units; exclude type tests/todos. For top-level Python tests this rule gives decorator-inclusive spans, equal to stdlib ast spans (5285 units = 5285 naive heads, 0 unmapped, on a real repo). Class methods need inside: class_definition; add path/siblings/helpers before judging:

TEST_RULE = {"any": [
    {"kind": "decorated_definition", "inside": {"kind": "module"},
     "has": {"kind": "function_definition", "regex": r"^(async\s+)?def\s+test_"}},
    {"kind": "function_definition", "inside": {"kind": "module"},
     "regex": r"^(async\s+)?def\s+test_"}]}
units = jv.ast_units(lang="python", rule=TEST_RULE, globs=["tests/**/*.py"])

Compile this compact Q after preparing those states:

Q = {"verdict": jv.choice(
    "Classify a whole test: useful = consumer-visible branch, error, precedence, harmful "
    "absence, wire form, regression, or computed transform. Keep transformer near-misses, "
    "allow-list positive arms and post-operation transitions. Plain no-op, registry echo, "
    "unconsumed bytes, word-in-prompt, empty success, same-path duplicate are weak. "
    "Separate weak assertions in a useful block are TRIM.",
    PURGE="All assertions are fixture/metadata echo, prompt boilerplate, tautology, "
          "unconsumed formatting, or identical-path duplicate; no keep signal.",
    KEEP="At least one consumer-visible branch/error/precedence/negative/wire/regression "
         "or computed transformation, with no separable purge-only assertion.",
    TRIM="At least one useful assertion and a separable purge-only assertion; retain the unit.")}

Escalate verdict.PURGE ≥ 0.7, TRIM, errors/fallbacks; read each purge and calibrate. With deletion authority dry-run jv.delete_spans(units, ids, dry_run=True) (verified on a copy: file untouched, unknown ids reported; the real run removed exactly the chosen tests with their decorators and left other decorators intact), inspect spans, apply confirmed deletions, hand-edit TRIM, re-extract/re-judge SAME Q. Assertion helpers count; an absent keyword alone isn't grounds to delete.

Promote to a write-time rule

When requested, write <repo>/.omp/rules/test-value.md or ~/.omp/agent/rules/test-value.md: one yes/no question, gated by scope/condition.

---
description: Warn when a changed test asserts only incidental implementation details
question: Does this test edit add assertions that only echo fixtures, inspect wiring, or check successful completion without pinning a plausible consumer-visible failure?
scope: "tool:edit(*.test.ts), tool:write(*.test.ts)"
condition: "expect\\(|assert"
---
Read the test: replace incidental assertions with a consumer contract, or remove them; preserve useful assertions.

TTSR judges {output,content} (≤60k chars) after completion, never interrupts. P(yes) ≥ 0.7 delivers the body as a hidden warning; failure delivers nothing. ttsr.judge: auto requires native TypeSafe Jev. Optional globs, regex condition or ast-grep astCondition gate cost; scopes include text, thinking, tool and tool:<name>(<glob>). omp ttsr list shows questions; omp ttsr test/scan do not exercise the judge. /omfg drafts rules; rules do not replace bulk sweeps.

Guardrails and handoff

• Rubric first; cover extraction before judgment; read flagged source, not just scores.

Choice probability is not correctness; scale scores are weighted level indices.

• Preserve Q's hash across full/after-action runs; any wording change means re-judge all.

Inspect errors and fallback rows; never turn them into automatic edits.

• Calibrate before bulk/destructive action, inspect disagreements and dry-run spans.

Review authority is not deletion authority. Never stash, reset, commit or touch unrelated work.

• Report counts first: extracted/covered, dropped/truncated, piloted/judged, errors,

models, flags, calibration, changed/remaining; separate estimates from reviewed source.