malevrigns/agent-jev

Weights: https://huggingface.co/aimeigaoshou/agent-jev

The job   ·   Primitives   ·   Forward pass   ·   Results   ·   Latency   ·   Run it   ·   HTTP   ·   Repository

AgentJev-0.6B does not write. You hand it a state and questions you have already phrased. One forward pass returns a probability for every option. The number of decoded tokens is zero.

Put it where an agent loop needs a gate, a route, or a score. Leave the prose to a larger model.

A coding agent, a triage bot, or a workflow spends most of its steps on questions that are not writing problems. Did the tests pass. Which tool is next. Is this command safe to run. The usual answer is to ask a 27B–70B model to talk its way to a boolean, then parse the talk.

_{The state stays unstructured. The questions are typed. The answer is the distribution.}

The state can be a diff, a stack trace, a ticket thread, or a table. Serialize an object and it is sent as stable JSON. Question ids exist so you can match the response. They are not shown to the model. The meaning has to live in the question and in the option text.

Three shapes. Every one of them returns the full distribution, not only the winner.

margin is the gap between the best option and the second. Choice is a preference inside the set you passed. It is not an independent probability that the action will succeed. If you need that, ask a boolean per action and calibrate it on your own outcomes.

The backbone is Qwen3-0.6B with the language-model head removed. Each candidate is read at its last token. A small permutation-equivariant head then scores the set: the order of options does not smuggle in a ranking. Softmax is per question.

flowchart LR
  state["State"] --> enc["Qwen3-0.6B"]
  questions["Boolean, choice, score"] --> enc
  enc --> head["Candidate head"]
  head --> dist["Distribution"]

Three properties are worth the implementation, not the slogan.

Nothing is decoded. Hidden states go to logits. There is no output vocabulary step, so there is no JSON to repair.

The context is 2,048 tokens, and over-length input is refused. A diff or a trace that does not fit is an error, not a silent crop of the question or a candidate.

A shared prefix is reused across candidates of the same question. On one fixed load — 64 choice options plus one boolean, 66 paths, 33,547 path tokens — the unshared median was 609.65 ms. The shared-prefix path was 298.91 ms. The largest probability difference was 0.000508, and the chosen option did not change. Generated tokens were 0 either way. That figure is this load, measured after warmup, not a promise about every prompt.

_{The state is encoded once. Candidates branch from that prefix.}

Different questions do not yet share a state cache. Training also supports encoder_impl: tree: TreeEncoder merges identical token prefixes within each question, uses ancestor-only attention and path-depth positions, and backpropagates through shared nodes. CPU collate precomputes tree topology; automatic routing uses native segmented causal SDPA for wide trees, a bounded dense mask for fragmented trees, and independent paths for small batches. Skewed batches use a compact forest when it saves both padding and mask space. Existing path batches and checkpoints remain compatible, and the default encoder remains path. Full-attention Qwen3 with eager or SDPA is required. The timing above measures the serving KV runtime; see the separate TreeEncoder benchmark report for training-encoder timings, numerical error, memory, and reproduction commands.

Official test split of Typed Decisions: 400 cases, 2,000 questions, five questions over one state, four workflows. Accuracy is agreement with the public teacher argmax. It is not a measured coding-agent success rate.

Rows without a soft-CE number are copied from the dataset card. They were not re-scored in this repository, and that card does not publish soft cross-entropy. Brier, ECE, and score MAE for those rows use the card's definitions.

Against Laya, on this split, the accuracy gap is +2.25 points. A case-level bootstrap over the 400 cases gives a 95% interval of [+0.65, +3.90]. Against the phase-4 weights this run started from, the gap is +40.55 points, interval [+37.35, +43.50].

_{The gap, counted in questions. Overall 1,585 against 1,540. Invoice 431 against 406. Customer service 411 against 382.}

Five hundred questions each. AgentJev is the calibrated checkpoint. Laya is the same published specialist checkpoint as the table above.

Specialist and generalist are not the same measurement. The dataset card says so, and the table marks it. Jev 1.13.0 answered these schemas zero-shot. AgentJev, Laya, ModernBERT, and MiniLM were fit on this benchmark.

Laya's published checkpoint trained on all 1,200 official training cases. This run held out 120 development cases and 120 calibration cases, and selected the step-600 checkpoint by development soft cross-entropy before the test split was opened. Temperature is one positive scalar per primitive, fit only on the calibration cases. The loss was soft cross-entropy plus 0.1 times sum-of-candidates Brier. Seed 20260921. Dataset revision ea9306458d6e9563628369a3d1e72e362fb381d2.

Targets are teacher distributions, including synthetic cases. Beating a row here does not mean a pull request merged, an incident was contained, or an invoice was paid.

Full numbers: typed_decisions/comparison.json, typed_decisions/protocol.json, typed_decisions/REPORT_zh.md.

The trade-off against Laya is clear: Laya is smaller on single short questions; AgentJev scales under wide candidate sets.

Laya's ModernBERT backbone has no causal prefix seam: each candidate in a 64-option question requires a complete forward pass over the state text. AgentJev caches the prompt prefix tokens once and scores all candidate branches against that single KV context, dropping redundant backbone token operations from 33,547 to 2,551 (92.4% reduction).

_{Wide candidate load only. On a short five-question case, Laya's smaller encoder is still the faster one.}

The v1 weights are the safetensors state dict at the immutable v1 revision. This git tree has the code. The server wants a torch checkpoint, so the release files are fetched and wrapped once. Install a PyTorch build for your hardware with the official selector before the requirements below; requirements.txt requires torch>=2.0.0.

git clone https://github.com/malevrigns/agent-jev.git
cd agent-jev
python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install huggingface_hub safetensors
python fetch_v1_weights.py

fetch_v1_weights.py pins revision 7d433994fbde17a3f0993c2f2b02fe8ca1370db1, checks the size and SHA-256 of both release files, confirms every tensor is bfloat16, writes agentjev_v1.pt, copies temperatures.json beside it, and fetches the Qwen3 base model into models/Qwen3-0.6B. The tokenizer is loaded with local_files_only=True, so a bare repository id for --model-path only works when that model is already cached. The script prints the command it has prepared:

python -m jev_service.server \
  --checkpoint agentjev_v1.pt \
  --model-path models/Qwen3-0.6B \
  --temperatures temperatures.json \
  --port 8149

model.safetensors is the full module, backbone plus candidate head. It is not a causal language model, and AutoModelForCausalLM will not load it. AgentJevModel builds the Qwen3 skeleton, then load_state_dict(..., strict=True) replaces it. Load it at dtype=torch.bfloat16: at the pinned revision all 343 tensors are bfloat16, 1,196,881,242 bytes over 598,418,689 parameters. The dtype follows the revision — the repository's default branch now serves a later checkpoint that is float32 and twice the size — which is why the quickstart downloads the pinned revision instead of the default branch.

The process binds 127.0.0.1 only. The workbench is http://127.0.0.1:8149/. GET /health and GET /api/info return the loaded checkpoint. The benchmark checkpoint selected for the table above was step 600. These published tensors are that run.

--temperatures takes scalars fit on held-out calibration cases. --page swaps the workbench HTML. --device defaults to cuda:0. --max-tokens defaults to 2048.

agentjev_client.py speaks to that server. No extra dependencies beyond the standard library.

from agentjev_client import AgentJev

jev = AgentJev("http://127.0.0.1:8149")

state = {
    "task": "Fix NullPointerException in UserAuthService.verifyToken",
    "test_output": "Tests run: 14, Failures: 1 — test_expired_token",
}

gate = jev.decide_boolean(
    state,
    "Have all tests passed?",
    criteria={
        "true": "The suite is green and the project builds.",
        "false": "Any test is still failing.",
    },
)

route = jev.decide_choice(
    state,
    "What should the agent do next?",
    options={
        "read_failed_test": "Open test_expired_token and read the assertion.",
        "rewrite_file": "Ask a larger model to rewrite the service.",
        "commit": "Commit the current diff anyway.",
        "retry": "Rerun the suite without changing the code.",
    },
)

risk = jev.score(
    state,
    "How much operational risk does this change carry?",
    levels=[
        "Isolated change, no external behavior.",
        "A unit assertion moved.",
        "A public signature changed.",
        "Authentication behavior may be bypassed.",
    ],
)

gate carries decision, prob_true, prob_false, confidence, wall_ms. route carries best_action, probability, margin, distribution. risk carries level and expected_score.

evaluate(state, questions) is the lower-level call when one state needs several questions in the same request. Several states go in the HTTP body as requests, up to 32.

Scripts that assume a server already listening on port 8149:

python run_practical_test.py
python test_coding_scenarios.py
python test_game_suite.py

The first two walk software-engineering scenes. The third walks customer-service, maze, snake, and ViZDoom-shaped decisions. They are demonstrations against a live server, not an offline unit suite.

POST /api/evaluate

{
  "state": "23 tests passed, 1 failed.",
  "questions": [
    {
      "id": "done",
      "type": "boolean",
      "question": "Are all tests passing?",
      "criteria": {
        "true": "The suite is green.",
        "false": "At least one test failed."
      }
    },
    {
      "id": "next",
      "type": "choice",
      "question": "What is the useful next action?",
      "options": {
        "debug_failure": "Read the failing assertion.",
        "submit_patch": "Open a pull request now."
      }
    }
  ]
}

{
  "api_version": "agentjev.decision.v1",
  "results": [
    {
      "id": "0",
      "answers": [
        {
          "id": "done",
          "type": "boolean",
          "probability": 0.08,
          "value": false,
          "distribution": { "true": 0.08, "false": 0.92 }
        },
        {
          "id": "next",
          "type": "choice",
          "value": "debug_failure",
          "top_probability": 0.87,
          "margin": 0.74,
          "distribution": { "debug_failure": 0.87, "submit_patch": 0.13 }
        }
      ]
    }
  ],
  "usage": { "generated_tokens": 0 }
}

The probabilities above illustrate the shape. A score answer adds score, level, and legend. Batch with {"requests": [{"id", "state", "questions"}, ...]}. Limits on one call: 32 states, 128 questions, 1,024 candidate paths, and a body between 1 and 1,000,000 bytes. Candidate descriptions inside one question must be distinct.

agentjev_hook.py is a Claude Code PreToolUse command hook for Bash, Write, and Edit. It posts the tool payload to http://127.0.0.1:8149/api/evaluate, asks a boolean and a four-level score, and prints a decision.

It blocks only when the score is level 3 and the boolean says the action is not safe. If the server does not answer, the hook exits 0 and the tool proceeds. Point the hook at the script, and keep the server on port 8149. The endpoint is fixed in the file.

_{The hook judges the call. It does not replace the agent that writes the patch.}

assets/decision_arena.html is a static arena you can open in a browser. The page served on port 8149 is the live workbench, bound to the loaded checkpoint.

The training cases and the 400 test cases are split by case id. Every question of a case stays in one split. Factors, gold labels, and case ids are not part of the model input. The test split was not used to pick the checkpoint or the temperatures.

That is hygiene for this benchmark. It is not a claim about any other dataset.

Serving notes that are easy to miss live in jev_service/README.md.

@misc{agentjev2026,
  title        = {AgentJev: A 0.6B System One Decision Model},
  author       = {malevrigns},
  year         = {2026},
  howpublished = {\url{https://github.com/malevrigns/agent-jev}},
  license      = {Apache-2.0}
}

• Linux.Do — 本项目非常认可 LinuxDo 社区！

Apache-2.0. See LICENSE.