A free GitHub project directory for the TypeSafe Jev / System One model.Explore official SDKs, agent tools, open-source alternatives and Jev news.

Data updated

All tags

Projects tagged with “llm-evaluation”

Browse Jev GitHub projects tagged with llm-evaluation. Filter, sort and explore their source repositories.

  • 🔥🔥 Papers, open reproductions and independent evaluations behind System One models and Jev.

    OmniJev/awesome-jev-gallery · Python

    49443
    awesomeawesome-listcalibrated-probabilitiescalibrationdecision-modelllmllm-evaluation
  • Agent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya.

    openlayer-ai/jevals · Python

    12013
    ai-agentsevaluationguardrailsllmllm-evaluationragas
  • Independent, evidence-based map of when TypeSafe's Jev actually holds up vs. breaks down — real API-call receipts, not a leaderboard. 中文為主的雙語 repo。

    Zaious/jev-capability-atlas · Python

    306
    ai-agentsbenchmarkcalibrationllm-evaluationmachine-learningzh-tw
  • Typed decisions (choice / score / yes-no) from local Qwen models on Apple Silicon. Probabilities come straight from the logits, no text generation. TypeSafe-compatible HTTP API, runs on MLX.

    Micha0827/snapjudge · Python

    140
    apple-siliconclassificationdecision-makingfastapillmllm-evaluationlocal-llm
  • What can Jev actually do? Reproducible experiments and research reports exploring its capabilities and limits.

    keta1930/what-the-jev · Python

    131
    benchmarkdecision-modelevaluationllm-evaluation
  • Independent, reproducible benchmark: a decision model (jev), eight constrained LLMs, and deterministic baselines on typed decisions - accuracy, calibration, latency, cost, failure modes

    nibzard/decision-model-benchmark · HTML

    121
    benchmarkcalibrationdecision-modelllm-evaluationreproducibilityhtml
  • Build and regression-test conversational state machines powered by Jev. Inspect decisions, capture failing conversations as tests, and export runnable TypeScript for your app.

    priyankark/jev-state · TypeScript

    101
    ai-agentscode-generationconversational-aideveloper-toolsevaluationllm-evaluationlocal-first
  • A Jev-driven general-purpose agent harness for faster, lower-cost execution, with built-in side-by-side experiments against LLM-only agents.

    parkavenue9639/jevloop · Python

    100
    agent-runtimeai-agentsdockerllm-evaluationobservabilitypythonsandbox
  • Semantic assertions for pytest: test what your LLM app's output means, judged by TypeSafe's Jev.

    allebee/pytest-jev · Python

    81
    llmllm-evaluationpytestpytest-pluginpythontesting
  • Pre-registered independent eval of TypeSafe Jev against a nano-class LLM, a frontier LLM, and a supervised encoder (Banking77 + CLINC150 zero-shot)

    ickma2311/jev-baselines-eval · Python

    60
    intent-classificationllm-cascadellm-evaluationpreregistrationpython
  • An instrumented 2048 web lab where every move is a Jev (TypeSafe AI System One) Choice, with no heuristic fallback | 用 Jev 决策模型驱动每一步的 2048 网页实验台,概率、置信度、延迟与成本全部摊开可见,且刻意不做启发式兜底

    ARCJ137442/jev-2048 · TypeScript

    51
    2048llm-evaluationtyped-decisionstypescript
  • Fifty real-world financial use cases for TypeSafe's Jev model: typed, structured LLM answers over ledgers, fraud, portfolios, trades and filings, each graded against data where the right answer is known.

    IslamBaraka90/jev-typesafe-real-financial-use-cases · JavaScript

    51
    ai-agentsbacktestingfinancial-datafintechjavascriptllmllm-evaluation
  • Evaluation tools for Jev-compatible decision models, covering accuracy, calibration, hidden instructions, padding and consistency.

    mertkayacs/jevoss · Python

    51
    benchmarkcalibrationdecision-modelevaluationllm-evaluationprompt-injectionpython
  • Practical agent skills and examples for building with Jev. API setup, routing, ranking, and evidence checks.

    laguagu/jev-skills

    41
    agent-pluginsagent-skillsclaude-codecodexevidence-verificationjavascriptllm-evaluation
  • Code for 'When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models' (arXiv:2610.29217)

    Rushab1/gr-jev · Python

    30
    arxivcalibrationchaosnlidecision-modelecthrhotpotqallm-evaluation
  • Free English RAG benchmark for TypeSafe Jev 1.13 — frozen candidate pools, calibration, paired bootstrap CIs, $0 runs. Compared with OpenJev and NVIDIA cross-encoders.

    emretheus/jev-rag-benchmark · Python

    30
    benchmarkcalibrationinformation-retrievalllm-evaluationnvidiaopenjevrag
  • Independent Jev / System One reference with API examples, implementation guidance and reusable prompts for engineers and coding assistants.

    sebastianbennis/jev-system-one-reference

    30
    ai-agentsapi-referenceclassificationcoding-assistantllmllm-evaluationprompt-engineering
  • Grade the agent sessions already on your disk. Claude Code, Codex, opencode, Gemini CLI and Antigravity, scored with Jev for cents.

    killerz3/jevalyzer · TypeScript

    30
    ai-agentsbunclaude-codeclicodexgemini-clillm-evaluation
  • Jev vs GPT-4.1 as synthetic survey respondents on Twin-2K-500. How you ask mattered more than which model you used.

    jjd-lab/jev-synthetic-survey · Python

    20
    behavioral-economicsbenchmarkcalibrationdigital-twindigital-twinsllmllm-evaluation
  • LangChain + Jev (TypeSafe) tutorial: a support-ops agent whose small decisions (triage, model routing, tool guarding, evals) are made by Jev. Real run outputs included.

    PromptEngineer48/langchain-jev-tutorial · Python

    21
    ai-agentslangchainlangsmithllm-evaluationpython
  • 1073tally
    Turn a pile of comments into an auditable distribution: Jev judges each item, code counts the percentages. React workspace + Python engine with pluggable data sources.

    adlternative/tally · Python

    20
    data-analysisi18nllmllm-evaluationpythonreact
  • 1137decide
    Bulk decisions for AI agents. Jev classifies files and logs; your agent reviews exceptions. Reproducible cost and quality benchmarks.

    alsoleg89/decide · Python

    10
    ai-agentsclassificationclaude-codecodexcontext-engineeringllm-evaluationmcp
  • A Jev-powered quality gate for GitHub issues, pull requests, and commits.

    Bnymn1306/jev-github-quality-gate · TypeScript

    10
    ai-evaluationdeveloper-toolsgithubllm-evaluationquality-gate
  • O Jev (TypeSafe AI) como guardrail em tempo real e como pré-avaliador de agentes conversacionais de negócio. Next.js + LangChain via Vercel AI Gateway, com estudo de calibração em português.

    JoaoClemer/jev-lab · TypeScript

    10
    ai-agentsguardrailslangchainllm-as-judgellm-evaluationnextjs
  • Hands-on research lab for TypeSafe's Jev (System One model): reproducible benchmarks of Noul/Choice/Score primitives, confidence gating, fan-out latency, agent control — plus a living audit of the Jev ecosystem.

    llt22/jev-lab · Python

    10
    ai-agentsbenchmarkconfidence-calibrationevaluationllmllm-evaluationllm-observability
  • 发明 RLHF 的人,这次做了个不会说话的模型:Jev 独立研究报告。52 页 PDF + 50 条中文实测复现包 + 143 条可回溯数据表

    HackSing/jev-report · Python

    10
    pythonai-researchchinesellm-evaluation
  • Jev skill for Claude Code and Codex: TypeSafe System One guide, Python CLI, 5 tested recipes, and findings from 19,367 Jev calls.

    marcodicesare-dev/jev-skill · Python

    10
    agent-skillsai-agentsclassificationclaude-codecodexdecision-modelllm-evaluation
  • Independent benchmark data for TypeSafe's Jev (System One model) vs LLMs: accuracy, calibration, cost. Boards + per-decision logs, CC-BY-4.0

    Jevals/jevals-data

    12
    benchmarkcalibrationdatasetleaderboardllm-evaluation
  • A hands-on educational lab for DeepEval + Jev: inspect real recorded judgments, explore release gates and drift playback, and run fresh evaluations on your own workloads.

    shyamsridhar123/JudgeJev · Python

    10
    deepevaldrift-detectionenterprise-aievaluationllm-evaluationllmopspython
  • 1528s1kit
    Test, calibrate and route System One decision models (Jev, Laya, kev, llama.cpp, Ollama) on your own data.

    Eliot5566/s1kit · Python

    10
    calibrationclidecision-modelgithub-actionslayallm-evaluationllm-routing
  • Recorded comparison of Jev, Terra, and Opus on 100 synthetic security-triage cases, five passes each, with a static inspectable dashboard

    Robertzu43/system-one-security-triage · TypeScript

    10
    ai-securitybenchmarkllm-evaluationsecuritytypescript
  • Typed decision tools for AI agents, powered by TypeSafe Jev

    Renwang-Huang/arbitype · Python

    01
    ai-agentsclassificationclaudecursordecision-makingguardrailsllm-evaluation
  • A 60-game benchmark of TypeSafe's Jev evaluation model playing Battleship. The model matches plain code; it does not beat it.

    ickas/battleship-vs-jev · TypeScript

    00
    battleshipbenchmarkevaluation-metricsllm-evaluationtypescript
  • Use Jev instead of an LLM for RAG decisions: relevance grading, query routing, hallucination checks and LLM-as-a-judge evals. Faster, cheaper, measured first.

    davidzna/better-cheaper-llm · Python

    00
    cost-optimizationgroundednessguardrailshallucination-detectionllmllm-as-a-judgellm-evaluation
  • An evolution simulation whose only law of nature is an AI judgment model's yes/no answers. Islands evolve what common sense expects; a predator-prey arms race appears only when predators alone select prey, tested against no-interaction, fixed-predator and shuffled controls.

    isaacbitran/darwins-world · Python

    00
    ai-researchartificial-lifecoevolutionevolutionevolutionary-simulationjudgment-modelllm-evaluation
  • Semantic ORDER BY for DuckDB, backed by TypeSafe AI's Jev model. Ships with independent calibration numbers.

    yodablocks/duckdb-jev · Python

    00
    brier-scorecalibrationduckdbexpected-calibration-errorllm-evaluationpythonsql
  • JEV evaluation (FM-JEV-01): replay-only, advisory-only evaluation of Foreman-style supervision with TypeSafe Jev. Measures how much safety comes from the model versus a deterministic evidence gate. No worker authority.

    MahdiHedhli/foreman-jev-evaluation · Python

    00
    agent-supervisionai-safetyevaluation-harnessllm-evaluationpython
  • ¿Jev entiende tu acento? Pre-registered audit of TypeSafe AI's Jev on Spanish — accuracy, calibration and token cost — plus a CLI to run the same comparison on your own labelled data.

    marcosmartinez/jev-acento · Python

    01
    benchmarkcalibrationexpected-calibration-errorllm-evaluationnlppre-registrationreproducible-research
  • Independent calibration audit of TypeSafe AI's Jev: when it says 80%, is it right 80% of the time? Tested on EIA weekly petroleum reports against an LLM classifier via LangChain, with exact ground truth from EIA data.

    n9omi/jev-calibration-audit · Python

    00
    calibrationcommoditieseiaenergy-marketsforecast-verificationlangchainllm-evaluation

1–50 of 55