Projects tagged with “llm-evaluation”
Browse Jev GitHub projects tagged with llm-evaluation. Filter, sort and explore their source repositories.
- 🔥🔥 Papers, open reproductions and independent evaluations behind System One models and Jev.
OmniJev/awesome-jev-gallery · Python
49443awesomeawesome-listcalibrated-probabilitiescalibrationdecision-modelllmllm-evaluation - 123jevalsAgent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya.
openlayer-ai/jevals · Python
12013ai-agentsevaluationguardrailsllmllm-evaluationragas - Independent, evidence-based map of when TypeSafe's Jev actually holds up vs. breaks down — real API-call receipts, not a leaderboard. 中文為主的雙語 repo。
Zaious/jev-capability-atlas · Python
306ai-agentsbenchmarkcalibrationllm-evaluationmachine-learningzh-tw - 378snapjudgeTyped decisions (choice / score / yes-no) from local Qwen models on Apple Silicon. Probabilities come straight from the logits, no text generation. TypeSafe-compatible HTTP API, runs on MLX.
Micha0827/snapjudge · Python
140apple-siliconclassificationdecision-makingfastapillmllm-evaluationlocal-llm - 401what-the-jevWhat can Jev actually do? Reproducible experiments and research reports exploring its capabilities and limits.
keta1930/what-the-jev · Python
131benchmarkdecision-modelevaluationllm-evaluation - Independent, reproducible benchmark: a decision model (jev), eight constrained LLMs, and deterministic baselines on typed decisions - accuracy, calibration, latency, cost, failure modes
nibzard/decision-model-benchmark · HTML
121benchmarkcalibrationdecision-modelllm-evaluationreproducibilityhtml - 447jev-stateBuild and regression-test conversational state machines powered by Jev. Inspect decisions, capture failing conversations as tests, and export runnable TypeScript for your app.
priyankark/jev-state · TypeScript
101ai-agentscode-generationconversational-aideveloper-toolsevaluationllm-evaluationlocal-first - 450jevloopA Jev-driven general-purpose agent harness for faster, lower-cost execution, with built-in side-by-side experiments against LLM-only agents.
parkavenue9639/jevloop · Python
100agent-runtimeai-agentsdockerllm-evaluationobservabilitypythonsandbox - 520pytest-jevSemantic assertions for pytest: test what your LLM app's output means, judged by TypeSafe's Jev.
allebee/pytest-jev · Python
81llmllm-evaluationpytestpytest-pluginpythontesting - Pre-registered independent eval of TypeSafe Jev against a nano-class LLM, a frontier LLM, and a supervised encoder (Banking77 + CLINC150 zero-shot)
ickma2311/jev-baselines-eval · Python
60intent-classificationllm-cascadellm-evaluationpreregistrationpython - 620jev-2048An instrumented 2048 web lab where every move is a Jev (TypeSafe AI System One) Choice, with no heuristic fallback | 用 Jev 决策模型驱动每一步的 2048 网页实验台,概率、置信度、延迟与成本全部摊开可见,且刻意不做启发式兜底
ARCJ137442/jev-2048 · TypeScript
512048llm-evaluationtyped-decisionstypescript - Fifty real-world financial use cases for TypeSafe's Jev model: typed, structured LLM answers over ledgers, fraud, portfolios, trades and filings, each graded against data where the right answer is known.
IslamBaraka90/jev-typesafe-real-financial-use-cases · JavaScript
51ai-agentsbacktestingfinancial-datafintechjavascriptllmllm-evaluation - 645jevossEvaluation tools for Jev-compatible decision models, covering accuracy, calibration, hidden instructions, padding and consistency.
mertkayacs/jevoss · Python
51benchmarkcalibrationdecision-modelevaluationllm-evaluationprompt-injectionpython - 709jev-skillsPractical agent skills and examples for building with Jev. API setup, routing, ranking, and evidence checks.
laguagu/jev-skills
41agent-pluginsagent-skillsclaude-codecodexevidence-verificationjavascriptllm-evaluation - 774gr-jevCode for 'When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models' (arXiv:2610.29217)
Rushab1/gr-jev · Python
30arxivcalibrationchaosnlidecision-modelecthrhotpotqallm-evaluation - Free English RAG benchmark for TypeSafe Jev 1.13 — frozen candidate pools, calibration, paired bootstrap CIs, $0 runs. Compared with OpenJev and NVIDIA cross-encoders.
emretheus/jev-rag-benchmark · Python
30benchmarkcalibrationinformation-retrievalllm-evaluationnvidiaopenjevrag - Independent Jev / System One reference with API examples, implementation guidance and reusable prompts for engineers and coding assistants.
sebastianbennis/jev-system-one-reference
30ai-agentsapi-referenceclassificationcoding-assistantllmllm-evaluationprompt-engineering - 839jevalyzerGrade the agent sessions already on your disk. Claude Code, Codex, opencode, Gemini CLI and Antigravity, scored with Jev for cents.
killerz3/jevalyzer · TypeScript
30ai-agentsbunclaude-codeclicodexgemini-clillm-evaluation - Jev vs GPT-4.1 as synthetic survey respondents on Twin-2K-500. How you ask mattered more than which model you used.
jjd-lab/jev-synthetic-survey · Python
20behavioral-economicsbenchmarkcalibrationdigital-twindigital-twinsllmllm-evaluation - LangChain + Jev (TypeSafe) tutorial: a support-ops agent whose small decisions (triage, model routing, tool guarding, evals) are made by Jev. Real run outputs included.
PromptEngineer48/langchain-jev-tutorial · Python
21ai-agentslangchainlangsmithllm-evaluationpython - 1073tallyTurn a pile of comments into an auditable distribution: Jev judges each item, code counts the percentages. React workspace + Python engine with pluggable data sources.
adlternative/tally · Python
20data-analysisi18nllmllm-evaluationpythonreact - 1137decideBulk decisions for AI agents. Jev classifies files and logs; your agent reviews exceptions. Reproducible cost and quality benchmarks.
alsoleg89/decide · Python
10ai-agentsclassificationclaude-codecodexcontext-engineeringllm-evaluationmcp - A Jev-powered quality gate for GitHub issues, pull requests, and commits.
Bnymn1306/jev-github-quality-gate · TypeScript
10ai-evaluationdeveloper-toolsgithubllm-evaluationquality-gate - 1318jev-labO Jev (TypeSafe AI) como guardrail em tempo real e como pré-avaliador de agentes conversacionais de negócio. Next.js + LangChain via Vercel AI Gateway, com estudo de calibração em português.
JoaoClemer/jev-lab · TypeScript
10ai-agentsguardrailslangchainllm-as-judgellm-evaluationnextjs - 1321jev-labHands-on research lab for TypeSafe's Jev (System One model): reproducible benchmarks of Noul/Choice/Score primitives, confidence gating, fan-out latency, agent control — plus a living audit of the Jev ecosystem.
llt22/jev-lab · Python
10ai-agentsbenchmarkconfidence-calibrationevaluationllmllm-evaluationllm-observability - 1359jev-report发明 RLHF 的人,这次做了个不会说话的模型:Jev 独立研究报告。52 页 PDF + 50 条中文实测复现包 + 143 条可回溯数据表
HackSing/jev-report · Python
10pythonai-researchchinesellm-evaluation - 1371jev-skillJev skill for Claude Code and Codex: TypeSafe System One guide, Python CLI, 5 tested recipes, and findings from 19,367 Jev calls.
marcodicesare-dev/jev-skill · Python
10agent-skillsai-agentsclassificationclaude-codecodexdecision-modelllm-evaluation - 1416jevals-dataIndependent benchmark data for TypeSafe's Jev (System One model) vs LLMs: accuracy, calibration, cost. Boards + per-decision logs, CC-BY-4.0
Jevals/jevals-data
12benchmarkcalibrationdatasetleaderboardllm-evaluation - 1457JudgeJevA hands-on educational lab for DeepEval + Jev: inspect real recorded judgments, explore release gates and drift playback, and run fresh evaluations on your own workloads.
shyamsridhar123/JudgeJev · Python
10deepevaldrift-detectionenterprise-aievaluationllm-evaluationllmopspython - 1528s1kitTest, calibrate and route System One decision models (Jev, Laya, kev, llama.cpp, Ollama) on your own data.
Eliot5566/s1kit · Python
10calibrationclidecision-modelgithub-actionslayallm-evaluationllm-routing - Recorded comparison of Jev, Terra, and Opus on 100 synthetic security-triage cases, five passes each, with a static inspectable dashboard
Robertzu43/system-one-security-triage · TypeScript
10ai-securitybenchmarkllm-evaluationsecuritytypescript - 1635arbitypeTyped decision tools for AI agents, powered by TypeSafe Jev
Renwang-Huang/arbitype · Python
01ai-agentsclassificationclaudecursordecision-makingguardrailsllm-evaluation - A 60-game benchmark of TypeSafe's Jev evaluation model playing Battleship. The model matches plain code; it does not beat it.
ickas/battleship-vs-jev · TypeScript
00battleshipbenchmarkevaluation-metricsllm-evaluationtypescript - Use Jev instead of an LLM for RAG decisions: relevance grading, query routing, hallucination checks and LLM-as-a-judge evals. Faster, cheaper, measured first.
davidzna/better-cheaper-llm · Python
00cost-optimizationgroundednessguardrailshallucination-detectionllmllm-as-a-judgellm-evaluation - 1746darwins-worldAn evolution simulation whose only law of nature is an AI judgment model's yes/no answers. Islands evolve what common sense expects; a predator-prey arms race appears only when predators alone select prey, tested against no-interaction, fixed-predator and shuffled controls.
isaacbitran/darwins-world · Python
00ai-researchartificial-lifecoevolutionevolutionevolutionary-simulationjudgment-modelllm-evaluation - 1790duckdb-jevSemantic ORDER BY for DuckDB, backed by TypeSafe AI's Jev model. Ships with independent calibration numbers.
yodablocks/duckdb-jev · Python
00brier-scorecalibrationduckdbexpected-calibration-errorllm-evaluationpythonsql - JEV evaluation (FM-JEV-01): replay-only, advisory-only evaluation of Foreman-style supervision with TypeSafe Jev. Measures how much safety comes from the model versus a deterministic evidence gate. No worker authority.
MahdiHedhli/foreman-jev-evaluation · Python
00agent-supervisionai-safetyevaluation-harnessllm-evaluationpython - 1929jev-acento¿Jev entiende tu acento? Pre-registered audit of TypeSafe AI's Jev on Spanish — accuracy, calibration and token cost — plus a CLI to run the same comparison on your own labelled data.
marcosmartinez/jev-acento · Python
01benchmarkcalibrationexpected-calibration-errorllm-evaluationnlppre-registrationreproducible-research - Independent calibration audit of TypeSafe AI's Jev: when it says 80%, is it right 80% of the time? Tested on EIA weekly petroleum reports against an LLM classifier via LangChain, with exact ground truth from EIA data.
n9omi/jev-calibration-audit · Python
00calibrationcommoditieseiaenergy-marketsforecast-verificationlangchainllm-evaluation
1–50 of 55