Projects tagged with “evaluation”
Browse Jev GitHub projects tagged with evaluation. Filter, sort and explore their source repositories.
- 123jevalsAgent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya.
openlayer-ai/jevals · Python
12013ai-agentsevaluationguardrailsllmllm-evaluationragas - Probability-aware evaluation for typed decision models: calibration, selective risk, latency, and reproducible benchmarks.
AbdelStark/jev-benchmarks · Python
285pythonbenchmarkingcalibrationevaluation - 323typed_evalsFast, typed, calibrated evaluations for LLM and agent outputs, powered by Jev — with simple, framework-agnostic Python APIs
TrustifAI/typed_evals · Python
191ai-agentscrewaievaluationevaluation-frameworkevaluation-metrics - 357jev-harnessDecision harness for TypeSafe Jev — confidence gates, shadow mode, recipes, and evals. Claude CLI 48.9s → Jev 1.3s on the same row-filter job.
AntonioCoppe/jev-harness · TypeScript
152ai-agentsconfidencedecisionevaluationharnessllm - 401what-the-jevWhat can Jev actually do? Reproducible experiments and research reports exploring its capabilities and limits.
keta1930/what-the-jev · Python
131benchmarkdecision-modelevaluationllm-evaluation - 447jev-stateBuild and regression-test conversational state machines powered by Jev. Inspect decisions, capture failing conversations as tests, and export runnable TypeScript for your app.
priyankark/jev-state · TypeScript
101ai-agentscode-generationconversational-aideveloper-toolsevaluationllm-evaluationlocal-first - 601sysone-benchFirst independent head-to-head benchmark of System One decision models (Laya vs Jev) on byte-identical inputs
instax-dutta/sysone-bench · Python
63benchmarkcalibrationdecision-modelevaluationlayallm-eval - 637jev-testA test harness for TypeSafe's Jev :Confidence audit, Hallucination judge, Jev for RAG
mity-prodgen/jev-test · Python
51evaluationmodelragpython - 645jevossEvaluation tools for Jev-compatible decision models, covering accuracy, calibration, hidden instructions, padding and consistency.
mertkayacs/jevoss · Python
51benchmarkcalibrationdecision-modelevaluationllm-evaluationprompt-injectionpython - Ten runnable Jev examples for agent decisions: memory conflicts, tool-result checks, recovery, context selection, and handoffs. JavaScript, zero dependencies.
ReallyArtificial/jev-by-example · JavaScript
41agent-memoryai-agentsevaluationexamplesgood-first-issuehacktoberfesthelp-wanted - Three measured experiments on RAG hallucination: quote-checking, TypeSafe's Jev, and IBM's STAIR. 850+ graded questions, raw responses included.
aryanchauhanoffical/no-hallucination · Python
40benchmarkevaluationhallucinationllmragretrieval - 727openevalsFast and cheap agent evals. jev as judge.
memovai/openevals · TypeScript
40ai-agentsevaluationlangchainlangfuseobservabilitytypescript - A reproducible, self-hosted benchmark for comparing JEV and LLM decision-making on your own datasets, models, and evaluation policies.
Tenkei/jev-decision-bench · Python
30ai-agentsanthropicbenchmarkdeepseekevaluationllmopenai - Standalone TypeSafe AI/Jev semantic-evidence adapter for bounded Agent-Workflow decisions, explicit uncertainty, compatibility evaluation, and host-owned policy.
ngallodev-software/agent-workflow-typesafe-ai · Python
10agentic-aiai-agentsdecision-supportdeveloper-toolsevaluationpythonsemantic-reasoning - 1199here-we-go-jevLocal playground and test bench for TypeSafe's Jev System One model: typed questions, calibrated-probability answers, and side-by-side comparison with an LLM baseline.
creativoma/here-we-go-jev · TypeScript
10buncalibrationclassificationevaluationllmopenrouter - 1233jev-benchBenchmark TypeSafe's Jev (System One model) vs Claude on structured decisions: accuracy, latency, calibration, cost
stern9/jev-bench · Python
10anthropicbenchmarkclaudeevaluationllmpython - 1258jev-crash-courseAn 11-level crash course on Jev, TypeSafe AI's System One decision model — runnable examples against the real API, plus a capstone project with unit tests and evals. Works with any LLM provider via LiteLLM.
nadeemcite/jev-crash-course · Python
10ai-agentscrash-courseevaluationlitellmllmpython - 1312jev-judgeYAML-based evaluation runner for recorded LLM outputs, with TypeSafe Jev API integration, offline mock mode, and CI reports.
00200200/jev-judge · Python
10ai-agentsci-cdevaluationllm-as-a-judgeragtesting - 1321jev-labHands-on research lab for TypeSafe's Jev (System One model): reproducible benchmarks of Noul/Choice/Score primitives, confidence gating, fan-out latency, agent control — plus a living audit of the Jev ecosystem.
llt22/jev-lab · Python
10ai-agentsbenchmarkconfidence-calibrationevaluationllmllm-evaluationllm-observability - Reproducible riichi mahjong benchmark for Jev, GPT, Mortal, and hybrid agents using MJAI and RiichiEnv.
hamakyo/jev-mahjong-bench · TypeScript
10benchmarkevaluationgame-aigpthybrid-agentsmahjongmahjong-ai - Applying Jev to Snips Natural Language Understanding Benchmark
will-rice/jev-snips-benchmark · Python
11benchmarkevaluationfew-shotintent-classificationnatural-language-understandingnluslot-filling - 1418jevbenchPreregistered, bias-corrected test of TypeSafe Jev's calibration under human disagreement (ChaosNLI, 100 labels per item)
GautamTalksDev/jevbench · Python
10calibrationchaosnlievaluationpreregistrationpython - 1457JudgeJevA hands-on educational lab for DeepEval + Jev: inspect real recorded judgments, explore release gates and drift playback, and run fresh evaluations on your own workloads.
shyamsridhar123/JudgeJev · Python
10deepevaldrift-detectionenterprise-aievaluationllm-evaluationllmopspython - 1488nocapNo evidence, no answer. 🧢 System One evidence gate for RAG & AI agents: Jev / Laya / any LLM judges before your LLM speaks. MCP server, Python SDK, CI route tests.
xi029/nocap · Python
10ai-agentsclaude-codedecision-modeldeepseekevaluationguardrailshallucination - 1540stuntdoubleDrop-in /v1/systemone proxy that shadows Jev with local decision models (Kev, Laya) and reports whether you can swap
ReallyArtificial/stuntdouble · JavaScript
10ai-agentsevaluationgood-first-issuehelp-wantedjavascriptkevlaya - 1736crosstalk-labFind where an API router confuses look-alike operations, rewrite the wording, and rerun. Runs offline in mock mode; live mode uses TypeSafe's Jev.
Chaddacus/crosstalk-lab · TypeScript
00data-visualizationevaluationllmopenapitool-routingtypescript - 1814fastgate-jevJev (TypeSafe AI) as a System One decision layer for a multilingual EN/UZ/RU RAG helpdesk, with an independent benchmark
rajantripathi/fastgate-jev · Python
00ai-agentsevaluationllm-routingmultilingualraguzbek - 1917jev_naicsClassify a business's description of operations into one of 1,012 NAICS 2022 industries, or hold it for a person. No LLM writes anything: Postgres search and code-owned gates do the work, and TypeSafe Jev only picks from a closed list. The hybrid: 0 wrong codes on its eval sets.
jmurray10/jev_naics · Python
00ai-agentscensusdockerevaluationfastapifull-text-searchindustry-classification - 1928jev-access-dayA learning scaffold for TypeSafe AI's System One models: eval harness plus a measured, plain-language comparison of the Jev decision model vs an LLM stand-in on 24 real operational decisions. All numbers reproducible from committed run files.
andreaserradev-gbj/jev-access-day · TypeScript
00benchmarkdecision-supportevaluationllmtype-safety - Evaluate typed AI decisions on labeled French-language cases.
gbesse/jev-banc-francais · JavaScript
00calibrationevaluationfrancejavascript - 1976jev-baseballTesting TypeSafe Jev (a decision-only model) on MLB judgement tasks against simple baselines
yasumorishima/jev-baseball · Python
00baseballevaluationllmmlbpython - 2057jev-crash-courseAn 11-level crash course on Jev, TypeSafe AI's System One decision model — runnable examples against the real API, plus a capstone project with unit tests and evals. Works with any LLM provider via LiteLLM.
nadyth/jev-crash-course · Python
00ai-agentscrash-courseevaluationlitellmllmpython - 2085jev-demoDecision Arena: TypeSafe's Jev vs open-source Laya playing highway-env, Snake and Blackjack with zero training, plus benchmarks and a Claude Code watchdog
nadeem4/jev-demo · TypeScript
00arenabenchmarkclaude-codedecision-modelevaluationhighway-env - 2123jev-eval-harnessLLM-as-a-judge evaluation harness: a confidence-gated cascade (fast typed judge → Claude) that scores LLM outputs and gates CI on a regression diff. TypeScript, real recorded transcripts, real benchmark numbers.
laszloblum/jev-eval-harness · TypeScript
00anthropicciclaudeeval-harnessevaluationllmllm-as-a-judge - 2144jev-field-trialA pre-registered field trial of Jev (TypeSafe's judgment model) on a second brain and Claude Code history: 20 tests, bars written first, failures included, and the tools to repeat it.
NaluKicks-808/jev-field-trial · Python
00ai-agentsclaude-codeevaluationsecond-brainpython - 2248jev-llm-judgeCan a decision model replace an LLM as an eval judge? Same rubrics, scored by Jev and by an LLM, side by side.
nadheesh/jev-llm-judge · Python
00agent-evaluationevaluationllm-as-a-judgellm-evaluationpython - 2302jev-packsEvidence-gated registry of Jev question packs — curated questions, golden cases and measured evidence for Jev-compatible decision endpoints
dtduc-git/jev-packs · Python
00ai-testingcalibrationevaluationgolden-datasetllm-evaluationquestion-packspython - Terminal-based Jev intent triage with typed decisions, explicit simulation mode, review policy and JSON exports.
suprkco/jev-serp-opportunity-lab · Python
00evaluationpythonseostructured-output - 2460jev-tool-callingApplying Jev to zero-shot tool calling on the Berkeley Function Calling Leaderboard (BFCL)
will-rice/jev-tool-calling · Python
01benchmarkbfclevaluationfunction-callingllm-agentstoken-classificationtool-calling
1–43 of 43