October 15, 2026

Beyond the leaderboard.
Decision models in a real workflow.

We tested TypeSafe Jev 1.13, Jev 1.14, OpenAI Decisions, and Microsoft Decision 1 in an existing enterprise workflow, comparing correctness and decision-pipeline latency.

Isotopes AIBy Prasanth Jayachandran

TypeSafe helped popularize decision models with the launch of Jev: give a model application context and a set of possible answers, and get back typed decisions with probabilities that code can use directly. Their post Lies, Damned Lies, and Benchmarks makes a related argument: public scores are no substitute for private evaluations on the work you actually need done. We wanted to see how these models fare inside our own enterprise application, where several decisions have to work together before a user gets an answer.

At Isotopes AI, we build aidnn, a system of intelligence over a company’s existing analytical and operational systems. It provides the infrastructure and execution layer for data agents that plan, execute, and verify work, with NeoCortex retaining business context and knowledge from previous work. People use that work through notebooks, dashboards, and data applications. Our Neutron Apps bring the data, interactive views, and follow-up questions together so people can investigate a finding without rebuilding the analysis elsewhere.

The workflow we test here starts with one of those follow-ups. Someone is looking at a view and asks to change the grouping, narrow the dates, or apply another filter. The app needs to interpret the question using the current context and decide whether it can answer from the data it already carries. This follow-up workflow predates our use of Jev. Here, we compare TypeSafe Jev, OpenAI Decisions and Microsoft Decision 1. We included Jev 1.14, announced on October 14, 2026, which was still in preview when we wrote this post.

How long do decisions take?

Lower is better · all supported attempts

Raw dataShow chart

Every run · milliseconds

Model Run Passed /88 Mean Median p95 Passing-only mean
TypeSafe Jev 1.13 1 88 842.375 791.0 1297 842.375
TypeSafe Jev 1.13 2 88 853.648 828.5 1305 853.648
TypeSafe Jev 1.13 3 88 844.466 829.0 1278 844.466
TypeSafe Jev 1.14 (preview) 1 81 895.5 835.5 1454 873.111
TypeSafe Jev 1.14 (preview) 2 85 905.318 845.5 1456 895.435
TypeSafe Jev 1.14 (preview) 3 81 882.682 839.0 1462 858.284
OpenAI Decisions 1 60 1859.466 1873.0 3115 1842.617
OpenAI Decisions 2 60 1678.841 1543.0 2903 1642.467
OpenAI Decisions 3 60 1615.841 1395.0 3034 1572.4
Microsoft Decision 1 1 55 2951.625 2781.0 5028 2966.509
Microsoft Decision 1 2 54 2704.182 2707.5 4560 2845.722
Microsoft Decision 1 3 55 2733.455 2620.0 5252 2905.6

Pooled and matched successful cases · milliseconds

Model All mean All median All p95 Passing mean Passing n Matched median Matched p95
TypeSafe Jev 1.13 846.83 806.5 1290 846.83 264 727.0 1236
TypeSafe Jev 1.14 (preview) 894.5 838.0 1456 875.931 247 743.0 1268
OpenAI Decisions 1718.049 1546.0 3034 1685.828 180 1417.5 2861
Microsoft Decision 1 2796.42 2707.5 4955 2906.311 164 2517.5 4332

The matched cohort is the same 38 scenarios passing every run for every provider: 114 observations per route. Passing-only means use different case populations.

TypeSafe Jev 1.13

TypeSafe Jev 1.14 (preview)

OpenAI Decisions

Microsoft Decision 1

Medianp95Small points: individual runs
The whiskers span the minimum and maximum across three runs. Timing includes requests and composition. It excludes final routing, runtime validation, result execution, and downstream fallback. Jev and OpenAI use direct APIs; Microsoft uses OpenRouter/Azure.

How a Neutron App answers a follow-up

Read the diagram from left to right. Someone asks a question about what is on screen. The Reflex path breaks that request into bounded choices, such as the source, metric, filters, and dates. A confidence check determines whether the app can proceed. When the request can be resolved, the app answers from the data it already carries in the browser. New or unresolved questions go to the full agent team.

How a Neutron App answers a follow-upYour systemsaidnn
Tap for the next questionrevenue by region · source: 1 of 5
  1. 01AskSomeone asks a follow-up about what's on screen.
  2. 02ChooseFast reflex models each pick from a fixed list: source, metric, filters, dates.
  3. 03CheckA confidence gate stops anything uncertain.
  4. 04AnswerThe app answers from the data it carries, in the browser.
  5. 05Hand offAnything new or unclear goes to the full Team of Rivals.
Analyze AI answers: Show support cases by support tier for the selected customer. The live result is a bar chart of Frontline, Escalation, and Engineering support cases.
A real follow-up in Analyze AI, captured from our product.

We don’t need an extra guardrail model on this path: constrained choices leave no free-text channel for prompt disclosure, and failed checks deopt to the full team’s existing guardrails.

We chose this workflow because its decisions depend on one another. Monthly totals within a date window need both the grouping and the date restriction. An early mistake can change the context for later decisions, even if those decisions carry high confidence. The confidence gate alone cannot guarantee a correct answer. We check the resulting decisions against the request’s requirements and measure decision and composition time.

As in our earlier model-provider comparison, we change the model integration while keeping the application fixed.

What we held fixed

Zero data retention was required before selecting providers. We used TypeSafe and OpenAI directly under our existing ZDR arrangements, and Microsoft Decision 1 through OpenRouter/Azure with ZDR enforced. Models unavailable under our account’s ZDR requirements were excluded, without a performance judgment. We also left Clef unevaluated: it wasn’t available through our OpenRouter setup, and Cloudflare’s direct setup required an existing Cloudflare account ID.

Application logic and the 0.50 confidence threshold stayed fixed. Starting context, instructions, choices, and ordering matched across API formats; later requests could diverge after different decisions. This comparison covers decisions and composition within the fast path. It does not time the full user request or the agent team’s work after a handoff.

What passed

Across three runs of 88 scenarios, Jev 1.13 passed 100%, Jev 1.14 passed 93.6%, OpenAI passed 68.2%, and Microsoft passed 62.1%. These are repetitions of the same scenarios, not independent tasks.

A pass requires the stated requirements and routing checks to succeed. A handoff fails a supported scenario because the fast path is expected to finish it.

Pass rates reflect the suite’s requirement and routing checks; we also checked returned rows against fixture data for a subset of passing outputs.

How long it took

The median decision and composition time was 0.807 seconds for Jev 1.13, 0.838 for Jev 1.14, 1.546 for OpenAI, and 2.708 for Microsoft.

These timings include network, serving, and composition across the chosen routes. They exclude final routing, runtime validation, result execution, and downstream fallback. A faster decision stage need not produce a faster complete answer if more requests require a handoff. Different model choices can also cause the pipeline to do different amounts of work.

Early failures can shorten a run. On the same 38 scenarios that passed every run for all four models, medians were 0.727 seconds for Jev 1.13, 0.743 for Jev 1.14, 1.418 for OpenAI, and 2.518 for Microsoft. Both cohorts are in the raw-data table.

Where attempts failed

The animation above shows the application workflow. The Sankeys show evaluation checkpoints for the same 88 scenarios: candidate creation, requirement assertions, and routing checks. Each scored attempt exits at its first failed check; these exits are not measured production handoffs.

“Requirements matched” means passing explicit assertions about scope, conditions, dates, and ordering. These checks run before the final router; another model does not judge the answer.

In one recorded OpenAI failure, paraphrased here, the request asked for monthly totals within a three-month window. The candidate kept the grouping but omitted the date restriction, despite confidence of 0.60. It failed the meaning check before the final router ran.

Clearing 0.50 did not make that answer correct. The requirement check caught the missing restriction.

Where do attempts fall through?

88 scenarios × 3 runs per model

Raw dataShow chart
Model Attempted Candidate produced Requirements matched Passed checks No candidate Requirements failed Final gate rejected Requirements failed ≥0.50
TypeSafe Jev 1.13 264 264 264 264 0 0 0 0
TypeSafe Jev 1.14 (preview) 264 261 261 247 3 0 14 0
OpenAI Decisions 264 216 204 180 48 12 24 9
Microsoft Decision 1 264 207 175 164 57 32 11 21

Every run · all 88 attempts retained

Model Run Attempted Candidate produced Requirements matched Passed checks No candidate Requirements failed Final gate rejected Requirements failed ≥0.50
TypeSafe Jev 1.13 1 88 88 88 88 0 0 0 0
TypeSafe Jev 1.13 2 88 88 88 88 0 0 0 0
TypeSafe Jev 1.13 3 88 88 88 88 0 0 0 0
TypeSafe Jev 1.14 (preview) 1 88 86 86 81 2 0 5 0
TypeSafe Jev 1.14 (preview) 2 88 88 88 85 0 0 3 0
TypeSafe Jev 1.14 (preview) 3 88 87 87 81 1 0 6 0
OpenAI Decisions 1 88 72 68 60 16 4 8 3
OpenAI Decisions 2 88 72 68 60 16 4 8 3
OpenAI Decisions 3 88 72 68 60 16 4 8 3
Microsoft Decision 1 1 88 71 59 55 17 12 4 8
Microsoft Decision 1 2 88 67 58 54 21 9 4 6
Microsoft Decision 1 3 88 69 58 55 19 11 3 7

TypeSafe Jev 1.13

TypeSafe Jev 1.14 (preview)

OpenAI Decisions

Microsoft Decision 1

Blue bands continue; gray branches stop at a grading checkpoint. Requirements are evaluation assertions, not proof of a production handoff. All providers share one thickness scale.

TypeSafe Jev 1.13

Jev 1.13 passed all 264 scorable attempts and all 57 safety/edge checks. Intermediate confidence scores varied without changing outcomes on the scorable scenarios.

TypeSafe Jev 1.14 (preview)

Jev 1.14 passed 247 of 264 supported attempts. Fourteen fell below the confidence gate and three could not produce a valid candidate. The numeric-bucket and entity-boundary scenarios fell back in all three runs. In the separate safety and edge checks, it passed 18 of 19 each run: the repeated failure omitted a requested grouping. Extra fallbacks and missing requirements are different failure modes; a failed attempt does not necessarily mean a wrong answer reached the user.

Jev 1.14 was in preview when we wrote this post. We hope these regressions are addressed before it leaves preview and will rerun the suite when it does.

OpenAI Decisions

Among scorable attempts, OpenAI produced no candidate in 48 attempts. Twelve candidates missed requirements such as conditions or ordering; nine had confidence at or above 0.50. The final router rejected another 24 for low confidence, leaving 180 passes.

The same 60 scorable scenarios passed each run, and matched requests returned identical recorded distributions. The mistakes were repeatable too. Lowering the gate would not fix missing requirements.

Microsoft Decision 1

Among scorable attempts, Microsoft produced no candidate in 57 attempts. Another 32 failed meaning checks through missing or extra conditions or incorrect scope; 21 had confidence at or above 0.50. The final router rejected another 11 for low confidence, leaving 164 passes.

Totals were close at 55, 54, and 55 passes, but ten scenarios changed between pass and fail. Scores and choices also changed for identical requests. The traces do not establish the cause.

Candidates with both incorrect meaning and low confidence count only at the meaning check, where the test stopped.

Confidence and repeatability

Calibration measures how well probabilities correspond to outcomes; repeatability asks whether identical requests return identical scores. TypeSafe’s RLCD training targets calibration, which does not establish endpoint repeatability. We did not measure calibration curves.

For one identical Jev 1.13 request, the same selected option’s confidence changed from 0.450 to 0.560. Every scorable scenario still passed: an intermediate score crossing the threshold need not change the final outcome.

Safety/edge scores were 19/19 each run for Jev 1.13, 18/19 for Jev 1.14, 15/19 for OpenAI, and 17/19, 17/19, and 16/19 for Microsoft. These include expected handoffs and positive edge cases. We retained fallback reasons, probabilities, confidence, and failed assertions to diagnose the failures.

Supporting data: safety and repeatability

Safety and edge checks

Model Run 1 /19 Run 2 /19 Run 3 /19 Safety total /57
TypeSafe Jev 1.13 19 19 19 57
TypeSafe Jev 1.14 (preview) 18 18 18 54
OpenAI Decisions 15 15 15 45
Microsoft Decision 1 17 17 16 50

Repeated identical requests

This request-level analysis covers Jev 1.13, OpenAI, and Microsoft. Jev 1.14’s workflow pass counts and latency for each run appear in the chart data above.

Model Question comparisons Probability ≥1 pp Choice changed Confidence crossed 0.50 Mixed graded workflows /88
TypeSafe Jev 1.13 7244 3091 46 73 0
OpenAI Decisions 7242 0 0 0 0
Microsoft Decision 1 6465 1458 76 109 10

We compare runs 1–2, 1–3, and 2–3 using recorded requests. These pairs overlap, so they are not independent observations. Matching requires the same case, request content, ordering, and returned model ID. These comparisons include supported, safety, and fallback calls. Changed downstream requests are excluded. The final column separately counts mixed outcomes among the 88 scorable supported scenarios.

Why the eval stays private

This is our existing regression and guardrail suite, which runs whenever we make changes, rather than a custom benchmark built for this post. Its cases encode product behavior, application context, and internal guardrails, so we’re keeping the suite private while sharing aggregate results and how we measured them.

Appendix: 20 example questions

Paraphrased examples from the suite, with names, dates, and amounts generalized. Application context and expected answers are omitted.

  1. Break down revenue by sales channel for one region.
  2. For each channel, show revenue, units sold, and the number of sales records.
  3. Which five regions generated the most revenue?
  4. How many distinct customers bought through our partner channel?
  5. Show monthly revenue for the first three months of the year.
  6. Show cumulative revenue by month.
  7. Within each region, which three sales channels generated the most revenue?
  8. Show refunds as a percentage of revenue for each region.
  9. For closed support cases, what was the median response time for each team?
  10. How much did response times vary for urgent cases, measured by standard deviation?
  11. List open billing cases that breached their service-level agreement.
  12. Within each priority level, which three teams had the slowest average response time for closed cases?
  13. What is the conversion rate for each marketing channel?
  14. Which campaigns exceeded a specified conversion-rate threshold?
  15. List campaigns with no marketing channel recorded.
  16. What was the inventory value in each warehouse on a given date?
  17. Which ten products had the lowest stock in a selected location on that date?
  18. Count service events within each of the application’s defined duration ranges.
  19. What was year-end ARR for customers due to renew in the following quarter?
  20. Show a selected customer’s open support cases, including case IDs and service products.
Method, limitations, and sources
  • Evaluation scope: The scored comparison covers 88 supported scenarios per run. We report the 19 safety/edge checks separately; the suite also includes five deterministic controls and one fallback integration check. All 12 displayed runs finished with zero HTTP or invalid-response failures. Production decision logic stayed unchanged. The same requirement and routing checks apply to every provider.
  • Three repetitions: Runs were collected on October 14, 2026, in sequential batches with model order rotated within each batch. Jev 1.14 was measured in a later batch that also repeated the 1.13 baseline; those additional 1.13 runs passed 263 of 264 attempts, with one low-confidence fallback. The charts retain the initial three 1.13 runs. This was not one interleaved four-model experiment. Local model caches and automatic test reruns were disabled. Source hashes stayed fixed. The latency whiskers show the observed minimum and maximum across runs. They are not confidence intervals.
  • Safety assertions: Safety checks accept explicit safe handoffs where appropriate, for every provider. Incorrect results and provider errors still fail. The confidence threshold stayed at 0.50.
  • Latency: Timing includes requests and composition, including early exits. It excludes final routing, runtime validation, result execution, and downstream fallback. Passing-only averages compare different case populations. The matched subset contains the same 38 scenarios passing every run for every provider, or 114 observations per route.
  • Repeatability: We paired identical recorded requests within each case and returned model identifier. Pairwise comparisons overlap. Changed downstream requests are excluded. Returned precision limits what differences we can observe; a model identifier alone does not prove that weights stayed fixed.
  • ZDR: TypeSafe and OpenAI used existing first-party agreements. Microsoft requests required Azure, ZDR, and no provider fallback. Excluded models receive no performance score.
  • Resolved models: TypeSafe Jev 1.13: jev-1.13.0; TypeSafe Jev 1.14 (preview): jev-1.14.0; OpenAI Decisions: gpt-6-luna; Microsoft Decision 1: microsoft/microsoft-decision-1-20261009. Aliases may resolve differently in future runs.
  • Provider claims: TypeSafe’s Jev launch describes RLCD. Microsoft’s Decision 1 announcement describes single-pass scoring and robustness to input perturbations. Our repeated-request analysis holds the recorded input fixed, which tests a different property. We do not assume every provider uses RLCD.
  • Private evidence: The raw-data views contain aggregates. We retain the cases, prompts, schemas, internal stage labels, orchestration, fallback reasons, probabilities, confidence scores, and failed assertions privately.

Jev 1.13 remains our default.

We’re keeping Jev 1.13 as our default for this workflow. It passed every scored attempt in the displayed evaluation and had the lowest measured decision and composition latency. Jev 1.14’s preview results gave us no reason to switch. Separately, our production experience has been roughly 2–5× faster for this workflow. That improvement is an operational observation, not a speedup measured by the provider comparison above. In production, a workflow that used to be one of the slowest parts of the application now accounts for a small share of response time. We’re happy with that improvement, and our performance work is now focused on the bottlenecks elsewhere.

Decision models are evolving quickly, with new launches and claims of better accuracy or speed arriving in quick succession. We’ll keep testing new models against the same workflow requirements. For us, a new provider needs to match or improve correctness across the workflow before a speed claim gives us a reason to switch.

aidnn is the most sophisticated AI agent for analytics, delivering verified, source-traced answers from wherever your data lives.

Talk to Sales