Jev 1.13
50/50100% exact-label accuracy
- Mean latency
- 481 ms
- Total batch cost
- $0.003405
Small decisions.
Fully inspectable.
Jev 1.13 vs GPT-4o-mini
50 synthetic classification cases
GPT-4o-mini
50/50100% exact-label accuracy
- Mean latency
- 742 ms
- Total batch cost
- $0.005311
Please refund the duplicate charge from this morning.
Expected labelrefund
Why this label?
The direct request is to return a duplicate charge, which is a refund request.
Jev 1.13
PredictionrefundCorrect
Latency474 ms
GPT-4o-mini
PredictionrefundCorrect
Latency974 ms
Behind the numbers
The method. And its limits.
A small, fully published pilot.
Evidence you can inspect and download.
How scoring worksExact-label accuracy: correct predictions divided by all 50 cases. One request per case and model, with no automatic retries. A case is either correct or incorrect; it has no percentage score.
What this establishesBoth models scored 50/50 on this fixed synthetic set. The sample is small, AI-authored and AI-reviewed. It does not establish general superiority, production accuracy, or statistical equivalence.
Dataset & scoring rubric 50 cases · 6 labels
All messages are AI-authored original synthetic text and AI-reviewed against the frozen rubric. No human or expert labeling is claimed. No real customer, Reddit, or scraped message is included.
Assign exactly one primary intent label to each short customer-support message. Negated requests are excluded. When supported intents overlap, apply this priority: refund → account issue → technical support → product inquiry → complaint. Use other when no supported intent remains.
refund
The user asks for money to be returned, a charge to be reversed, or a purchase-related refund.
technical support
The user needs help diagnosing or fixing an active software, device, integration, or service malfunction.
account issue
The user needs help with account access, credentials, security, profile details, login identity, or account administration.
product inquiry
The user asks what a product, plan, feature, policy, eligibility rule, or coverage option offers or supports.
complaint
The user expresses nonspecific dissatisfaction and provides no more actionable supported request.
other
A greeting, thanks-only message, or out-of-scope message with none of the five supported intents.
The models receive the original message and shared rubric. Expected labels, rationales, translations and case IDs are excluded from the prompts.
Run provenance & measurements Exact served revisions
- Run ID
- 20260920T100629126Z-536fff3c
- Dataset ID
- decivo-jev-classification-pilot-v1
- Started / completed (UTC)
- 2026-09-20T10:06:29.127Z / 2026-09-20T10:07:30.453Z
Jev 1.13
Served: typesafe/jev-1.13-20260917
Upstream provider: TypeSafe
Median latency: 451.211 ms
P95 latency: 629.001 ms
Reported batch cost: $0.003405024
GPT-4o-mini
Served: openai/gpt-4o-mini-2024-07-18
Upstream provider: OpenAI
Median latency: 708.225 ms
P95 latency: 973.680 ms
Reported batch cost: $0.005311200
Latency is end-to-end request time and includes network and provider overhead. Costs are the API-reported usage charges for this batch. A single run does not characterize latency variability or future prices.