Jev is live on MindsHub Inference. Here’s how we’re testing it inside MindsHub Agents.
Jev, TypeSafe’s decision model, is now available through MindsHub Inference.
After adding support, we asked the obvious follow-up: where would we use it ourselves?
We started with a decision MindsHub Agents makes on every message: can it answer directly, or does it need tools?
Across 300 test runs (30 hand-written prompts, each run ten times), Jev decided about 2.5 times faster than our router on agent-bound requests, never timed out, and was right on every call where it chose the agent. Accuracy was otherwise comparable: it disagreed with our router on one prompt, consistently, and both broke our own routing rule on two others. That makes Jev a credible shortcut for starting agent work sooner, not yet a proven routing upgrade.
Why routing matters
A direct answer takes one model call. A request that needs tools starts an isolated environment where the agent can browse, run code and create files.
“What does HTTP 429 mean?” can be answered directly. “Search the web for our latest announcement and turn it into a one-page PDF” needs the agent. The router is a single model call. It decides the route, and on direct requests, the same call writes the answer. Jev classifies only, so it can run ahead of the router rather than replace it.
Choosing the wrong path has a cost. Starting an agent unnecessarily makes a simple answer slower and more expensive. Answering directly when a request needs live information or a file can leave the task incomplete or produce an unsupported answer.
Testing alongside our existing router
Jev is running in shadow mode: it makes its own decision, but the existing router still controls what happens. MindsHub Agents does not wait for the shadow result before proceeding.
We ask Jev to choose between answer_directly and needs_agent, using our routing rules. Those rules cover requests for tools, files, current information, calculations and multi-step plans. Questions about MindsHub Agents or the assistant’s runtime also go to the agent. The exact abbreviated prompt is in the appendix.
We tested straightforward questions and harder cases, repeating each prompt ten times in shuffled order. Repetition helped distinguish recurring mistakes from results that changed between runs. These are synthetic tests, not a representative sample of customer conversations. The comparison isn’t fully even: the router’s prompt has been tuned over weeks while Jev’s criteria were written in a day, router timeouts count as correct when the agent fallback happens to be right, and we tested against a single router model, the MindsHub Air model, our free default.
Faster decisions, comparable routing
Across the 136 runs where both systems chose the agent and the router returned a decision, the median time was about 1.5 seconds for the router and 0.6 seconds for Jev.
That is a promising gap before agent work starts. It is not yet a measured reduction in users’ waiting time. On direct requests, the router also writes the answer, so comparing its full response time with Jev’s classification time would be misleading.
We scored routes against labels assigned before testing. Router timeouts fall back to the agent; the table counts that as a match when the expected route is agent.
| Test set | Existing router, including fallback | Jev |
|---|---|---|
| Straightforward questions | 196/200 | 190/200 |
| Harder questions | 83/100 | 81/100 |
| Total matching expected routes | 279/300 | 271/300 |
The router’s total includes 24 timeouts that fell back to the agent and matched by default. Outside the two disputed rows, both systems were about 97% right (275/280 and 270/280). Jev’s ten misses outside those rows are all on the spreadsheet prompt; the router’s five are spread across four prompts.
The repeated examples explain more than the totals:
- The spreadsheet request exposed a consistent Jev mistake. It chose a direct answer in all ten runs, even though the user requested a spreadsheet. The router chose the agent every time.
- Compound interest exposed router variability. Jev chose the expected agent route every time. The router answered directly twice, chose the agent three times and timed out five times.
- Some harder questions exposed ambiguity in our rules. Our policy sends even simple calculations to the agent. The router answered the seconds-in-a-week question directly every time; Jev did so in nine of ten runs. The quicksort question was also disputed because “today’s data” could mean a general question or a request to inspect a dataset.
Confidence helped identify some uncertain cases, but it was not a guarantee. Jev’s median confidence on the spreadsheet was 0.60. On the disputed quicksort question, it consistently chose direct with a median confidence of 1.00. We cannot choose a reliable threshold from those examples alone. The next step is to sweep a cutoff on Jev’s returned probability against a few hundred independently labeled conversations, with a held-out set to check it.
What could this change in MindsHub Agents?
The approach we want to test is a shortcut: when Jev confidently identifies a request that needs tools, start the agent sooner. In these runs, all 161 of Jev’s needs_agent decisions matched the expected route, so the shortcut would not have started a single unnecessary sandbox.
That distinction matters. Jev’s spreadsheet mistake would still pass through the router, which caught it in this experiment. But if Jev mistakenly sent a simple question straight to the agent, it could create an unnecessary sandbox run. We need to evaluate the combined system, not just the two classifiers separately.
Running Jev first would add about 0.6 seconds before requests that still need the router. Running both concurrently could avoid that sequential delay, but their actual latency and billing effects need to be measured.
Cost is another reason to investigate. Using token counts and the rates in the Cost section below, a million Jev calls would cost about $24, compared with about $128 for router delegation calls. A direct router answer would cost more, but it also produces the response that Jev cannot write.
Avoiding an unnecessary agent task may matter more than the classifier’s bill. One PDF task’s recorded usage, valued at our assumed model rates, came to about $0.0066 before sandbox compute. That is an illustration from one task, not a typical agent cost.
What we learned
- Jev was about 2.5 times faster than the router on agent-bound requests (0.6 seconds vs. 1.5), and the gap held across all 30 prompts.
- Jev returned a classification on all 300 runs. The router timed out on 25, each time spending its full 2-second budget before falling back to the agent.
- All 161 of Jev’s
needs_agentdecisions matched the expected route. The proposed shortcut acts only on those, so on this set it would not have started a single unnecessary agent session. - Jev’s one mistake was systematic: the same prompt, all ten runs, at its lowest confidence outside the two disputed rows. The router’s mistakes varied from run to run on the same prompt.
- Outside the two rows where our own policy is disputed, accuracy was comparable: 275/280 for the router and 270/280 for Jev.
Speed alone is not enough. The router matched slightly more expected routes, Jev repeated one clear mistake consistently, and some disagreements revealed ambiguity in our routing policy. We’re keeping Jev in shadow mode while we test the proposed shortcut on representative conversations with independently reviewed labels. The next question is whether it reduces waiting without starting unnecessary agent sessions. That is the result we need before changing how MindsHub Agents routes requests.
Methods and limitations
We tested 30 handwritten prompts, 20 straightforward and 10 harder, 10 times each, for 300 turns. Requests ran in shuffled order, two seconds apart, each in a fresh conversation. This test repeated behavior on a small set of prompts; it does not provide 300 different scenarios or test normal conversation history.
Expected labels were assigned by the author before testing and were not independently reviewed. Two harder labels are disputed, and one follow-up prompt is artificial without prior conversation. Scores describe agreement with these labels, not independently established answer quality.
The router used mindshub_air; Jev reported model version jev-1.13.0. Both timers ran server-side inside MindsHub Agents. Router timings include answer generation on direct turns; Jev timings include a new connection, our inference gateway, and the provider request. We did not control caching or verify the reuse of router connections. Router timeouts are excluded from its timing summaries but included in routing outcomes through the fallback. The confidence figures below are medians, not probabilities of being correct. These runs logged only confidence; the probe now logs the probability distribution as well.
Cost
Router costs use the MindsHub Air model’s published rates at the time of testing: $0.20 per million input tokens and $1.20 per million output tokens. Jev has no MindsHub list rate; the table values it at the rate used in the test, about $0.04 per million input tokens. Both models are in the MindsHub free tier, so within the fair-use policy neither call is billed to MindsHub users.
| Call | Measured tokens (median) | Cost per call | Per million calls |
|---|---|---|---|
| Router, deciding to delegate | 344 in, 49 out | $0.000128 | about $128 |
| Router, writing a direct answer | 343 in, 155 out | $0.000255 | about $255 |
| Jev | 570 in, no output charge | $0.000024 | about $24 |
Appendix: full results
Each row summarizes ten runs. “Matches / timeouts” reports the number of final routes matching the expected label, followed by the number of router timeouts. Timeouts fall back to the agent and can therefore count as matches. Router times exclude timeouts and include answer writing when it replies directly. All times below are milliseconds; confidence is the median across runs. Prompts are quoted as sent, so some say Cowork, the name of the MindsHub Agents workspace mode.
Straightforward questions
| # | Prompt | Expected | Router: matches / timeouts | Router ms: median (range) | Jev: matches, confidence | Jev ms: median (range) |
|---|---|---|---|---|---|---|
| 1 | What’s the difference between a list and a tuple in Python? | direct | 10/10, 0 | 2860 (2627–3481) | 10/10, 1.00 | 608 (523–990) |
| 2 | Explain what a race condition is in concurrent programming. | direct | 10/10, 0 | 2984 (2644–3485) | 10/10, 1.00 | 594 (544–728) |
| 3 | What does HTTP status code 429 mean? | direct | 10/10, 0 | 2167 (1876–3346) | 10/10, 1.00 | 576 (530–634) |
| 4 | Summarize the CAP theorem in two sentences. | direct | 9/10, 1 | 1997 (1718–2320) | 10/10, 1.00 | 602 (530–673) |
| 5 | What’s the time complexity of binary search? | direct | 10/10, 0 | 2192 (1616–2732) | 10/10, 1.00 | 594 (535–684) |
| 6 | What’s the difference between TCP and UDP? | direct | 10/10, 0 | 3048 (2592–5358) | 10/10, 1.00 | 626 (570–1092) |
| 7 | Explain what idempotency means in the context of an API. | direct | 10/10, 0 | 3358 (2745–5013) | 10/10, 1.00 | 623 (550–1069) |
| 8 | What is a deadlock in operating systems? | direct | 10/10, 0 | 2872 (2611–5508) | 10/10, 1.00 | 613 (553–1083) |
| 9 | What’s the difference between a process and a thread? | direct | 10/10, 0 | 3204 (3092–3731) | 10/10, 1.00 | 592 (556–670) |
| 10 | Explain the difference between authentication and authorization. | direct | 10/10, 0 | 2076 (1977–2502) | 10/10, 1.00 | 581 (536–819) |
| 11 | What’s the current price of Bitcoin in USD? | agent | 10/10, 1 | 1486 (1042–1702) | 10/10, 1.00 | 590 (527–1048) |
| 12 | Search the web for MindsDB’s latest product announcement and summarize it in three bullet points. | agent | 10/10, 1 | 1465 (1092–1932) | 10/10, 1.00 | 606 (561–1134) |
| 13 | Create a one-page PDF summary of the benefits of vector databases. | agent | 10/10, 0 | 1573 (1214–1895) | 10/10, 0.72 | 594 (548–675) |
| 14 | Calculate the compound interest on $10,000 at 5% annually for 7 years. | agent | 8/10, 5 | 1567 (1256–2462) | 10/10, 0.78 | 594 (539–922) |
| 15 | What’s tomorrow’s weather forecast for San Francisco? | agent | 10/10, 0 | 1515 (1166–1637) | 10/10, 1.00 | 606 (551–668) |
| 16 | What’s 847293 divided by 37, rounded to two decimal places? | agent | 9/10, 0 | 1679 (1311–2106) | 10/10, 0.75 | 579 (538–637) |
| 17 | Search the web for today’s top tech news headline. | agent | 10/10, 4 | 1456 (1174–1932) | 10/10, 1.00 | 610 (530–718) |
| 18 | What’s my current account balance? | agent | 10/10, 0 | 1552 (1281–1627) | 10/10, 1.00 | 588 (547–642) |
| 19 | How many unread emails do I have right now? | agent | 10/10, 0 | 1522 (1151–1746) | 10/10, 1.00 | 586 (490–710) |
| 20 | Create a spreadsheet listing the prime numbers between 1 and 100. | agent | 10/10, 0 | 1436 (1212–1986) | 0/10, 0.60 | 600 (527–689) |
The router matched 196/200 expected routes, including 11 correct timeout fallbacks. It timed out 12 times; the CAP theorem timeout produced the one incorrect agent fallback. It gave three incorrect direct answers. Jev matched 190/200 routes, with all ten errors on the spreadsheet prompt. The two systems agreed on 175 of the 188 runs where the router returned a decision.
In the 76 runs where both chose agent and the router returned a decision, the reported median timings were 1.52 seconds for the router (range 1.04–1.93) and 0.60 seconds for Jev (0.49–1.13).
Harder questions
| # | Prompt | Expected | Router: matches / timeouts | Router ms: median (range) | Jev: matches, confidence | Jev ms: median (range) |
|---|---|---|---|---|---|---|
| 1 | Are you GPT-4, or a different model? | agent | 10/10, 2 | 1556 (1181–1739) | 10/10, 0.97 | 595 (528–1464) |
| 2 | Does Cowork support connecting to Slack? | agent | 10/10, 0 | 1430 (1331–1878) | 10/10, 0.96 | 593 (543–649) |
| 3 | What’s the latest stable release of PostgreSQL? | agent | 10/10, 2 | 1482 (1235–1565) | 10/10, 0.93 | 568 (512–1071) |
| 4 | Roughly how many seconds are in a week? | agent by policy, disputed | 0/10, 0 | 1631 (1248–2104) | 1/10, 0.05 | 609 (521–660) |
| 5 | Explain what a hash table is, and check if this repo has one in its data structures folder. | agent | 10/10, 0 | 1490 (1117–1831) | 10/10, 0.96 | 616 (540–667) |
| 6 | If I were to reverse a linked list of length n, would that take O(n) or O(n log n)? | direct | 10/10, 0 | 2060 (1464–2556) | 10/10, 1.00 | 609 (539–640) |
| 7 | Can Cowork read files from my Google Drive, and what’s the size limit per file? | agent | 10/10, 1 | 1510 (1361–1761) | 10/10, 0.93 | 583 (555–974) |
| 8 | Following up on what we discussed, is that still accurate today? | agent, ambiguous; artificial, since a fresh conversation has no prior context | 9/10, 2 | 1776 (1557–2222) | 10/10, 0.92 | 589 (533–669) |
| 9 | What is Cowork’s pricing, roughly, for a small team? | agent | 10/10, 2 | 1530 (1062–1908) | 10/10, 1.00 | 580 (523–1037) |
| 10 | Is quicksort’s worst case O(n^2) or O(n log n), and would that change if I ran it on today’s data? | agent by policy, disputed | 4/10, 4 | 3206 (2934–3662) | 0/10, 1.00 | 606 (540–736) |
The router matched 83/100 expected routes, including 13 correct timeout fallbacks; Jev matched 81/100. The router’s 17 direct-route mismatches occurred on the seconds-in-a-week question (ten), quicksort (six), and the context-free follow-up (one). Jev’s 19 occurred on seconds in a week (nine) and quicksort (ten). The two systems agreed on 85 of the 87 runs where the router returned a decision.
The seconds-in-a-week label follows the calculation rule literally. The quicksort label assumes the user wants data inspected, an interpretation that needs review. The follow-up has no prior conversation to refer to. These annotations should be resolved before using the test set to select a production threshold.
Routing question
This is the abbreviated question used in the article. The full configuration also sends questions about MindsHub Agents and the assistant’s runtime to the agent.
_ROUTE_QUESTION = {
"type": "choice",
"instructions": (
"Classify what is required to produce the assistant's next reply. "
"Return exactly one of: answer_directly or needs_agent. Classify as "
"needs_agent if any condition listed for it applies; otherwise "
"classify as answer_directly. When uncertain, choose needs_agent."
),
"criteria": {
"answer_directly": (
"The reply can be produced solely from text already in the "
"conversation and stable general knowledge."
),
"needs_agent": (
"The reply requires any tool use: browsing or external retrieval; "
"access to an attachment, file, account, or dataset; executing or "
"testing code; verifying potentially time-sensitive information; "
"a calculation; or a multi-step plan."
),
},
}