I wanted this to work. I’m saying that up front, because the rest of this reads like a complaint. It’s meant as a post-mortem.
Datris is built so the model is a setting rather than a load-bearing beam. Bring your own keys, pick your provider, change your mind on a Tuesday. So when open-weights models started topping leaderboards, and Kimi K3 became the model everyone was talking about, opening the door wider was the obvious next move. We built the integration: hundreds of models, one config change. We filtered the catalog down to the ones that declare what an agent needs, meaning tool calling, structured outputs, a large context window. Kimi K3 cleared that filter without trouble. On paper it was ready.
Then I pointed it at the Datris Assistant and gave it a softball. Pull a one-day weather forecast for the ten biggest US cities off a free public API, load it into MongoDB, run it nightly. No auth. Ten rows a night. With a frontier model the Assistant does this in a handful of calls.
It started well, which honestly made the rest worse. It checked what already existed in my environment before building anything, and it asked good scope questions: which source, what data, what cadence, what destination. Stop the tape there and you’d think the leaderboards had it right.
Then it started building. Its first attempt at registering the shape of the data was broken, which is fine, that happens. What came next wasn’t. Instead of pausing to work out how the platform handles JSON, it began guessing out loud at full speed. It rewrote the ingestion script from JSON to CSV, then to hand-rolled plain text, then back to JSON. It created the pipeline three separate times. At one point it deleted the pipeline outright, on the reasoning that it had no data in it anyway, and built a new one.
Every failure arrived with a fresh diagnosis delivered in a confident voice, and every diagnosis blamed a different piece of my platform. The schema detector wasn’t picking up the field names. The CSV parser was wrapping the output in extra quotes. The stream parser was mangling CSV headers. None of that was true. The real cause never moved from the first minute to the last: it had fed the system a malformed sample, and the platform has one simple, discoverable convention for JSON data that the model never stopped to learn until it had exhausted every available way of not learning it. Then it stamped the finished report with a date more than a year in the past.
Here’s the part that complicates the story. It eventually got it right. Ten forecasts loaded, verified in the database, handed back to me in a tidy table with sensible highlights. If your metric is whether the task completed, Kimi K3 passed my test.
But I sat through the whole thing. I watched it thrash and misdiagnose and delete and rebuild, and that ride is what a user is actually buying. A frontier model on this task feels like someone who has run the job before. This felt like a very bright intern with nobody supervising him, narrating guesses at full confidence while rearranging my infrastructure. You don’t hand your data platform to that intern a second time.
The gap isn’t intelligence. The single-shot work was good, all of it: the scope questions, the final table, the script itself. The gap is something else, and I think it’s the interesting part, because a lot of teams are about to pick a model off a leaderboard that doesn’t measure it.
What an agent actually asks of a model
A chat question is one turn. Prompt in, answer out. Judge the answer, score the model.
An agent session doesn’t work like that. The Datris Assistant hands the model a long set of standing instructions and a couple dozen tools, each with its own argument schema, then holds it inside a loop that can run for dozens of steps. At every step the model has to decide whether to call a tool or talk to the human, pick the right tool, fill in the arguments exactly right with no invented fields and no almost-correct names, read what comes back, recover if what came back is an error, and keep every instruction from turn one alive through turn forty. Kick off a pipeline run, poll it until it finishes, work out what the result means, decide what to do next.
Each of those steps is a chance to be slightly wrong, and slight wrongness compounds. Ninety-five percent reliability per step sounds fine right up until you’re forty steps deep, at which point “usually right” has quietly turned into “usually derails.” Single-shot benchmarks never surface this. They never make a model live with its own earlier mistakes.
“It worked eventually” doesn’t get you out of the math, either. An agent that brute-forces its way to the goal has still spent your time, your tokens, and a piece of infrastructure that got deleted and rebuilt on the way. It has also spent most of your willingness to let it run unsupervised. Users don’t score the ending. They sit through the session.
The training these models haven’t had
There are, roughly, two phases in how a modern model learns.
The first is imitation. The model reads an enormous amount of text and learns to predict what comes next. Book-smart, basically. It has absorbed how code looks, how answers sound, how an explanation gets put together. This phase is what makes a model knowledgeable and articulate, and it’s most of what leaderboard benchmarks reward. Open-weights labs are very good at it.
The second phase is learning by doing, and it’s what people mean when they say RL, reinforcement learning. Instead of predicting text, the model gets dropped into real tasks. Here are your tools, here’s the goal, go. It tries, it fails, sometimes it succeeds, and the reward only lands when the whole job comes out right, not when an individual sentence sounds plausible. Run that millions of times across long, messy, multi-step tasks and what the model picks up is habits rather than facts. Learn the system’s conventions before building on top of a guess. When something errors, read the error, and suspect your own last move before you blame the tools. Don’t tear down and rebuild what a small fix would have handled. Don’t quietly drop an instruction because it scrolled off a while ago.
Imitation gets you someone who aced the exams. Reinforcement on real tasks gets you someone who has run the job a thousand times and stopped making rookie mistakes. For a data platform you want the second person.
The frontier closed labs spend enormous compute on that second phase, and specifically on this shape of work: long tool loops, strict schemas, recovery from errors. Their flagship product is the agent experience now, so they have to. Most open-weights models get much less of it, and much less of it aimed at long-horizon tool use in particular. It isn’t that they get none. The problem is that this kind of training is expensive, it needs infrastructure that doesn’t come free with a web crawl (task environments, graders, simulated tools), and the results barely register on the benchmarks everyone screenshots. A model can gain almost nothing on a leaderboard and still be far better inside an agent. So the leaderboard stays the headline and the gap stays invisible until you run the loop yourself.
What we did about it
We shelved the integration. The code works. It passed every test we threw at it, end to end. But the point of a door is what walks through it, and what walked through wasn’t good enough to put in front of Datris users. An agent-native data platform whose assistant finds its way by deleting and rebuilding your pipelines isn’t agent-native.
The door isn’t welded shut. Open-weights labs know the gap is there, a few are starting to spend on exactly this kind of training, and the day an open model runs our real loop reliably, this becomes the fastest revival in the project’s history. I’d like that day to arrive. An open model you can run wherever you want, operating your own data infrastructure, is a good future. It just isn’t this month.
If you’re picking a model for agent work
Stop reading the leaderboard and run your own loop. Your system prompt, your tools, your schemas, a real multi-step session, twenty times through. And don’t only check whether the task completed. Watch the ride. Count the wrong turns, the confident misdiagnoses, the retries, whatever got deleted along the way. A session that ends in success after four rewrites and a teardown is not a success your users will forgive you for.
When a model everyone is raving about disappoints you in that test, it isn’t quite hype versus reality. The raving and your test are measuring two different kinds of competence. There’s plenty of book-smart on the market now. Job-smart is still rare, and agents need the second one.
Todd Fearn is the founder of Datris.ai, an open-source, agent-native data platform built on the Model Context Protocol, and he runs IData Corporation, a data engineering consultancy for financial services firms. He has spent about thirty years building production data infrastructure inside institutions like Goldman Sachs, Bridgewater Associates, Deutsche Bank, and Freddie Mac.